search

swiftlang / swift-package-manager

The Package Manager for the Swift Programming Language
Apache License 2.0
9.74k stars 1.34k forks source link

`swift test list` unexpected sorting behavior #6786

Closed MahdiBM closed 6 months ago

MahdiBM commented 1 year ago

Description

swift test list's output doesn't keep the sorting that a project's test classes/functions already have.

Reverse Function Order

...
PennyTests.CoinHandlerTests/testUserAtTheEndAndCoinSignAtTheBeginning
PennyTests.GHHooksTests/testEventHandler
PennyTests.GHHooksTests/testHeadingFinder
PennyTests.GHHooksTests/testMakeReleaseBody
PennyTests.GHHooksTests/testMarkdownFormatting
PennyTests.GHHooksTests/testParseCodeOwners
PennyTests.GHHooksTests/testSemVerBump
PennyTests.GHHooksTests/testUnicodesPrefix
PennyTests.GatewayProcessingTests/testAutoFaqsCommand
...

The order of all functions of a class are completely reversed compared to the source files. This is not the case when Xcode shows the tests list, but it makes the SSWG VSCode extension show the tests in reverse order. (cc @adam-fowler) For example the correct order for GHHooksTests functions is:

PennyTests.GHHooksTests/testUnicodesPrefix
PennyTests.GHHooksTests/testSemVerBump
PennyTests.GHHooksTests/testParseCodeOwners
PennyTests.GHHooksTests/testMarkdownFormatting
PennyTests.GHHooksTests/testMakeReleaseBody
PennyTests.GHHooksTests/testHeadingFinder
PennyTests.GHHooksTests/testEventHandler

Wrong Class Order

The order of class can also be incorrect compared to the source file orders that both Xcode and VSCode use:

....
PennyTests.CoinHandlerTests/testUserAtTheEndAndCoinSignAtTheBeginning
PennyTests.GHHooksTests/testEventHandler
PennyTests.GHHooksTests/testSemVerBump
PennyTests.GHHooksTests/testUnicodesPrefix
PennyTests.GatewayProcessingTests/testAutoFaqsCommand
PennyTests.GatewayProcessingTests/testAutoPings
PennyTests.GatewayProcessingTests/testBotStateManagerReceivesSignal
PennyTests.GatewayProcessingTests/testCommandsRegisterOnStartup
PennyTests.GatewayProcessingTests/testServerBoostCoins
PennyTests.LeafRenderTests/testAutoPingsHelp
...

As you can see, the PennyTests.GHHooksTests class appears before PennyTests.GatewayProcessingTests. This is the reverse of what Xcode and VSCode show. The rest of the class-orders is correct. I've confirmed changing the class's name from GHHo... to GhHo... fixes the order of the list command output, which makes me think the problem is with the priority that the sorting algorithm of the list command gives to uppercase letters.

Expected behavior

Sorting should be consistent with sorting of the source files in IDEs and sorting of the functions that user has chosen in their XCTest classes.

Actual behavior

The sorting mechanism has some unexpected behavior as explained.

Steps to reproduce

You can use the same project to run your tests on: Penny.

Swift Package Manager version/commit hash

Swift Package Manager - Swift 5.9.0

Swift & OS version (output of swift --version ; uname -a)

swift-driver version: 1.87 Apple Swift version 5.9 (swiftlang-5.9.0.124.4 clang-1500.0.38.1) Target: arm64-apple-macosx14.0 Darwin Macbook.local 23.0.0 Darwin Kernel Version 23.0.0: Tue Jul 18 20:35:35 PDT 2023; root:xnu-10002.0.199.505.1~3/RELEASE_ARM64_T6000 arm64

neonichu commented 1 year ago

I don't think I agree with the premise here that swift test should be consistent with Xcode and/or VSCode. If we think a different sort order is better in isolation, I'm open to it, but not because some other tool does it differently.

MahdiBM commented 1 year ago

So the second Wrong Class Order issue does kind of ask to be inline with how other IDEs work, but that would also be how Finder or probably most other places sort stuff, so it's suboptimal that SPM has chosen to be inconsistent.

Now that i check more, i see the current order of class names seems to be inline with lexicographical ordering, so i guess you could say SPM is at least following something for that.

The first Reverse Function Order issue though is less subjective. The order just doesn't match how users declare their functions.

adam-fowler commented 1 year ago

@MahdiBM The ordering of tests in VS Code test will be resolved by this https://github.com/apple/sourcekit-lsp/issues/611

grynspan commented 6 months ago

swift test does not make any guarantees about the order of reported tests. When using XCTest on macOS (as shipped with Xcode), source location information simply isn't available. SwiftPM theoretically does know the source locations of tests when building with swift-corelibs-xctest, however it is not a goal for that library to implement features that are not available in XCTest-proper.