No tests found when passing test suite to multi_scan using the only_testing param

dvataksi16 commented 3 years ago

New Issue Checklist

[x] Updated fastlane-plugin-test_center to the latest version
[x] I read the README.md
[x] I reviewed the example(s) for the action(s) I am using
[x] I have removed any sensitive data such as passwords, authentication tokens, or anything else I do not want to world to see

If you love this fastlane plugin, consider sponsoring it or asking your company to sponsor it. I would really appreciate any gesture: https://github.com/sponsors/lyndsey-ferguson. 😍

Issue Description

We're using multi_scan in my project to run iOS UI tests. As of the latest version, when I pass in a filename of an XCTestCase to the "only_testing" field, none of the tests get detected. This used to work on previous versions! This is our lane:

multi_scan(
    workspace: $workspace,
    scheme: $uiTestsScheme,
    devices: [:iphone],
    output_types: "junit",
    output_files: "ui_test_iphone_report.xml",
    result_bundle: true,
    only_testing: ['UITests/UserPermissionAcceptanceTests'], // UITests is a folder containing all of our ui tests. UserPermissionAcceptanceTests is an XCTestCase containing a subset of tests.
    derived_data_path: "./ui_test_derived_data",
    test_without_building: true,
    sdk: "iphonesimulator",
    fail_build: false,
    try_count: 5,
    code_coverage: true,
    skip_slack: true,
    parallel_testrun_count: 4,
    quit_simulators: true
  )

This is the result:

[11:21:54]: ------------------------
[11:21:54]: --- Step: multi_scan ---
[11:21:54]: ------------------------
[11:21:54]: As of Xcode 11, test_result bundles created in the output directory are actually symbolic links to an xcresult bundle

+------------------------+---------------------+
| Summary for multi_scan (test_center v3.14.4) |
+------------------------+---------------------+
| try_count              | 5                   |
| parallel_testrun_count | 4                   |
| quit_simulators        | true                |
+------------------------+---------------------+

[11:21:54]: Using deprecated option: '--custom_report_file_name' (Use `--output_files` instead)
[11:21:54]: Using deprecated option: '--custom_report_file_name' (Use `--output_files` instead)
[11:21:54]: Resolving Swift Package Manager dependencies...
[11:21:54]: $ xcodebuild -resolvePackageDependencies -workspace S.xcworkspace -scheme UITests -derivedDataPath ./ui_test_derived_data
[11:21:55]: ▸ Command line invocation:
[11:21:55]: ▸     "/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild" -resolvePackageDependencies -workspace S.xcworkspace -scheme UITests -derivedDataPath ./ui_test_derived_data
[11:21:55]: ▸ User defaults from command line:
[11:21:55]: ▸     IDEDerivedDataPathOverride = /Users/nil/nil/iphone/ui_test_derived_data
[11:21:56]: ▸ resolved source packages:
[11:21:56]: $ xcodebuild -showBuildSettings -workspace S.xcworkspace -scheme UITests -derivedDataPath ./ui_test_derived_data
[11:21:59]: Command timed out after 3 seconds on try 1 of 4, trying again with a 6 second timeout...
[11:22:03]: :parallel_testrun_count greater than the number of tests (0). Reducing to that number.

+-------------------+------+
|    multi_scan results    |
+-------------------+------+
| result            | true |
| total_tests       | 0    |
| passing_testcount | 0    |
| failed_testcount  | 0    |
| failed_tests      | []   |
| total_retry_count | 0    |
| report_files      | []   |
+-------------------+------+
+------------------+-------------------------------------+
|                      Lane Context                      |
+------------------+-------------------------------------+
| DEFAULT_PLATFORM | ios                                 |
| PLATFORM_NAME    | ios                                 |
| LANE_NAME        | ios ui_test_iphone_user_permissions |
+------------------+-------------------------------------+

The following multi_scan lane IS successful when I do not pass in a specific file but just a directory for the only_testing field:

multi_scan(
    workspace: $workspace,
    scheme: $uiTestsScheme,
    devices: [:iphone],
    output_types: "junit",
    output_files: "ui_test_iphone_report.xml",
    result_bundle: true,
    only_testing: ['UITests'], // NOTICE HERE I'M NOT PASSING A SPECIFIC FILE JUST A DIRECTORY
    derived_data_path: "./ui_test_derived_data",
    test_without_building: true,
    sdk: "iphonesimulator",
    fail_build: false,
    try_count: 5,
    code_coverage: true,
    skip_slack: true,
    parallel_testrun_count: 4,
    quit_simulators: true
  )

It's important for us to be able to break apart our tests.

I think my issue is similar to https://github.com/lyndsey-ferguson/fastlane-plugin-test_center/issues/281. I'm not using cucumberish tests. I tried altering the value of "test_without_building:" to false and adding invocation_based_tests: true, as suggested in the thread.

Environment

✅ fastlane environment ✅

### Stack

| Key                         | Value                                         |
| --------------------------- | --------------------------------------------- |
| OS                          | 10.15.5                                       |
| Ruby                        | 2.7.1                                         |
| Bundler?                    | false                                         |
| Git                         | git version 2.17.1                            |
| Installation Source         | ~/.rbenv/versions/2.7.1/bin/fastlane          |
| Host                        | Mac OS X 10.15.5 (19F101)                     |
| Ruby Lib Dir                | ~/.rbenv/versions/2.7.1/lib                   |
| OpenSSL Version             | OpenSSL 1.1.1g  21 Apr 2020                   |
| Is contained                | false                                         |
| Is homebrew                 | false                                         |
| Is installed via Fabric.app | false                                         |
| Xcode Path                  | /Applications/Xcode 2.app/Contents/Developer/ |
| Xcode Version               | 11.7                                          |

### System Locale

| Variable | Value       |   |
| -------- | ----------- | - |
| LANG     | en_US.UTF-8 | ✅ |
| LC_ALL   |             |   |
| LANGUAGE |             |   |

# More information about multiple platforms in fastlane: https://github.com/fastlane/fastlane/blob/master/docs/Platforms.md
# All available actions: https://github.com/fastlane/fastlane/blob/master/fastlane/docs/Actions.md

# fastlane reports which actions are used
# No personal data is recorded. Learn more at https://github.com/fastlane/enhancer

```

`./fastlane/Appfile`

```ruby
app_identifier "" # The bundle identifier of your app
apple_id "" # Your Apple email address

# you can even provide different app identifiers, Apple IDs and team names per lane:
# More information: https://github.com/fastlane/fastlane/blob/master/fastlane/docs/Appfile.md

```

fastlane gems



Gem
Version
Update-Status




fastlane
2.162.0
✅ Up-To-Date



Loaded fastlane plugins:



Plugin
Version
Update-Status




fastlane-plugin-test_center
3.14.4
✅ Up-To-Date


fastlane-plugin-trainer
0.4.1
✅ Up-To-Date


fastlane-plugin-versioning
0.4.4
✅ Up-To-Date


fastlane-plugin-appcenter
1.10.0
✅ Up-To-Date



Loaded gems

| Gem                         | Version      |
| --------------------------- | ------------ |
| did_you_mean                | 1.4.0        |
| slack-notifier              | 2.3.2        |
| rouge                       | 2.0.7        |
| xcpretty                    | 0.3.0        |
| terminal-notifier           | 2.0.0        |
| unicode-display_width       | 1.7.0        |
| terminal-table              | 1.8.0        |
| plist                       | 3.5.0        |
| CFPropertyList              | 3.0.2        |
| addressable                 | 2.7.0        |
| multipart-post              | 2.0.0        |
| word_wrap                   | 1.0.0        |
| tty-cursor                  | 0.7.1        |
| tty-spinner                 | 0.9.3        |
| colored                     | 1.2          |
| highline                    | 1.7.10       |
| commander-fastlane          | 4.4.6        |
| faraday                     | 1.0.1        |
| faraday_middleware          | 1.0.0        |
| gh_inspector                | 1.1.3        |
| mini_magick                 | 4.10.1       |
| rubyzip                     | 2.3.0        |
| security                    | 0.1.3        |
| xcpretty-travis-formatter   | 1.0.0        |
| naturally                   | 2.2.0        |
| simctl                      | 1.6.8        |
| uber                        | 0.1.0        |
| declarative-option          | 0.1.0        |
| representable               | 3.0.4        |
| retriable                   | 3.1.2        |
| mini_mime                   | 1.0.2        |
| signet                      | 0.14.0       |
| httpclient                  | 2.8.3        |
| google-api-client           | 0.38.0       |
| uri                         | 0.10.0       |
| nanaimo                     | 0.3.0        |
| colored2                    | 3.1.2        |
| claide                      | 1.0.3        |
| atomos                      | 0.1.3        |
| xcodeproj                   | 1.18.0       |
| public_suffix               | 4.0.6        |
| tty-screen                  | 0.8.1        |
| babosa                      | 1.0.4        |
| excon                       | 0.76.0       |
| unf_ext                     | 0.0.7.7      |
| unf                         | 0.1.4        |
| domain_name                 | 0.5.20190701 |
| http-cookie                 | 1.0.3        |
| faraday-cookie_jar          | 0.0.7        |
| fastimage                   | 2.2.0        |
| json                        | 2.3.1        |
| dotenv                      | 2.7.6        |
| bundler                     | 2.1.4        |
| jwt                         | 2.2.2        |
| declarative                 | 0.0.20       |
| multi_json                  | 1.15.0       |
| os                          | 1.1.1        |
| memoist                     | 0.16.2       |
| googleauth                  | 0.13.1       |
| rake                        | 13.0.1       |
| digest-crc                  | 0.6.1        |
| google-cloud-errors         | 1.0.1        |
| google-cloud-env            | 1.3.3        |
| google-cloud-core           | 1.5.0        |
| google-cloud-storage        | 1.29.1       |
| emoji_regex                 | 3.0.0        |
| aws-eventstream             | 1.1.0        |
| aws-sigv4                   | 1.2.2        |
| aws-partitions              | 1.380.0      |
| jmespath                    | 1.4.0        |
| aws-sdk-core                | 3.109.1      |
| aws-sdk-kms                 | 1.39.0       |
| aws-sdk-s3                  | 1.83.0       |
| forwardable                 | 1.3.1        |
| logger                      | 1.4.2        |
| cgi                         | 0.1.0        |
| timeout                     | 0.1.0        |
| stringio                    | 0.1.0        |
| ipaddr                      | 1.2.2        |
| openssl                     | 2.1.2        |
| ostruct                     | 0.2.0        |
| strscan                     | 1.0.3        |
| date                        | 3.0.0        |
| delegate                    | 0.1.0        |
| fileutils                   | 1.4.1        |
| io-console                  | 0.5.6        |
| zlib                        | 1.1.0        |
| singleton                   | 0.1.0        |
| rexml                       | 3.2.3        |
| open3                       | 0.1.0        |
| yaml                        | 0.1.0        |
| psych                       | 3.1.0        |
| mutex_m                     | 0.1.0        |
| webrick                     | 1.6.0        |
| trainer                     | 0.9.1        |
| xctest_list                 | 1.2.1        |
| colorize                    | 0.8.1        |
| fastlane-plugin-test_center | 3.14.4       |
| fastlane-plugin-trainer     | 0.4.1        |
| fastlane-plugin-versioning  | 0.4.4        |
| fastlane-plugin-appcenter   | 1.10.0       |
| csv                         | 3.1.2        |

generated on: 2020-10-09

Gem	Version	Update-Status
fastlane	2.162.0	✅ Up-To-Date

Plugin	Version	Update-Status
fastlane-plugin-test_center	3.14.4	✅ Up-To-Date
fastlane-plugin-trainer	0.4.1	✅ Up-To-Date
fastlane-plugin-versioning	0.4.4	✅ Up-To-Date
fastlane-plugin-appcenter	1.10.0	✅ Up-To-Date

lyndsey-ferguson commented 3 years ago

Hello @dvataksi16 , this may be a difference of terminology, the only_testing doesn't accept folders or files. It accepts test targets (sometimes called testbundles, or testables), test targets+ test suites, and test targets + test suites + test cases.

This stackoverflow discussion may clarify what I mean.

If you did in fact meant to write testbundle/testsuite in the first example, and testbundle in the second, then the first thing I would ask from you is a full log generated by running your lane with the --verbose flag: make sure that you obfuscate any data that is sensitive (project names, security tokens, passwords, maybe test names).

lyndsey-ferguson commented 3 years ago

Also, it would help to know for which version multi_scan was working for you.

dvataksi16 commented 3 years ago

Hello, Yes I did mean testbundle/testsuite in my example. I cannot pass a testsuite to the only_testing flag. Thank you for pointing that out. I was not aware of the terminology.

This is the output with the verbose flag:

DEBUG [2020-10-12 09:27:43.76]: Checking if there are any plugins that should be loaded...
DEBUG [2020-10-12 09:27:43.76]: Loading 'fastlane-plugin-test_center' plugin
INFO [2020-10-12 09:27:43.76]: [32mgem 'fastlane-plugin-test_center' is already installed[0m
DEBUG [2020-10-12 09:27:44.39]: Loading 'fastlane-plugin-trainer' plugin
INFO [2020-10-12 09:27:44.40]: [0;32;49mgem 'fastlane-plugin-trainer' is already installed[0m
DEBUG [2020-10-12 09:27:44.40]: Loading 'fastlane-plugin-versioning' plugin
INFO [2020-10-12 09:27:44.40]: [0;32;49mgem 'fastlane-plugin-versioning' is already installed[0m
DEBUG [2020-10-12 09:27:44.41]: Loading 'fastlane-plugin-appcenter' plugin
INFO [2020-10-12 09:27:44.41]: [0;32;49mgem 'fastlane-plugin-appcenter' is already installed[0m
+-----------------------------+---------+------------------------------------------+
|                                   [0;32;49mUsed plugins[0m                                   |
+-----------------------------+---------+------------------------------------------+
| Plugin                      | Version | Action                                   |
+-----------------------------+---------+------------------------------------------+
| fastlane-plugin-test_center | 3.14.4  | suppressed_tests suppress_tests          |
|                             |         | tests_from_xcresult                      |
|                             |         | suppress_tests_from_junit                |
|                             |         | tests_from_xctestrun                     |
|                             |         | quit_core_simulator_service              |
|                             |         | collate_test_result_bundles              |
|                             |         | tests_from_junit multi_scan              |
|                             |         | test_options_from_testplan               |
|                             |         | collate_html_reports                     |
|                             |         | collate_junit_reports                    |
|                             |         | collate_json_reports                     |
|                             |         | testplans_from_scheme collate_xcresults  |
| fastlane-plugin-trainer     | 0.4.1   | trainer                                  |
| fastlane-plugin-versioning  | 0.4.3   | get_version_number_from_git_branch       |
|                             |         | increment_build_number_in_xcodeproj      |
|                             |         | get_version_number_from_plist            |
|                             |         | get_build_number_from_plist              |
|                             |         | ci_build_number                          |
|                             |         | get_version_number_from_xcodeproj        |
|                             |         | increment_version_number_in_xcodeproj    |
|                             |         | increment_version_number_in_plist        |
|                             |         | get_app_store_version_number             |
|                             |         | get_info_plist_path                      |
|                             |         | get_build_number_from_xcodeproj          |
|                             |         | increment_build_number_in_plist          |
| fastlane-plugin-appcenter   | 1.10.0  | appcenter_fetch_version_number           |
|                             |         | appcenter_fetch_devices appcenter_upload |
+-----------------------------+---------+------------------------------------------+

[0;33;49mSuccessfully loaded Appfile at path '/Users/user/repos/iphone/fastlane/Appfile'[0m
- [0;36;49mapp_identifier[0m: '[0;32;49mcom.company[0m'
- [0;36;49mapple_id[0m: '[0;32;49memail[0m'
-------
INFO [2020-10-12 09:27:44.83]: [0;32;49m----------------------------------------[0m
INFO [2020-10-12 09:27:44.83]: [0;32;49m--- Step: Verifying fastlane version ---[0m
INFO [2020-10-12 09:27:44.83]: [0;32;49m----------------------------------------[0m
INFO [2020-10-12 09:27:44.83]: Your fastlane version 2.162.0 matches the minimum requirement of 2.125.2  ✅
INFO [2020-10-12 09:27:44.83]: [0;32;49m------------------------------[0m
INFO [2020-10-12 09:27:44.83]: [0;32;49m--- Step: default_platform ---[0m
INFO [2020-10-12 09:27:44.83]: [0;32;49m------------------------------[0m
INFO [2020-10-12 09:27:44.83]: [0;32;49mDriving the lane 'ios ui_test_iphone_user_permissions' 🚀[0m
INFO [2020-10-12 09:27:44.83]: Environment Tools Version
INFO [2020-10-12 09:27:44.83]: [0;32;49m---------------------------------[0m
INFO [2020-10-12 09:27:44.83]: [0;32;49m--- Step: xcodebuild -version ---[0m
INFO [2020-10-12 09:27:44.83]: [0;32;49m---------------------------------[0m
INFO [2020-10-12 09:27:44.83]: [0;36;49m$ xcodebuild -version[0m
INFO [2020-10-12 09:27:44.91]: ▸ [0;35;49mXcode 11.7[0m
INFO [2020-10-12 09:27:44.92]: ▸ [0;35;49mBuild version 11E801a[0m
INFO [2020-10-12 09:27:44.92]: [0;32;49m---------------------[0m
INFO [2020-10-12 09:27:44.92]: [0;32;49m--- Step: ruby -v ---[0m
INFO [2020-10-12 09:27:44.92]: [0;32;49m---------------------[0m
INFO [2020-10-12 09:27:44.92]: [0;36;49m$ ruby -v[0m
INFO [2020-10-12 09:27:44.93]: ▸ [0;35;49mruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-darwin18][0m
INFO [2020-10-12 09:27:44.93]: Cleaning up old code coverage reports...
INFO [2020-10-12 09:27:44.94]: [0;32;49m---------------------------------------------------------------------------[0m
INFO [2020-10-12 09:27:44.94]: [0;32;49m--- Step: xcrun simctl uninstall booted com.company || true ---[0m
INFO [2020-10-12 09:27:44.94]: [0;32;49m---------------------------------------------------------------------------[0m
INFO [2020-10-12 09:27:44.94]: [0;36;49m$ xcrun simctl uninstall booted com.company || true[0m
INFO [2020-10-12 09:27:46.20]: [0;32;49m------------------------[0m
INFO [2020-10-12 09:27:46.20]: [0;32;49m--- Step: multi_scan ---[0m
INFO [2020-10-12 09:27:46.20]: [0;32;49m------------------------[0m
WARN [2020-10-12 09:27:46.29]: [0;33;49mAs of Xcode 11, test_result bundles created in the output directory are actually symbolic links to an xcresult bundle[0m

+------------------------+---------------------+
| [0;32;49mSummary for multi_scan (test_center v3.14.4)[0m |
+------------------------+---------------------+
| try_count              | 5                   |
| parallel_testrun_count | 4                   |
| quit_simulators        | true                |
+------------------------+---------------------+

ERROR [2020-10-12 09:27:46.31]: [1;34;49mUsing deprecated option: '--custom_report_file_name' (Use `--output_files` instead)[0m
DEBUG [2020-10-12 09:27:46.31]: Preparing Scan config options for multi_scan testing
ERROR [2020-10-12 09:27:46.32]: [1;34;49mUsing deprecated option: '--custom_report_file_name' (Use `--output_files` instead)[0m
WARN [2020-10-12 09:27:46.36]: [0;33;49mResolving Swift Package Manager dependencies...[0m
INFO [2020-10-12 09:27:46.36]: [0;36;49m$ xcodebuild -resolvePackageDependencies -workspace S.xcworkspace -scheme SUITests -derivedDataPath ./ui_test_derived_data[0m
INFO [2020-10-12 09:27:46.79]: ▸ [0;35;49mCommand line invocation:[0m
INFO [2020-10-12 09:27:46.79]: ▸ [0;35;49m    "/Applications/Xcode 2.app/Contents/Developer/usr/bin/xcodebuild" -resolvePackageDependencies -workspace S.xcworkspace -scheme SUITests -derivedDataPath ./ui_test_derived_data[0m
INFO [2020-10-12 09:27:46.79]: ▸ [0;35;49mUser defaults from command line:[0m
INFO [2020-10-12 09:27:46.79]: ▸ [0;35;49m    IDEDerivedDataPathOverride = /Users/user/repos/iphone/ui_test_derived_data[0m
INFO [2020-10-12 09:27:48.20]: ▸ [0;35;49mresolved source packages: [0m
INFO [2020-10-12 09:27:48.21]: [0;36;49m$ xcodebuild -showBuildSettings -workspace S.xcworkspace -scheme SUITests -derivedDataPath ./ui_test_derived_data[0m
WARN [2020-10-12 09:27:51.22]: [0;33;49mCommand timed out after 3 seconds on try 1 of 4, trying again with a 6 second timeout...[0m
DEBUG [2020-10-12 09:27:55.04]: Fetching available simulator devices
DEBUG [2020-10-12 09:27:55.48]: Getting tests from xctestrun file at './ui_test_derived_data/Build/Products/SUITests_iphonesimulator13.7-x86_64.xctestrun'
DEBUG [2020-10-12 09:27:55.70]: Found the following tests: SMediaAlbumTests/test_ui_createMediaAlbumMaterialInGroup
    SMediaAlbumTests/test_ui_createMediaAlbumMaterialInCSLParentSection
    SMediaAlbumTests/test_ui_createEmptyMediaAlbumMaterialInCSLParentSection
    SMediaAlbumTests/test_ui_cancelCreatingMediaAlbumInCSLParentSection
    SLoginAcceptanceTests/test_ui_webViewLoginBackButtonReturnsToMainLogin
    SLoginAcceptanceTests/test_ui_successfulEmailLogin
    SLoginAcceptanceTests/test_ui_successfulNativeLoginToExternalAuthSchool
    SLoginAcceptanceTests/test_ui_successfulNativeLoginToExternalBrowserAuthSchool
    SLoginAcceptanceTests/test_ui_externalBasicAuthLogin
    SLoginAcceptanceTests/test_ui_successfulUsernameLogin
    SLoginAcceptanceTests/test_ui_viewHelpCenterFromLoginScreen
    SLoginAcceptanceTests/test_ui_successfulRemoteAuthLogin
    SLoginAcceptanceTests/test_ui_cancelExternalBasicAuthLogin
    SLoginAcceptanceTests/test_ui_failsWithIncorrectUsernameLogin
    SLoginAcceptanceTests/test_ui_failsWithIncorrectPasswordUsernameLogin
    SLoginAcceptanceTests/test_ui_successfulLDAPLogin
    SLoginAcceptanceTests/test_ui_successfulSAMLLogin
    SLoginAcceptanceTests/test_ui_failsWithUnknownEmail
    SLoginAcceptanceTests/test_ui_failsWithIncorrectPassword
    SLoginAcceptanceTests/test_ui_successfulMicrosoftSSOLogin
    SLoginAcceptanceTests/test_ui_leftMenuTopButtonsAccessible
    SLoginAcceptanceTests/test_ui_denyCameraAccessPermissionQRLogin
    SLoginAcceptanceTests/test_ui_authorizeCameraAccessInAppSettings
    SLoginAcceptanceTests/test_ui_authorizeCameraAccessPermissionQRLogin
    SGradesAcceptanceTests/test_ui_viewStudentGradeReportInChildFromGradesTab
    SGradesAcceptanceTests/test_ui_viewStudentGradeReportInParentFromGradesTab
    SGradesAcceptanceTests/test_ui_viewStudentGradeReportWithOnlyLetterGrades
    SGradesAcceptanceTests/test_ui_viewStudentGradeReportWithLetterAndPercentageGrades
    SGradesAcceptanceTests/test_ui_viewStudentGradeReportForMaterialsWithNoGradingPeriodInChildSectionFromGradesTab
    SGradesAcceptanceTests/test_ui_viewStudentGradeReportForMaterialsWithNoGradingPeriodInParentSectionFromGradesTab
    SGradesAcceptanceTests/test_ui_viewStudentGradeReportWithPointsScale
    SGradesAcceptanceTests/test_ui_viewStudentGradeReportFromChildSectionProfile
    SGradesAcceptanceTests/test_ui_viewStudentGradeReportFromParentSectionProfile
    SGradesAcceptanceTests/test_ui_viewStudentGradeReportForMaterialsWithNoGradingPeriodInChildSection
    SGradesAcceptanceTests/test_ui_viewStudentGradeReportForMaterialsWithNoGradingPeriodInParentSection
    SMessageAcceptanceTests/test_ui_cancelMessage
    SMessageAcceptanceTests/test_ui_composeMessageInstructor
    SMessageAcceptanceTests/test_ui_composeMessageParent
    SMessageAcceptanceTests/test_ui_composeMessageStudent
    SRequestsAcceptanceTests/test_ui_receiveAndAcceptGroupRequestResultsInRecipientBeingGroupMember
    SRequestsAcceptanceTests/test_ui_receiveAndDenyGroupRequest
    SRequestsAcceptanceTests/test_ui_receiveAndDenyConnectionRequest
    SRequestsAcceptanceTests/test_ui_receiveAndAcceptConnectionRequest
    SNativeGradedMaterialsTests/test_ui_createGradedDiscussionFromCourseProfile
    SNativeGradedMaterialsTests/test_ui_openAssignmentWithEmbeddedMedia
    SNativeGradedMaterialsTests/test_ui_gradeAssignmentWithRubricViaGradebook
    SNativeGradedMaterialsTests/test_ui_createUngradedDiscussionFromCourseProfile
    SNativeGradedMaterialsTests/test_ui_submitToAssignmentFromGoogleDriveResourceApp
    SNativeGradedMaterialsTests/test_ui_createSubmitAndGradeCourseAssignmentAsAdminLinkedSection
    SNativeGradedMaterialsTests/test_ui_createSubmitAndGradeCourseAssignmentAsAdminUnlinkedSection
    SUserProfileAcceptanceTests/test_ui_viewUserBadges
    SUserProfileAcceptanceTests/test_ui_badgesTabNotVisibleForOtherUser
    SUserProfileAcceptanceTests/test_ui_userAndSchoolProfilesHaveCorrectTabs
    SGroupProfileAcceptanceTests/test_ui_groupProfileHasCorrectTabsForAdmin
    SGroupProfileAcceptanceTests/test_ui_groupProfileHasCorrectTabsForMember
    SGroupProfileAcceptanceTests/test_ui_createGroupUpdateAsAdminAndViewAsMember
    SGroupProfileAcceptanceTests/test_ui_createAndViewGroupUpdateAsMember
    SGroupProfileAcceptanceTests/test_ui_createDiscussionFromGroupProfile
    SCourseProfileAcceptanceTests/test_ui_courseProfileHasCorrectTabsForAdmin
    SCourseProfileAcceptanceTests/test_ui_courseProfileHasCorrectTabsForMember
    SCourseProfileAcceptanceTests/test_ui_createCourseUpdateAsAdminInCSLParentSection
    SCourseProfileAcceptanceTests/test_ui_navigateCourseMaterialsSubfolders
    SCourseProfileAcceptanceTests/test_ui_createAndViewCourseUpdateInCSLChildSection
    SCourseProfileAcceptanceTests/test_ui_createAndViewCourseUpdateInUnlinkedSection
    SCourseProfileAcceptanceTests/test_ui_editTitleOfFile
    SCourseProfileAcceptanceTests/test_ui_verifyPageMaterial
    SCourseProfileAcceptanceTests/test_ui_addFilesAndDiscardChanges
    SCourseProfileAcceptanceTests/test_ui_clickOnFolderMultipleTimes
    SCourseProfileAcceptanceTests/test_ui_addLinkCancelAndDiscardChanges
    SCourseProfileAcceptanceTests/test_ui_createLinkmaterialInCSLParentSection
    SCourseProfileAcceptanceTests/test_ui_submitToTestQuizAsStudentInCSLChildSection
    SCourseProfileAcceptanceTests/test_ui_submitToTestQuizAsStudentInCSLParentSection
    SRecentActivityAcceptanceTests/test_ui_createAssignmentFromRecentActivity
    SRecentActivityAcceptanceTests/test_ui_createGradedDiscussionFromRecentActivity
    SRecentActivityAcceptanceTests/test_ui_createUngradedDiscussionFromRecentActivity
    SRecentActivityAcceptanceTests/test_ui_createAndViewNonCSLUpdateFromRecentActivity
    SRecentActivityAcceptanceTests/test_ui_createAndViewChildCSLUpdateFromRecentActivity
    SRecentActivityAcceptanceTests/test_ui_createAndViewParentCSLUpdateFromRecentActivity
    SRecentActivityAcceptanceTests/test_ui_createGroupEventWithRSVPDisabledFromRecentActivityResultsInEventInCalendar
    SRecentActivityAcceptanceTests/test_ui_createPersonalEventWithRSVPDisabledFromRecentActivityResultsInEventInCalendar
    SRecentActivityAcceptanceTests/test_ui_createNonCSLCourseEventWithRSVPDisabledFromRecentActivityResultsInEventInCalendar
    SRecentActivityAcceptanceTests/test_ui_createAllDayNonCSLCourseEventWithRSVPDisabledFromRecentActivityResultsInEventInCalendar
    SRecentActivityAcceptanceTests/test_ui_createNonCSLCourseEventWithEndTime
    SRecentActivityAcceptanceTests/test_ui_openAndNavigateAccountSettingsScreen
    UserPermissionsAcceptanceTests/test_ui_enterpriseFacultyUserHasEnterpriseFeatures
    UserPermissionsAcceptanceTests/test_ui_enterpriseNonFacultyUserHasEnterpriseFeatures
    GoogleSSOAcceptanceTests/test_ui_failsGoogleSSOLoginWhenCanceling
    GoogleSSOAcceptanceTests/test_ui_successfulGoogleSSOLogin
WARN [2020-10-12 09:27:55.70]: [0;33;49m:parallel_testrun_count greater than the number of tests (0). Reducing to that number.[0m
DEBUG [2020-10-12 09:27:55.70]: > setup_logcollection
DEBUG [2020-10-12 09:27:55.70]: < done in TestCenter::Helper::MultiScanManager.initialize
DEBUG [2020-10-12 09:27:55.70]: Fetching available simulator devices
DEBUG [2020-10-12 09:27:56.14]: Results for each test run: []

+-------------------+------+
|    [0;32;49mmulti_scan results[0m    |
+-------------------+------+
| result            | true |
| total_tests       | 0    |
| passing_testcount | 0    |
| failed_testcount  | 0    |
| failed_tests      | []   |
| total_retry_count | 0    |
| report_files      | []   |
+-------------------+------+

INFO [2020-10-12 09:27:56.14]: Generating Unit Test Report...
WARN [2020-10-12 09:27:56.14]: [0;33;49mLane Context:[0m
INFO [2020-10-12 09:27:56.14]: {:DEFAULT_PLATFORM=>:ios, :PLATFORM_NAME=>:ios, :LANE_NAME=>"ios ui_test_iphone_user_permissions"}
ERROR [2020-10-12 09:27:56.14]: [0;31;49mError in your Fastfile at line 150[0m
ERROR [2020-10-12 09:27:56.15]: [0;31;49m    148:    uiTestPattern = /test_output\/report.xcresult$/[0m
ERROR [2020-10-12 09:27:56.15]: [0;31;49m    149:    xcresultPattern = (scheme == $SUITestsScheme) ? uiTestPattern : unitTestPattern[0m
ERROR [2020-10-12 09:27:56.15]: [0;31;49m => 150:    File.expand_path(Dir['test_output/**/*'].select{|g| g =~ xcresultPattern }.first)[0m
ERROR [2020-10-12 09:27:56.15]: [0;31;49m    151:  end[0m
ERROR [2020-10-12 09:27:56.15]: [0;31;49m    152:  [0m
ERROR [2020-10-12 09:27:56.15]: [0;31;49mno implicit conversion of nil into String[0m
INFO [2020-10-12 09:27:56.15]: [0;32;49mSuccessfully generated documentation at path '/Users/user/repos/iphone/fastlane/README.md'[0m

+------+---------------------------------+-------------+
|                   [0;32;49mfastlane summary[0m                   |
+------+---------------------------------+-------------+
| Step | Action                          | Time (in s) |
+------+---------------------------------+-------------+
| 1    | Verifying fastlane version      | 0           |
| 2    | default_platform                | 0           |
| 3    | xcodebuild -version             | 0           |
| 4    | ruby -v                         | 0           |
| 5    | xcrun simctl uninstall booted   | 1           |
|      | com.company || true |             |
| 6    | multi_scan                      | 9           |
+------+---------------------------------+-------------+

+------------+--------------+----------------+
|          [0;33;49mPlugin updates available[0m          |
+------------+--------------+----------------+
| Plugin     | Your Version | Latest Version |
+------------+--------------+----------------+
| versioning | [0;31;49m0.4.3[0m        | [0;32;49m0.4.4[0m          |
+------------+--------------+----------------+
INFO [2020-10-12 09:27:56.15]: To update all plugins, just run
INFO [2020-10-12 09:27:56.15]: [0;36;49m$ bundle exec fastlane update_plugins[0m

ERROR [2020-10-12 09:27:56.15]: [0;31;49mfastlane finished with errors[0m

Looking for related GitHub issues on fastlane/fastlane...
Search query: no implicit conversion of nil into String

URL: https://api.github.com/search/issues?q=no%20implicit%20conversion%20of%20nil%20into%20String+repo:fastlane/fastlane
➡️  [0;33;49mlatest_testflight_build_number receiving error: [!] no implicit conversion of Integer into String[0m
    https://github.com/fastlane/fastlane/issues/17161 [[0;32;49mclosed[0m] 10 💬
    5 weeks ago

➡️  [0;33;49mlatest_testflight_build_number receiving error: [!] no implicit conversion of Integer into String[0m
    https://github.com/fastlane/fastlane/issues/17162 [[0;32;49mclosed[0m] 1 💬
    5 weeks ago

➡️  [0;33;49mgoogle_cloud_storage TypeError: [!] no implicit conversion of nil into String[0m
    https://github.com/fastlane/fastlane/issues/16836 [[0;32;49mclosed[0m] 10 💬
    21 Aug 2020

and 86 more at: https://github.com/fastlane/fastlane/search?q=no%20implicit%20conversion%20of%20nil%20into%20String&type=Issues&utf8=✓

🔗  You can ⌘ + double-click on links to open them directly in your browser.

This was working for us when using test center v 3.14.3.

AlexTrott commented 3 years ago

Can confirm I'm seeing the same issue since updating to v3.14.4

lyndsey-ferguson commented 3 years ago

Thank you, the logs were very helpful and I believe I've fixed the problem.

@dvataksi16 and @AlexTrott, can you modify your Pluginfile per my instructions below, run bundle install, and then run your fastlane again (with the --verbose flag)?

Pluginfile:

gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-293-tests-not-found"

If there are still problems, please let me know and attach the console output as a text file to this issue (makes it easier for me to review). If it works, please let me know.

dvataksi16 commented 3 years ago

This update fixes the issue for me! Thank you 💯

lyndsey-ferguson commented 3 years ago

Fixed in v3.14.5

moglistree commented 3 years ago

I am seeing a similar issue with version 3.15.3. Trying to run a scheme with 71 tests and it only identifies the first 10 tests when I use test_without_building as true. But if I do and build and run of the tests is identifies all 71 tests and runs them no problem.

lyndsey-ferguson commented 3 years ago

@moglistree can you please create a new Bug Issue? Please pay particular attention to the template to provide the information I need to look into the problem.

lyndsey-ferguson / fastlane-plugin-test_center