[Bug] multi_scan - Unable to get a successful run

AlexTrott commented 1 year ago

New Issue Checklist

[x] Updated fastlane-plugin-test_center to the latest version
[x] I read the README.md
[x] I reviewed the example(s) for the action(s) I am using
[x] I have removed any sensitive data such as passwords, authentication tokens, or anything else I do not want to world to see
[x] I have reviewed the Discussions forum to see my question has already been addressed.

Issue Description

I've just tried updating my fastlane version (pre trainer removal) and test_center version (pre trainer removal) to the latest on both, and I'm unable to get multi-scan to pass.

I've created a sample application, which is just exciting scan and just executing multi-scan https://github.com/AlexTrott/SampleApp-TestCenter-Issue/blob/main/fastlane/Fastfile

Complete output when running fastlane, including the stack trace and command used

Output from executing multi_scan

ERROR [2022-08-25 13:21:00.84]: Cannot find .xcresult in derived data which is needed to determine test results. This is an issue within scan. File an issue on GitHub or try setting option `result_bundle: true`
INFO [2022-08-25 13:21:00.84]: Successfully generated documentation at path '/Users/alextrott/Developer/github/SampleApp-TestCenter-Issue/fastlane/README.md'

+------+------------------+-------------+
|           fastlane summary            |
+------+------------------+-------------+
| Step | Action           | Time (in s) |
+------+------------------+-------------+
| 1    | default_platform | 0           |
| 💥   | multi_scan       | 10          |
+------+------------------+-------------+

DEBUG [2022-08-25 13:21:00.84]: All plugins are up to date
ERROR [2022-08-25 13:21:00.84]: fastlane finished with errors
DEBUG [2022-08-25 13:21:00.84]: All plugins are up to date
/Users/alextrott/.rvm/gems/ruby-3.0.0/gems/fastlane-2.209.1/fastlane_core/lib/fastlane_core/ui/interface.rb:163:in `build_failure!': \e[0;31;49m[!] Cannot find .xcresult in derived data which is needed to determine test results. This is an issue within scan. File an issue on GitHub or try setting option `result_bundle: true`\e[0m (FastlaneCore::Interface::FastlaneBuildFailure)
    from /Users/alextrott/.rvm/gems/ruby-3.0.0/gems/fastlane-2.209.1/fastlane_core/lib/fastlane_core/ui/ui.rb:17:in `method_missing'
    from /Users/alextrott/.rvm/gems/ruby-3.0.0/gems/fastlane-2.209.1/scan/lib/scan/runner.rb:204:in `trainer_test_results'
    from /Users/alextrott/.rvm/gems/ruby-3.0.0/gems/fastlane-2.209.1/scan/lib/scan/runner.rb:268:in `handle_results'
    from /Users/alextrott/.rvm/gems/ruby-3.0.0/gems/fastlane-2.209.1/scan/lib/scan/runner.rb:24:in `run'
    from /Users/alextrott/.rvm/gems/ruby-3.0.0/gems/fastlane-plugin-test_center-3.19.1/lib/fastlane/plugin/test_center/helper/multi_scan_manager/retrying_scan.rb:20:in `run'

Environment

[INSERT OUTPUT HERE]

✅ fastlane environment ✅

### Stack | Key | Value | | --------------------------- | -------------------------------------------------- | | OS | 12.4 | | Ruby | 3.0.0 | | Bundler? | true | | Git | git version 2.32.1 (Apple Git-133) | | Installation Source | ~/.rvm/gems/ruby-3.0.0/bin/fastlane | | Host | macOS 12.4 (21F79) | | Ruby Lib Dir | ~/.rvm/rubies/ruby-3.0.0/lib | | OpenSSL Version | OpenSSL 1.1.1l 24 Aug 2021 | | Is contained | false | | Is homebrew | false | | Is installed via Fabric.app | false | | Xcode Path | /Applications/Xcode-13.4.0.app/Contents/Developer/ | | Xcode Version | 13.4 | | Swift Version | 5.6.1 | ### System Locale | Variable | Value | | | -------- | ----------- | - | | LANG | | | | LC_ALL | en_GB.UTF-8 | ✅ | | LANGUAGE | | | ### fastlane files:

`./fastlane/Fastfile`

```ruby # This file contains the fastlane.tools configuration # You can find the documentation at https://docs.fastlane.tools # # For a list of all available actions, check out # # https://docs.fastlane.tools/actions # # For a list of all available plugins, check out # # https://docs.fastlane.tools/plugins/available-plugins # # Uncomment the line if you want fastlane to automatically update itself # update_fastlane default_platform(:ios) platform :ios do lane :this_passes do scan( result_bundle: true ) end lane :this_fails do multi_scan( result_bundle: true ) end end ```

`./fastlane/Appfile`

```ruby # app_identifier("[[APP_IDENTIFIER]]") # The bundle identifier of your app # apple_id("[[APPLE_ID]]") # Your Apple Developer Portal username # For more information about the Appfile, see: # https://docs.fastlane.tools/advanced/#appfile ```

### fastlane gems | Gem | Version | Update-Status | | -------- | ------- | ------------- | | fastlane | 2.209.1 | ✅ Up-To-Date | ### Loaded fastlane plugins: | Plugin | Version | Update-Status | | --------------------------- | ------- | ------------- | | fastlane-plugin-test_center | 3.19.1 | ✅ Up-To-Date |

Loaded gems

| Gem | Version | | ------------------------------- | ------------ | | did_you_mean | 1.5.0 | | pathname | 0.1.0 | | set | 1.0.1 | | tsort | 0.1.0 | | rake | 13.0.6 | | rexml | 3.2.5 | | CFPropertyList | 3.0.5 | | public_suffix | 5.0.0 | | addressable | 2.8.1 | | artifactory | 3.0.15 | | atomos | 0.1.3 | | aws-eventstream | 1.2.0 | | aws-partitions | 1.621.0 | | aws-sigv4 | 1.5.1 | | jmespath | 1.6.1 | | aws-sdk-core | 3.135.0 | | aws-sdk-kms | 1.58.0 | | aws-sdk-s3 | 1.114.0 | | babosa | 1.0.4 | | bundler | 2.2.3 | | claide | 1.1.0 | | colored | 1.2 | | colored2 | 3.1.2 | | colorize | 0.8.1 | | highline | 2.0.3 | | commander | 4.6.0 | | declarative | 0.0.20 | | digest-crc | 0.6.4 | | unf_ext | 0.0.8.2 | | unf | 0.1.4 | | domain_name | 0.5.20190701 | | dotenv | 2.8.1 | | emoji_regex | 3.2.3 | | excon | 0.92.4 | | faraday-em_http | 1.0.0 | | faraday-em_synchrony | 1.0.0 | | faraday-excon | 1.1.0 | | faraday-httpclient | 1.0.1 | | multipart-post | 2.0.0 | | faraday-multipart | 1.0.4 | | faraday-net_http | 1.0.1 | | faraday-net_http_persistent | 1.2.0 | | faraday-patron | 1.0.0 | | faraday-rack | 1.0.0 | | faraday-retry | 1.0.3 | | ruby2_keywords | 0.0.5 | | faraday | 1.10.2 | | http-cookie | 1.0.5 | | faraday-cookie_jar | 0.0.7 | | faraday_middleware | 1.2.0 | | fastimage | 2.2.6 | | gh_inspector | 1.1.3 | | jwt | 2.4.1 | | memoist | 0.16.2 | | multi_json | 1.15.0 | | os | 1.1.4 | | signet | 0.17.0 | | googleauth | 1.2.0 | | httpclient | 2.8.3 | | mini_mime | 1.1.2 | | trailblazer-option | 0.1.2 | | uber | 0.1.0 | | representable | 3.2.0 | | retriable | 3.1.2 | | webrick | 1.7.0 | | google-apis-core | 0.7.0 | | google-apis-androidpublisher_v3 | 0.25.0 | | google-apis-playcustomapp_v1 | 0.10.0 | | google-apis-iamcredentials_v1 | 0.13.0 | | google-apis-storage_v1 | 0.17.0 | | google-cloud-env | 1.6.0 | | google-cloud-errors | 1.2.0 | | google-cloud-core | 1.6.0 | | google-cloud-storage | 1.39.0 | | json | 2.6.2 | | mini_magick | 4.11.0 | | naturally | 2.2.1 | | optparse | 0.1.1 | | plist | 3.6.0 | | rubyzip | 2.3.2 | | security | 0.1.3 | | simctl | 1.6.8 | | terminal-notifier | 2.0.0 | | unicode-display_width | 1.8.0 | | terminal-table | 1.8.0 | | tty-screen | 0.8.1 | | tty-cursor | 0.7.1 | | tty-spinner | 0.9.3 | | word_wrap | 1.0.0 | | nanaimo | 0.3.0 | | xcodeproj | 1.22.0 | | rouge | 2.0.7 | | xcpretty | 0.3.0 | | xcpretty-travis-formatter | 1.0.1 | | xctest_list | 1.2.1 | | fastlane-plugin-test_center | 3.19.1 |

*generated on:* **2022-08-25**

Sample project I created to demonstrate the issue, it's as clean as it can be https://github.com/AlexTrott/SampleApp-TestCenter-Issue I could be missing something obvious, but I would have expected just running multi-scan would work

AlexTrott commented 1 year ago

@lyndsey-ferguson so I've been looking into this, and I think I missed something obvious, these are the "Summaries from Scan" and I noticed that when it's called from multi scan result_bundle is removed, I'll keep looking for a solution, but if see this, and can point me in the next direction, that'd be great, I'll try to resolve this bug over the weekend!

Summary for scan from scan

+------------------------------------------------+-----------------------------------------------------+
|                                       Summary for scan 2.209.1                                       |
+------------------------------------------------+-----------------------------------------------------+
| result_bundle                                  | true                                                |
| project                                        | ./SampleApp.xcodeproj                               |
| scheme                                         | SampleApp                                           |
| skip_detect_devices                            | false                                               |
| ensure_devices_found                           | false                                               |
| force_quit_simulator                           | false                                               |
| reset_simulator                                | false                                               |
| disable_slide_to_type                          | true                                                |
| reinstall_app                                  | false                                               |
| clean                                          | false                                               |
| open_report                                    | false                                               |
| output_directory                               | ./fastlane/test_output                              |
| output_types                                   | html,junit                                          |
| buildlog_path                                  | ~/Library/Logs/scan                                 |
| include_simulator_logs                         | false                                               |
| xcodebuild_formatter                           | xcpretty                                            |
| output_remove_retry_attempts                   | false                                               |
| derived_data_path                              | ./derived_data                                      |
| should_zip_build_products                      | false                                               |
| output_xctestrun                               | false                                               |
| use_clang_report_name                          | false                                               |
| disable_concurrent_testing                     | false                                               |
| skip_build                                     | false                                               |
| slack_use_webhook_configured_username_and_icon | false                                               |
| slack_username                                 | fastlane                                            |
| slack_icon_url                                 | https://fastlane.tools/assets/img/fastlane_icon.png |
| skip_slack                                     | false                                               |
| slack_only_on_failure                          | false                                               |
| xcodebuild_command                             | env NSUnbufferedIO=YES xcodebuild                   |
| skip_package_dependencies_resolution           | false                                               |
| disable_package_automatic_updates              | false                                               |
| use_system_scm                                 | false                                               |
| number_of_retries                              | 0                                                   |
| fail_build                                     | true                                                |
| xcode_path                                     | /Applications/Xcode-13.4.0.app                      |
+------------------------------------------------+-----------------------------------------------------+

Summary for scan from multiscan

+------------------------------------------------+-----------------------------------------------------+
|                                       Summary for scan 2.209.1                                       |
+------------------------------------------------+-----------------------------------------------------+
| output_types                                   | html,junit                                          |
| project                                        | ./SampleApp.xcodeproj                               |
| scheme                                         | SampleApp                                           |
| skip_detect_devices                            | false                                               |
| ensure_devices_found                           | false                                               |
| force_quit_simulator                           | false                                               |
| reset_simulator                                | false                                               |
| disable_slide_to_type                          | true                                                |
| reinstall_app                                  | false                                               |
| clean                                          | false                                               |
| open_report                                    | false                                               |
| output_directory                               | ./fastlane/test_output                              |
| buildlog_path                                  | ~/Library/Logs/scan                                 |
| xcodebuild_formatter                           | xcpretty                                            |
| output_remove_retry_attempts                   | false                                               |
| derived_data_path                              | ./derived_data                                      |
| should_zip_build_products                      | false                                               |
| output_xctestrun                               | false                                               |
| use_clang_report_name                          | false                                               |
| disable_concurrent_testing                     | false                                               |
| build_for_testing                              | true                                                |
| slack_use_webhook_configured_username_and_icon | false                                               |
| slack_username                                 | fastlane                                            |
| slack_icon_url                                 | https://fastlane.tools/assets/img/fastlane_icon.png |
| skip_slack                                     | false                                               |
| slack_only_on_failure                          | false                                               |
| xcodebuild_command                             | env NSUnbufferedIO=YES xcodebuild                   |
| skip_package_dependencies_resolution           | false                                               |
| disable_package_automatic_updates              | false                                               |
| use_system_scm                                 | false                                               |
| number_of_retries                              | 0                                                   |
| fail_build                                     | true                                                |
| include_simulator_logs                         | false                                               |
| skip_build                                     | false                                               |
| xcode_path                                     | /Applications/Xcode-13.4.0.app                      |
+------------------------------------------------+-----------------------------------------------------+

lyndsey-ferguson commented 1 year ago

@AlexTrott I have had to remove the resultBundle item because back in the day, the scan action would use the default location. I was providing my own location....but now with the later versions of scan it needs that for the cache I believe....I have been working off and on on this, but I have not spent a lot of time on it.

AlexTrott commented 1 year ago

I think I'm close to the root of the issue, I think it's around the operations done here: https://github.com/lyndsey-ferguson/fastlane-plugin-test_center/blob/master/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb#L20-L22

I'll keep you posted and hopefully raise a PR for it

AlexTrott commented 1 year ago

Just adding some more information, incase anyone else wants to jump in, I've found that when this is called in fastlane

# path = ./fastlane/test_output/SampleApp.xcresult
Scan.cache[:result_bundle_path] = path

https://github.com/fastlane/fastlane/blob/master/scan/lib/scan/test_command_generator.rb#L247

Which is obviously wrong, as multi scan sets the name to be report.xcresult but this is just part of the issue.

The other part is when Scan.cache[:result_bundle_path] in the future, over at

# Scan.cache[:result_bundle_path] returns nil
      result_bundle_path = Scan.cache[:result_bundle_path]

      # Looks for xcresult file in derived data if not specifically set
      if result_bundle_path.nil?
        xcresults = find_xcresults_in_derived_data
        new_xcresults = xcresults - @xcresults_before_run

        if new_xcresults.size != 1
          UI.build_failure!("Cannot find .xcresult in derived data which is needed to determine test results. This is an issue within scan. File an issue on GitHub or try setting option `result_bundle: true`")
        end

        result_bundle_path = new_xcresults.first
        Scan.cache[:result_bundle_path] = result_bundle_path
      end

https://github.com/fastlane/fastlane/blob/master/scan/lib/scan/runner.rb#L196-L209

Which is why I'm seeing the error I posted above.

So just to try some things if I force result_bundle_path to be the path multiscan report.xcresult like this

result_bundle_path = "./fastlane/test_output/report.xcresult"

Which works for when all the test pass, but if multiple runs go through, it fails out at the end, which is interesting

hactar commented 1 year ago

We're seeing this too and have reverted to the old version of testcenter for now.

AlexTrott commented 1 year ago

@lyndsey-ferguson @hactar I think I've found a solution if you set the following variables

SCAN_DERIVED_DATA_PATH='./derived_data'
SCAN_RESULT_BUNDLE=true
SCAN_OUTPUT_DIRECTORY='./derived_data/Logs/Test'

Scan will be able to find the the generated .xcresult as right now the logic is the following:

    def find_xcresults_in_derived_data
      derived_data_path = Scan.config[:derived_data_path]
      return [] if derived_data_path.nil? # Swift packages might not have derived data

      xcresults_path = File.join(derived_data_path, "Logs", "Test", "*.xcresult")
      return Dir[xcresults_path]
    end

https://github.com/fastlane/fastlane/blob/ff41f3556f1baf8e700477e50f65be728780aa94/scan/lib/scan/runner.rb#L176-L182

So if it can't find the xcresult in the default scan location, it'll look in : derived_data_path/Logs/Test/*xcresult

So I wonder, what do you think, should I raise a PR for fastlane to support custom XCResult path, or should I raise a PR to update test_center to put the reports in derived_data_path and then after move them to a desirable location?

lyndsey-ferguson commented 1 year ago

Hello @AlexTrott , thank you for looking into this. You could do both. If you update multi_scan to set those env vars, then move the xcresult file to where the client requested it, that would fix the issue for people using a version of fastlane prior to your PR. You could also add a check in multi_scan for the version of fastlane where your PR with the custom xcresult path change was actually released.

piotrsmajek commented 1 year ago

@lyndsey-ferguson Is it possible to fix it in shortly time?

andyjohns commented 1 year ago

I seem to be encountering this issue as well, looking for a workaround. Setting environment variables as described above does not seem to help.

AlexTrott commented 1 year ago

I personally haven’t had chance to write a fix to help out, but my workaround seems to resolve the issue for my project.

@andyjohns you got an example of it not working? Or an error code you can copy and paste so I can see why it’s not working for you

andyjohns commented 1 year ago

Thanks @AlexTrott. Here's a snippet of my FastFile:

    multi_scan(
      scheme: scheme,
      workspace: "Example/#{workspace}",
      devices: ["iPhone 14 Pro"],
      code_coverage: true,
      clean: true,
      prelaunch_simulator: true, 
      pre_delete_cloned_simulators: true,
      quit_simulators: true,
      retry_test_runner_failures: true,
      output_directory: "fastlane/output",
      parallel_testrun_count: 1,
      collate_reports: false,
      try_count: 2,
      output_types: "junit, xcresult"
  )

I'm using Xcode 14. I'm using test_center version v3.19.1, but also have tried going back to v3.15.3 and get the same results. I've also tried Fastlane v2.210.1 and v2.210.0

Attached is part of my testing run with verbose logging enabled, I've search/replaced my actual product name to privatize.

Thanks!

multiscan-fails.txt

AlexTrott commented 1 year ago

@andyjohns I think I've found you're issue:

| derived_data_path                              | /Users/user/Library/Developer/Xcode/DerivedData/ThisProjectExample  |
|                                                | -foybrrinptpuemfpcacqzlwwkurh
| output_directory                               | /Users/user/Developer/this-project/fastlane/output/ThisProject  |
|                                                | ExampleTests-batch-1

You want scan to see these values as the following:

derived_data_path:  /Users/user/Library/Developer/Xcode/DerivedData/ThisProjectExample
output_directory: /Users/user/Library/Developer/Xcode/DerivedData/ThisProjectExample/Logs/Test

This is because since scan made a change to trainer, it's expectation on the .xcresult have changed, so where previous multi_scan could do it's thing, now scan is trying to finalise something and it looks for an .xcresult in the derived data location. Reference: https://github.com/fastlane/fastlane/blob/ff41f3556f1baf8e700477e50f65be728780aa94/scan/lib/scan/runner.rb#L176-L182

So if you set the following in your .env file

SCAN_DERIVED_DATA_PATH='./derived_data'
SCAN_RESULT_BUNDLE=true
SCAN_OUTPUT_DIRECTORY='./derived_data/Logs/Test'

or the following in multi_scan if you prefer

derived_data_path = 'fastlane/output/derived_data/' # This can be where ever you want, I keep it inside the folder because of CI workspace
multi_scan(
    scheme: scheme,
    workspace: "Example/#{workspace}",
    devices: ["iPhone 14 Pro"],
    code_coverage: true,
    clean: true,
    prelaunch_simulator: true, 
    pre_delete_cloned_simulators: true,
    quit_simulators: true,
    retry_test_runner_failures: true,
    derived_data_path: derived_data_path, # Add derived_data_path
    output_directory: "#{derived_data_path}/Logs/Test", # Appended /Logs/Tests/ as it's what fastlane expects 🙃
    parallel_testrun_count: 1,
    collate_reports: false,
    try_count: 2,
    output_types: "junit, xcresult"
)

I hope all this makes sense, but I think you may have tried my original solution, but not noticed you were overriding output_directory

andyjohns commented 1 year ago

Thanks for the help @AlexTrott. Unfortunately neither of those changes have helped. You can see in this bit of the log that the output_directory is set by multi_scan to include a subdirectory for the "batch":

DEBUG [2022-09-22 14:08:12.24]: Setting output_directory to /Users/user/Developer/this-project/derived_data/Logs/Test/ThisProjectExampleTests-batch-1

I believe that happens here:

https://github.com/lyndsey-ferguson/fastlane-plugin-test_center/blob/46deffc686bd589ee56cb82b2415de6919c1395e/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb#L88

Setting the batch_count, try_count, and parallel_testrun_count all to 1 does not seem to change this.

andyjohns commented 1 year ago

Ok, as soon as I post that, I realize it's running two batches too, even thought it's specifically set to run a single batch. Further, I see the first batch is the unit tests, the second batch is the UI tests. Perhaps I'll see if there's a way to run them as separate commands, as that may do it.

AlexTrott commented 1 year ago

@andyjohns ah man that sucks, I'll try to replicate it all better tomorrow, I don't think it's try count as I use that, I don't use batch though, so the issue could lie there.

For reference, due to my CICD setup I use only_testing and do splitting based logic on the previous runs junit timing

AlexTrott commented 1 year ago

@andyjohns just tried to replicate your issue, and I'm not seeing it on Xcode 13.4.1, going to download Xcode 14 and see if I can replicate it there

Cyberbeni commented 1 year ago

Gemfile:

gem "fastlane", :git => 'git@github.com:Cyberbeni/fastlane.git' , :branch => 'scan-result-bundle'

Pluginfile:

gem 'fastlane-plugin-test_center', :git => 'git@github.com:Cyberbeni/fastlane-plugin-test_center.git' , :branch => 'scan-result-bundle-path'

Patrick-iOS commented 1 year ago

We have the same issue. It would be great to get that fixed

piotrsmajek commented 1 year ago

@lyndsey-ferguson We have the same issue. We are blocked from upgrade fastlane version due this issue :(

jseidman95 commented 1 year ago

@lyndsey-ferguson same here

Cyberbeni commented 1 year ago

The PR in fastlane has been merged: https://github.com/fastlane/fastlane/commit/c5a95c21e5af2d2388f04d73836d4e649585fd30

So after the next fastlane release, this issue can be resolved as well. Anyone who wants to try out the fixes before that, just edit Gemfile/Pluginfile according to my previous comment.

jseidman95 commented 1 year ago

That's just for scan though or works for multi scan?

Cyberbeni commented 1 year ago

test_center passed in the -resultBundlePath through xcargs, which resulted in scan not finding it because it was looking for it at the default location, now we will set the new option: https://github.com/lyndsey-ferguson/fastlane-plugin-test_center/pull/387/files#diff-841f7fcfdc288929b9f92ec68ae08020eaa1bcbb47da6614b98e4baefc7d101cL182-R182

Cyberbeni commented 1 year ago

scan was originally using xcpretty (which parses the xcodebuild CLI output) for getting the test results, now they are using trainer (which parses the result bundle), that's why this wasn't a problem before.

lyndsey-ferguson / fastlane-plugin-test_center