search

lyndsey-ferguson / fastlane-plugin-test_center

🎯 The best fastlane plugin to understand and tame misbehaving iOS tests 🎉
MIT License
285 stars 66 forks source link

collating test_result bundles in Xcode 11 crashes #145

Closed jineshqa closed 5 years ago

jineshqa commented 5 years ago

New Issue Checklist

Issue Description

I am trying to run tests using multi_scan, at the end of parallel test run collate actions fail. Please see the output for more details.

Complete output when running fastlane, including the stack trace and command used
You can ⌘ + double-click on links to open them directly in your browser.
[11:22:21]: Deleting iPhone Xs 13 Clone 1 TestCenter::Helper::MultiScanManager::SimulatorHelper<70323043884920>
[11:22:27]: Deleting iPhone Xs 13 Clone 2 TestCenter::Helper::MultiScanManager::SimulatorHelper<70323043884920>
/Users/jdoshi/.gems/gems/fastlane-plugin-test_center-3.8.4/lib/fastlane/plugin/test_center/actions/collate_test_result_bundles.rb:97:in `collate_infoplist': [!] undefined method `zip' for nil:NilClass (NoMethodError)
    from /Users/jdoshi/.gems/gems/fastlane-plugin-test_center-3.8.4/lib/fastlane/plugin/test_center/actions/collate_test_result_bundles.rb:25:in `block in collate_bundles'
    from /Users/jdoshi/.gems/gems/fastlane-plugin-test_center-3.8.4/lib/fastlane/plugin/test_center/actions/collate_test_result_bundles.rb:23:in `foreach'
    from /Users/jdoshi/.gems/gems/fastlane-plugin-test_center-3.8.4/lib/fastlane/plugin/test_center/actions/collate_test_result_bundles.rb:23:in `collate_bundles'
    from /Users/jdoshi/.gems/gems/fastlane-plugin-test_center-3.8.4/lib/fastlane/plugin/test_center/actions/collate_test_result_bundles.rb:14:in `block in run'
    from /Users/jdoshi/.gems/gems/fastlane-plugin-test_center-3.8.4/lib/fastlane/plugin/test_center/actions/collate_test_result_bundles.rb:13:in `each'
    from /Users/jdoshi/.gems/gems/fastlane-plugin-test_center-3.8.4/lib/fastlane/plugin/test_center/actions/collate_test_result_bundles.rb:13:in `run'
    from /Users/jdoshi/.gems/gems/fastlane-plugin-test_center-3.8.4/lib/fastlane/plugin/test_center/helper/multi_scan_manager/report_collator.rb:127:in `collate_test_result_bundles'
    from /Users/jdoshi/.gems/gems/fastlane-plugin-test_center-3.8.4/lib/fastlane/plugin/test_center/helper/multi_scan_manager/report_collator.rb:24:in `collate'
    from /Users/jdoshi/.gems/gems/fastlane-plugin-test_center-3.8.4/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:204:in `collate_batched_reports_for_testable'
    from /Users/jdoshi/.gems/gems/fastlane-plugin-test_center-3.8.4/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:165:in `block in collate_batched_reports'
    from /Users/jdoshi/.gems/gems/fastlane-plugin-test_center-3.8.4/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:164:in `each'
    from /Users/jdoshi/.gems/gems/fastlane-plugin-test_center-3.8.4/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:164:in `collate_batched_reports'
    from /Users/jdoshi/.gems/gems/fastlane-plugin-test_center-3.8.4/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:146:in `run_test_batches'
    from /Users/jdoshi/.gems/gems/fastlane-plugin-test_center-3.8.4/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:53:in `run'
    from /Users/jdoshi/.gems/gems/fastlane-plugin-test_center-3.8.4/lib/fastlane/plugin/test_center/actions/multi_scan.rb:23:in `run'
    from /Users/jdoshi/.gems/gems/fastlane-2.131.0/fastlane/lib/fastlane/runner.rb:256:in `block (2 levels) in execute_action'
    from /Users/jdoshi/.gems/gems/fastlane-2.131.0/fastlane/lib/fastlane/actions/actions_helper.rb:50:in `execute_action'
    from /Users/jdoshi/.gems/gems/fastlane-2.131.0/fastlane/lib/fastlane/runner.rb:235:in `block in execute_action'
    from /Users/jdoshi/.gems/gems/fastlane-2.131.0/fastlane/lib/fastlane/runner.rb:227:in `chdir'
    from /Users/jdoshi/.gems/gems/fastlane-2.131.0/fastlane/lib/fastlane/runner.rb:227:in `execute_action'
    from /Users/jdoshi/.gems/gems/fastlane-2.131.0/fastlane/lib/fastlane/runner.rb:157:in `trigger_action_by_name'
    from /Users/jdoshi/.gems/gems/fastlane-2.131.0/fastlane/lib/fastlane/fast_file.rb:154:in `method_missing'
    from Fastfile:597:in `block (2 levels) in parsing_binding'
    from /Users/jdoshi/.gems/gems/fastlane-2.131.0/fastlane/lib/fastlane/lane.rb:33:in `call'
    from /Users/jdoshi/.gems/gems/fastlane-2.131.0/fastlane/lib/fastlane/runner.rb:49:in `block in execute'
    from /Users/jdoshi/.gems/gems/fastlane-2.131.0/fastlane/lib/fastlane/runner.rb:45:in `chdir'
    from /Users/jdoshi/.gems/gems/fastlane-2.131.0/fastlane/lib/fastlane/runner.rb:45:in `execute'
    from /Users/jdoshi/.gems/gems/fastlane-2.131.0/fastlane/lib/fastlane/lane_manager.rb:56:in `cruise_lane'
    from /Users/jdoshi/.gems/gems/fastlane-2.131.0/fastlane/lib/fastlane/command_line_handler.rb:36:in `handle'
    from /Users/jdoshi/.gems/gems/fastlane-2.131.0/fastlane/lib/fastlane/commands_generator.rb:108:in `block (2 levels) in run'
    from /Library/Ruby/Gems/2.3.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:178:in `call'
    from /Library/Ruby/Gems/2.3.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:153:in `run'
    from /Library/Ruby/Gems/2.3.0/gems/commander-fastlane-4.4.6/lib/commander/runner.rb:476:in `run_active_command'
    from /Users/jdoshi/.gems/gems/fastlane-2.131.0/fastlane_core/lib/fastlane_core/ui/fastlane_runner.rb:76:in `run!'
    from /Library/Ruby/Gems/2.3.0/gems/commander-fastlane-4.4.6/lib/commander/delegates.rb:15:in `run!'
    from /Users/jdoshi/.gems/gems/fastlane-2.131.0/fastlane/lib/fastlane/commands_generator.rb:349:in `run'
    from /Users/jdoshi/.gems/gems/fastlane-2.131.0/fastlane/lib/fastlane/commands_generator.rb:41:in `start'
    from /Users/jdoshi/.gems/gems/fastlane-2.131.0/fastlane/lib/fastlane/cli_tools_distributor.rb:119:in `take_off'
    from /Users/jdoshi/.gems/gems/fastlane-2.131.0/bin/fastlane:23:in `'
    from /Users/jdoshi/.gems/bin/fastlane:22:in `load'
    from /Users/jdoshi/.gems/bin/fastlane:22:in `
'

Environment

✅ fastlane environment ✅

### Stack

| Key                         | Value                                                          |
| --------------------------- | -------------------------------------------------------------- |
| OS                          | 10.14.5                                                        |
| Ruby                        | 2.3.7                                                          |
| Bundler?                    | false                                                          |
| Git                         | git version 2.19.0                                             |
| Installation Source         | ~/.gems/bin/fastlane                                           |
| Host                        | Mac OS X 10.14.5 (18F132)                                      |
| Ruby Lib Dir                | /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib |
| OpenSSL Version             | LibreSSL 2.6.5                                                 |
| Is contained                | false                                                          |
| Is homebrew                 | false                                                          |
| Is installed via Fabric.app | false                                                          |
| Xcode Path                  | /Applications/Xcode11.app/Contents/Developer/                  |
| Xcode Version               | 11.0                                                           |

### System Locale

| Variable | Value       |   |
| -------- | ----------- | - |
| LANG     | en_US.UTF-8 | ✅ |
| LC_ALL   |             |   |
| LANGUAGE |             |   |

### fastlane gems

| Gem      | Version | Update-Status |
| -------- | ------- | ------------- |
| fastlane | 2.131.0 | ✅ Up-To-Date  |

### Loaded fastlane plugins:

| Plugin                      | Version | Update-Status |
| --------------------------- | ------- | ------------- |
| fastlane-plugin-test_center | 3.8.4   | ✅ Up-To-Date  |


Loaded gems

| Gem                         | Version      |
| --------------------------- | ------------ |
| did_you_mean                | 1.0.0        |
| slack-notifier              | 2.3.2        |
| rouge                       | 2.0.7        |
| xcpretty                    | 0.3.0        |
| terminal-notifier           | 2.0.0        |
| terminal-table              | 1.8.0        |
| multipart-post              | 2.0.0        |
| word_wrap                   | 1.0.0        |
| public_suffix               | 2.0.5        |
| babosa                      | 1.0.2        |
| colored                     | 1.2          |
| highline                    | 1.7.10       |
| commander-fastlane          | 4.4.6        |
| http-cookie                 | 1.0.3        |
| faraday-cookie_jar          | 0.0.6        |
| gh_inspector                | 1.1.3        |
| mini_magick                 | 4.9.5        |
| multi_xml                   | 0.6.0        |
| security                    | 0.1.3        |
| xcpretty-travis-formatter   | 1.0.0        |
| naturally                   | 2.2.0        |
| simctl                      | 1.6.5        |
| jwt                         | 2.1.0        |
| uber                        | 0.1.0        |
| declarative                 | 0.0.10       |
| declarative-option          | 0.1.0        |
| representable               | 3.0.4        |
| retriable                   | 3.1.2        |
| mime-types                  | 3.2.2        |
| httpclient                  | 2.8.3        |
| google-api-client           | 0.23.9       |
| tty-cursor                  | 0.7.0        |
| tty-spinner                 | 0.9.1        |
| tty-screen                  | 0.7.0        |
| faraday                     | 0.15.4       |
| json                        | 2.2.0        |
| io-console                  | 0.4.5        |
| excon                       | 0.66.0       |
| rubyzip                     | 1.2.3        |
| CFPropertyList              | 3.0.1        |
| libxml-ruby                 | 2.9.0        |
| plist                       | 3.5.0        |
| atomos                      | 0.1.3        |
| colored2                    | 3.1.2        |
| nanaimo                     | 0.2.6        |
| xcodeproj                   | 1.12.0       |
| claide                      | 1.0.3        |
| google-cloud-env            | 1.2.1        |
| google-cloud-core           | 1.3.1        |
| digest-crc                  | 0.4.1        |
| google-cloud-storage        | 1.16.0       |
| memoist                     | 0.16.0       |
| multi_json                  | 1.13.1       |
| googleauth                  | 0.6.7        |
| unf                         | 0.1.4        |
| domain_name                 | 0.5.20190701 |
| faraday_middleware          | 0.13.1       |
| signet                      | 0.11.0       |
| fastimage                   | 2.1.5        |
| bundler                     | 2.0.2        |
| unicode-display_width       | 1.6.0        |
| addressable                 | 2.6.0        |
| os                          | 1.0.1        |
| psych                       | 2.1.0.1      |
| mime-types-data             | 3.2019.0331  |
| xctest_list                 | 1.1.8        |
| colorize                    | 0.8.1        |
| fastlane-plugin-test_center | 3.8.4        |



*generated on:* **2019-09-25**
lyndsey-ferguson commented 5 years ago

This works fine for Xcode 10?

jineshqa commented 5 years ago

Yes it does. Also, take a look at this as well.

https://github.com/fastlane/fastlane/issues/15138

lyndsey-ferguson commented 5 years ago

@jineshqa do you have a theory that they are somehow related? This issue that you created seems to indicate that there is a problem collating test result bundles. The issue that you reference fastlane/fastlane#15138 indicates that there is a crash after uploading to testflight.

lyndsey-ferguson commented 5 years ago

I saw that you commented in the issue. According to the stack trace of the crash, these are not related issues.

lyndsey-ferguson commented 5 years ago

As this fails in both Xcode 10 and Xcode 11, this is probably not related to the versions. I haven't changed anything in the collation of test result bundles of v3.8.4 vs. v.3.8.x.

You must have something in your project that is catching an edge case that I have not encountered before. I need either the test result bundles to see why the collation fails, or a sample project that you can submit freely that exhibits the problem.

🙏

jineshqa commented 5 years ago

Hey Lyndsey,

This does not fail on xcode 10.

Thanks, Jinesh

lyndsey-ferguson commented 5 years ago

Okay. Can you provide the test result bundles?

lyndsey-ferguson commented 5 years ago

Oh, and another question, if you remove the request to create test_result bundles, does it still crash?

jineshqa commented 5 years ago

I am not sure I can try.

jineshqa commented 5 years ago

Also, will get you sample project.

jineshqa commented 5 years ago

I am not sure I can try.

This is didn't help.

jineshqa commented 5 years ago

Also, will get you sample project.

Sample project is located here

Test output is attached. test_outout.zip

jineshqa commented 5 years ago

@lyndsey-ferguson is above info good enough.

lyndsey-ferguson commented 5 years ago

@jineshqa yes, I was able to fix the crash. I have a PR open.

Can you modify your Pluginfile per my instructions below, run bundle install, and then run your fastlane again (with the --verbose flag)?

Pluginfile:

gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-145-crash-when-collating-test-bundles"

If there are still problems, please let me know and attach the console output as a text file to this issue (makes it easier for me to review). If it works, please let me know.

lyndsey-ferguson commented 5 years ago

However, I do not believe that collation now works. The test summaries no longer exist and the tests failures and passes are no longer listed.

lyndsey-ferguson commented 5 years ago

It looks like there is a new format: https://github.com/fastlane-community/trainer/issues/32

lyndsey-ferguson commented 5 years ago

Ok, this looks like a lot of work.

lyndsey-ferguson commented 5 years ago

I'm going to change this issue to a Feature Request to support Xcode 11 format.

lyndsey-ferguson commented 5 years ago

Actually, I can issue a fix for now until test_result bundles can be collated with the new format.

lyndsey-ferguson commented 5 years ago

@jineshqa try running the branch I suggested above with multi_scan. It will use version 2 of the test result bundle path

jineshqa commented 5 years ago

This should have everything you need.

console-verbose.log console.log

lyndsey-ferguson commented 5 years ago

Please try again.

Delete your Gemfile.lock file, make sure that you have modified your Pluginfile per my instructions below, run bundle install, and then run your fastlane again (with the --verbose flag).

Pluginfile:

gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-145-crash-when-collating-test-bundles"

If there are still problems, please let me know and attach the complete console output as a text file to this issue (makes it easier for me to review). If it works, please let me know.

jineshqa commented 5 years ago

Monday Oct 7th EOD, promise.

lyndsey-ferguson commented 5 years ago

@jineshqa I got your console.log from the closed issue. To answer the question, no, there is an edge-case that I overlooked. Thanks for providing the log, super helpful.

console_verbose.log

lyndsey-ferguson commented 5 years ago

@jineshqa @szymon-kazmierczak I believe (:pray:) that I have fixed the issue.

Can you modify your Pluginfile per my instructions below, (if you already have, please delete your Gemfile.lock file), run bundle install, and then run your fastlane again (with the --verbose flag)?

Pluginfile:

gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-145-crash-when-collating-test-bundles"

If there are still problems, please let me know and attach the console output as a text file to this issue (makes it easier for me to review). If it works, please let me know.

lyndsey-ferguson commented 5 years ago

@jineshqa @szymon-kazmierczak any update?

jineshqa commented 5 years ago

console_verbose.log

I still see the error. I followed your instructions to the dot. Below is my fastlane env output if it helps.

🚫 fastlane environment 🚫 ### Stack | Key | Value | | --------------------------- | -------------------------------------------------------------- | | OS | 10.14.6 | | Ruby | 2.3.7 | | Bundler? | false | | Git | git version 2.19.0 | | Installation Source | ~/.gems/bin/fastlane | | Host | Mac OS X 10.14.6 (18G95) | | Ruby Lib Dir | /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib | | OpenSSL Version | LibreSSL 2.6.5 | | Is contained | false | | Is homebrew | false | | Is installed via Fabric.app | false | | Xcode Path | /Applications/Xcode11.app/Contents/Developer/ | | Xcode Version | 11.1 | ### System Locale | Variable | Value | | | -------- | ----------- | - | | LANG | en_US.UTF-8 | ✅ | | LC_ALL | | | | LANGUAGE | | | ### fastlane files:
`./fastlane/Fastfile` ```ruby # This file contains the fastlane.tools configuration # You can find the documentation at https://docs.fastlane.tools # # For a list of all available actions, check out # # https://docs.fastlane.tools/actions # # For a list of all available plugins, check out # # https://docs.fastlane.tools/plugins/available-plugins # # Uncomment the line if you want fastlane to automatically update itself # update_fastlane default_platform(:ios) platform :ios do lane :sweep do begin sh 'rm -rf ./test_output' rescue end test_run_block = lambda do |testrun_info| failed_test_count = testrun_info[:failed].size if failed_test_count > 0 UI.important('The run of tests would finish with failures due to fragile tests here.') try_attempt = testrun_info[:try_count] if try_attempt < 3 UI.header('Since we are using :multi_scan, we can re-run just those failing tests!') end end end result = multi_scan( try_count: 3, fail_build: false, scheme: 'collate errors', result_bundle: true, device: "iPhone 11 (13.1)", testrun_completed_block: test_run_block, output_directory: './fastlane/test_output', output_types: 'junit,html', buildlog_path: './fastlane/test_output/', code_coverage: true, invocation_based_tests: false, parallel_testrun_count: 2 ) unless result[:failed_testcount].zero? UI.message("There are #{result[:failed_testcount]} legitimate failing tests") end end end ```
`./fastlane/Appfile` ```ruby # app_identifier("[[APP_IDENTIFIER]]") # The bundle identifier of your app # apple_id("[[APPLE_ID]]") # Your Apple email address # For more information about the Appfile, see: # https://docs.fastlane.tools/advanced/#appfile ```
### fastlane gems | Gem | Version | Update-Status | | -------- | ------- | ------------- | | fastlane | 2.133.0 | ✅ Up-To-Date | ### Loaded fastlane plugins: | Plugin | Version | Update-Status | | --------------------------- | ------- | ------------------ | | fastlane-plugin-test_center | 3.8.5 | 🚫 Update available |
Loaded gems | Gem | Version | | --------------------------- | ------------ | | did_you_mean | 1.0.0 | | slack-notifier | 2.3.2 | | rouge | 2.0.7 | | xcpretty | 0.3.0 | | terminal-notifier | 2.0.0 | | terminal-table | 1.8.0 | | multipart-post | 2.0.0 | | word_wrap | 1.0.0 | | public_suffix | 2.0.5 | | colored | 1.2 | | highline | 1.7.10 | | commander-fastlane | 4.4.6 | | http-cookie | 1.0.3 | | faraday-cookie_jar | 0.0.6 | | gh_inspector | 1.1.3 | | mini_magick | 4.9.5 | | multi_xml | 0.6.0 | | rubyzip | 1.3.0 | | security | 0.1.3 | | xcpretty-travis-formatter | 1.0.0 | | jwt | 2.1.0 | | uber | 0.1.0 | | declarative | 0.0.10 | | declarative-option | 0.1.0 | | representable | 3.0.4 | | retriable | 3.1.2 | | httpclient | 2.8.3 | | google-api-client | 0.23.9 | | tty-cursor | 0.7.0 | | tty-spinner | 0.9.1 | | tty-screen | 0.7.0 | | faraday | 0.15.4 | | json | 2.2.0 | | io-console | 0.4.5 | | excon | 0.67.0 | | CFPropertyList | 3.0.1 | | libxml-ruby | 2.9.0 | | plist | 3.5.0 | | atomos | 0.1.3 | | colored2 | 3.1.2 | | nanaimo | 0.2.6 | | xcodeproj | 1.12.0 | | claide | 1.0.3 | | google-cloud-env | 1.2.1 | | google-cloud-core | 1.3.1 | | digest-crc | 0.4.1 | | google-cloud-storage | 1.16.0 | | memoist | 0.16.0 | | multi_json | 1.13.1 | | googleauth | 0.6.7 | | unf | 0.1.4 | | domain_name | 0.5.20190701 | | faraday_middleware | 0.13.1 | | signet | 0.11.0 | | babosa | 1.0.3 | | fastimage | 2.1.7 | | bundler | 2.0.2 | | unicode-display_width | 1.6.0 | | addressable | 2.7.0 | | os | 1.0.1 | | psych | 2.1.0.1 | | mime-types | 3.3 | | mime-types-data | 3.2019.1009 | | xctest_list | 1.1.8 | | colorize | 0.8.1 | | fastlane-plugin-test_center | 3.8.5 |
*generated on:* **2019-10-12**
jineshqa commented 5 years ago

My plugin file looks like this,

# Autogenerated by fastlane
#
# Ensure this file is checked in to source control!

gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-145-crash-when-collating-test-bundles"
lyndsey-ferguson commented 5 years ago

For some reason, it is failing to pre-delete xcresult files. When xcodebuild tries to create a new one, it fails because it already exists.

lyndsey-ferguson commented 5 years ago

Can you remove your Gemfile.lock file, bundle install, and then re-run your lane with the --verbose flag?

lyndsey-ferguson commented 5 years ago

I've added more logging

jineshqa commented 5 years ago

console_verbose.log

There you go captain. @lyndsey-ferguson

szymon-kazmierczak commented 5 years ago

@lyndsey-ferguson seems to be working for me! Btw, with multiscan parallel support my test run went down from 1 hour 20 minutes to 26 minutes with only 3 simulators 😂 Amazing work on the plugin 🤘

lyndsey-ferguson commented 5 years ago

@szymon-kazmierczak thank you for testing and confirming the fix. I truly appreciate that you shared the good news and the joy 😄.

Now I just have to figure out what is going on with @jineshqa 's system.

lyndsey-ferguson commented 5 years ago

@jineshqa can you please remove your Gemfile.lock, modify your Pluginfile per my instructions below, run bundle install, and confirm that the revision SHA for the fastlane-plugin-test_center is add995823a105d1b4b257a33fc7efaf92fa79f78?

The section in the Gemfile.lock file for the plugin should look like this:

GIT
  remote: https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git
  revision: add995823a105d1b4b257a33fc7efaf92fa79f78
  branch: issue-145-crash-when-collating-test-bundles
  specs:
    fastlane-plugin-test_center (3.8.5)
      colorize
      json
      plist
      xcodeproj
      xctest_list (>= 1.1.8)

Once you've confirmed that the Gemfile.lock is pointing to the correct source, run your fastlane again (with the --verbose flag).

Pluginfile:

gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-145-crash-when-collating-test-bundles"

If there are still problems, please let me know and attach the console output as a text file to this issue (makes it easier for me to review). If it works, please let me know.

jineshqa commented 5 years ago

Gemlock file screenshot

image

Pluginfile

gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-145-crash-when-collating-test-bundles"

Bundle list output image

Console log console_verbose.txt

lyndsey-ferguson commented 5 years ago

@jineshqa, the logs indicate that it was not run with the latest changes. For example, in the logs I see this:

[worker 2] DEBUG [2019-10-14 10:17:32.10]: Deleting xcresults:
[worker 2] DEBUG [2019-10-14 10:17:32.10]: Restarting Simulator D7EF0888

Where on my local, I see this:

[worker 1] DEBUG [2019-10-14 14:01:37.48]: About to check if we are using Xcdoe 11
[worker 1] DEBUG [2019-10-14 14:01:37.48]: Also looking in '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/AtomicBoyTests-batch-1' for xcresult bundles
[worker 1] DEBUG [2019-10-14 14:01:37.48]: Deleting xcresults:
[worker 1] DEBUG [2019-10-14 14:01:37.48]:   /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/AtomicBoyTests-batch-1/AtomicBoy.test_result.xcresult
[worker 1] DEBUG [2019-10-14 14:01:37.49]: Restarting Simulator EF4C5F39
lyndsey-ferguson commented 5 years ago

The difference above tells me that what you are running is not using the latest code.

lyndsey-ferguson commented 5 years ago

Is this running on CI and your bundle install was performed locally? CI must have the same Gemfile.lock and Pluginfile in order to get those changes.

jineshqa commented 5 years ago

@lyndsey-ferguson I am running it on local and I do bundle install manually. What you are suggesting is that somehow I am not running the latest code from the branch that you told me to add in plugin file? If yes, kindly tell me how to make sure that I am running the latest. I checked the SHA and it's same as you mentioned. Also, I uninstalled the fastlane-plugin-test_center gem before I ran bundle install just to be sure. What else should I be doing?

lyndsey-ferguson commented 5 years ago

Let's put a breakpoint in to see if it is the same code. I will assume that you have not done debugging of Ruby code before, so please be patient and follow these instructions (and thank you for your participation!!):

  1. Add gem 'pry-byebug' to your Gemfile
  2. Run bundle install
  3. In your Fastfile add this to the top: require 'pry-byebug'
  4. In your Fastfile, above the line where you call multi_scan add byebug.
  5. Remove the option to run parallel_testrun_count (otherwise your code will lock up waiting for you to respond to the debug console in another process that you do not have access to).
  6. Run your fastlane with the --verbose flag
  7. The code should stop and put you into a byebug debug session.
  8. Add a breakpoint in the method that deletes the xcresults by typing: 
    b ::TestCenter::Helper::MultiScanManager::RetryingScanHelper#delete_xcresults
  9. Type c to continue
    1. When the code stops at the breakpoint, copy the code and paste it here.

For example, it will look like this:

[35, 44] in /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/lib/fastlane/plugin/test_center/helper/multi_scan_manager/retrying_scan_helper.rb
   35:               `xcrun simctl boot #{udid} 2>/dev/null`
   36:             end
   37:           end
   38:         end
   39: 
=> 40:         def delete_xcresults
   41:           derived_data_path = File.expand_path(@options[:derived_data_path] || Scan.config[:derived_data_path])
   42:           xcresults = Dir.glob("#{derived_data_path}/Logs/Test/*.xcresult")
   43:           FastlaneCore::UI.verbose("About to check if we are using Xcdoe 11")
   44:           if FastlaneCore::Helper.xcode_at_least?('11.0.0')
  1. Type c to continue if your code looks like the above. Once a test-run completes, it will stop here again. Press n to follow the code to see what is going on here. If it does not stop here, we'll figure out what to do next.
lyndsey-ferguson commented 5 years ago

Please be sure to include the file path as I did. I want to make sure it is using the correct gem.

lyndsey-ferguson commented 5 years ago

One other thing, you do not have to uninstall the gem. But you do need to remove the Gemfile.lock file. Also, and this is important, are you running fastlane with the bundle exec command? It makes sure that the gems that are being used are the ones you specified in your Gemfile.

Very important

i.e.:

bundle exec fastlane <your_lane_name_here> --verbose
jineshqa commented 5 years ago

@lyndsey-ferguson Apologies for the delay. I am dealing with a stupid setup issue.

INFO [2019-10-15 16:26:51.12]: ▸ Loading...
INFO [2019-10-15 16:26:51.38]: ▸ Ignoring byebug-11.0.1 because its extensions are not built.  Try: gem pristine byebug --version 11.0.1
INFO [2019-10-15 16:26:51.64]: ▸ Could not find byebug-11.0.1 in any of the sources
INFO [2019-10-15 16:26:51.64]: ▸ Run `bundle install` to install missing gems.

Even though byebug is installed.

  ~ bundle list | grep 'byebug'
  * byebug (11.0.1)
  * pry-byebug (3.7.0)

Any idea?

lyndsey-ferguson commented 5 years ago

How are you calling it? This is what I would do if I were you:

  1. Add the pry-byebug gem to your Gemfile.
  2. Run bundle install
  3. Run bundle exec fastlane your_lane_name --verbose

This is when you've put the require 'pry-byebug' in your Fastfile right?

If you ran gem install pry-byebug that may not install it in the right place for when you run bundle exec fastlane ....

jineshqa commented 5 years ago

That is exactly what I am trying to do but when I execute bundle exec fastlane sweep --verbose command fails with the below error.


INFO [2019-10-15 16:26:51.12]: ▸ Loading...
INFO [2019-10-15 16:26:51.38]: ▸ Ignoring byebug-11.0.1 because its extensions are not built.  Try: gem pristine byebug --version 11.0.1
INFO [2019-10-15 16:26:51.64]: ▸ Could not find byebug-11.0.1 in any of the sources
INFO [2019-10-15 16:26:51.64]: ▸ Run `bundle install` to install missing gems.```
lyndsey-ferguson commented 5 years ago

Ah, did you already gem pristine byebug --version 11.0.1? Otherwise I would gem cleanup byebug; gem cleanup pry-byebug

jineshqa commented 5 years ago

Now my entire ruby installation is messed up. The default one on the OS.

lyndsey-ferguson commented 5 years ago

What did you do? I cannot imagine that running those commands would cause the problem.

Unless, you ran other commands with sudo. That would be very dangerous.

lyndsey-ferguson commented 5 years ago

I highly recommend using a virtual version manager such as rbenv. That way you will not break your system Ruby and you can install any version of Ruby that you want.