undefined method `value' for nil:NilClass

hactar commented 3 years ago

New Issue Checklist

[x] Updated fastlane-plugin-test_center to the latest version
[x] I read the README.md
[x] I reviewed the example(s) for the action(s) I am using
[x] I have removed any sensitive data such as passwords, authentication tokens, or anything else I do not want to world to see
[x] I have reviewed the Discussions forum to see my question has already been addressed.

Issue Description

multi_scan crashes when it encounters a junit file that does not include a "name" in its root xml object. I don't know why such a junit file was generated in the first place, I have attached it to this document. So multi_scan crashes when collating the tests. My guess is there are two things to do: 1) figure out why a "wrong" junit is generated in the first place and 2) make collate_junit_reports.rb more robust to ignore such junits in case they do occur.

Complete output when running fastlane, including the stack trace and command used

I'm not gonna include the full output but the parts I think are relevant, but can provide more if required:

Tail of the output:

[0;36;49m[worker 4] [0mDEBUG [2021-05-15 11:16:30.23]: ReportCollator collating
[0;36;49m[worker 4] [0mDEBUG [2021-05-15 11:16:30.23]: All tests passed for batch 4? true
DEBUG [2021-05-15 11:19:13.01]: Collating results for all batches
DEBUG [2021-05-15 11:19:13.01]: ReportCollator collating
DEBUG [2021-05-15 11:19:13.01]: Collating junit report files ["/Volumes/extra/jenkins/workspace/re_IOS-3692-testing-improvements/fastlane/test_output/BikemapTests-batch-2/report.junit", "/Volumes/extra/jenkins/workspace/re_IOS-3692-testing-improvements/fastlane/test_output/BikemapTests-batch-4/report.junit", "/Volumes/extra/jenkins/workspace/re_IOS-3692-testing-improvements/fastlane/test_output/BikemapTests-batch-3/report.junit", "/Volumes/extra/jenkins/workspace/re_IOS-3692-testing-improvements/fastlane/test_output/BikemapTests-batch-1/report.junit"]
DEBUG [2021-05-15 11:19:13.01]: collate_junit_reports with ["/Volumes/extra/jenkins/workspace/re_IOS-3692-testing-improvements/fastlane/test_output/BikemapTests-batch-2/report.junit", "/Volumes/extra/jenkins/workspace/re_IOS-3692-testing-improvements/fastlane/test_output/BikemapTests-batch-4/report.junit", "/Volumes/extra/jenkins/workspace/re_IOS-3692-testing-improvements/fastlane/test_output/BikemapTests-batch-3/report.junit", "/Volumes/extra/jenkins/workspace/re_IOS-3692-testing-improvements/fastlane/test_output/BikemapTests-batch-1/report.junit"]
WARN [2021-05-15 11:19:13.02]: [0;33;49mLane Context:[0m
INFO [2021-05-15 11:19:13.02]: {:DEFAULT_PLATFORM=>:ios, :PLATFORM_NAME=>:ios, :LANE_NAME=>"ios sweep"}
INFO [2021-05-15 11:19:13.02]: [0;32;49mSuccessfully generated documentation at path '/Volumes/extra/jenkins/workspace/re_IOS-3692-testing-improvements/fastlane/README.md'[0m

+------+----------------------------+-------------+
|                [0;32;49mfastlane summary[0m                 |
+------+----------------------------+-------------+
| Step | Action                     | Time (in s) |
+------+----------------------------+-------------+
| 1    | Verifying fastlane version | 0           |
| 2    | default_platform           | 0           |
| 💥   | [0;31;49mmulti_scan[0m                 | 638         |
+------+----------------------------+-------------+

DEBUG [2021-05-15 11:19:13.06]: All plugins are up to date
ERROR [2021-05-15 11:19:13.06]: [0;31;49mfastlane finished with errors[0m

Looking for related GitHub issues on fastlane/fastlane...
Search query: undefined method `value' for nil

URL: https://api.github.com/search/issues?q=undefined%20method%20%60value%27%20for%20nil+repo:fastlane/fastlane
➡️  [0;33;49mfastlane multi_scan: undefined method `length' for nil:NilClass[0m
    https://github.com/fastlane/fastlane/issues/18639 [[0;31;49mopen[0m] 0 💬
    a week ago

➡️  [0;33;49mUndefined method `status' for nil:NilClass (NoMethodError)[0m
    https://github.com/fastlane/fastlane/issues/17910 [[0;31;49mopen[0m] 33 💬
    3 weeks ago

➡️  [0;33;49mfastlane deliver init fails with undefined method `get_app_info_localizations' for nil:NilClass[0m
    https://github.com/fastlane/fastlane/issues/18565 [[0;31;49mopen[0m] 1 💬
    3 weeks ago

and 329 more at: https://github.com/fastlane/fastlane/search?q=undefined%20method%20%60value%27%20for%20nil&type=Issues&utf8=✓

🔗  You can ⌘ + double-click on links to open them directly in your browser.
DEBUG [2021-05-15 11:19:14.86]: All plugins are up to date
INFO [2021-05-15 11:19:14.86]: Deleting iPhone 8 Clone 1 TestCenter::Helper::MultiScanManager::SimulatorHelper<70244558031000>
INFO [2021-05-15 11:19:14.97]: Deleting iPhone 8 Clone 2 TestCenter::Helper::MultiScanManager::SimulatorHelper<70244558031000>
INFO [2021-05-15 11:19:15.08]: Deleting iPhone 8 Clone 3 TestCenter::Helper::MultiScanManager::SimulatorHelper<70244558031000>
INFO [2021-05-15 11:19:15.19]: Deleting iPhone 8 Clone 4 TestCenter::Helper::MultiScanManager::SimulatorHelper<70244558031000>
INFO [2021-05-15 11:19:15.30]: Shutting down ED6B82AD-C9F5-4EBC-9AEA-94D98BBB17C7
/Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/actions/collate_junit_reports.rb:13:in `block in run': [0;31;49m[!] undefined method `value' for nil:NilClass[0m (NoMethodError)
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/actions/collate_junit_reports.rb:13:in `map'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/actions/collate_junit_reports.rb:13:in `run'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/helper/multi_scan_manager/report_collator.rb:66:in `collate_junit_reports'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/helper/multi_scan_manager/report_collator.rb:28:in `collate'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:395:in `collate_batched_reports_for_testable'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:281:in `block in collate_batched_reports'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:280:in `each'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:280:in `collate_batched_reports'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:252:in `run_test_batches'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:110:in `block in run'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:137:in `block in setup_run_tests_for_each_device'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:126:in `each'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:126:in `each_with_index'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:126:in `setup_run_tests_for_each_device'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:108:in `run'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/actions/multi_scan.rb:38:in `run'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/runner.rb:263:in `block (2 levels) in execute_action'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/actions/actions_helper.rb:69:in `execute_action'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/runner.rb:255:in `block in execute_action'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/runner.rb:229:in `chdir'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/runner.rb:229:in `execute_action'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/runner.rb:157:in `trigger_action_by_name'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/fast_file.rb:159:in `method_missing'
    from Fastfile:156:in `block (2 levels) in parsing_binding'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/lane.rb:33:in `call'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/runner.rb:49:in `block in execute'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/runner.rb:45:in `chdir'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/runner.rb:45:in `execute'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/lane_manager.rb:47:in `cruise_lane'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/command_line_handler.rb:36:in `handle'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/commands_generator.rb:109:in `block (2 levels) in run'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/commander-4.6.0/lib/commander/command.rb:187:in `call'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/commander-4.6.0/lib/commander/command.rb:157:in `run'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/commander-4.6.0/lib/commander/runner.rb:444:in `run_active_command'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane_core/lib/fastlane_core/ui/fastlane_runner.rb:76:in `run!'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/commander-4.6.0/lib/commander/delegates.rb:18:in `run!'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/commands_generator.rb:353:in `run'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/commands_generator.rb:42:in `start'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/cli_tools_distributor.rb:122:in `take_off'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/bin/fastlane:23:in `'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/bin/fastlane:23:in `load'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/bin/fastlane:23:in `'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/bin/ruby_executable_hooks:24:in `eval'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/bin/ruby_executable_hooks:24:in `'
/Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/actions/collate_junit_reports.rb:13:in `block in run': undefined method `value' for nil:NilClass (NoMethodError)
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/actions/collate_junit_reports.rb:13:in `map'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/actions/collate_junit_reports.rb:13:in `run'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/helper/multi_scan_manager/report_collator.rb:66:in `collate_junit_reports'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/helper/multi_scan_manager/report_collator.rb:28:in `collate'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:395:in `collate_batched_reports_for_testable'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:281:in `block in collate_batched_reports'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:280:in `each'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:280:in `collate_batched_reports'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:252:in `run_test_batches'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:110:in `block in run'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:137:in `block in setup_run_tests_for_each_device'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:126:in `each'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:126:in `each_with_index'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:126:in `setup_run_tests_for_each_device'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:108:in `run'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-plugin-test_center-3.15.3/lib/fastlane/plugin/test_center/actions/multi_scan.rb:38:in `run'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/runner.rb:263:in `block (2 levels) in execute_action'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/actions/actions_helper.rb:69:in `execute_action'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/runner.rb:255:in `block in execute_action'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/runner.rb:229:in `chdir'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/runner.rb:229:in `execute_action'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/runner.rb:157:in `trigger_action_by_name'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/fast_file.rb:159:in `method_missing'
    from Fastfile:156:in `block (2 levels) in parsing_binding'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/lane.rb:33:in `call'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/runner.rb:49:in `block in execute'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/runner.rb:45:in `chdir'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/runner.rb:45:in `execute'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/lane_manager.rb:47:in `cruise_lane'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/command_line_handler.rb:36:in `handle'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/commands_generator.rb:109:in `block (2 levels) in run'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/commander-4.6.0/lib/commander/command.rb:187:in `call'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/commander-4.6.0/lib/commander/command.rb:157:in `run'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/commander-4.6.0/lib/commander/runner.rb:444:in `run_active_command'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane_core/lib/fastlane_core/ui/fastlane_runner.rb:76:in `run!'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/commander-4.6.0/lib/commander/delegates.rb:18:in `run!'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/commands_generator.rb:353:in `run'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/commands_generator.rb:42:in `start'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/fastlane/lib/fastlane/cli_tools_distributor.rb:122:in `take_off'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/gems/fastlane-2.183.0/bin/fastlane:23:in `'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/bin/fastlane:23:in `load'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/bin/fastlane:23:in `'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/bin/ruby_executable_hooks:24:in `eval'
    from /Users/bikemap/.rvm/gems/ruby-2.6.5/bin/ruby_executable_hooks:24:in `'

multi_scan settings:

+-------------------------+--------------------------------------------------+
|                [0;32;49mSummary for multi_scan (test_center v3.15.3)[0m                |
+-------------------------+--------------------------------------------------+
| try_count               | 3                                                |
| testrun_completed_block | # |
| parallel_testrun_count  | 4                                                |
+-------------------------+--------------------------------------------------+

ERROR [2021-05-15 11:08:34.86]: [1;34;49mUsing deprecated option: '--custom_report_file_name' (Use `--output_files` instead)[0m
WARN [2021-05-15 11:08:34.87]: [0;33;49mScanfile found: overriding multi_scan options with it's values.[0m
DEBUG [2021-05-15 11:08:34.87]: Building the project in preparation for multi_scan testing
ERROR [2021-05-15 11:08:34.88]: [1;34;49mUsing deprecated option: '--custom_report_file_name' (Use `--output_files` instead)[0m
INFO [2021-05-15 11:08:34.89]: [0;32;49mSuccessfully loaded '/Volumes/extra/jenkins/workspace/re_IOS-3692-testing-improvements/fastlane/Scanfile' 📄[0m

+-------------------+------------------------+
| Detected Values from './fastlane/Scanfile' |
+-------------------+------------------------+
| scheme            | Bikemap                |
| open_report       | false                  |
| devices           | ["iPhone 8"]           |
| clean             | false                  |
| derived_data_path | build/DerivedData      |
| output_types      | junit                  |
| xcargs            | -UseNewBuildSystem=YES |
+-------------------+------------------------+

How multi_scan is called:

lane :sweep do
  test_run_block = lambda do |testrun_info|
    sh("set-simulator-location -c 51.510279 -0.138947")

    failed_test_count = testrun_info[:failed].size

    if failed_test_count > 0
      UI.important('The run of tests would finish with failures due to fragile tests here.')

      try_attempt = testrun_info[:try_count]
      if try_attempt < 3
        UI.header("Using :multi_scan - rerunning failing tests #{try_attempt}... ")
      end
    end
  end

  result = multi_scan(
    workspace: MY_XCODE_WORKSPACE_FILEPATH,
    try_count: 3,
    fail_build: true,
    scheme: 'Bikemap',
    testrun_completed_block: test_run_block,
    parallel_testrun_count: 4,
  )
  unless result[:failed_testcount].zero?
    UI.message("There are #{result[:failed_testcount]} legitimate failing tests")
  end
end

The report.junit the plugin is choking on (note the other 3 junit files look ok, and this does not happen on every run, sometimes all junit files are ok):

<?xml version='1.0' encoding='UTF-8'?>
<testsuites tests='0' failures='0'/>

🚫 fastlane environment 🚫

### Stack

| Key                         | Value                                       |
| --------------------------- | ------------------------------------------- |
| OS                          | 11.3                                        |
| Ruby                        | 2.6.5                                       |
| Bundler?                    | true                                        |
| Git                         | git version 2.28.0                          |
| Installation Source         | ~/.rvm/gems/ruby-2.6.5/bin/fastlane         |
| Host                        | macOS 11.3 (20E232)                         |
| Ruby Lib Dir                | ~/.rvm/rubies/ruby-2.6.5/lib                |
| OpenSSL Version             | OpenSSL 1.1.1d  10 Sep 2019                 |
| Is contained                | false                                       |
| Is homebrew                 | false                                       |
| Is installed via Fabric.app | false                                       |
| Xcode Path                  | /Applications/Xcode.app/Contents/Developer/ |
| Xcode Version               | 12.5                                        |

### System Locale

| Error                       |
| --------------------------- |
| No Locale with UTF8 found 🚫 |

### fastlane gems

| Gem      | Version | Update-Status |
| -------- | ------- | ------------- |
| fastlane | 2.183.0 | ✅ Up-To-Date  |

### Loaded fastlane plugins:

| Plugin                      | Version | Update-Status |
| --------------------------- | ------- | ------------- |
| fastlane-plugin-test_center | 3.15.3  | ✅ Up-To-Date  |

Loaded gems

| Gem                             | Version      |
| ------------------------------- | ------------ |
| did_you_mean                    | 1.4.0        |
| rake                            | 13.0.3       |
| CFPropertyList                  | 3.0.3        |
| public_suffix                   | 4.0.6        |
| addressable                     | 2.7.0        |
| artifactory                     | 3.0.15       |
| atomos                          | 0.1.3        |
| aws-eventstream                 | 1.1.1        |
| aws-partitions                  | 1.456.0      |
| aws-sigv4                       | 1.2.3        |
| jmespath                        | 1.4.0        |
| aws-sdk-core                    | 3.114.0      |
| aws-sdk-kms                     | 1.43.0       |
| aws-sdk-s3                      | 1.94.1       |
| babosa                          | 1.0.4        |
| bundler                         | 2.1.4        |
| claide                          | 1.0.3        |
| colored                         | 1.2          |
| colored2                        | 3.1.2        |
| colorize                        | 0.8.1        |
| highline                        | 2.0.3        |
| commander                       | 4.6.0        |
| declarative                     | 0.0.20       |
| digest-crc                      | 0.6.3        |
| unf_ext                         | 0.0.7.7      |
| unf                             | 0.1.4        |
| domain_name                     | 0.5.20190701 |
| dotenv                          | 2.7.6        |
| emoji_regex                     | 3.2.2        |
| excon                           | 0.81.0       |
| faraday-excon                   | 1.1.0        |
| faraday-net_http                | 1.0.1        |
| faraday-net_http_persistent     | 1.1.0        |
| multipart-post                  | 2.0.0        |
| ruby2_keywords                  | 0.0.4        |
| faraday                         | 1.4.1        |
| http-cookie                     | 1.0.3        |
| faraday-cookie_jar              | 0.0.7        |
| faraday_middleware              | 1.0.0        |
| fastimage                       | 2.2.3        |
| gh_inspector                    | 1.1.3        |
| jwt                             | 2.2.3        |
| memoist                         | 0.16.2       |
| multi_json                      | 1.15.0       |
| os                              | 1.1.1        |
| signet                          | 0.15.0       |
| googleauth                      | 0.16.2       |
| httpclient                      | 2.8.3        |
| mini_mime                       | 1.1.0        |
| trailblazer-option              | 0.1.1        |
| uber                            | 0.1.0        |
| representable                   | 3.1.1        |
| retriable                       | 3.1.2        |
| rexml                           | 3.2.5        |
| webrick                         | 1.7.0        |
| google-apis-core                | 0.3.0        |
| google-apis-androidpublisher_v3 | 0.2.0        |
| google-apis-playcustomapp_v1    | 0.2.0        |
| google-apis-iamcredentials_v1   | 0.3.0        |
| google-apis-storage_v1          | 0.3.0        |
| google-cloud-env                | 1.5.0        |
| google-cloud-errors             | 1.1.0        |
| google-cloud-core               | 1.6.0        |
| google-cloud-storage            | 1.31.0       |
| json                            | 2.5.1        |
| mini_magick                     | 4.11.0       |
| naturally                       | 2.2.1        |
| plist                           | 3.6.0        |
| rubyzip                         | 2.3.0        |
| security                        | 0.1.3        |
| simctl                          | 1.6.8        |
| terminal-notifier               | 2.0.0        |
| unicode-display_width           | 1.7.0        |
| terminal-table                  | 1.8.0        |
| tty-screen                      | 0.8.1        |
| tty-cursor                      | 0.7.1        |
| tty-spinner                     | 0.9.3        |
| word_wrap                       | 1.0.0        |
| nanaimo                         | 0.3.0        |
| xcodeproj                       | 1.19.0       |
| rouge                           | 2.0.7        |
| xcpretty                        | 0.3.0        |
| xcpretty-travis-formatter       | 1.0.1        |
| trainer                         | 0.9.1        |
| xctest_list                     | 1.2.1        |
| fastlane-plugin-test_center     | 3.15.3       |


*generated on:* **2021-05-15**

lyndsey-ferguson commented 3 years ago

Hi @hactar; I won't be able to guess why this bad junit is being made. Have you reviewed it? Does it look valid? I see by the output that you did provide that you used the --verbose flag. I add a lot of output around junits and HTML reporting, do you see any clues there?

Are there any crashes in the test runner that you can see? Usually, full logs have more information before the problem that help me figure out what is going on, or at least let me add more logging around areas that I suspect are the culprit.

So, two things:

Can you provide the full logs? If you're worried about advertising sensitive data to the world, I the new issue does provide a way for you to encrypt the logs.
If you cannot provide the logs, give me a walkthrough of anything that you see in the logs before the problem: are there crashes in a test run? Is the junit valid? Are there any odd log outputs that indicate that bad data would be written to the logs?

hactar commented 3 years ago

Hey @lyndsey-ferguson, thanks, a few updates: I posted the junit already (BikemapTests-batch-3. The others, BikemapTests-batch-1, BikemapTests-batch-2 and BikemapTests-batch-4 look normal)

<?xml version='1.0' encoding='UTF-8'?>
<testsuites tests='0' failures='0'/>

Thats it, it does not contain anything other than this. I've gone deeper into the logs, and yes it appears that a test runner crashed during the run:

``` [0;36;49m[worker 4] [0mDEBUG [2021-05-15 11:14:32.45]: Batch #4 incrementing retry count to 1 [0;36;49m[worker 4] [0mDEBUG [2021-05-15 11:14:32.45]: Scan passed the tests for batch #4 [0;36;49m[worker 4] [0mINFO [2021-05-15 11:14:32.45]: [0;32;49m-----------------------------------------------------[0m [0;36;49m[worker 4] [0mINFO [2021-05-15 11:14:32.45]: [0;32;49mStep: set-simulator-location -c 51.510279 -0.138947[0m [0;36;49m[worker 4] [0mINFO [2021-05-15 11:14:32.45]: [0;32;49m-----------------------------------------------------[0m [0;36;49m[worker 4] [0mINFO [2021-05-15 11:14:32.45]: [0;36;49m$ set-simulator-location -c 51.510279 -0.138947[0m [0;36;49m[worker 4] [0mINFO [2021-05-15 11:14:32.77]: ▸ [0;35;49mSetting location to 51.510279 -0.138947[0m [0;36;49m[worker 4] [0mDEBUG [2021-05-15 11:14:32.77]: ReportCollator collating [0;36;49m[worker 4] [0mDEBUG [2021-05-15 11:14:32.77]: All tests passed for batch 4? true INFO [2021-05-15 11:14:32.79]: Starting test run 6 [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:36.29]: Deleting xcresults: [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:36.29]: Restarting Simulator 2FA81A51-DB84-4A34-BD97-FD0A2FB39353 [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.30]: retrying_scan #update_scan_options [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.30]: Setting workspace to /Volumes/extra/jenkins/workspace/re_IOS-3692-testing-improvements/Bikemap.xcworkspace [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.30]: Setting fail_build to true [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.30]: Setting scheme to Bikemap [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.30]: Setting output_types to junit [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.30]: Setting skip_detect_devices to false [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.30]: Setting ensure_devices_found to false [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.30]: Setting reset_simulator to false [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.30]: Setting disable_slide_to_type to true [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.30]: Setting prelaunch_simulator to true [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.30]: Setting reinstall_app to false [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.30]: Setting app_identifier to com.toursprung.bikemap [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.30]: Setting only_testing to ["BikemapTests/BMDeepLinkTest/testDiscover", "BikemapTests/BMDeepLinkTest/testMyRoutes", "BikemapTests/BMDeepLinkTest/testRecorded", "BikemapTests/BMDeepLinkTest/testSavedRoutes", "BikemapTests/BMDeepLinkTest/testRoutePlanner", "BikemapTests/BMDeepLinkTest/testPlannedRoutes", "BikemapTests/BMDeepLinkTest/testShareLocation", "BikemapTests/BMDeepLinkTest/testStartFreeRide", "BikemapTests/BMDeepLinkTest/testNavigationSettings"] [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.30]: Setting xctestrun to build/DerivedData/Build/Products/Bikemap_iphonesimulator14.5-x86_64.xctestrun [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.30]: Setting clean to false [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.30]: Setting open_report to false [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.30]: Setting output_directory to /Volumes/extra/jenkins/workspace/re_IOS-3692-testing-improvements/fastlane/test_output/BikemapTests-batch-3 [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.30]: Setting output_files to report.junit [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.31]: Setting buildlog_path to ~/Library/Logs/scan/worker-3-logs [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.31]: Setting include_simulator_logs to false [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.31]: Setting derived_data_path to /var/folders/k9/2frl09_n32vfltrncztxx4_m0000gn/T/derived_data_path20210515-3664-1vg76t8-worker-3 [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.31]: Setting should_zip_build_products to false [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.31]: Setting output_xctestrun to false [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.31]: Setting result_bundle to false [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.31]: Setting use_clang_report_name to false [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.31]: Setting disable_concurrent_testing to true [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.31]: Setting build_for_testing to false [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.31]: Setting xcargs to -UseNewBuildSystem=YES -parallel-testing-enabled NO [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.31]: Setting slack_use_webhook_configured_username_and_icon to false [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.31]: Setting slack_username to fastlane [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.31]: Setting slack_icon_url to https://fastlane.tools/assets/img/fastlane_icon.png [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.31]: Setting skip_slack to false [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.31]: Setting slack_only_on_failure to false [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.31]: Setting destination to ["platform=iOS Simulator,id=2FA81A51-DB84-4A34-BD97-FD0A2FB39353"] [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.31]: Setting xcodebuild_command to env NSUnbufferedIO=YES xcodebuild [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.31]: Setting skip_package_dependencies_resolution to false [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.31]: Setting disable_package_automatic_updates to false [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.31]: Setting use_system_scm to false [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.31]: Setting number_of_retries to 0 [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.31]: Setting skip_build to false [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.31]: Setting Scan.devices to ["iPhone 8 Clone 3 TestCenter::Helper::MultiScanManager::SimulatorHelper<70244558031000>"] [0;35;49m[worker 3] [0m [0;35;49m[worker 3] [0m+------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ [0;35;49m[worker 3] [0m| [0;32;49mSummary for scan 2.183.0[0m | [0;35;49m[worker 3] [0m+------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ [0;35;49m[worker 3] [0m| workspace | /Volumes/extra/jenkins/workspace/re_IOS-3692-testing-improvements/Bikemap.xcworkspace | [0;35;49m[worker 3] [0m| fail_build | true | [0;35;49m[worker 3] [0m| scheme | Bikemap | [0;35;49m[worker 3] [0m| output_types | junit | [0;35;49m[worker 3] [0m| skip_detect_devices | false | [0;35;49m[worker 3] [0m| ensure_devices_found | false | [0;35;49m[worker 3] [0m| force_quit_simulator | false | [0;35;49m[worker 3] [0m| reset_simulator | false | [0;35;49m[worker 3] [0m| disable_slide_to_type | true | [0;35;49m[worker 3] [0m| prelaunch_simulator | true | [0;35;49m[worker 3] [0m| reinstall_app | false | [0;35;49m[worker 3] [0m| app_identifier | com.toursprung.bikemap | [0;35;49m[worker 3] [0m| only_testing | ["BikemapTests/BMDeepLinkTest/testDiscover", "BikemapTests/BMDeepLinkTest/testMyRoutes", "BikemapTests/BMDeepLinkTest/testRecorded", "BikemapTests/BMDeepLinkTest/testSavedRoutes", "BikemapTests/BMDeepLinkTest/testRoutePlanner", "BikemapTests/BMDeepLinkTest/testPlannedRoutes", "BikemapTests/BMDeepLinkTest/testShareLocation", "BikemapTests/BMDeepLinkTest/testStartFreeRide", "BikemapTests/BMDeepLinkTest/testNavigationSettings"] | [0;35;49m[worker 3] [0m| xctestrun | build/DerivedData/Build/Products/Bikemap_iphonesimulator14.5-x86_64.xctestrun | [0;35;49m[worker 3] [0m| clean | false | [0;35;49m[worker 3] [0m| open_report | false | [0;35;49m[worker 3] [0m| output_directory | /Volumes/extra/jenkins/workspace/re_IOS-3692-testing-improvements/fastlane/test_output/BikemapTests-batch-3 | [0;35;49m[worker 3] [0m| output_files | report.junit | [0;35;49m[worker 3] [0m| buildlog_path | ~/Library/Logs/scan/worker-3-logs | [0;35;49m[worker 3] [0m| derived_data_path | /var/folders/k9/2frl09_n32vfltrncztxx4_m0000gn/T/derived_data_path20210515-3664-1vg76t8-worker-3 | [0;35;49m[worker 3] [0m| should_zip_build_products | false | [0;35;49m[worker 3] [0m| output_xctestrun | false | [0;35;49m[worker 3] [0m| use_clang_report_name | false | [0;35;49m[worker 3] [0m| disable_concurrent_testing | true | [0;35;49m[worker 3] [0m| xcargs | -UseNewBuildSystem=YES -parallel-testing-enabled NO | [0;35;49m[worker 3] [0m| slack_use_webhook_configured_username_and_icon | false | [0;35;49m[worker 3] [0m| slack_username | fastlane | [0;35;49m[worker 3] [0m| slack_icon_url | https://fastlane.tools/assets/img/fastlane_icon.png | [0;35;49m[worker 3] [0m| skip_slack | false | [0;35;49m[worker 3] [0m| slack_only_on_failure | false | [0;35;49m[worker 3] [0m| xcodebuild_command | env NSUnbufferedIO=YES xcodebuild | [0;35;49m[worker 3] [0m| skip_package_dependencies_resolution | false | [0;35;49m[worker 3] [0m| disable_package_automatic_updates | false | [0;35;49m[worker 3] [0m| use_system_scm | false | [0;35;49m[worker 3] [0m| number_of_retries | 0 | [0;35;49m[worker 3] [0m| include_simulator_logs | false | [0;35;49m[worker 3] [0m| skip_build | false | [0;35;49m[worker 3] [0m| xcode_path | /Applications/Xcode.app | [0;35;49m[worker 3] [0m| result_bundle | false | [0;35;49m[worker 3] [0m| build_for_testing | false | [0;35;49m[worker 3] [0m| devices | ["iPhone 8"] | [0;35;49m[worker 3] [0m+------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ [0;35;49m[worker 3] [0m [0;35;49m[worker 3] [0mINFO [2021-05-15 11:13:44.40]: Starting scan #1 with 9 tests for batch #3. [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:13:44.40]: Fetching available simulator devices [0;35;49m[worker 3] [0mINFO [2021-05-15 11:13:45.83]: Disabling 'Slide to Type' iPhone 8 Clone 3 TestCenter::Helper::MultiScanManager::SimulatorHelper<70244558031000> [0;35;49m[worker 3] [0mINFO [2021-05-15 11:13:45.83]: [0;36;49m$ /usr/libexec/PlistBuddy -c "Add :KeyboardContinuousPathEnabled bool false" /Users/bikemap/Library/Developer/CoreSimulator/Devices/2FA81A51-DB84-4A34-BD97-FD0A2FB39353/data/Library/Preferences/com.apple.keyboard.ContinuousPath.plist >/dev/null 2>&1[0m [0;35;49m[worker 3] [0mINFO [2021-05-15 11:13:45.91]: [0;36;49m$ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -destination 'platform=iOS Simulator,id=2FA81A51-DB84-4A34-BD97-FD0A2FB39353' -derivedDataPath /var/folders/k9/2frl09_n32vfltrncztxx4_m0000gn/T/derived_data_path20210515-3664-1vg76t8-worker-3 -disable-concurrent-testing -xctestrun 'build/DerivedData/Build/Products/Bikemap_iphonesimulator14.5-x86_64.xctestrun' -UseNewBuildSystem=YES -parallel-testing-enabled NO -only-testing:BikemapTests/BMDeepLinkTest/testDiscover -only-testing:BikemapTests/BMDeepLinkTest/testMyRoutes -only-testing:BikemapTests/BMDeepLinkTest/testRecorded -only-testing:BikemapTests/BMDeepLinkTest/testSavedRoutes -only-testing:BikemapTests/BMDeepLinkTest/testRoutePlanner -only-testing:BikemapTests/BMDeepLinkTest/testPlannedRoutes -only-testing:BikemapTests/BMDeepLinkTest/testShareLocation -only-testing:BikemapTests/BMDeepLinkTest/testStartFreeRide -only-testing:BikemapTests/BMDeepLinkTest/testNavigationSettings test-without-building | tee '/Users/bikemap/Library/Logs/scan/worker-3-logs/Bikemap-Bikemap.log' | xcpretty --report junit --output '/Volumes/extra/jenkins/workspace/re_IOS-3692-testing-improvements/fastlane/test_output/BikemapTests-batch-3/report.junit' --report junit --output '/var/folders/k9/2frl09_n32vfltrncztxx4_m0000gn/T/junit_report20210515-10273-w6yguz' [0m [0;35;49m[worker 3] [0mINFO [2021-05-15 11:13:45.91]: ▸ [0;35;49mLoading...[0m [0;35;49m[worker 3] [0mINFO [2021-05-15 11:15:13.08]: ▸ [0;35;49m2021-05-15 11:15:13.087 xcodebuild[10456:44374984] [MT] IDETestOperationsObserverDebug: 84.435 elapsed -- Testing started completed.[0m [0;35;49m[worker 3] [0mINFO [2021-05-15 11:15:13.08]: ▸ [0;35;49m2021-05-15 11:15:13.087 xcodebuild[10456:44374984] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start[0m [0;35;49m[worker 3] [0mINFO [2021-05-15 11:15:13.08]: ▸ [0;35;49m2021-05-15 11:15:13.087 xcodebuild[10456:44374984] [MT] IDETestOperationsObserverDebug: 84.435 sec, +84.435 sec -- end[0m [0;35;49m[worker 3] [0mINFO [2021-05-15 11:15:14.48]: ▸ [0;35;49mTesting failed:[0m [0;35;49m[worker 3] [0mINFO [2021-05-15 11:15:14.48]: ▸ [0;35;49m BikemapTests:[0m [0;35;49m[worker 3] [0mINFO [2021-05-15 11:15:14.48]: ▸ [0;35;49m The test runner encountered an error (Failed to establish communication with the test runner. If you believe this error represents a bug, please attach the result bundle at /var/folders/k9/2frl09_n32vfltrncztxx4_m0000gn/T/derived_data_path20210515-3664-1vg76t8-worker-3/Logs/Test/Test-Transient Testing-2021.05.15_11-13-48-+0200.xcresult. (Underlying Error: The operation couldn’t be completed. (DTXProxyChannel error 1.)))[0m [0;35;49m[worker 3] [0mINFO [2021-05-15 11:15:14.48]: ▸ [0;35;49m** TEST EXECUTE FAILED **[0m [0;35;49m[worker 3] [0m2021-05-15 11:15:13.087 xcodebuild[10456:44374984] [MT] IDETestOperationsObserverDebug: 84.435 elapsed -- Testing started completed. [0;35;49m[worker 3] [0m2021-05-15 11:15:13.087 xcodebuild[10456:44374984] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start [0;35;49m[worker 3] [0m2021-05-15 11:15:13.087 xcodebuild[10456:44374984] [MT] IDETestOperationsObserverDebug: 84.435 sec, +84.435 sec -- end [0;35;49m[worker 3] [0mTesting failed: [0;35;49m[worker 3] [0m BikemapTests: [0;35;49m[worker 3] [0m The test runner encountered an error (Failed to establish communication with the test runner. If you believe this error represents a bug, please attach the result bundle at /var/folders/k9/2frl09_n32vfltrncztxx4_m0000gn/T/derived_data_path20210515-3664-1vg76t8-worker-3/Logs/Test/Test-Transient Testing-2021.05.15_11-13-48-+0200.xcresult. (Underlying Error: The operation couldn’t be completed. (DTXProxyChannel error 1.))) [0;35;49m[worker 3] [0m [0;35;49m[worker 3] [0m** TEST EXECUTE FAILED ** [0;35;49m[worker 3] [0mERROR [2021-05-15 11:15:20.65]: [0;31;49mExit status: 65[0m [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:15:20.65]: Batch #3 incrementing retry count to 1 [0;35;49m[worker 3] [0mDEBUG [2021-05-15 11:15:20.65]: Scan unable to test for batch #3 ```

I took a look at Test-Transient Testing-2021.05.15_11-13-48-+0200.xcresult. but it does not appear to contain anything useful, see screenshot. The build log file that it is referencing does not exist.

Here's the full log, encrypted: build-multi-scan-enc.zip

hactar commented 3 years ago

Would a good workaround be adding something to collate_junit_reports.rb above line 13 which filters out xml files where the root object did not contain a .name value?

lyndsey-ferguson commented 3 years ago

@hactar that would be a workaround, but I wonder why the file doesn't exist?

lyndsey-ferguson commented 3 years ago

I have to move this to the backlog again, personal life has gotten too busy to try and pick this up.

hactar commented 3 years ago

So just to be clear, the file does exist, its just malformed: its not a full report, but only contains two lines because the test runner crashed:

<?xml version='1.0' encoding='UTF-8'?>
<testsuites tests='0' failures='0'/>

I tried the work around, but its not complete, because the tests that were supposed to run in that testrunner are never run or rerun, so multi_scan completes with the workaround, but because only 38 of the 47 tests or so where run, it marks the result as a failure.

We have found a different workaround, we have removed parallel_testrun_count from our multi_scan settings. By turning off the parallel test run we no longer have crashing test runners, and therefore the tests get run and rerun reliably - at the cost of parallelism.

lyndsey-ferguson commented 3 years ago

Sure, it is malformed, but we don't know why. That's what concerns me.

multi_scan uses the results of those junit files to determine which tests passed so it doesn't have to run them again. Without a valid junit, the program doesn't know.

Thanks for the note about the parallelism. There were problems when running tests in parallel and extra params were sent via a Scanfile that broke things: I believe I fixed them, but make sure that you're not using a Scanfile, or it is relatively empty to test if that is the problem.

lyndsey-ferguson commented 3 years ago

@hactar I'm trying to decrypt the log files that you sent me. When I run the command to decrypt the key that you encrypted with my public key, I am using this command:

openssl rsautl -decrypt -ssl -inkey ~/.ssh/id_rsa -in secret.txt.key.enc -out secret.txt.key

When I try to read the secret.txt.key, my computer tells me that it is a binary key (which it shouldn't).

Did you encrypt that key with my public key like this?

# download https://github.com/lyndsey-ferguson/fastlane-plugin-test_center/files/5577804/lyndsey-ferguson-id_rsa.pub.pkcs8.zip -> lyndsey-ferguson-id_rsa.pub
openssl rsautl -encrypt -pubin -inkey <my public key> -in secret.txt.key. -out secret.txt.key.enc

And if so, did you create the secret.txt.key and encrypt the log file with these commands?

hactar commented 3 years ago

The exact command I ran was this: openssl rsautl -encrypt -pubin -inkey lyndsey-ferguson-id_rsa.pub.pkcs8 -in secret.txt.key -out secret.txt.key.enc

I created the key using openssl aes-256-cbc -in build-multi-scan.txt -out build-multi-scan.txt.enc -pass file:secret.txt.key

Thanks for the tip with the Scanfile, we do still have a scanfile around, will see if removing it changes anything (the disabling of parallel testing is holding up though, the issue has not occured since)

tejassharma96 commented 3 years ago

Isn't it just the one generated by running build-without-testing? I'm having the same issue and wasn't able to resolve it by setting parallel test count to 1.

lyndsey-ferguson commented 2 years ago

@hactar, I have been working on making multi_scan use xcresult files instead of junit files, if you're interested in testing (alpha quality), follow the issue for the feature request

hactar commented 2 years ago

@lyndsey-ferguson Cool thanks! we're busy with our iOS 15 release but will test it once we find a moment.

tachtevrenidis commented 2 years ago

same issue here, trying the workaround

multi_scan(workspace: ENV["WORKSPACE"],
              devices: [DEFAULT_SIMULATOR],
              scheme: ENV["UI_TEST_SCHEME"],
              skip_slack: true,
              result_bundle: true,
              include_simulator_logs: false, # does not work in lab (causes Error 74)
              output_types: "junit",
              try_count: 5,
              # parallel_testrun_count: 3, /* DISABLING AS A WORK AROUND TO THIS ISSUE https://github.com/lyndsey-ferguson/fastlane-plugin-test_center/issues/339 */
              collate_reports: true)

lyndsey-ferguson commented 2 years ago

I want to apologize that I have not responded to this issue. A family health issue is requiring me to narrow my focus on the essentials and I don't have the time I need to focus on the plugin more than a minute. If you can get some other contributors to this project to help with this, I would consider merging a resulting PR.

lyndsey-ferguson / fastlane-plugin-test_center