Cannot view screenshots of failed tests

craig-jamieson commented 2 years ago

New Issue Checklist

[x] Updated fastlane-plugin-test_center to the latest version
[x] I read the README.md
[x] I reviewed the example(s) for the action(s) I am using
[x] I have removed any sensitive data such as passwords, authentication tokens, or anything else I do not want to world to see
[x] I have reviewed the Discussions forum to see my question has already been addressed.

begin
      multi_scan(
        workspace: "xcworkspace",
        scheme: "Stage",
        try_count: 2,
        result_bundle: true,
        parallel_testrun_count: 3,
        clean: clean,
        configuration: "Stage Debug",
        destination: "platform=iOS Simulator,OS=#{IOS_VERSION},id=CE009200-82BE-4C99-94C5-B97705398B6C",
        reset_simulator: true,
        include_simulator_logs: false,
        skip_slack: true,
        code_coverage: true,
        only_testing: ["UITests"],
        derived_data_path: DERIVED_DATA_FOLDER,
        output_directory: TEST_RESULTS_FOLDER,
        output_files: "report.html,junit.xml"
        )
    rescue => ex
      report_result = build_failing_tests_report(report_identifier: "Report ID", test_results_folder: TEST_RESULTS_FOLDER)
      total_tests_count = 0
      total_failed_tests = 0
      for failing_test_suite in report_result
        if failing_test_suite[:failures] == "0"
          next
        end
        e2e_error_message = ""
        class_name = ""
        total_tests_count += failing_test_suite[:tests].to_i
        total_failed_tests += failing_test_suite[:failures].to_i
        for test_summary in failing_test_suite[:total_tests]
          e2e_error_message += "#{test_summary[:test_name]} \n "
          class_name = test_summary[:class_name]
        end
        branch = git_branch
        final_error_message += "#{class_name} - #{e2e_error_message} \n"
      end
      final_error_message = "Branch - #{branch} \n \n #{final_error_message}"
      tests_failed = true
    end

    copy_artifacts(
      target_path: "artifacts/test-results",
      artifacts: ["#{TEST_RESULTS_FOLDER}/*.*"]
      )

    copy_artifacts(
      target_path: "e2e_tests/test-results/archive",
      artifacts: ["#{DERIVED_DATA_FOLDER}/Logs/Test/*.xcresult"]
      )

    zip(
      path: "e2e_tests/test-results/archive",
      output_path: "artifacts/test-results/archivedResults.zip",
      verbose: false
      )

    clear_derived_data(derived_data_path: DERIVED_DATA_FOLDER)
Environment
platform :ios do
  before_all do
    # Allow up to 6 attempts for starting xcodebuild
    ENV["FASTLANE_XCODEBUILD_SETTINGS_RETRIES"] = "5"

    # Prepare CircleCi configuration, including a usable Keychain
    setup_circle_ci

    # This uses the .xcode-version file in the project directory to ensure we
    # use a consistent Xcode version to ship and on the CI to test.
    # If upgrading to a new Xcode, change the `.xcode-version` file.
    ensure_xcode_version
  end

  import("./Fastfiles/Setup.rb")

  import("./Fastfiles/Testing.rb")  

  import("./Fastfiles/Release.rb")  

  import("./Fastfiles/EmergencyRelease.rb")    

  import("./Fastfiles/Utilities.rb")      

  import("./Fastfiles/Build.rb")

  # CircleCI exclusive lanes. Only imported if IS_CI_MACHINE
  if ENV["IS_CI_MACHINE"] == "TRUE" 
    import("./Fastfiles/CircleCI.rb")
  end  
end

# Path resolvers
def build_output_path
  File.expand_path("../#{BUILD_OUTPUT_FOLDER}")
end

def ipa_file_path
  File.join(build_output_path, IPA_FILENAME)
end

def dsyms_file_path
  File.join(build_output_path, DSYM_FILENAME)
end

```
`./fastlane/Appfile`
```ruby app_identifier "io.b2a.BankProd" # The bundle identifier of your app apple_id "itc@monzo.com" # Your Apple email address # For more information about the Appfile, see: # https://docs.fastlane.tools/advanced/#appfile ```

fastlane gems

Gem Version Update-Status

fastlane 2.188.0 🚫 Update available

Loaded fastlane plugins:

Plugin Version Update-Status

fastlane-plugin-aws_s3 2.0.2 🚫 Update available

fastlane-plugin-firebase_app_distribution 0.2.9 🚫 Update available

fastlane-plugin-versioning 0.4.5 🚫 Update available

fastlane-plugin-test_center 3.15.3 ✅ Up-To-Date

Loaded gems
| Gem | Version | | ----------------------------------------- | ------------ | | did_you_mean | 1.3.0 | | bundler | 2.1.4 | | rake | 13.0.6 | | CFPropertyList | 3.0.3 | | concurrent-ruby | 1.1.9 | | i18n | 1.8.10 | | minitest | 5.14.4 | | thread_safe | 0.3.6 | | tzinfo | 1.2.9 | | activesupport | 5.2.6 | | public_suffix | 4.0.6 | | addressable | 2.8.0 | | httpclient | 2.8.3 | | json | 2.5.1 | | algoliasearch | 1.27.5 | | rubyzip | 2.3.2 | | apktools | 0.7.4 | | artifactory | 3.0.15 | | atomos | 0.1.3 | | aws-eventstream | 1.1.1 | | aws-partitions | 1.479.0 | | aws-sigv4 | 1.2.4 | | jmespath | 1.4.0 | | aws-sdk-core | 3.117.0 | | aws-sdk-kms | 1.44.0 | | aws-sdk-s3 | 1.96.2 | | babosa | 1.0.4 | | claide | 1.0.3 | | colored2 | 3.1.2 | | cork | 0.3.0 | | nap | 1.1.0 | | open4 | 1.3.4 | | claide-plugins | 0.9.2 | | fuzzy_match | 2.0.4 | | netrc | 0.11.0 | | ffi | 1.15.3 | | ethon | 0.14.0 | | typhoeus | 1.4.0 | | cocoapods-core | 1.10.1 | | cocoapods-deintegrate | 1.0.4 | | cocoapods-downloader | 1.4.0 | | cocoapods-plugins | 1.0.0 | | cocoapods-search | 1.0.0 | | cocoapods-trunk | 1.5.0 | | cocoapods-try | 1.2.0 | | escape | 0.0.4 | | fourflusher | 2.3.1 | | gh_inspector | 1.1.3 | | molinillo | 0.6.6 | | ruby-macho | 1.4.0 | | nanaimo | 0.3.0 | | rexml | 3.2.5 | | xcodeproj | 1.20.0 | | cocoapods | 1.10.1 | | colored | 1.2 | | colorize | 0.8.1 | | highline | 2.0.3 | | commander | 4.6.0 | | faraday-em_http | 1.0.0 | | faraday-em_synchrony | 1.0.0 | | faraday-excon | 1.1.0 | | faraday-httpclient | 1.0.1 | | faraday-net_http | 1.0.1 | | faraday-net_http_persistent | 1.2.0 | | faraday-patron | 1.0.0 | | multipart-post | 2.0.0 | | ruby2_keywords | 0.0.5 | | faraday | 1.5.1 | | faraday-http-cache | 2.2.0 | | rchardet | 1.8.0 | | git | 1.8.1 | | kramdown | 2.3.1 | | kramdown-parser-gfm | 1.1.0 | | no_proxy_fix | 0.1.2 | | sawyer | 0.8.2 | | octokit | 4.21.0 | | unicode-display_width | 1.7.0 | | terminal-table | 1.8.0 | | danger | 8.2.3 | | thor | 0.20.3 | | danger-swiftlint | 0.24.5 | | declarative | 0.0.20 | | digest-crc | 0.6.4 | | unf_ext | 0.0.7.7 | | unf | 0.1.4 | | domain_name | 0.5.20190701 | | dotenv | 2.7.6 | | emoji_regex | 3.2.2 | | excon | 0.85.0 | | http-cookie | 1.0.4 | | faraday-cookie_jar | 0.0.7 | | faraday_middleware | 1.0.0 | | fastimage | 2.2.4 | | jwt | 2.2.3 | | memoist | 0.16.2 | | multi_json | 1.15.0 | | os | 1.1.1 | | signet | 0.15.0 | | googleauth | 0.16.2 | | mini_mime | 1.1.0 | | trailblazer-option | 0.1.1 | | uber | 0.1.0 | | representable | 3.1.1 | | retriable | 3.1.2 | | webrick | 1.7.0 | | google-apis-core | 0.4.1 | | google-apis-androidpublisher_v3 | 0.9.0 | | google-apis-playcustomapp_v1 | 0.5.0 | | google-apis-iamcredentials_v1 | 0.6.0 | | google-apis-storage_v1 | 0.6.0 | | google-cloud-env | 1.5.0 | | google-cloud-errors | 1.1.0 | | google-cloud-core | 1.6.0 | | google-cloud-storage | 1.34.1 | | mini_magick | 4.11.0 | | naturally | 2.2.1 | | plist | 3.6.0 | | security | 0.1.3 | | simctl | 1.6.8 | | terminal-notifier | 2.0.0 | | tty-screen | 0.8.1 | | tty-cursor | 0.7.1 | | tty-spinner | 0.9.3 | | word_wrap | 1.0.0 | | rouge | 2.0.7 | | xcpretty | 0.3.0 | | xcpretty-travis-formatter | 1.0.1 | | mime-types-data | 3.2021.0704 | | mime-types | 3.3.1 | | fastlane-plugin-aws_s3 | 2.0.2 | | fastlane-plugin-firebase_app_distribution | 0.2.9 | | trainer | 0.9.1 | | xctest_list | 1.2.1 | | fastlane-plugin-test_center | 3.15.3 | | fastlane-plugin-versioning | 0.4.5 | | mini_portile2 | 2.6.1 | | racc | 1.5.2 | | nokogiri | 1.12.5 | | xcode-build-times | 0.1.1 | | xcode-install | 2.6.2 |

Gem	Version	Update-Status
fastlane	2.188.0	🚫 Update available

Plugin	Version	Update-Status
fastlane-plugin-aws_s3	2.0.2	🚫 Update available
fastlane-plugin-firebase_app_distribution	0.2.9	🚫 Update available
fastlane-plugin-versioning	0.4.5	🚫 Update available
fastlane-plugin-test_center	3.15.3	✅ Up-To-Date

lyndsey-ferguson commented 2 years ago

Are you stating that the derived data directory is empty?

lyndsey-ferguson commented 2 years ago

Can you include the logs using the --verbose flag? This may help determine when that directory is being emptied.

craig-jamieson commented 2 years ago

@lyndsey-ferguson I'm having trouble using the verbose flag on our pipeline, but I've attached the logs from one of the runs, without using multi_scan we currently:

Set the derived data for the ui tests
Copy the test results file from the derived data into the test artefacts, and zip the xcresult file into a archivedResults file

However, now the archivedResults folder is empty, I think that perhaps the derived data file path is not being read correctly when using multi_scan

0.txt

lyndsey-ferguson commented 2 years ago

I noticed this in the logs: https://github.com/lyndsey-ferguson/fastlane-plugin-test_center/issues/179 can you try not using CircleCI (GitHub Actions maybe), or not setting parallel runners?

craig-jamieson commented 2 years ago

When I don't set parallel runners I can view the screenshots as expected, however, I haven't had an issue with the simulators being cloned on CircleCI, it's only when viewing the archivedResults

lyndsey-ferguson commented 2 years ago

It will be a while before I can look at this. Perhaps the collation of test results is wiping out the screenshots, can you add collate_reports: false as one of the options for multi_scan?

lyndsey-ferguson commented 2 years ago

I want to apologize that I have not responded to this issue. A family health issue is requiring me to narrow my focus on the essentials and I don't have the time I need to focus on the plugin more than a minute. If you can get some other contributors to this project to help with this, I would consider merging a resulting PR.

lyndsey-ferguson / fastlane-plugin-test_center

Cannot view screenshots of failed tests #361

New Issue Checklist

Environment

fastlane gems

Loaded fastlane plugins: