Job not fail on tests failure

Zaszczyk commented 4 years ago

I use this plugin and it is very well. However I cannot configure or it could be a bug: the job not fail on tests failure. I have such logs:

[13:04:48]: ▸ Test [matrix-xyz] has been created in the Google Cloud.
[13:04:48]: ▸ Firebase Test Lab will execute your instrumentation test on 2 device(s).
[13:04:48]: ▸ Creating individual test executions...
[13:05:00]: ▸ ...........................................................done.
[13:05:00]: ▸ Test results will be streamed to [https://console.firebase.google.com/project/xyz/testlab/histories/bh.xyz/matrices/xyz].
[13:11:02]: ▸ 13:11:02 Test matrix status: Finished:2
[13:11:02]: ▸ Instrumentation testing complete.
[13:11:02]: ▸ More details are available at [https://console.firebase.google.com/project/xyz/testlab/histories/bh.xyz/matrices/xyz].
[13:11:04]: Test status: [{"axis_value"=>"Nexus6P-23-pl_PL-portrait", "outcome"=>"Failed", "test_details"=>"1 test cases failed, 1 passed"}, {"axis_value"=>"Pixel2-28-en_US-portrait", "outcome"=>"Failed", "test_details"=>"1 test cases failed, 1 passed"}]
[13:11:04]: Finishing...

In logs I see tests failed, but the result of job is: passed. How can I configure this plugin, to fail job when tests not passed?

My lane"

lane :firebase_instrumentation_test do

        apkPath = "path"

    firebase_test_lab_android(
      project_id: "fitqbeapp",                                    # Your Firebase project name.
      gcloud_service_key_file: "#{ENV['GCP_KEY']}",         # File path containing the gcloud auth key.
      # gcloud_components_channel: "alpha",                            # If you use gcloud component channel option (alpha/beta).
      type: "instrumentation",                                                   # Optional: Test type (robo/instrumentation).
      devices: [                                                      # Devices
        {
          model: "Nexus6P",
          version: "23",
          locale: "pl_PL",
          orientation: "portrait"
        },
        {
          model: "Pixel2",
          version: "28"
        }
      ],
      app_apk: apkPath,                                       # The path for your android app apk.
      app_test_apk: "/app-debug-androidTest.apk",       # The path for your android instrumentation test apk.
      console_log_file_name: "fastlane/console_output.log",
      timeout: "30m",

    )
  end

Zaszczyk commented 4 years ago

I found fork, which implements job failure when tests not pass: https://github.com/cats-oss/fastlane-plugin-firebase_test_lab_android/compare/master...dubsmash:master there are commits from Dec 24, 2019. @cats-oss could you merge it to this repo?

ghost commented 3 years ago

It'll be really nice to fail the lane when the tests fail to avoid parsing the output and checking for "outcome =>Failure".

Now I'm reading the JSON output manually and failing the lane in case the tests have failed.

NinjaRat84 commented 3 years ago

I would love this feature as well, still up for merging the request?

ziadtawfeek commented 3 years ago

@cats-oss Thank you so much for this plugin 🙏 It has proved to be useful for me! Do you have any plans to merge this soon?

thereallukesimpson commented 3 years ago

Thanks for the great plugin, but it usefullness is limited when it does not trigger a failed pipeline on test failure. Is there anything I can do to help get this feature merged in?

The PR above https://github.com/cats-oss/fastlane-plugin-firebase_test_lab_android/pull/17 also contains the addition of the --num-unform-shards param which would also be super useful! Anyone know why it was closed?

cats-oss / fastlane-plugin-firebase_test_lab_android

Job not fail on tests failure #14