search

allure-framework / allure-kotlin

Allure integrations for test frameworks targeting Kotlin and Java with 1.6 source compatibility.
Apache License 2.0
57 stars 21 forks source link

Error when running tests in Firebase Test Lab using Flank on API 30 and above #76

Open ramzyabushaaban opened 2 years ago

ramzyabushaaban commented 2 years ago

I'm submitting a ...

What is the current behavior?

When running the tests on my local emulator (API 30) I can see the Allure report files generated correctly with no problem, however, I get the below exception when running the same tests with the same configs in Firebase Test Lab using Flank (Pixel2 API 30)

androidx.test.services.storage.TestStorageException: No content provider registered for: content://androidx.test.services.storage.outputfiles/allure-results/72ee8821-c17d-4341-abb4-36ef6f1513cd-result.json. Are all test services apks installed? 07-04 17:57:18.561: E/TestRunner(12260): at androidx.test.services.storage.internal.TestStorageUtil.makeContentProviderClient(TestStorageUtil.java:108) 07-04 17:57:18.561: E/TestRunner(12260): at androidx.test.services.storage.internal.TestStorageUtil.getOutputStream(TestStorageUtil.java:88) 07-04 17:57:18.561: E/TestRunner(12260): at androidx.test.services.storage.TestStorage.openOutputFile(TestStorage.java:223) 07-04 17:57:18.561: E/TestRunner(12260): at androidx.test.services.storage.TestStorage.openOutputFile(TestStorage.java:203) 07-04 17:57:18.561: E/TestRunner(12260): at io.qameta.allure.android.writer.TestStorageResultsWriter$outputStreamResultsWriter$1.invoke(TestStorageResultsWriter.kt:16) 07-04 17:57:18.561: E/TestRunner(12260): at io.qameta.allure.android.writer.TestStorageResultsWriter$outputStreamResultsWriter$1.invoke(TestStorageResultsWriter.kt:15) 07-04 17:57:18.561: E/TestRunner(12260): at io.qameta.allure.kotlin.OutputStreamResultsWriter.write(OutputStreamResultsWriter.kt:23) 07-04 17:57:18.561: E/TestRunner(12260): at io.qameta.allure.android.writer.TestStorageResultsWriter.write(TestStorageResultsWriter.kt:20) 07-04 17:57:18.561: E/TestRunner(12260): at io.qameta.allure.kotlin.AllureLifecycle.writeTestCase(AllureLifecycle.kt:330) 07-04 17:57:18.561: E/TestRunner(12260): at io.qameta.allure.kotlin.junit4.AllureJunit4.testFinished(AllureJunit4.kt:60) 07-04 17:57:18.561: E/TestRunner(12260): at org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) 07-04 17:57:18.561: E/TestRunner(12260): at org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) 07-04 17:57:18.561: E/TestRunner(12260): at org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) 07-04 17:57:18.561: E/TestRunner(12260): at org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) 07-04 17:57:18.561: E/TestRunner(12260): at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) 07-04 17:57:18.561: E/TestRunner(12260): at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) 07-04 17:57:18.561: E/TestRunner(12260): at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) 07-04 17:57:18.561: E/TestRunner(12260): at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) 07-04 17:57:18.561: E/TestRunner(12260): at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) 07-04 17:57:18.561: E/TestRunner(12260): at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) 07-04 17:57:18.561: E/TestRunner(12260): at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) 07-04 17:57:18.561: E/TestRunner(12260): at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) 07-04 17:57:18.561: E/TestRunner(12260): at io.victoralbertos.device_animation_test_rule.DeviceAnimationTestRule$1.evaluate(DeviceAnimationTestRule.java:49) 07-04 17:57:18.561: E/TestRunner(12260): at org.junit.rules.RunRules.evaluate(RunRules.java:20) 07-04 17:57:18.561: E/TestRunner(12260): at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) 07-04 17:57:18.561: E/TestRunner(12260): at org.junit.runners.ParentRunner.run(ParentRunner.java:413) 07-04 17:57:18.561: E/TestRunner(12260): at org.junit.runners.Suite.runChild(Suite.java:128) 07-04 17:57:18.561: E/TestRunner(12260): at org.junit.runners.Suite.runChild(Suite.java:27) 07-04 17:57:18.561: E/TestRunner(12260): at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) 07-04 17:57:18.561: E/TestRunner(12260): at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) 07-04 17:57:18.561: E/TestRunner(12260): at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) 07-04 17:57:18.561: E/TestRunner(12260): at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) 07-04 17:57:18.561: E/TestRunner(12260): at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) 07-04 17:57:18.561: E/TestRunner(12260): at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) 07-04 17:57:18.561: E/TestRunner(12260): at org.junit.runners.ParentRunner.run(ParentRunner.java:413) 07-04 17:57:18.561: E/TestRunner(12260): at org.junit.runner.JUnitCore.run(JUnitCore.java:137) 07-04 17:57:18.561: E/TestRunner(12260): at org.junit.runner.JUnitCore.run(JUnitCore.java:115) 07-04 17:57:18.561: E/TestRunner(12260): at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:56) 07-04 17:57:18.561: E/TestRunner(12260): at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:395) 07-04 17:57:18.561: E/TestRunner(12260): at com.deliveryhero.pandora.test.PandoraTestRunner.onStart(PandoraTestRunner.kt:76) 07-04 17:57:18.561: E/TestRunner(12260): at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2205)

Please tell us about your environment:

My grade

testInstrumentationRunner = "io.qameta.allure.android.runners.AllureAndroidJUnitRunner" testInstrumentationRunnerArguments += mapOf("clearPackageData" to "true", "useTestStorageService" to "true")

I also use this property allure.results.useTestStorage=true

Flank Configs `gcloud: results-bucket: android-ta results-dir: 139727 record-video: true timeout: 30m async: false client-details: network-profile: null results-history-name: null

Android gcloud

  app: /bitrise/deploy/app.apk
  test: /bitrise/deploy/apptest.apk
  additional-apks:
  auto-google-login: false
  use-orchestrator: true
  directories-to-pull:
    - /sdcard/
  grant-permissions: all
  type: null
  other-files:
  scenario-numbers:
  scenario-labels:
  obb-files:
  obb-names:
  performance-metrics: false
  num-uniform-shards: null
  test-runner-class: null
  test-targets:
    - annotation de.foodora.android.util.CriticalTest
  robo-directives:
  robo-script: null
  device:
    - model: Pixel2
      version: 30
      locale: en_US
      orientation: portrait
  num-flaky-test-attempts: 1
  test-targets-for-shard:
  fail-fast: false
flank:
  max-test-shards: 8
  shard-time: 30
  num-test-runs: 1
  smart-flank-gcs-path: gs://android-ta/android_app_ta.xml
  smart-flank-disable-upload: false
  default-test-time: 120.0
  use-average-test-time-for-new-tests: false
  files-to-download:
  test-targets-always-run:
  disable-sharding: false
  project: [REDACTED]
  local-result-dir: results
  full-junit-result: false
  # Android Flank Yml
  keep-file-path: false
  additional-app-test-apks:
  run-timeout: -1
  legacy-junit-result: false
  ignore-failed-tests: false
  output-style: multi
  disable-results-upload: false
  default-class-test-time: 240.0
  disable-usage-statistics: false
  output-report: none
  skip-config-validation: false
  custom-sharding-json: `

| Allure version | 2.4.0 |

PrimankaDEN commented 1 year ago

I hope it would be helpful for someone. I think it is firebase problem. I faced with similar problem on microsoft appcenter. The reason is appcenter does not support orchestrator. This content provider is part of https://mvnrepository.com/artifact/androidx.test.services/test-services You should install orchestrator.apk and test-services.apk to run tests. When you run tests on your local emulator, gradle script does it for you, but appcenter does not