search

bitrise-steplib / steps-virtual-device-testing-for-android

MIT License
22 stars 23 forks source link

coverage.ec not found in downloaded directory #67

Closed omuomugin closed 2 years ago

omuomugin commented 3 years ago

Troubleshooting

Useful information

Issue description

I need exec file from the android testing result but it turned out that no *.exec or *.ec file were included in $VDTESTING_DOWNLOADED_FILES_DIR directory.

According to the instruction guide from Firebase Test Lab it should generate artifacts/coverage.ec for coverage report with this command

gcloud firebase test android run \
  --type instrumentation \
  --app your-app.apk \
  --test your-app-test.apk \
  --device model=TestDevice,version=AndroidVersion  \
  --environment-variables coverage=true,coverageFile="/sdcard/coverage.ec" \ # I guess this is doing the trick
  --directories-to-pull /sdcard

see also https://firebase.google.com/docs/test-lab/android/command-line#code_coverage_reports_for_instrumentation_tests

as I read this discussion https://discuss.bitrise.io/t/collect-code-coverage-during-virtual-android-device-testing/3002/12 it may be worked for some time but not now.

my config is below

...
    - virtual-device-testing-for-android@1:
        is_skippable: true
        inputs:
        - test_devices: Pixel2,29,ja,portrait
        - directories_to_pull: "/sdcard/"
        - download_test_results: 'true'
        - environment_variables: |-
            coverage=true
            coverageFile=/sdcard/coverage.ec
        - test_type: instrumentation
...

when I run 

ls -R $VDTESTING_DOWNLOADED_FILES_DIR

I get 

Pixel2-29-ja-portrait_artifacts_5eb3c240877fe387.uid
Pixel2-29-ja-portrait_artifacts_hangouts_incoming_call.ogg
Pixel2-29-ja-portrait_artifacts_hangouts_message.ogg
Pixel2-29-ja-portrait_artifacts_sdcard_Android_data_com.google.android.youtube_cache_exo_5eb3c240877fe387.uid
Pixel2-29-ja-portrait_artifacts_sdcard_Android_media_com.google.android.talk_Notifications_hangouts_message.ogg
Pixel2-29-ja-portrait_artifacts_sdcard_Android_media_com.google.android.talk_Ringtones_hangouts_incoming_call.ogg
Pixel2-29-ja-portrait_bugreport.txt
Pixel2-29-ja-portrait_instrumentation.results
Pixel2-29-ja-portrait_logcat
Pixel2-29-ja-portrait_test_cases_0000_logcat
Pixel2-29-ja-portrait_test_cases_0001_logcat
Pixel2-29-ja-portrait_test_cases_0002_logcat
Pixel2-29-ja-portrait_test_cases_0003_logcat
Pixel2-29-ja-portrait_test_cases_0004_logcat
Pixel2-29-ja-portrait_test_result_1.xml
Pixel2-29-ja-portrait_video.mp4

and no *.exec or *.ec found

My question is

Bitrise info

Steps to reproduce

  1. set up [BETA] Virtual Device Testing for Android workflow
  2. set below as config 
    ...
- virtual-device-testing-for-android@1:
    is_skippable: true
    inputs:
    - test_devices: Pixel2,29,ja,portrait
    - directories_to_pull: "/sdcard/"
    - download_test_results: 'true'
    - environment_variables: |-
        coverage=true
        coverageFile=/sdcard/coverage.ec
    - test_type: instrumentation
...
  3. won't see any *.ec or *.exec file for coverage report
bitce commented 3 years ago

Hi @omuomugin! Sorry for the delay on your report, but thanks for raising it.

The issue is known as you suspected but the solution is not implemented yet unfortunately. We'll do our best to communicate the request and raise the priority of the problem as much as we can.

omuomugin commented 3 years ago

@bitce I got that 👍 Waiting for the issue to be fixed! or is there anything that I can do as an individual developer? I mean if the missing implementation is clear, maybe I can try fixing this ?

Thanks

Roland-Bak commented 3 years ago

Hello there @omuomugin 👋

Thank you for offering to help, it is very much appreciated. Unfortunately the developers have a lot on their hands meaning sometimes they need to prioritise in a way that some issues are attended to at later times.

But you can fork the step and create a Pull Request with a proposed solution, which is very welcome, as it would speed up the process of implementing a fix, so a win-win for everyone 🙂

carloswabreu1 commented 3 years ago

Hello there, any update on this issue?

Roland-Bak commented 3 years ago

Hello @carloswabreu1 👋

I checked in with the developers to see the status of this task, and there are still discussions on how to best approach this, but the process will take more time than expected. As soon as there is an update, we'll let you know! 🙂

carloswabreu1 commented 3 years ago

Thanks for the swift reply, @Roland-Bak. It's my understanding that this is a critical missing piece if we want to report test coverage coming from instrumented tests. Is there any other way we could be using e.g. the Sonarqube step to report test coverage that doesn't need this fix?

ofalvai commented 3 years ago

For those having this problem, please check if the logcat file downloaded from the test device contains an error like this:

06-25 02:14:42.416: E/CoverageListener(20344): Failed to generate Emma/JaCoCo coverage. 
06-25 02:14:42.416: E/CoverageListener(20344): java.lang.reflect.InvocationTargetException
06-25 02:14:42.416: E/CoverageListener(20344):  at java.lang.reflect.Method.invoke(Native Method)
06-25 02:14:42.416: E/CoverageListener(20344):  at androidx.test.internal.runner.listener.CoverageListener.generateCoverageReport(CoverageListener.java:101)
06-25 02:14:42.416: E/CoverageListener(20344):  at androidx.test.internal.runner.listener.CoverageListener.instrumentationRunFinished(CoverageListener.java:70)
06-25 02:14:42.416: E/CoverageListener(20344):  at androidx.test.internal.runner.TestExecutor.reportRunEnded(TestExecutor.java:92)
06-25 02:14:42.416: E/CoverageListener(20344):  at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:65)
06-25 02:14:42.416: E/CoverageListener(20344):  at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:395)
06-25 02:14:42.416: E/CoverageListener(20344):  at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2145)
06-25 02:14:42.416: E/CoverageListener(20344): Caused by: java.io.FileNotFoundException: /sdcard/coverage.ec (Permission denied)
06-25 02:14:42.416: E/CoverageListener(20344):  at java.io.FileOutputStream.open0(Native Method)
06-25 02:14:42.416: E/CoverageListener(20344):  at java.io.FileOutputStream.open(FileOutputStream.java:308)
06-25 02:14:42.416: E/CoverageListener(20344):  at java.io.FileOutputStream.<init>(FileOutputStream.java:238)
06-25 02:14:42.416: E/CoverageListener(20344):  at com.vladium.emma.rt.RT.dumpCoverageData(RT.java:50)
06-25 02:14:42.416: E/CoverageListener(20344):  ... 7 more

It's a common issue in my experience, the permission denied error means that the test application doesn't have permission to write to the sdcard folder. This Stackoverflow question has some solutions for granting that permission during the tests.

You might also run into problems with granting android.permission.WRITE_EXTERNAL_STORAGE because of the scoped storage enforcement in recent Android releases. Check this issue for a possible solution.

bitrise-coresteps-bot commented 3 years ago

Hello there, I'm a bot. On behalf of the community I thank you for opening this issue.

To help our human contributors focus on the most relevant reports, I check up on old issues to see if they're still relevant. This issue has had no activity for 90 days, so I marked it as stale.

The community would appreciate if you could check if the issue still persists. If it isn't, please close it. If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me".

If no comment left within 21 days, this issue will be closed.

bitrise-coresteps-bot commented 2 years ago

I'll close this issue as it doesn't seem to be relevant anymore. We believe an old issue probably has a bunch of context that's no longer relevant, therefore, if the problem still persists, please open a new issue.