MarathonLabs / marathon

Cross-platform test runner
https://docs.marathonlabs.io
GNU General Public License v2.0
580 stars 121 forks source link

Allure integration and allure-results #356

Closed QAutomatron closed 3 years ago

QAutomatron commented 4 years ago

How does integration with allure work?

I see that an allure-results directory is created with reports inside, but they do not include steps or links from annotation like @TmsLink. Is there any configuration to enable these features or it is planned for development?

Malinskiy commented 4 years ago

Right now we generate the allure-results on the marathon's side. The annotations are parsed using the dex-test-parser and are populated in the report here https://github.com/Malinskiy/marathon/blob/develop/core/src/main/kotlin/com/malinskiy/marathon/report/allure/AllureReporter.kt

@TestLink specifically - https://github.com/Malinskiy/marathon/blob/develop/core/src/main/kotlin/com/malinskiy/marathon/report/allure/AllureReporter.kt#L103

Are you using the FQDN io.qameta.allure.TmsLink for your @TmsLink?

Malinskiy commented 4 years ago

We plan to support the on-device generation of allure-results, but this is a work in progress and requires several fixes from allure's side which are currently being worked on. If you would like to test these out - feel free to check the branch https://github.com/Malinskiy/marathon/tree/feature/allure-steps

QAutomatron commented 4 years ago

Got it, thanks for the quick reply.

Are you using the FQDN io.qameta.allure.TmsLink for your @TmsLink?

I used the latest Allure library for Android(which creates reports on the device), version 2.0.3 and the @TmsLink annotation from io.qameta.allure.android.annotations.TmsLink so maybe this is the issue. Will try to switch to "classic" Allure.

QAutomatron commented 4 years ago

Using io.qameta.allure:allure-java-commons instead of allure-android works for links.

But it would be good to implement the usage of allure-android (https://github.com/allure-framework/allure-android) and pulling results from the device.

Malinskiy commented 3 years ago

This (as in integratino with on-device allure framework) is already possible in 0.6.0 with the allure-kotlin support (allure-android is deprecated). See the docs https://malinskiy.github.io/marathon/ven/android.html#allure-kotlin-support

Closing the issue, feel free to reopen if you have more questions around the integration.

stMayhem commented 3 years ago

@Malinskiy Hello. Should marathon 0.6.1, include steps in the allure report after enabling allureConfiguration.enabled = true? I use Kaspresso+Marathon. Now I get reports like this:

538aa94b-7b57-4bd1-867d-1efa7ce186a1-result.txt [extension and file paths have been changed]

Malinskiy commented 3 years ago

@stMayhem Just want to confirm that you're using the file from the device and not marathon's own generated report, this is described in the docs:

Enabling this option effectively creates two allure reports for each test run:

one from the point of view of the marathon test runner one from the point of view of on-device test execution

All allure output from devices will be collected under $output/allure-device-results folder.

stMayhem commented 3 years ago

Yes, I’m using alure-device-results

Malinskiy commented 3 years ago

@stMayhem Is there something wrong during the file transfer from device to running host? If not then it's an issue for allure implementation and should be reported to the maintainer

stMayhem commented 3 years ago

I think the problem is that I am using kaspresso steps, with allure steps (io.qameta.allure.kotlin.Allure.step) it works fine.