Provide Global hooks RunListener after\before all tests (like testRunStarted, testRunFinished in RunListener)

[abay-ibrayev]

Is your feature request related to a problem? Please describe.

We are using custom implementation of org.junit.runner.notification.RunListener in our Android project.

In testRunStarted hook we create new test run in TestRail TMS (1 time for 1 test run), and then in testFinished hook we send to created test run - test result (success/fail)

Problem: on each separated test execution usingMarathon it creates separated test run, so testRunStarted executes for each test instead of executing it only 1 time

Describe the solution you'd like If possible to provide interface to work with this global test run hooks (before marathon test run start, and after finish)

[tagantroy]

Hello @ibrayevabay,

I don’t think that it is possible to provide hook for global runlistener. Probably it is better to add new or adapt existing API for this use case and integrate on test runner side. I was thinking about adding integration for https://reportportal.io/ long time ago, but abandoned my draft. @Malinskiy what do you think about adding support for external real-time reporting tools? Not sure if it will be easy to build integration tests for this integration :(

[Malinskiy]

The problem here is that running tests using marathon inherently means you have 1+ runs in terms of JUnit. So the before all tests will be called multiple times (e.g. you have 2 devices and there are 2 instances of JUnitCore that bootstraps running tests).

My suggestion is to implement the analytics solution on marathon's side: this is the only place where you have full visibility over the test run. The best solution for this is to implement it by mapping the execution result into the metrics that you want to send: General interface is called Reporter.

There is an example of this approach in AllureReporter. The idea is really simple: after all the tests are finished executing, all implementations of Reporter's are invoked. One of those implementations could be TestRail or whatever else you need in terms of reporting. Happy to work together on this.

[tagantroy]

My suggestion is to implement the analytics solution on marathon's side: this is the only place where you have full visibility over the test run. The best solution for this is to implement it by mapping the execution result into the metrics that you want to send: General interface is called Reporter..

Reporter interface doesn’t support realtime reporting. Should we come up with new API?

[Malinskiy]

We have a real-time interface, but allure reporter doesn’t need realtime tracking. Does this use-case need real-time reporting or after-test-run is okay @ibrayevabay ?

[abay-ibrayev]

We have a real-time interface, but allure reporter doesn’t need realtime tracking. Does this use-case need real-time reporting or after-test-run is okay @ibrayevabay ?

Hello, hmm, actually for my task I need real-time after-each-test hook interface with ability to access executed test's details. To explain it more clearly please look at example below:
We are using executed test's description to get custom annotation for each autotest in test method signature we have custom annotation @TestRail(id = "925") (something like this, and sending result directly to this test case id)

So I think ur solution using Reporter is not good for my task :c, but thanks anyway!

Actually I need only before-all-tests hook, and somehow send test run id for the each emulator's test execution instance

[Malinskiy]

@ibrayevabay if it’s the real-time reporting you’re after then you’ll have to extend the https://github.com/MarathonLabs/marathon/blob/5964ce78feb90c785b46ff44b1718f782c77d9b4/core/src/main/kotlin/com/malinskiy/marathon/analytics/internal/sub/TrackerInternalAdapter.kt and extend trackTest method. There already examples in the codebase that do this task for graphite, influx and influx2.

The only problem that you should be aware of is that these stores are accessed synchroniously to the test execution flow: the next test batch will not be executed until the API calls will finish. For internal TSDB storages that might not be a problem since latency there will be negligible. For external 3party reporting this might prove to be challenging and affect the testing time though. Please keep this in mind.

My suggestion would be to still use the approach outlined above is your best bet since it will allow you to batch send all the metrics for a particular test run (no idea if testrail supports this, but it should) instead of wasting round-trip for each individual test metric.

[abay-ibrayev]

Thanks for the comprehensive answer to my question, I will look at this solutions and also investigate if will be possible to use after-test-run hooks with TestRail API.