search

Xray-App / xray-junit-extensions

Enhanced JUnit 5 integration with Xray Test Management for Jira
Eclipse Public License 2.0
16 stars 8 forks source link

New xml results file created for every test when running with Orchestrator #60

Open joehoag opened 1 week ago

joehoag commented 1 week ago

Hello. I am attempting to use xray-junit-extensions for an Android project. When we enable Orchestrator, I think that each test is interpreted as an individual test plan, and so we get a separate XML results file for each test / test case. So if we ran 1000 tests, we would get 1000 xml results files.

When we disable Orchestrator, we get the expected result: a single XML results file that contains all test cases.

We need to have Orchestrator enabled. Can you give me any advice as to how to overcome the one-results-file-per-test-case problem?

bitcoder commented 1 week ago

What is Orchestrator? I'm not aware of it

joehoag commented 1 week ago

https://developer.android.com/training/testing/instrumented-tests/androidx-test-libraries/runner#use-android

Orchestrator clears out all app state before each test, and runs each test in its own instrumentation "sandbox". It is used to make sure that each test is run "fresh" and is not affected by leftover state from previous tests.

bitcoder commented 1 week ago

One thing that you could do is to merge the Junit XML reports together. I remember the junit-merge tool but I haven't tested in a while. There are other though. Maybe that addresses your goal. I have no experience with Orchestractor.

joehoag commented 1 week ago

Thanks. We would pursue a merging solution as a last resort. Right now I'm working on finding some sort of hook that tells me when all of my tests have completed running. With that, I could tweak EnhancedLegacyXmlReportGeneratingListener.java to only produce the results file once all tests have completed.

bitcoder commented 1 week ago

@joehoag if you find it, that would be great if it could generalized and eventually included on this project. Can you get a sample and simple project that showcases the current behaviour so I can eventually try to replicate it?