Add JUnit reporter

[pettero]

:pushpin: References

• Issue: https://github.com/pedrovgs/Shot/issues/345

:tophat: What is the goal?

Add JUnit test reports so that test results can be picked up by our pipeline. This also allows us to get statistics on failed screenshots to find flaky tests.

The main goal here is to get a list of failed tests. Azure Devops will get support for attachments in the future and then it would make sense to add failed screenshots in the test reports as well.

How is it being implemented?

Extracting the HTML reporter into a trait moving the implementation into to HTMLExecutionReporter. This move also allows us to add more ExecutionReporters to the project.

How can it be tested?

Run the tests, besides the html report there should also be a file called TEST-shot.xml that contains the test result in a JUnit compliant way.

How to test

The test error that are reported are:

• Screenshot not found
• Different screenshots
• Different image dimensions
• Other errors uses the same error message as the HTMLExecutionReporter

[pettero]

@pedrovgs any thoughts on this?

[pedrovgs]

Sorry for the late response @pettero . Let me enable the CI check first and later review the code. Thanks!

[pettero]

Hmm, any suggestions on what to do with the failing screenshots?

com.karumi.ui.view.CursorActivityTest.cursorIsNotVisibleOnScreenshot
com.karumi.ui.view.ScrollActivityTest.verticalScrollIsNotVisibleOnScreenshot
com.karumi.ui.view.ScrollActivityTest.horizontalScrollIsNotVisibleOnScreenshot

One thing that we have seen is that something changed around Christmas. The symptoms was that we had tests that were stable and then suddenly became unstable.

The tests became flaky were tests where we had interactions with the UI (clicks) and screenshots.

What happened was that we were starting to see buttons in a pressed state in the tests and starting failing.

We did not update shot at that time and we are running the tests in Firebase TestLab so my feeling is that something has changed with the emulator which makes the disableFlakyComponentsAndWaitForIdle not to be as effective as it has been.

[pettero]

@pedrovgs Now it looks like the tests are passing.

[pedrovgs]

Let's merge and release a new version then @pettero 😃

[pedrovgs]

@pettero can you please confirm version 6.1.0 containing junit reports feature is working as expected?

[jnepormosenofreenow]

Hi! I'm really interested in this feature. We just tested it in my team and apparently no XML report is generated. Running the following command at the root of the project shows the file isn't generated (unless I got something wrong, that seems to be the expected name):

find . -name 'TEST-shot.xml'

The HTML report is generated without issue, but not the XML one. I don't see any errors in the logs either.

I'm trying to find the issue, but I don't know enough Scala. If I can help in any way, please tell me and I'll do! Thanks!

[jnepormosenofreenow]

Hi, sorry for the tag @pedrovgs and @pettero 🙇 , but did you manage to test this? Using the latest version, it still doesn't generate the XML report for us.

I've gone over the code over and over and it doesn't seem like it requires any configuration. It should just work, but for some reason it doesn't. Did someone test it successfully or should I open another issue?

[pedrovgs]

@pettero can you please confirm the feature is not working for you either?

[sergio-sastre]

Hi, sorry for the tag @pedrovgs and @pettero 🙇 , but did you manage to test this? Using the latest version, it still doesn't generate the XML report for us.

I've gone over the code over and over and it doesn't seem like it requires any configuration. It should just work, but for some reason it doesn't. Did someone test it successfully or should I open another issue?

@jnepormosenofreenow @pedrovgs I can confirm the report is generated, but only after executing the verify task, under debug/verification (same location as the html report). When executing record, it doesn't.