[Question] Reconsider copying required assets to monocart reporter directory

edumserrano commented 2 months ago

Question

Would you reconsider the current approach that monocart reporter has of referencing assets directly from Playwright's test output dir as opposed to copying the necessary assets to the monocart reporter output dir and then referencing them from there?

Reason

I've started a new Playwright tests project recently and I fell again into the problem I reported a long time ago in #34. For this new project I configured Playwright so that:

./test-results: is the Playwright output dir.
./test-reporters: is the output dir for all test reporters. I also have jest tests so even some jest reporters go into this directory. The structure of this directory looks something like:

root-dir/
└── test-reporters/
    ├── jest/
    │   ├── code-coverage
    │   ├── html-report
    │   └── junit-report
    └── playwright/
        ├── junit-reporter
        └── monocart-reporter

For the reasons discussed in #34 this configuration will mean that when serving the monocart html report from ./test-reporters/playwright/monocart-reporter the attachments for tests that fail visual comparisons are not shown because they are located outside the directory of the html file being served (the attachments are at ./test-results).

It took me a while till I remembered what the problem was. It also took me longer to remember because the first thing I tried was to use the built in html reporter and setting its output to ./test-reporters/playwright/html-reporter and everything worked.

The main difference of course is that the built-in HTML reporter copies everything it requires from the Playwright test output dir into the HTML reporter output dir. This way serving the html file from the HTML reporter output dir will always work regardless of where you choose to set the output dir to.

Conclusion

I know that in the past, in #34, you mentioned it would be a waste to copy the required assets from the Playwright test output dir to the monocart-reporter dir. I would like to ask you if you would be open to reconsider this?

Alternative small improvement for user experience

If you're against copying the assets to the monocart-reporter dir, could you consider at least adding a warning to the monocart-reporter to tell the user that his configuration is incorrect in cases where the directory of the html file for the monocart-reporter is not the same as Playwright test output dir?

This would be similar to how the built-in HTML reporter warns the user if the user tries to set the HTML reporter output dir to be inside the Playwright test output dir. If you try this you get this warning:

Configuration Error: HTML reporter output folder clashes with the tests output folder:

html reporter folder: \test-results\html-reporter test results folder: \test-results

HTML reporter will clear its output directory prior to being generated, which will lead to the artifact loss.

cenfun commented 2 months ago

Please try monocart-reporter@2.8.0 with new option copyAttachments=true

In fact, copyAttachments is set to true by default, which means all attachments are copied by default, thus providing better support for multiple playwright reports.

edumserrano commented 1 month ago

Hi @cenfun , sorry for my delayed response.

I tested this and the attachments are indeed copied by default. Thank you! 👏

cenfun / monocart-reporter