unindented / react-styleguidist-visual

Automated visual testing for React Styleguidist, using Puppeteer and pixelmatch.
MIT License
125 stars 23 forks source link

CI-friendly console output #32

Open kot-lex opened 6 years ago

kot-lex commented 6 years ago

It would be great if there were an ability to use custom test reporters. In our team we use Teamcity to run tests. It requires some extra console output so log could be recognized as tests.

I see two general ways of implementing this:

The first is to wrap entire module in some test framework like Mocha. In that case screenshots will be taken in before() hook and the comparison should look like individual test per screenshot. Pros: We could use full power of Mocha including all existing reporters Cons: Requires a lot of work

The second approach is in wrapping existing info, success, failure functions into single reporter object, call it defaultReporter and provide an ability to include custom user reporters via cli options.

Pros: Much easier to implement Won't break current architecture and backward compatibility Cons: Inability of using existing reporters A bit of Reinventing the wheel here.

unindented commented 6 years ago

The project is already using the debug package. If you run the tool with DEBUG=react-styleguidist-visual you'll get CI-friendly statements in stdout. If DEBUG or CI env variables are set, spinners won't show up.

If there are more statements that you think should be using debug, feel free to create a PR. But I'd totally avoid creating a custom logging solution.

kot-lex commented 6 years ago

debug is great in terms of log analysis, but it doesn't provide a way to add any custom output. I'm talking about providing an "which output format to use" option. It could be either XML for Jenkins, metadata for Teamcity or even HTML report for comparing screenshots side by side. It seems like the ideal solution would be to add some plugin system.

kot-lex commented 6 years ago

Btw, there is the third (probably the easiest) way besides the ones mentioned before: Update the test function so it will return an array with test results. That will allow us to create wrappers with reporting logic.

unindented commented 6 years ago

That third option sounds better than parsing logs. Feel free to open a PR.