New output generator: Test summary to file (or allow custom output generators)

testower commented 4 years ago

Use-cases

We are planning to use the fuzzy-tester to create smoke tests for our geocoder after building a new index. The current output generators are a bit limiting in terms of machine-analyzing the test results. Upon test failure, and rollback of our index, we would like to report to developers which tests failed, without making them read the logs of the container running the test. Ideally, this should be reported in a structured machine-readable way.

Attempted Solutions

The csv output is very limited (I don't understand how to make use of it), and the json output very verbose, and neither give succinct info about which tests are failing and succeeding in summary, like the terminal output does. We can of course enumerate json files to generate a test failure report, but still missing a summary like how many tests are in total, execution time etc.

Proposal

Create a new output generator which generates a machine-readable file with test results summaries.

An ok alternative could be to allow custom output generator, in a way similar to what eslint allows custom formatters: https://eslint.org/docs/developer-guide/working-with-custom-formatters

orangejulius commented 4 years ago

Hi @testower, Thanks for raising this issue, we definitely could use a round of improvements on the output on the fuzzy tester.

The CSV and JSON output generators have not been used much recently, and there are lots of inconsistencies and bugs in both of them.

Personally I think our best shot is to improve the JSON output generator and deprecate the CSV output generator (I've never used it and have no idea what it does or what it is supposed to do).

Some things that we could add to the JSON output generator that would all be relatively small and easy to tackle individually, but add up to a lot more usability:

Summary information, as you suggested
Allow running autocomplete mode (currently this is a bit of a hack and is implemented as an output generator, when it should be a separate flag)
Prune the output a bit so that more useful data is emphasized and irrelevant internal data from running the fuzzy-tester is omitted (as I recall, the JSON output basically prints the full state of the fuzzy-tester for a given test case, and is therefore quite verbose and more analogous to a core dump)

The core team would be happy to provide some guidance to anyone who wants to start working on some of this, and we can help out a bit ourselves as well!

CC @blackmad who has also recently been thinking about ways to better utilize the output of the fuzzy-tester, which would benefit from machine readable output as well.

missinglink commented 4 years ago

Something I've found useful in the past (and may not be relevant here but is worth considering) is that you can detect additional file descriptors opened on a process and log different data to each, if that makes sense to do.

https://github.com/pelias/interpolation/blob/master/stream/address/lookup.js#L8

testower commented 4 years ago

Hey thanks, sounds like we're on the same page. I'd be happy to contribute on this. Do you want to split this up in smaller issues or how to you want to proceed?

pelias / fuzzy-tester