search

fediverse-devnet / feditest

A testing framework for distributed, heterogeneous systems communicating with complex protocols, such as the Fediverse
https://feditest.org/
MIT License
32 stars 6 forks source link

Create a full test run transcript in JSON #103

Closed jernst closed 4 months ago

jernst commented 4 months ago

Right now, we have to re-run a TestRun to format a report differently. That's not a good idea given how expensive TestRuns are. Also, reports generally don't show all information that was collected during a TestRun, and if a user decides they needed more detail than a particular report shows, they have to re-run the TestPlan.

Instead:

  1. During a TestRun, silently log everything.
  2. Create an optional output format that dumps all that was recorded into a JSON file.
  3. Create a sub-command that can parse such a JSON log and generate other forms of report from it, without re-running the tests.

There's also the advantage of being able to run several different reports from the same TestRun, like a web page and a summary.

steve-bate commented 4 months ago

Is this effectively the same as what I suggested in the feditest-tests-fediverses repo (CTRF)? This is what I was thinking when I entered that issue. Generate to JSON and then report formatting could be done repeatedly on that data.

https://github.com/fediverse-devnet/feditest-tests-fediverse/issues/60

jernst commented 4 months ago

Some people are a little slower on the uptake (me!) Yep!

In the one minute I looked at CTRF I figured we have a bunch of data that they don't really have fields for, in particular the constellations. So my thinking is we simply dump the object hierarchy that I'm creating internally as-is.

steve-bate commented 4 months ago

I tried doing a simple JSON serialization and there's an issue because the objects are not a hierarchy. They are a cyclic graph. I could write custom serialization code to address that issue, but I'd need to write a custom deserializer too (needed even with a simple hierarchy). There might be an general object-to-JSON mapper library somewhere, but I haven't searched very hard for it at this point.

I'm wondering if we should use the built-in pickle instead for this purpose. We dump the test results into a binary format (pickle) and then have a report subcommand for generating HTML with whatever template we choose.