Mark-Simulacrum
commented
2 years ago I think that's one route, but I personally prefer that we just drop the all.html summary; the raw data is still going to be present in JSON (results.json) and in the all tarball; I think that's sufficient.
Also adjusted so that logs for everything present in the summary report are included.
This avoids a whole bunch of time writing logs for every crate out there. We could optimize the writing itself (e.g., doing it in parallel would likely be a huge win), but in practice there's no real need for each individual crate to have its own file in S3, and that has other costs that would be nice to avoid (e.g., compression is much less practical).
We can add support at a later point for the HTML reports to link to some server (or lambda, whatever) that processes the tarball and serves results from it. It takes around ~3 seconds to cat an all-crate zstd-compressed tarball, which takes up 400 MB. 3 seconds is probably too long for point queries to be viable on a re-download and decompress, but that kind of speed means that it's not infeasible for us to query on demand with some lightweight improvements to that process.