RFC: fancier index with per-directory summaries and collapsible directory tree

[arvidj]

This MR makes the index.html page in the generated documentation easier to read for larger projects:

It adds:

1. points stats to each line (makes it easier to deduce the size of a file)
2. per-directory summary, with the average coverage of the directory and the (visited / total) points
3. collapsible sections per directory

I've built these three things on top of each other, so it'd be quite easy to cherry-pick parts of the proposal.

As it is, the MR is in a little bit of a mess, I'd like to check first if there is interest in such a proposal. Notably:

• I didn't make an effort to make the generated html nicely indented
• I haven't made an attempt to indent the OCaml code according to the project conventions 😬
• haven't yet tested on a larger project
• haven't adapted or added any tests

The last commit contains an example project (as in the image above), and a script for generating the report.

If you like it, I'll start cleaning up.

[aantron]

Thanks. It looks like a good improvement!

I'd request two additional things:

• Hiding the nested tree part of this behind an option, like bisect-ppx-report html --tree or similar. For small projects, the tree adds noise/confusion, especially if they have only one source directory. So, I think the PR should add the per-file statistics to all output, but format the output as a tree only for large projects, when asked to. If you'd like, I can help out with this.
• Adapting the CSS to the dark theme (you can force generation with the dark theme with bisect-ppx-report html --theme dark, or set your browser's theme to dark).

@ejgallego If you have time, could you try this on Coq? It's the only other really large project I know of that is using/considering Bisect_ppx (https://github.com/coq/coq/pull/12259).

[arvidj]

Hi Anton, thanks for your thoughtful comments!

Hiding the nested tree part of this behind an option, like bisect-ppx-report html --tree or similar. For small projects, the tree adds noise/confusion, especially if they have only one source directory. So, I think the PR should add the per-file statistics to all output, but format the output as a tree only for large projects, when asked to. If you'd like, I can help out with this.

Sure! I was thinking about it, and you could actually implement it as a theme. Then the markup would always be "tree-compatible" but depending on the theme you choose, you either see the "classic reports" or you get a report with collapsible directories. The disadvantage is that you would have to have 4 themes if you want (light / dark + classic / tree). The advantage is that there is no need to adapt the markup. On the other hand, programming in OCaml is nicer than programming in CSS.

Adapting the CSS to the dark theme (you can force generation with the dark theme with bisect-ppx-report html --theme dark, or set your browser's theme to dark).

Sure!

[aantron]

On the other hand, programming in OCaml is nicer than programming in CSS.

Indeed. It's probably best to actually emit different markup rather than do this with themes. Doing it with themes might get more and more difficult to maintain if the number of themes ever starts growing.

[arvidj]

Here's an example of a larger report: https://arvidj.eu/_tezos_coverage.2022-02-04/

[aantron]

Here's an example of a larger report: https://arvidj.eu/_tezos_coverage.2022-02-04/

Thanks. It's pretty clear that this PR's kind of feature is useful :)

[arvidj]

I've added the tree option and I've made an adaption for the dark theme.

What remains to do:

• make sure the tests pass
• play around with the javascript on the collapsible markers to make it more intuitive (although it's not very discoverable that shift-clicking the plus/minus signs does something different than just clicking).
• clean up the history. any specific thoughts about this or do you think it could pass as a single commit PR?

[aantron]

• clean up the history. any specific thoughts about this or do you think it could pass as a single commit PR?

No need to do this. I will squash this PR to one commit.

[arvidj]

Okay, I think this is ready for review now. I've fixed the existing test (I only had to modify one cram test) and I added a cram test for the html-tree.

For the open/close buttons, I've added two easter eggs:

• shift-clicking toggles the state of all immediate sub-foldres
• control-clicking toggles the state of all sub-folders I'm not sure it's intuitive, but it's useful once you know how it works.

[aantron]

Could you please fix the numbers so that there aren't underscores in the middle?

[arvidj]

Could you please fix the numbers so that there aren't underscores in the middle?

For sure, just did that. The reason I added it was to make large numbers a bit more readable, at the request of a pointy-haired boss hehe. An alternative would to use:

• spaces: 300 999/555 555 or
• dots: 300.999/555.555
• commas: 300,999/555,555

I'd prefer spaces in this case but I don't feel strongly about this (or the need to have any digit grouping whatsoever).

[aantron]

Part of the issue with these separators is they all either look strange (non-standard), or conflict with the various standard but local conventions for decimal and thousands delimiters, which differ from country to country. Given that Bisect is rarely run on files with thousands of instrumentation points, and even more rarely tens of thousands, but probably never millions, I think it's best to just not use separators. We can come back to this later, though, upon more feedback :)

[aantron]

Thank you for this PR!