search

istio / istio.io

Source for the istio.io site
https://istio.io/
Apache License 2.0
763 stars 1.52k forks source link

Produce coverage reports of doc tests #8352

Open ssuchter opened 3 years ago

ssuchter commented 3 years ago

We have a cool and novel testing system for technical content (e.g. bash commands) in our istio.io docs. It'd be valuable for guiding the community to ensure completeness if there was a CI system that repeatedly ran these doc tests, and produced not just pass/fail but also coverage reports.

I'm imagining both coverage in terms of files as well as "lines" in the files. (The lines wouldn't cover prose, but rather technical snippets.)

ssuchter commented 3 years ago

@justinpettit @nmittler @frankbu

justinpettit commented 3 years ago

I'm not sure if you're discussing coverage of the source code or the docs. Do you mean knowing what amount of the Istio source code is being executed by the doc test? Or what percentage of the doc code snippets are being run by the test. If the latter, we would hope 100%, but I think there's the potential to miss a snippet, but we could potentially catch that with a lint stage. I'd guess you mean the former, but just wanted to make sure.

ssuchter commented 3 years ago

Great question. I'm definitely talking about what percentage of the doc code snippets are being run by the test, and what percentage of docs with code snippets are being tested. I am not interested at all in trying to make sure that we cover all the Istio codebase with the doc tests, I wouldn't try to optimize that. I don't even know what I'd do with any number, aside from being worried if it was too high.

On Fri, Oct 23, 2020 at 4:20 PM Justin Pettit notifications@github.com wrote:

I'm not sure if you're discussing coverage of the source code or the docs. Do you mean knowing what amount of the Istio source code is being executed by the doc test? Or what percentage of the doc code snippets are being run by the test. If the latter, we would hope 100%, but I think there's the potential to miss a snippet, but we could potentially catch that with a lint stage. I'd guess you mean the former, but just wanted to make sure.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/istio/istio.io/issues/8352#issuecomment-715629416, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOYJW5FB6BYKYFSKTLMFWLSMIFVLANCNFSM4S475PXA .

frankbu commented 3 years ago

Aside from the percentage of lines that are covered in a test, which as @justinpettit said, we hope is 100%, we already have a couple of things to help understand the coverage:

  1. Doc pages have an indicator on the page to say if they are tested or not. See https://github.com/istio/istio.io/blob/master/README.md#testing-document-content

  2. You can run the command make test_status in the istio/istio.io root directory to get a report of what docs are/aren't tested. I suppose this could potentially be beefed up a little, run periodically, and the results published somewhere on the site.

Checking what lines are/aren't covered by a given test sounds like a reasonable addition, but there are several valid reasons why some tests will not be 100%. It wouldn't be bad to do this as a lint check, requiring the test author to explicitly annotate intentional exceptions so we can formally check that nothing has been accidentally missed by the test.