Generate Implementation Dashboard

[laurentsenta]

Base Bricks

• [x] https://github.com/ipfs/gateway-conformance/pull/125
• [x] https://github.com/ipfs/gateway-conformance/pull/145 | 1 day
• [x] https://github.com/ipfs/gateway-conformance/pull/146
  • [x] Update workflows to expose their JSON outputs & retrieve them from gateway-conformance | 1 day
  • [x] Try in experiment repositories
  • [x] Implement the artifact-retrieval script
  • [x] Update client repositories
    • https://github.com/ipfs/kubo/pull/10092
    • https://github.com/ipfs/boxo/pull/442
    • https://github.com/ipfs/bifrost-gateway/pull/195

Phase 1: Example output

Generate Dashboard - Base Structure

We'll generate a dashboard in gh-pages, under a sub-path like project.io/dashboard.

• [x] Define the Dashboard sitemap | 1 day
• [x] https://github.com/ipfs/gateway-conformance/pull/150
  • [x] Output to gh-pages and setup a single URL | 1 day
  • [x] Generate the dashboard daily using outputs from other repos | 1 day
• [x] https://github.com/ipfs/gateway-conformance/pull/152 | 3 days
  • [x] /(index)
  • [x] /dashboard
  • [x] render gateway version
  • [x] render implementation
  • [x] render results
• [x] https://github.com/ipfs/gateway-conformance/pull/156 to be used in gateway checker | 1 day

Generate Dashboard - Drill Down

We'll let the user drill down test results, and link to test outputs / github action results

• [x] Create pages to drill-down tests results | 2 day
  • [x] per version
  • [x] per implementation
  • [x] per test / sub test / etc
• [x] Render the execution logs in tests | 1 day
• [x] https://github.com/ipfs/gateway-conformance/pull/166#issue-1898102599 | 1 day

Generate Dashboard - Specs Relationships

• [x] https://github.com/ipfs/gateway-conformance/pull/159
• [x] https://github.com/ipfs/gateway-conformance/pull/163
  • [x] Add https:// to the specs URL + add Specs field for multiple specs
  • [x] Create spec -> tests & tests -> spec indexes | 2 days

Implementation Dashboard - Going Public | 5 days

Generate Dashboard - 💅

• [ ] Export Hints in the test outputs as well
• [ ] If possible, add links to the test definition sources (it should be possible to extract the test line number and file, etc and create permalink to github's conformance source)
• [ ] Nicer Design | 1 day
• [ ] Edit Content | 2 day

Follow Ups

• [ ] fix the redirects for /tests and /results (not critical if we host at the root)
• [ ] Deal with matrix in client jobs (like https://github.com/ipfs/bifrost-gateway/actions/runs/5925256845)
• [ ] Validate specs (check the page and anchors exists)
• [ ] Generate a spec coverage table (detect how many specs pages + anchors we have and how much we test)

Related Discussions

https://github.com/ipfs/gateway-conformance/pull/116 https://github.com/ipfs/gateway-conformance/pull/116#pullrequestreview-1550977269

[BigLep]

Awesome project breakdown @laurentsenta !

[laurentsenta]

Updated with notes from @BigLep review's and discussion with @lidel + reorganized to support https://github.com/ipfs/public-gateway-checker/issues/442 more quickly.