[coverage] Coverage Measurements

[alphan]

Creating this issue to capture the plan for coverage measurements. This will be a long journey, so we are taking a pragmatic approach by limiting the scope and intentionally including some temporary workarounds to unblock the functionality we need sooner. After this we can think about additional enhancements such as differential coverage for PRs and local changes, dashboard, and other quality of life enhancements.

• [ ] Coverage measurements
  • [ ] On-target profile data
  • [x] Initial experiments, PoC, and foundation (#3080)
  • [ ] Enable coverage instrumentation using config settings and compiler flags as needed (#16947 , #16951)
    • This is a temporary measure to unblock progress until we can integrate our changes with bazel. Some resources: 1, 2, 3.
    • [x] Coverage runtime (#16762)
    • [ ] Coverage instrumentation (#16944 , #16951)
    • [x] Compiler flags for silicon creator targets (#16951)
    • [x] Coverage measurements from functional tests (#16951)
      • Measured coverage of (daily) commits between 2022-12-20 - 2022-12-31.
    • [ ] Coverage measurements from ROM E2E tests
      • This requires some investigation since instrumented binary does not fit in 32 KiB when built with -O0. We must support coverage with optimization and be careful about what we are instrumenting.
      • [ ] Reduce coverage instrumentation size overhead.
      • [ ] Transfer profile buffer before ROM hands over execution (either to ROM_EXT or shutdown).
  • [x] Coverage buffer transfer (#16762)
    • Mostly there, we should be more clear about endianness, increase buffer size, and break long lines to prevent buffer issues.
  • [x] Update vendored llvm_clang_rt_profile to llvmorg-13.0.1 (#16762)
    • We collect raw profile data from our target and there are no cross-version compatibility guarantees.
  • [x] Parse, index, and merge raw profile data in test logs (#16951)
  • [x] Remove vendored repo and use bazel to pull llvm_clang_rt_profile (#16973)
    • llvm-project releases a smaller archive that includes the part we need.
  • [x] Off-target profile data (unit tests)
  • ~Bazel should support this out of the box. We can use this to unblock CI integration and work on it in parallel.~
  • ~Depending on the profile data format that bazel uses we may need to convert to/from llvm profile data.~
  • Bazel has only primitive support for clang's source-based code coverage. We need to merge the profile data, process object files and generate reports.
  • [x] bazel config changes for coverage measurements using clang (#16891)
  • [x] build fixes for clang (#16890)
  • [x] Add clang-{lowRISC RISC-V toolchain version} to opentitan container Dockerfile (#16888).
    • lowRISC RISC-V toolchain version is 13.0.1.
  • [x] Determine the libraries that we are interested in (mostly silicon_creator modules, drivers, and rom code). (#16891)
  • [ ] Reports
  • See here.
  • [ ] On-target
    • [x] Functests (silicon creator modules and drivers)
    • [ ] ROM e2e tests
    • Also consider gdb-based e2e tests. Find out if it's possible to measure coverage in asm files.
    • [ ] Merged on-target coverage measurements
  • [x] Off-target unit tests
  • [x] Merged on-/off-target coverage measurements
  • [ ] Additional columns in the merged report with data from unit/func/e2e.
  • [ ] Additional reports such as mock and *_unittest.cc coverage.
  • [x] ~Refactor current implementation for multiple coverage configs, e.g. queries in hjson logic in py.~
    • I decided not to pursue this path in favor of better bazel integration because it turns into a "workflow orchestration" project which is essentially bazel.
  • [x] Update landing page.
  • [ ] Report generation and publication
  • [x] Public GCS bucket
    • sw-coverage under opentitan.org/ot-fpga-runner for now.
  • [x] Bucket layout:
    • <head_timestamp>-<head_hash>/<coverage_type>/{merged.profiledata, merged.so, report.txt, html}
  • [x] Index page
    • A minimalist page (gist) for now.
    • Should be updated as we update the layout of the GCS bucket to include additional report types.
    • [x] Results table
      • Dynamically generated.
    • [x] Trend chart
      • Added a line chart (chart.js). The index page looks pretty nice now.
  • [ ] Nightly runs
    • We can start working on this sooner using off-target coverage measurements.
    • [x] Local workstation
      • Started running unit test coverage script and uploading results to GCS on 2022-12-21 (using gsutil for now).
    • [ ] CI
      • [ ] Build and Run
      • [ ] Authenticate to GCP with OIDC/WIF and publish
• [ ] Improve coverage and coverage infrastructure
  • [ ] Add dashboard pages to the repo
  • [ ] Unit tests for rom.c (See #16022)
  • [ ] False positives possible?
    • What does llvm-cov do if we have the profile data but not the object file?
  • [ ] static inline change required by optimizations eliminates hash-mismatch errors. Check if still need to create the merged libraries.
  • [ ] Can we integrate with bazel and bazel coverage better?
    • [ ] @dmcardle's comment
    • [ ] @dmcardle's comment

cc @cfrantz @arunthomas @milesdai @drewmacrae @dmcardle

[alphan]

llvm-size output for ROM (local/fork 2163b7436 on tree 706863347)

Without instrumentation:

/tools/riscv/bin/llvm-size -A bazel-out/k8-fastbuild-ST-2cc462681f62/bin/sw/device/silicon_creator/rom/rom_with_fake_keys_fpga_cw310.elf 
bazel-out/k8-fastbuild-ST-2cc462681f62/bin/sw/device/silicon_creator/rom/rom_with_fake_keys_fpga_cw310.elf  :
section                size        addr
.vectors                384       32768
.crt                   1664       33152
.init_array               4       34816
.text                 25348       34820
.shutdown               132       60168
.rodata                4944       60320
.static_critical       8132   268435456
.bss                  43208   268443592
.logs.fields             80           0
.riscv.attributes        38           0
.debug_info           87908           0
.debug_abbrev         19600           0
.debug_aranges          168           0
.debug_line           59215           0
.debug_loc            51588           0
.debug_ranges          5032           0
.debug_str            34447           0
.debug_frame           6100           0
Total                347992