[TASK] Setup and maintain embedded CI on Thales servers

[JeanRochCoulon]

What Provide code coverage metrics in an automated way without consuming too much energy.

Howto First do not run heavy tests if smoke tests fail. Then add another workflow running on master once a week, to provide metrics and their history.

Current status There are currently no metrics available.

Risks The amount of code and infrastructure to change might be significant.

Prerequisites Complete CI simplification before adding new things.

KPI None

Description of done The metrics dashboard is available.

[cathales]

• CI simplification done, with a distinct workflow to run on master once a week
• PR #1517 merged
  • do not run heavy tests if a smoke test fails.
  • update conditions to run jobs.
  • switch configurations to embedded.
• "Code coverage reports" part not started yet.

[JeanRochCoulon]

Would be useful to refer the related PR.

[cathales]

• Fewer jobs are run thanks to https://github.com/openhwgroup/cva6/pull/1535 which changes PR pipelines to be dev instead of regress. However there is no job status checker in dev pipelines yet: added in https://github.com/openhwgroup/cva6/pull/1548
• Two remaining steps to significantly reduce latency, see this comment: https://github.com/openhwgroup/cva6/pull/1535#pullrequestreview-1679757493
• regress pipeline is still running on master branch
• Scheduled verif pipeline to run every Sunday, however some verif jobs may be timing out. It will be possible to fix that later.
• Synthesis only runs when core/ or corev_apu/ is changed: https://github.com/openhwgroup/cva6/pull/1550

[cathales]

This description includes notion of code coverage, which is the topic of https://github.com/openhwgroup/cva6/issues/1436

However the current task focuses on "when is each job run?".

[cathales]

Notes from meeting:

• 1436: when we have coverage results on several weeks, add a graph with coverage result in the dashboard

• Add a new color (gray?) for ignored tests: they are run for information but they are allowed to fail without making the whole pipeline fail. (Needs to update report_generator.py for report generation and the dashboard for report rendering)

[cathales]

For first point from last message see https://github.com/openhwgroup/cva6/issues/1436

For second point, we need to discuss how we want to specify which tests we want to ignore.

[cathales]

We wanted to ignore tests that are known to be failing on master. As we do not run these tests before merging PRs, do we still want to ignore them?

Note: the failing tests are in the compliance and riscv-tests-p jobs. Actually these tests are "crashing", not only "failing". This implies that subsequent tests in the same job are not run. So we thought that 2 tests were failing (one in each job) while more are crashing! I had to list them to debug my own stuff so I share it with you below.

List of crashing tests on c430c6c34be0e24175e03d3c91478e349bfaaf4e (2024-01-05):

• In compliance (verif/tests/testlist_riscv-compliance-cv32a6_embedded.yaml)
  • rv32uc-rvc
  • rv32ui-jal
  • rv32ui-bge
  • rv32ui-blt
  • rv32ui-bgeu
  • rv32ui-sw
  • rv32ui-lbu
  • rv32ui-sb
  • rv32ui-slti Note: this test, compiling slti.S, is actually named rv32ui-sw in the test list. Looks like a typo.
  • rv32ui-sra
  • rv32ui-srl
  • rv32ui-sh
  • rv32ui-lw
  • rv32ui-andi
  • rv32ui-srli
  • rv32ui-slli
  • rv32ui-beq
  • rv32ui-sll
  • rv32ui-addi
  • rv32ui-lh
  • rv32ui-and
  • rv32ui-xori
  • rv32ui-sub
  • rv32ui-slt
  • rv32ui-lb
  • rv32ui-or
  • rv32ui-lui
  • rv32ui-ori
  • rv32ui-bltu
  • rv32ui-fence_i
  • rv32ui-auipc
  • rv32ui-srai
  • rv32ui-jalr
  • rv32ui-xor
  • rv32ui-simple
  • rv32ui-lhu
  • rv32ui-bne
  • rv32ui-add
  • rv32ui-sltiu
  • rv32ui-sltu
  • rv32si-sbreak
  • rv32si-scall
  • rv32si-ma_fetch
  • rv32si-wfi
  • rv32si-csr
  • Note that there are 2 failing tests, rv32mi-scall and rv32mi-csr, but at least the job runs to completion after removing the above tests.
• Inriscv-tests-p (verif/tests/testlist_riscv-tests-cv32a6_embedded-p.yaml)
  • rv32mi-p-scall
  • rv32si-p-csr
  • rv32si-p-ma_fetch
  • rv32si-p-scall
  • rv32si-p-wfi
  • rv32si-p-sbreak
  • Note that there is 1 failing test, rv32mi-p-csr, but at least the job runs to completion after removing the above tests.

The crashes seem related to the two letters after rv32.

[cathales]

Can we consider this task as done?

[JeanRochCoulon]

Yes, this task is completed but the tests which are crashed are to be fixed. A new GitHub Issue shall be open for this issue.

[cathales]

Done #1944