search

riscv-software-src / riscv-unified-db

Machine-readable database of the RISC-V specification, and tools to generate various views
Other
13 stars 12 forks source link

Need regression tests to ensure artifacts (e.g., manual, profiles, CRDs) aren't broken by changes to main #193

Open james-ball-qualcomm opened 1 week ago

james-ball-qualcomm commented 1 week ago

Now that we are telling people to look at the artifacts (especially the MC-1 CRD), our tolerance for breaking these artifacts is low enough that we need proper regression tests that fail if the artifacts are changed in unintentional ways. For a good example of this, see issue #192.

dhower-qc commented 1 week ago

how do you want to check it, aside from just a successful build? Store known-good generated files?

AFOliveira commented 1 week ago

I was thinking of proposing to add a validation check to ensure generated encodings stay consistent with riscv-opcodes. I’m working on a few final updates to make this possible, and I think this could help keep encoding tight and accurate. This would also be possible for pseudoinstructions.

james-ball-qualcomm commented 1 week ago

Yes, storing known good artifacts and comparing against them. At least for the CRDs and the profiles we can compare the Asciidoc files that are generated. Hopefully we can have some reasonable GUI diff tool and ake it easy to update the expected file after the comparison has been done.

We can also have different kinds of regression tests depending on the branch. For example, we can have an integration branch for CRDs and Profiles that the bulk of development affecting them is done in branches off the integration branch. When a PR is performed to the integration branch the regression test doesn't bother checking the CRD and Profiles artifacts against good known ones since they changing often. However when a PR is performed to the main branch the regression test does check against the expected artifacts to prevent different users from breaking artifacts not related to their direct work. Then we can do releases off the main branch as we do now or just copy them over to GitHub pages.

+james+

From: Afonso Oliveira @.> Sent: Friday, October 25, 2024 11:40:10 AM To: riscv-software-src/riscv-unified-db @.> Cc: James Ball (QUIC) @.>; Author @.> Subject: Re: [riscv-software-src/riscv-unified-db] Need regression tests to ensure artifacts (e.g., manual, profiles, CRDs) aren't broken by changes to main (Issue #193)

WARNING: This email originated from outside of Qualcomm. Please be wary of any links or attachments, and do not enable macros.

I was thinking of proposing to add a validation check to ensure generated encodings stay consistent with riscv-opcodes. I’m working on a few final updates to make this possible, and I think this could help keep encoding tight and accurate. This would also be possible for pseudoinstructions.

— Reply to this email directly, view it on GitHubhttps://github.com/riscv-software-src/riscv-unified-db/issues/193#issuecomment-2438569509, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BBRBTGBLCQDSFN26U5VWAP3Z5KGAVAVCNFSM6AAAAABQTT57O6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZYGU3DSNJQHE. You are receiving this because you authored the thread.Message ID: @.***>