SV2V QA test suite - Githubissues

chili-chips-ba commented 2 weeks ago

While trying SV2V, we quickly realized how good it was compared to both Synlig and Yosys-Slang.

Yet, having run into some here-and-there issues, we believe that the SV2V project would benefit from adding our two SV RTL repos to its regression tests:

zachjs commented 2 weeks ago

Although sv2v has extensive test coverage (you can check its code coverage with make coverage), this repository doesn't include regression tests, per se. sv-tests already covers a handful of large open source SystemVerilog projects. My gut feeling is that it would be best for the community to focus our regression testing efforts into the sv-tests project. What do you think?

chili-chips-ba commented 2 weeks ago

While we are not sure about the internal workings of sv2v QA process, we were under the impression that you were taking only occasional and cursory look at sv-tests

"... indeed sv2v has hundreds of its own tests, generally focusing on just a few features at a time. I do use the sv-tests dashboard from time to time to check in on its assessment of sv2v..."

We also agree that moving the focus of your QA process to sv-tests is a worthy idea. However, to get the most out of it, the sv-tests would need to be divided into two sub-categories:

for RTL and Synth
for Sim and DV, perhaps even with UVM sub-sub-category

or provide a mechanism for marking selected SV features (along with tests that validate them) as NA. Those should then show greyed out and not contribute to your total score. We assume that all that the sv2v cares about at the moment is the RTL/Synth subset of the language.

In the meantime, the life goes on, and your private library of test designs may need to grow with it 😉...

zachjs commented 1 week ago

We also agree that moving the focus of your QA process to sv-tests is a worthy idea. However, to get the most out of it, the sv-tests would need to be divided into two sub-categories:

for RTL and Synth

for Sim and DV, perhaps even with UVM sub-sub-category

or provide a mechanism for marking selected SV features (along with tests that validate them) as NA. Those should then show greyed out and not contribute to your total score. We assume that all that the sv2v cares about at the moment is the RTL/Synth subset of the language.

sv-tests provides some value to me even though it complains about features that are out of scope for sv2v. I know these features are out of scope and can focus my efforts elsewhere. Certainly it would be great for sv-tests to incorporate your suggestions to improve its signal to noise ratio, but I don't have the bandwidth to take on that project.

In the meantime, the life goes on, and your private library of test designs may need to grow with it 😉...

I wouldn't call the tests private! They're all committed into the repo. I'm happy for contributors to add more test coverage in line with the guidelines (https://github.com/zachjs/sv2v/blob/master/test/README.md). Indeed, some of the tests come directly from outside contributors or are based on test cases provided in bug reports. If you find a bug in sv2v, please report that bug!

chili-chips-ba commented 1 week ago

@hzeller anything you can do about dividing sv-tests into two sub-categories (1) RTL/Synth; (2) DV/Sim, possible with UVM as a sub-sub-category?!

zachjs / sv2v

SV2V QA test suite #285