Beyond unit testing

[imd1]

What is your question? I noticed that are hundreds of unit tests, but I wonder how much testing is performed at the "system" level, i.e. real world combinations of rules?

Is there any merit in running VSG on a real world set of files and at least checking it is able to auto-fix them without error?

[jeremiah-c-leary]

Hey @imd1 ,

That has been the biggest hurdle with development, getting access to a wide range of VHDL code.

I do have a set of tests under vsg/tests/styles that run over some code I pulled down from open cores. They do not cover all the rules however.

Any ideas to improve this would be appreciated.

--Jeremy

[imd1]

@jeremiah-c-leary

Searched for trending github sites with vhdl tags:

• https://github.com/trending/vhdl

A couple standout:

• https://github.com/ghdl/ghdl
• https://github.com/OSVVM/OSVVM

Could run the tool on these sites, and attempt to fix their code to flush out VSG crash bugs at least?

[imd1]

Could repeat search in gitlab?

[jeremiah-c-leary]

Could run the tool on these sites, and attempt to fix their code to flush out VSG crash bugs at least?

That is a good idea. I will look into it.

[jeremiah-c-leary]

I downloaded OSVVM and ran into an issue with procedure_410. If there was more than one space before a : on a single line parameter list, it would combine the procedure keyword and the procedure identifier.

Very strange.

[jeremiah-c-leary]

I have made it through OSVVM and UVVM.

[imd1]

Good news...important to keep re running this as part of your testing, and ensure you go back to the original content to check VSG copes with fixing etc

[jeremiah-c-leary]

I made it through the code at this site: https://github.com/VLSI-EDA/PoC

important to keep re running this as part of your testing, and ensure you go back to the original content to check VSG copes with fixing etc

I wonder how I can keep testing against these code bases. There are a lot of files and some of them are over 5000 lines. Maybe it is something I could run periodically....but then it would have to be local. I have not quite figured out the best way to incorporate it.

I have been focusing on parser errors at the moment and will have to go back and check on the style.

[jeremiah-c-leary]

I was able to get through the following code bases (I pulled the list from rust_hdl's github page):

https://github.com/VUnit/vunit https://github.com/OSVVM/OSVVM https://github.com/UVVM/UVVM https://github.com/VLSI-EDA/PoC https://github.com/FPHDL/fphdl/ https://github.com/inforichland/freezing-spice https://github.com/sergeykhbr/riscv_vhdl https://github.com/fabioperez/space-invaders-vhdl https://github.com/BG2BKK/img_process_vhdl https://github.com/kevinpt/vhdl-extras https://github.com/xesscorp/VHDL_Lib

I think I will merge my pull request and do a release this weekend.

[jeremiah-c-leary]

Maybe I could have a file with a list of github links. Then I could have a script pull everything down and run VSG against it. Possibly before every release?

What do you think?

[imd1]

Maybe I could have a file with a list of github links. Then I could have a script pull everything down and run VSG against it. Possibly before every release?

What do you think?

Many options...you could run something on your CI server, maybe once a week on the master branch, or maybe run it on every merge/commit to master?

[jeremiah-c-leary]

Many options...you could run something on your CI server, maybe once a week on the master branch, or maybe run it on every merge/commit to master?

What if I did this:

1) Clone code repo twice a) First would be called the reference code b) Second I would apply a style and commit the changes. This would be called the fixed code. 2) Create a test a) Include the repo URL for the reference and fixed code b) Pull down reference code and fixed code URLs c) Run VSG with style to reference code d) Compare fixed reference to fixed code

I worry this could take a lot of effort, but maybe it is what needs to be done.

[imd1]

The main problems I've found with "real" code is VSG crashing or VSG completing but leaving malformed VHDL behind. I think your testing should concentrate on these two problems