search

enjoy-digital / litex

Build your hardware, easily!
Other
3k stars 569 forks source link

Add a proper CI test for the various configurations #763

Open enjoy-digital opened 3 years ago

enjoy-digital commented 3 years ago

Previously we were testing some configurations in test_targets (that has been removed with https://github.com/enjoy-digital/litex/commit/f31f9a20f0b9134af25e08d1d768c72bc07e473f since required rework/improvements). We should write a proper test that would verify that various configurations are building correctly and when possible run a simulation with Verilator or try to build the bitstream with the open-source toolchains (that we can install on CI).

The aim is not to cover every possible configuration but to have a quick way to detect and fix possible regressions.

enjoy-digital commented 3 years ago

While working on this, we should make sure to test the different CPU that can be simulated with Verilator (so also cover https://github.com/enjoy-digital/litex/issues/586).

navan93 commented 3 years ago

Hi @enjoy-digital ,

I was working on this same topic in my repo when I stumbled on this open issue. I have done some initial work on adding tests for checking that the CPUs work in simulation and a basic structure is working in my CI.

I tried a couple of approaches:

  1. Try to use the bios as it is and interact with it through python subprocess and wait for some string pattern to say pass/fail. But the code to run this test started becoming complicated and not very elegant (handling the interactive console).
  2. Create a simple app which will just print an output like "Hello World" and then stop the simulation by calling sim_finish() and have this app loaded by bios. This again involved quite a bit of changes in the litex build steps.
  3. Add BIOS_TEST_MODE as a constant in the litex_sim.py and if that is set then the bios would print a message and then exit the simulation. I found this to be the path of least resistance and involved minimal changes. Using this I have setup unit tests which will run the simulation on all the supported CPUs (currently only RiscV favours) and return a pass/fail. Moreover I feel this method can be extended in future to add more software based tests like CPU benchmarks etc.

I would like to know your thoughts on this and if you have any other ideas I will try them out.

Thanks Navaneeth

mithro commented 3 years ago

@navan93 -- I think a lot of this sounds pretty awesome but the implementation details can be hard.

On the topic of testing on real hardware, me and @antmicro are working on a few things at https://docs.google.com/document/d/1H3x34bdztWy-uN8EYe-FxJL2YojIjd-0feL9d8BMtN4/edit#heading=h.d8vkktb69rj3 and https://antmicro.com/blog/2021/08/open-source-github-actions-runners-with-gcp-and-terraform/

We are also looking at testing in and using Renode. There is an example of the work @antmicro is doing for the Zephyr project at https://zephyr-dashboard.renode.io/

navan93 commented 3 years ago

@mithro @enjoy-digital please take a loot at https://github.com/navan93/litex/commit/1d2e1058336408b6503a9799c63ec849aefe74f3 and https://github.com/navan93/litex/actions/runs/1375178782 , only Ibex is failing now but that should get fixed with #1074. Having hardware in the loop testing would be the ideal test, at least for the most used boards, but I guess pulling it off and maintaining it is gonna be a challenge.