Axi test harness

[nathanielnrn]

Thought it would be useful to open a draft PR to allow time to get feedback while classes are starting and I'm less available. This is nearly done.

Pull request allows for runt testing of cocotb simulation which outputs the execution of a kernel based on examples/dahlia .data and .expected files. Also introduces a subdirectory specifically for cocotb tests.

DRAFT: Regarding the reason this is a draft. Currently, running the make commands from runt does not behave the same way as running each command from a single shell one by one. Specifically, the Makefile does not use a new toplevel.v in spite of the -B flag. This means that a single computation (i.e dot-product) is performed on all data (i.e vectorized add data).

Additionally, make outputs some text which is specific to the file system running the command. For example, make outputs the commands it performs. Is there a way to selectively gather this text within runt? Just adding some tail and head commands to our output in the rust.toml?

[nathanielnrn]

Currently changing this to only test a single calyx program: vectorized-add. I think one test is better than no test, and runt behaves strangely when supplied more than one directory.

(It doesn't seem to fetch the correct toplevel, but will perform same computation on different input data, i.e dot product on vadd data).

Additionally, I am getting this weird unclosed file error which is polluting runt's expected output. As soon as that is fixed, this should be ready for deployment.

[rachitnigam]

The unclosed file error is sadly expected. It is a problem with cocotb and they haven’t fixed it

[rachitnigam]

hmm, runt should correctly work with multiple directories. Not sure why it’s failing to do thag

[rachitnigam]

Can you try changing the runt command to cd into the directory and run make from there?

[nathanielnrn]

So (finally!) had some time to fix this! Apologies for taking so long. I believe the most recent push should fix the one-computation-kernel issue

[rachitnigam]

@nathanielnrn seems like the tests are failing?

[nathanielnrn]

@nathanielnrn seems like the tests are failing?

Has to do with the AxBurst ports I added to make cocotb happy, probably should have made another branch for this. Updated runt gold files fixes this.

[rachitnigam]

Hm, still seems to be failing

[rachitnigam]

Looks like everything is passing @nathanielnrn! Merge when ready!

[nathanielnrn]

Latest commit makes the toplevel.v and main.sv we use for cocotb be dynamically generated as expected. However, the runt tests we have aren't passing due to differences in a newline here and lack of a new line there. I'm not able to reproduce failing cases on locally built docker containers. Any thoughts from anyone by any chance? I'll keeping looking into it in the mean time.

[sampsyo]

Neat! Hella weird about the newline nonsense. I guess I suspect some vagary about the test's shell pipeline…

May I recommend using fud -v or whatever it is to get verbose output from Fud instead of that progress indicator thingy ⠋⠙?

[rachitnigam]

Usually we use -q with fud in tests to make it stop producing any execution related output

[rachitnigam]

Okay, my changes should fix the silly fud problem @nathanielnrn was running into. Once this is merged, we should check off tasks in #1104 and create a new issue to turn the cocotb hackery embedded in the runt commands into a fud stage so we can start running lots of different Calyx kernels with this harness

[rachitnigam]

@nathanielnrn if you'd like, feel free to press the big green button! Congratulations on a successful summer research project!