Testing on hardware

[elms]

Brief explanation

Develop a hardware setup that provides good coverage for features and platforms supported. Ideally this can be incorporated into a CI approach in the future. It may start as a manual process or provide minimal coverage, but the approach needs to scale.

Expected results

An easy and relatively inexpensive setup that can be reproduced at different scales (different subsets of hardware and tests). A full scale setup would be connected to the CI system and report failures. I'm not sure if it's practical to incorporate into presubmit checks.

Detailed Explanation

Initial sketch:

• Cheap FPGA DUT (Device Under Test)
• Cheap "logic analyzer" type board (use fx2lafw)
• Connect them with an adapter board
• software to drive tests:
  • build image
  • load image
  • capture output of sigrok and UART
  • check against a expected result

It should probably target the most complex design for a specific architecture (eg PicoSoC), but maybe a good first intermediate would be iceram or ram_test with a change.

Simplifications to get started:

• start with UART only, maybe UART echo (repeat output)
• iCE40 boards are cheaper

Scaling

Additional target boards could be added with an additional set of DUT, "logic analyzer" board, and adapter board. I don't think the limit for this would be quite high, but requires specifying which UART/"logic analyzer" is connected to which hardware.

Probably shouldn't write to flash for all tests when it is scaled up.

Options

• Run without "logic analyzer"
• Run only specified DUT
• Run all or only specific test
• Some sanity check that hardware is there and not defunct in someway (not sure what to do here)

Prerequisites

• Decide initial target hardware
• Generate expected results
• Design adapter board (if supporting logic analyzer)

[mithro]

Thanks for the detailed description @elmsfu!

Using Glasgow from @whitequark for something like this might be another option. See https://github.com/whitequark/Glasgow

GitHub

whitequark/Glasgow

Scottish Army Knife for electronics. Contribute to whitequark/Glasgow development by creating an account on GitHub.

[elms]

I saw @daveshah1 add a repo for ECP5 testing https://github.com/daveshah1/hilotof

GitHub

daveshah1/hilotof

HiLoTOF -- Hardware-in-the-Loop Test framework for Open FPGAs - daveshah1/hilotof

[mkurc-ant]

I would go for something that can provide a stimulus to the DUT. It can be either the "Glasgow" project or eg. a FTDI breakout board. I guess a solution with FTDI might be cheaper and simpler.

[HackerFoo]

Another option is to use embedded CPU(s) (e.g. MicroBlaze) maybe one to stimulate the DUT and one to collect and summarize the results. Then exhaustive checks could be run for up to 32 bits in a few minutes.

[elms]

Another option is to use embedded CPU(s) (e.g. MicroBlaze) maybe one to stimulate the DUT and one to collect and summarize the results. Then exhaustive checks could be run for up to 32 bits in a few minutes.

I think an interesting aspect of this is for small tests. If we have the test driver be static and only reconfigure a region that contains a UUT (unit under test).