First class unit test support

[whitequark]

I'm currently copying this file between my projects:

import functools
from migen import *

__all__ = ["simulation_test"]

def simulation_test(case=None, **kwargs):
    def configure_wrapper(case):
        @functools.wraps(case)
        def wrapper(self):
            if hasattr(self, "configure"):
                self.configure(self.tb, **kwargs)
            def setup_wrapper():
                if hasattr(self, "simulationSetUp"):
                    yield from self.simulationSetUp(self.tb)
                yield from case(self, self.tb)
            run_simulation(self.tb, setup_wrapper(), vcd_name="test.vcd")
        return wrapper

    if case is None:
        return configure_wrapper
    else:
        return configure_wrapper(case)

It allows writing test cases like this:

class PCIeRXPHYTestbench(Module):
    def __init__(self):
        self.submodules.lane = PCIeSERDESInterface()
        self.submodules.phy  = PCIeRXPHY(self.lane)

    def do_finalize(self):
        self.states = {v: k for k, v in self.phy.fsm.encoding.items()}

    def phy_state(self):
        return self.states[(yield self.phy.fsm.state)]

    def transmit(self, symbols):
        for symbol in symbols:
            assert (yield self.phy.ts_error) == 0
            yield self.lane.rx_symbol.eq(symbol)
            yield

class PCIeRXPHYTestCase(unittest.TestCase):
    def setUp(self):
        self.tb = PCIeRXPHYTestbench()

    def simulationSetUp(self, tb):
        yield tb.lane.rx_valid.eq(1)

    def assertState(self, tb, state):
        self.assertEqual((yield from tb.phy_state()), state)

    def assertSignal(self, signal, value):
        self.assertEqual((yield signal), value)

    @simulation_test
    def test_rx_tsn_cycle_by_cycle(self, tb):
        yield tb.lane.rx_symbol.eq(K(28,5))
        yield
        yield from self.assertState(tb, "IDLE")
        yield tb.lane.rx_symbol.eq(D(1,0))

    # ...

Should something like this be integrated in Migen? Also, assertState and assertSignal would be handy there.

[sbourdeauducq]

There are many ways to write test benches. Does it help with the ARTIQ ones?

[whitequark]

Yes. In fact this style of writing testbenches is modelled directly after ARTIQ's more structured ones, since that was where I learned. I think that most if not all ARTIQ gateware testbenches can be converted to the style above, and many will become nicer, too.

[nakengelhardt]

I use something very similar. One thing I would like to request that I don't think will work with the above @simulation_test approach is that I almost always want to run multiple generators for one test.

Many of my modules have multiple independent interfaces, and it's very hard to control these from a single generator - much easier to have one generator per interface, and if needed they share some information via a common variable (e.g. an attribute of the testbench). Other modules are meant to connect to external interfaces (e.g. DRAM controller) that need to have their behavior simulated, and that's a generator that belongs to the interface, not whatever module happens to use it. I also like to add a gen_selfcheck to my modules to continuously check some invariants during integration tests (it helps me to get alerted to problems close to the source), and there's no reason not to have those running during the unit test too.

And one more generator I always add is the "timeout" generator (passive, calls self.fail after a specified number of cycles), because the usual failure mode is that the test never terminates. By the time I am sure that that is indeed what happened and interrupt it, the vcd is often so large that gtkwave crashes on loading... that is a generic enough problem that I think timeout could be an optional keyword argument somewhere.

[whitequark]

There are many ways to write test benches.

I agree. Adding this as suggested would be premature, and I am not certain of the best design here.