How to filter out test benches through the run() configuration?

[Ravenwater]

Question: our design flow organizes each standalone hardware block in its own repo. It will manage its design files in an hdl/ directory, and its testbench in a tb/ directory. The idea being that the repo is a self-contained development environment for the design, debug, test, and release of that hardware block.

When a hardware block is composed of dependent modules, we use an automation process where we specify the repo and the branch we want, and that gets cloned into the client alongside the repo content in a ext/ directory.

The problem we run into is that when you run pytest it will find the test benches in the ext/ directory and pytest will fail to build those test benches. We can use -k to get just the top-level repo test bench, but I am looking for a configuration of the run method of cocotb_test.simulator that simply ignores the test benches in the ext/ directory so that newbies to a repository can simply invoke pytest to run the top-level test bench.

Here is an example of a hardware block mv_tile_fixed that pulls in the dependent lut_power module defined in the hw_block repo that exhibits the problem of running the test bench of the external block.

(verify) stillwater@sw-desktop-300:~/dev/clones/mv_tile_fixed$ pytest
============================================================== test session starts ==============================================================platform linux -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/stillwater/dev/clones/mv_tile_fixed
plugins: forked-1.4.0, xdist-2.5.0, cocotb-test-0.2.2
collected 18 items

ext/hw_blocks/tb/test_lut_power.py F                                                                                                      [ 11%]
tb/test_mv_tile_fixed.py ................                                                                                                 [100%]

=================================================================== FAILURES ====================================================================_________________________________________________________ test_lut_power[8-16-0-4-1.5] __________________________________________________________
BASE = 1.5, WIDTH_IN = 4, IN_SIGNED = 0, WIDTH_OUT = 16, WIDTH_OUT_FRAC = 8

    @pytest.mark.parametrize("BASE",           [1.5])
    @pytest.mark.parametrize("WIDTH_IN",       [4  ])
    @pytest.mark.parametrize("IN_SIGNED",      [0  ])
    @pytest.mark.parametrize("WIDTH_OUT",      [16 ])
    @pytest.mark.parametrize("WIDTH_OUT_FRAC", [8  ])
    def test_lut_power(BASE, WIDTH_IN, IN_SIGNED, WIDTH_OUT, WIDTH_OUT_FRAC):

        dut = 'lut_power'

        local_d = dict(locals())
        parameters = {k: local_d[k] for k in local_d.keys() if k == k.upper()}
>       run(
            python_search=['tb/'],
            verilog_sources=glob.glob('hdl/*'),
            toplevel=dut,
            module=os.path.splitext(os.path.basename(__file__))[0],
            simulator='icarus',
            parameters=parameters,
            sim_build="sim_build/" + "__".join((f"{key}-eq-{value}" for key, value in parameters.items())),
            extra_env={f'PARAM_{k}': str(v) for k, v in parameters.items()},
        )
E       SystemExit: Process 'iverilog' termindated with error 1

ext/hw_blocks/tb/test_lut_power.py:63: SystemExit
--------------------------------------------------------------- Captured log call ---------------------------------------------------------------INFO     cocotb:simulator.py:291 Running command: iverilog -o /home/stillwater/dev/clones/mv_tile_fixed/sim_build/BASE-eq-1.5__WIDTH_IN-eq-4__IN_SIGNED-eq-0__WIDTH_OUT-eq-16__WIDTH_OUT_FRAC-eq-8/lut_power.vvp -D COCOTB_SIM=1 -s lut_power -g2012 -Plut_power.BASE=1.5 -Plut_power.WIDTH_IN=4 -Plut_power.IN_SIGNED=0 -Plut_power.WIDTH_OUT=16 -Plut_power.WIDTH_OUT_FRAC=8 /home/stillwater/dev/clones/mv_tile_fixed/hdl/mv_tile_fixed.sv
INFO     cocotb:simulator.py:302 error: Unable to find the root module "lut_power" in the Verilog source.
INFO     cocotb:simulator.py:302      : Perhaps ``-s lut_power'' is incorrect?
INFO     cocotb:simulator.py:302 1 error(s) during elaboration.
============================================================ short test summary info ============================================================
FAILED ext/hw_blocks/tb/test_lut_power.py::test_lut_power[8-16-0-4-1.5] - SystemExit: Process iverilog termindated with error 1
========================================================= 2 failed, 16 passed in 9.91s ==========================================================

The test benches for blocks that aggregate other modules are written as follows:

from cocotb_test.simulator import run
import pytest
import glob

@pytest.mark.parametrize("R", [4,2])
@pytest.mark.parametrize("C", [4,2])
@pytest.mark.parametrize("WIDTH_X", [8,7])
@pytest.mark.parametrize("WIDTH_K", [8,7])
def test_mv_tile_fixed(R, C, WIDTH_X, WIDTH_K):

    dut = \'mv_tile_fixed\'

    local_d = dict(locals())
    parameters = {k: local_d[k] for k in local_d.keys() if k == k.upper()}
    run(
        python_search=['tb/'],
        verilog_sources=glob.glob("hdl/*") + glob.glob("ext/*/hdl/*"),
        toplevel=dut,
        module=os.path.splitext(os.path.basename(__file__))[0],
        simulator='icarus',
        parameters=parameters,
        sim_build="sim_build/" + "__".join((f"{key}-eq-{value}" for key, value in parameters.items())),
        extra_env={f'PARAM_{k}': str(v) for k, v in parameters.items()},
    )

How do we configure run so that the test benches that exist in ext/<module>/tb are ignored?

[themperek]

Not sure if I fully understand the problem.

Looks to me the question is not cocotb-test specific but pytest.

If I am not wrong pytest by default will scan all directories and search for test_*.py files. You can change this default behaviour with configuration files (https://docs.pytest.org/en/6.2.x/customize.html) using testpaths variables.

You can use marking https://docs.pytest.org/en/6.2.x/example/markers.html

You can skip a test based on something (like where the test was started?) https://docs.pytest.org/en/7.1.x/how-to/skipping.html

More: https://docs.pytest.org/en/7.1.x/how-to/usage.html

[Ravenwater]

@themperek Yep, you are completely right: this filtering needs to happen through pytest configuration. Thank you for the pointer. I have created a pytest.ini file configuring the knowledge to look for tests just in the tb/ directory and the behavior I was after emerged. I can populate the ini file in the repo and steer a newbie around these issues.

Very happy, and thank you for your insight to direct me to the answer.