Doubts about the efficacy of fwk_io testing

[danielpieczko]

To reproduce on a Linux host, no hardware required:

1. Clone the head of the develop branch of fwk_io and initialise the submodules
2. Setup Python virtual environment and install requirements.txt
3. In fwk_io/test run pytest lib_i2c -n auto

Outcome: all tests that are run will fail with AttributeError: module 'Pyxsim.testers' has no attribute 'PytestComparisonTester'

The test_support submodule is version 2d6a6b9 (current head of develop) and searching through that repo there is no PytestComparisonTester method. Changing all instances of PytestComparisonTester to ComparisonTester results in the tests running successfully.

But the github actions job on the head of develop has not failed - I manually triggered a new job to make sure, and the tests all passed. I don't know enough about docker containers to understand what the contents really are. The github actions job launches the tests using the following command-line:

docker run --rm -w /fwk_io/test -v ${{github.workspace}}:/fwk_io ${FWK_IO_TESTER_IMAGE} bash -l run_tests.sh lib_i2c

I believe this is mounting the github workspace (which has checked out the head of develop of fwk_io with commit 2d6a6b9 for the test_support submodule) inside the docker container at the path /fwk_io, and then running the test script from the working directory /fwk_io/test - which should be the expected code that you want to run for the CI job. So how does that pass when a manual run outside of docker fails?

So there are (at least) two questions:

1. What version of the test_support submodule is actually in the docker container, allowing the tests to run?
2. If the version of the test_support submodule is not the expected one, then do we know that the version of fwk_io is correct inside the docker container? What if there is some old version of fwk_io and test_support already in the docker container and these are being tested, rather than the latest head of develop?

[ACascarino]

While I'm here - use AssertiveComparisonTester rather than ComparisonTester, as that's what PytestComparisonTester was renamed to

[mbanth]

This issue is now tracked in Jira AP-352. I have closed the issue here since it only pertains to XMOS internal testing.