Do testgen tests using STF files compile the P4 program once per test case?

[jafingerhut]

I have run a test like this manually after building p4c:

cd p4c/build
./testgen/testgen-p4c-bmv2-stf/fabric.p4.test -v

From the output it appears to behave as follows:

• run p4testgen once to generate up to 10 test scenarios
• For each of those 10 test scenarios, each of which has a separate STF file:
  • compile the P4 program to create a BMv2 JSON file
  • Run BMv2 with the resulting BMv2 JSON file, and the STF file for this test case

Thus it seems to compile the P4 program 10 times.

It seems like it should be straightforward to run such a test in a way that it compiles the P4 program only once, true?

[fruffy]

Yes, this is a limitation of the Python STF runner, not P4Testgen. P4Testgen only requires that state is being reset between tests.

I see two ways to fix this.

1. Change P4Testgen to consolidate all tests into a single STF file and make it reset switch state between each single tests. This will require adding a reset command to STF, similar to how PTF works.

2. Just add an option to the run-bmv2-test.py to take multiple tests as input. Only compile once but reinstantiate the switch between each test.