search

p4lang / p4c

P4_16 reference compiler
https://p4.org/
Apache License 2.0
683 stars 446 forks source link

Speed up STF tests by only compiling P4 program once for all test packets #4906

Open jafingerhut opened 2 months ago

jafingerhut commented 2 months ago

One possible way to address https://github.com/p4lang/p4c/issues/4571

The .p4.test script for an STF test currently does this:

(1) run p4testgen to create multiple STF tests, one packet per file (2) for each such STF test, compile the P4 program and run BMv2 on it

These changes modify step (2) to be:

(2a) for the first STF test, compile the P4 program, copy the BMv2 JSON file to a place outside the tmpXXXXX directory provided on the run-bmv2-test.py command line as a new option, and run BMv2 on it (2b) for the second and later STF tests, copy the BMv2 JSON file from the place outside the tmpXXXXX directory provided ont eh run-bmv2-test.py command line as a new option, skip running the P4 compiler because the BMv2 JSON file is already present, and run BMv2 on it

Basically we avoid running p4c-bm2-ss on the 2nd and later test packets. For some of the larger P4 test programs, they typically generate the maximum of 10 test packets, and today take 4-5 minutes on a 5-year-old x86_64 laptop I use. With these changes, that same test takes about 2 minutes. I think there are about 512 STF tests in the entire test suite, but most of them are shorter than that today, but I would not be surprised if this saved 30 minutes or so across all of them.

jafingerhut commented 2 months ago

@fruffy Is this something you think is worth merging as written? Or would you prefer that your suggested --stf-files [FILES] technique were implemented instead, and abandon the approach in this PR?

fruffy commented 2 months ago

@fruffy Is this something you think is worth merging as written? Or would you prefer that your suggested --stf-files [FILES] technique were implemented instead, and abandon the approach in this PR?

I have not been able to work on the other approach, so I can not say how involved it is. Although I think it requires only few changes. If the other approach is simple enough I'd prefer it because it requires fewer changes on the P4Testgen script side. Until then we can keep this open.

fruffy commented 1 month ago

Working on a rewrite of the run-bmv2-stf.py script, which has bitrotted a fair bit. After that rewrite I intend to extend the script to support running multiple files at once.