search

p4lang / p4c

P4_16 reference compiler
https://p4.org/
Apache License 2.0
669 stars 441 forks source link

Consider a CI/CD test run for DEBUG p4c build, instead of RELEASE #4824

Open jafingerhut opened 2 months ago

jafingerhut commented 2 months ago

One example of some recent failures in the automated CI tests for p4c that only failed with a DEBUG build, and not a RELEASE build, are mentioned in this issue: https://github.com/p4lang/p4c/issues/4822

If there is interest, I can add about 5 to 10 or so more tests that regularly fail with DEBUG builds, but pass with RELEASE builds. The exact set of such tests depends upon what combination of processor arch {x86_64, aarch64} and Ubuntu {20.04, 22.04, 24.04} you build and test with.

If adding one more DEBUG build as a prerequisite to all commits would be too many CI compute resources, perhaps having a nightly or weekly such build & test would be reasonable? I guess the main question is: are there such nightly/weekly CI tests running regularly now, and if so, how does one become notified if they fail?

asl commented 2 months ago

I think at least, if the DEBUG build is expensive in terms of test runtime, we should have Release configuration with assertions enabled.

Actually, I think all CI jobs must run with assertions enabled :)

vlstill commented 2 months ago

RelWithDebInfo (which might need to be properly configured) seems like the right build type for most of CI runs. We should preserve one Release build just to have chance at catching side effects in assertions. I don't see much benefit in using Debug build directly for CI.

If there is interest, I can add about 5 to 10 or so more tests that regularly fail with DEBUG builds, but pass with RELEASE builds. The exact set of such tests depends upon what combination of processor arch {x86_64, aarch64} and Ubuntu {20.04, 22.04, 24.04} you build and test with.

Please do, it would be definitely useful to know.

fruffy commented 2 months ago

We could add a nightly test for this, or let the sanitizer test build with DEBUG.

jafingerhut commented 2 months ago

Attached is a zip file with a README.md explaining how I got the lists of failing tests, and 6 files with the lists of failing tests for Ubuntu 20.04, 22.04, and 24.04, for DEBUG and RELEASE p4c builds. 2024-jul-20-testfails.zip