Closed rod-chapman closed 4 months ago
I agree that we should aim to expand the coverage of undefined behaviour. That said:
I am not sure what it would mean to "sound" in presence of undefined behaviour? CBMC does not, and does not claim to, detect all kinds of undefined behaviour. CBMC is able to synthesize assertions for some classes of undefined behaviour, and CBMC seeks to be sound for all input programs that do not exhibit undefined behaviour.
"Sound in the presence of undefined behaviour" is a meaningless claim I think. If CBMC does not detect and prevent all classes of UB, where is that documented? My copy of WG14's n1570 document (roughly "C11") Annex J.2 lists 203 cases of UB. Could you document exactly which of these are prevented by CBMC and which ones I still have to be worried about?
We should definitively avoid any claims for programs with undefined behaviour. We should strive to extend our checks for UBs, with the ultimate goal being that we cover them all.
@tautschnig: This one is easy when done in/near the C front-end, but hard when done late on the goto-program. This really needs to get done, and I'll start looking into it.
@kroening - in my experience, all C programs are littered with UB (every single signed integer operator has potential for UB for a start...), and just one instance of UB in a program is sufficient to nuke every other proof.
I can imagine CBMC having a "strict" mode where it really does get rid of all UB, either by subsetting or analysis. The user has to pay the price of some programs being rejected that lie outside of the analysable subset. This is how it works in Frama-C, SPARK, and probably ASTREE.
Please start by documenting exactly where we are now - a table of all 203+ cases would suffice.
For this particular instance, I am encouraged that clang manages to emit a warning. I am not sure, though, if clang warns for all such instances, or only some of them...
Can someone add the "Soundness" label to this issue please?
Can someone add the "Soundness" label to this issue please?
I've added the soundness label. That said, I agree with the above sentiments that it's hard to define "sound" and "unsound" for a behaviour that is "undefined". From the above it appears that a "fix"[?] for this "bug" may be to at least document (if not also warn) on undefined behaviours and how CBMC handles them. You mentioned a list of 203+, I don't suppose you have that list on hand or a link to it?
For me, any undefined behaviour always risks unsoundness. A correct (and possibly only) "fix" is to reject any and all UB. The problem is coming up with fast (and also sound!) algorithms to detect the UB in the first place without onerously subsetting the language in a way that just annoys users. The list of 200+ UB issues is in the ISO C standard (Appendix J last time I looked...)
Yeah... C standards are on the WG14 document log. For example https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3054.pdf is the current draft of the C'23 standard. See Annex J.2
Please take a look at https://github.com/diffblue/cbmc/pull/7880 -- this is an attempt to track down all UB (according to C11) systematically.
The goal would be to have checks for all dynamic UB (I'd probably declare the preprocessor-related ones outside of the scope of CBMC).
Note #8226.
Closing as with CBMC 6 we have all supported undefined-behaviour checks enabled by default. We still have more work to do as documented in doc/C/c11-undefined-behavior.html.
I am not convinced. "All supported UB checks" is a cop-out. What you're really saying is "the tool is still unsound for a bunch of things are are't supported." For this test case in particular: https://github.com/rod-chapman/cbmc-examples/blob/main/ub/ub1.c CBMC 6.0.1 is still demonstrably unsound, so I don't think you can claim victory just yet...
Happy to re-open this issue, but it will come pretty close to "won't fix": I don't see us rolling our own preprocessor anytime soon, which would be among the changes required to be able to tick all items on that list.
Well... do you have an analysis of which ones can and should be fixed? I am happy that a great many are out of scope and/or solved by some very simple subsetting.
I find CBMC 5.84 reports an unsound result for the following test case:
An attempt to verify shows that CBMC thinks that the final value of y is 2:
Compling with clang 14.0.3 (on Apple M1 macOS 13.3.1):
I would hope that CBMC would detect the undefined behaviour (as clang does) and reject the program.