Generated tests use UB in their calculations

[Lokathor]

Test code for struct offsets run checks like this:

    assert_eq!(
        unsafe { &(*(::core::ptr::null::<timeval>())).tv_sec as *const _ as usize },
        0usize,
        concat!(
            "Offset of field: ",
            stringify!(timeval),
            "::",
            stringify!(tv_sec)
        )
    );

But it's UB to dereference a null pointer, and it's also UB to create a &T with numeric value 0.

There is actually trivial to fix. Instead of using a null pointer, just create a value on the stack (using core::mem::zeroed(), which is safe since all C structs can be created as zeroed-memory), and then use a reference to that stack value, the fields of that stack value, and so on.

[Lokathor]

#[allocator_tag_for_miri]
fn my_fancy_allocator<T>() -> *const T { 0 as *const T }

This should make compiler happy and convert UB to potential runtime error?

That's just an allocator that fails all the time though. That doesn't fix the problem. You fundamentally cannot use the null pointer as the location of an object in Rust. It is simply not how Rust works. When an allocator returns null that means it failed to allocate.

[purew]

In case this is only about the warning, you could silence it from Rust, as we did in godot-rust/godot-rust#776. In short, it's:

#![allow(deref_nullptr)]

in the module that contains the generated symbols. Keep in mind that this lint will be unknown for older compiler versions.

Thanks, this answers my question. Agreed on bindgen should address this.

[Dushistov]

@Lokathor

That's just an allocator that fails all the time though. That doesn't fix the problem. You fundamentally cannot use the null pointer as the location of an object in Rust. It is simply not how Rust works. When an allocator returns null that means it failed to allocate.

0 is just example, if all ptr::null and ptr::is_null is waste and rustc can not handle 0 as valid address, what is by the way valid address on some ARM that I worked with (without OS and with virtual memory off).

it can be

#[allocator_tag_for_miri]
fn my_fancy_allocator<T>() -> *const T { 0x17000 as *const T }

 assert_eq!(
        let p = my_fancy_allocator::<timeval>();
        unsafe { &(*p).tv_sec as *const _ as usize },
        p as usize + 0usize,
        concat!(
            "Offset of field: ",
            stringify!(timeval),
            "::",
            stringify!(tv_sec)
        )
    );

[Lokathor]

It's not that 0 isn't ever a valid address, it's that 0 isn't ever a valid address within Rust. The way Rust is defined, 0 must be the null address, and there's never any live object at the null address. If your platform can use 0 as a valid address you must use inline assembly for accessing it.

But anyway, for almost all cases you don't need this fake pointer nonsense. Just put a zeroed struct on the stack, point at that struct, and then forget the struct at the end of the test.

(and beyond that, all these field offset tests that are generated are themselves fairly useless and really shouldn't be being generated anyway.)

[fzyzcjy]

Hi, is there any updates? It is giving warnings (I know it can be supressed - but it may not be the best practice)

warning: dereferencing a null pointer
  --> src/ffi_tflite_c_api.rs:65:19
   |
65 |         unsafe { &(*(::std::ptr::null::<TfLiteQuantizationParams>())).scale as *const _ as usize },
   |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this code causes undefined behavior when executed
   |
   = note: `#[warn(deref_nullptr)]` on by default

[kornelski]

@fzyzcjy Disable test generation in bindgen. These tests are not practically useful and even harm portability of the bindings.

[fzyzcjy]

@kornelski Interesting! I will disable them.

[Bromeon]

@kornelski

These tests are not practically useful and even harm portability of the bindings.

Can you elaborate? Isn't the purpose of the tests precisely to ensure that certain invariants are upheld even on other architectures?

[YaLTeR]

In one of my projects I bindgen for windows-gnu (because that's more convenient to run from my system), then the CI runs the layout tests on windows-msvc to verify that nothing breaks when compiling to the final target of windows-msvc. The tests are very much practically useful for this case.

[kornelski]

@Bromeon The tests bake in assumptions of the platform that generated the bindings. They're not testing Rust's actual behavior against C's actual behavior. They are testing bindgen's single-platform assumptions against Rust's #[repr(C)] platform-specific assumptions. Behavior of #[repr(C)] adjusts to the target platform, but tests don't. This leads to false alarms where the tests test for a wrong layout of a different platform. They're not finding real incompatibilities, but uncover bindgen's inflexibility.

There are very few cases where #[repr(C)] is not good enough and not portable enough, but there are plenty of trivial cases where bindgen's tests are non-portable. In practice that behavior forces generation of bindings for each platform individually, even when the bindings are identical for every platform (thanks to universality of #[repr(C)]), and only the tests differ.

[Lokathor]

Yes. To expand on that a little, the auto-generated tests are useless with C code.

(I'm told that they can cover some C++ cases well but I don't know the details there so I won't say either way.)

[YaLTeR]

In practice that behavior forces generation of bindings for each platform individually, even when the bindings are identical for every platform (thanks to universality of #[repr(C)]), and only the tests differ.

Hm, I guess that's exactly what I'm doing. I need to check whether the bindings actually differ.

[Lokathor]

The generated tests do not check if the bindings are different when moving to a new platform. Please do not rely on them for that.

[ilammy]

Chiming in with a new data point.

When generating bindings for --target aarch64-apple-ios:

warning: reference to packed field is unaligned
    --> [redacted]/boring/target/aarch64-apple-ios/debug/build/boring-sys-d38444b8ce3b96c5/out/bindings.rs:6006:18
     |
6006 |         unsafe { &(*(::std::ptr::null::<_OSUnalignedU64>())).__val as *const _ as usize },
     |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     |
     = warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
     = note: for more information, see issue #82523 <https://github.com/rust-lang/rust/issues/82523>
     = note: fields of packed structs are not properly aligned, and creating a misaligned reference is undefined behavior (even if that reference is never dereferenced)

That's a struct which is explicitly unaligned.

Generated code

```rust #[repr(C, packed)] #[derive(Debug, Default, Copy, Clone, PartialEq, Eq)] pub struct _OSUnalignedU64 { pub __val: u64, } #[test] fn bindgen_test_layout__OSUnalignedU64() { assert_eq!( ::std::mem::size_of::<_OSUnalignedU64>(), 8usize, concat!("Size of: ", stringify!(_OSUnalignedU64)) ); assert_eq!( ::std::mem::align_of::<_OSUnalignedU64>(), 1usize, concat!("Alignment of ", stringify!(_OSUnalignedU64)) ); assert_eq!( unsafe { &(*(::std::ptr::null::<_OSUnalignedU64>())).__val as *const _ as usize }, 0usize, concat!( "Offset of field: ", stringify!(_OSUnalignedU64), "::", stringify!(__val) ) ); } ```

[lu-zero]

https://doc.rust-lang.org/std/primitive.pointer.html#method.offset_from and https://doc.rust-lang.org/std/ptr/macro.addr_of.html are now available, the tests could use it with a mem::zeroed() struct I guess.

[saschanaz]

tests could use it with a mem::zeroed() struct I guess.

https://github.com/rust-lang/rust-bindgen/issues/1651#issuecomment-545015646 had some concern about that.

[Danielmelody]

At least this warning should be suppressed if there is no better way for now, this could cause more works to do while integrating with other crates.

[simlay]

It looks like https://github.com/rust-lang/rust/pull/95372 caused these warnings are now errors in nightly which might means it'll error on rust 1.62 (in ~10 weeks).

For the next person, it looks like quick work around (but definitely not the correct long term solution) is to use layout_tests(false) similar as seen with the iOS cross compilaion of boring.

[koutheir]

Disabling layout tests is only useful to remove the warning and undefined behavior, at the expense of more risks with regard to ABI changes. That's a compromise one needs to think about. The layout_tests(false) workaround simply hides the issue instead of solving it.

[Lokathor]

The layout tests can't detect that to begin with so there's no difference if you turn them off.

[koutheir]

If the layout tests can't detect ABI changes, then what are they supposed to validate?

[Lokathor]

They don't validate much because they don't call into C. It's just checking rust code against other rust code. All they actually end up doing is a clumsy check that your C type aliases are correct. Which you can do once per crate, instead of once per type, and get just as much benefit.

EDIT: even then, checking the type aliases once per crate is excessive probably. Really they should be checked once when defined (eg: in cty or std or wherever else) and then no downstream crate needs to double check after that.

[koutheir]

Does this mean that one way of actually solving this issue would be to remove the generation of these checks entirely?

[Lokathor]

that's my suggestion, yes.

[emilio]

This is not right. The layout tests are useful, because they test the actual generated rust code with the offsets that we get from libclang.

[emilio]

So even though it's not the same as calling into C and back, it's pretty close, and they've caught ABI issues in Firefox in the past, for example.

[Lokathor]

the tests could certainly detect some problems, but if all C type aliases are correct then that's about the extent of what they can check. For example, if a conditional compilation adds or changes a field in a C struct in a new C lib version, or on a different target, the rust code has no way to notice, and these tests won't help you. They are extremely un-portable tests.