rusticstuff / simdutf8

SIMD-accelerated UTF-8 validation for Rust.
Other
531 stars 26 forks source link

Miri reports UB with simd_bitmask (FW) #75

Closed Licenser closed 1 year ago

Licenser commented 1 year ago

A forward from https://github.com/simd-lite/simd-json/issues/264,

I managed to replicate the UB error from miri with this test:

cargo +nightly  miri test  --features public_imp utf8_validator

I could trigger it with avx2, sse4.2 and avx

#[cfg(all(feature = "public_imp", target_feature = "avx2"))]
#[test]
fn utf8_validator() {
    use simdutf8::basic::imp::ChunkedUtf8Validator;
    unsafe {
        let mut utf8_validator = simdutf8::basic::imp::x86::avx2::ChunkedUtf8ValidatorImp::new();
        let tmpbuf = [
            49, 46, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32,
            32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32,
            32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32,
        ];
        utf8_validator.update_from_chunks(&tmpbuf);
        assert!(utf8_validator.finalize(None).is_ok());
    }
}
Licenser commented 1 year ago

@Nilstrieb I forwarded your find

hkratz commented 1 year ago

@Licenser @Nilstrieb Thanks for reporting. Will have a look!

hkratz commented 1 year ago

This is not a simdutf8 issue. Instead it will be fixed once stdarch is updated in Rust to include the fix in https://github.com/rust-lang/stdarch/issues/1347.

Licenser commented 1 year ago

Sorry for the false positive :) I admit I didn't even think about looking into it being a issue with core rust, wonderful catch!