Closed saethlin closed 2 years ago
Found by running tests with RUSTFLAGS=-Zsanitizer=address
If we added this to CI (in some fashion), would it 100% reproduce the problem?
Yes, I believe so. I think the misalignment test always hits this code path.
I believe I see the error:
running 17 tests
=================================================================
==2509==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x607000003862 at pc 0x557d04ebb597 bp 0x7fe9783f6150 sp 0x7fe9783f5920
READ of size 16 at 0x607000003862 thread T8 (simd::test::mis)
#0 0x557d04ebb596 (/home/shep/rust/jetscii/target/debug/deps/jetscii-a76e6f5b4ca7b67e+0x185596) (BuildId: 78e10bd61c26c34e10acde62fcabeede608b512f)
#1 0x557d05567126 (/home/shep/rust/jetscii/target/debug/deps/jetscii-a76e6f5b4ca7b67e+0x831126) (BuildId: 78e10bd61c26c34e10acde62fcabeede608b512f)
#2 0x557d04fcd2d8 (/home/shep/rust/jetscii/target/debug/deps/jetscii-a76e6f5b4ca7b67e+0x2972d8) (BuildId: 78e10bd61c26c34e10acde62fcabeede608b512f)
#3 0x557d04f62fd1 (/home/shep/rust/jetscii/target/debug/deps/jetscii-a76e6f5b4ca7b67e+0x22cfd1) (BuildId: 78e10bd61c26c34e10acde62fcabeede608b512f)
#4 0x557d04f61d29 (/home/shep/rust/jetscii/target/debug/deps/jetscii-a76e6f5b4ca7b67e+0x22bd29) (BuildId: 78e10bd61c26c34e10acde62fcabeede608b512f)
#5 0x557d04f19a39 (/home/shep/rust/jetscii/target/debug/deps/jetscii-a76e6f5b4ca7b67e+0x1e3a39) (BuildId: 78e10bd61c26c34e10acde62fcabeede608b512f)
#6 0x557d04f45d33 (/home/shep/rust/jetscii/target/debug/deps/jetscii-a76e6f5b4ca7b67e+0x20fd33) (BuildId: 78e10bd61c26c34e10acde62fcabeede608b512f)
#7 0x557d04f0a9a9 (/home/shep/rust/jetscii/target/debug/deps/jetscii-a76e6f5b4ca7b67e+0x1d49a9) (BuildId: 78e10bd61c26c34e10acde62fcabeede608b512f)
#8 0x557d04fddbbf (/home/shep/rust/jetscii/target/debug/deps/jetscii-a76e6f5b4ca7b67e+0x2a7bbf) (BuildId: 78e10bd61c26c34e10acde62fcabeede608b512f)
#9 0x557d05029cc2 (/home/shep/rust/jetscii/target/debug/deps/jetscii-a76e6f5b4ca7b67e+0x2f3cc2) (BuildId: 78e10bd61c26c34e10acde62fcabeede608b512f)
#10 0x557d05028ac9 (/home/shep/rust/jetscii/target/debug/deps/jetscii-a76e6f5b4ca7b67e+0x2f2ac9) (BuildId: 78e10bd61c26c34e10acde62fcabeede608b512f)
#11 0x557d04ff4fac (/home/shep/rust/jetscii/target/debug/deps/jetscii-a76e6f5b4ca7b67e+0x2befac) (BuildId: 78e10bd61c26c34e10acde62fcabeede608b512f)
#12 0x557d04ffa9c7 (/home/shep/rust/jetscii/target/debug/deps/jetscii-a76e6f5b4ca7b67e+0x2c49c7) (BuildId: 78e10bd61c26c34e10acde62fcabeede608b512f)
#13 0x557d05613992 (/home/shep/rust/jetscii/target/debug/deps/jetscii-a76e6f5b4ca7b67e+0x8dd992) (BuildId: 78e10bd61c26c34e10acde62fcabeede608b512f)
#14 0x7fe97bb02608 (/lib/x86_64-linux-gnu/libpthread.so.0+0x8608) (BuildId: 7b4536f41cdaa5888408e82d0836e33dcf436466)
#15 0x7fe97b8d2132 (/lib/x86_64-linux-gnu/libc.so.6+0x11f132) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee)
0x607000003862 is located 0 bytes to the right of 66-byte region [0x607000003820,0x607000003862)
allocated by thread T8 (simd::test::mis) here:
#0 0x557d04ebc07e (/home/shep/rust/jetscii/target/debug/deps/jetscii-a76e6f5b4ca7b67e+0x18607e) (BuildId: 78e10bd61c26c34e10acde62fcabeede608b512f)
#1 0x557d04fec7eb (/home/shep/rust/jetscii/target/debug/deps/jetscii-a76e6f5b4ca7b67e+0x2b67eb) (BuildId: 78e10bd61c26c34e10acde62fcabeede608b512f)
#2 0x557d04feca97 (/home/shep/rust/jetscii/target/debug/deps/jetscii-a76e6f5b4ca7b67e+0x2b6a97) (BuildId: 78e10bd61c26c34e10acde62fcabeede608b512f)
#3 0x557d04fed470 (/home/shep/rust/jetscii/target/debug/deps/jetscii-a76e6f5b4ca7b67e+0x2b7470) (BuildId: 78e10bd61c26c34e10acde62fcabeede608b512f)
#4 0x557d04fec4f6 (/home/shep/rust/jetscii/target/debug/deps/jetscii-a76e6f5b4ca7b67e+0x2b64f6) (BuildId: 78e10bd61c26c34e10acde62fcabeede608b512f)
#5 0x557d04f4428c (/home/shep/rust/jetscii/target/debug/deps/jetscii-a76e6f5b4ca7b67e+0x20e28c) (BuildId: 78e10bd61c26c34e10acde62fcabeede608b512f)
#6 0x557d04f0a9a9 (/home/shep/rust/jetscii/target/debug/deps/jetscii-a76e6f5b4ca7b67e+0x1d49a9) (BuildId: 78e10bd61c26c34e10acde62fcabeede608b512f)
#7 0x557d05029cc2 (/home/shep/rust/jetscii/target/debug/deps/jetscii-a76e6f5b4ca7b67e+0x2f3cc2) (BuildId: 78e10bd61c26c34e10acde62fcabeede608b512f)
Thread T8 (simd::test::mis) created by T0 here:
#0 0x557d04ea57cc (/home/shep/rust/jetscii/target/debug/deps/jetscii-a76e6f5b4ca7b67e+0x16f7cc) (BuildId: 78e10bd61c26c34e10acde62fcabeede608b512f)
#1 0x557d05613801 (/home/shep/rust/jetscii/target/debug/deps/jetscii-a76e6f5b4ca7b67e+0x8dd801) (BuildId: 78e10bd61c26c34e10acde62fcabeede608b512f)
SUMMARY: AddressSanitizer: heap-buffer-overflow (/home/shep/rust/jetscii/target/debug/deps/jetscii-a76e6f5b4ca7b67e+0x185596) (BuildId: 78e10bd61c26c34e10acde62fcabeede608b512f)
Shadow bytes around the buggy address:
0x0c0e7fff86b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0e7fff86c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0e7fff86d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0e7fff86e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0e7fff86f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c0e7fff8700: fa fa fa fa 00 00 00 00 00 00 00 00[02]fa fa fa
0x0c0e7fff8710: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0e7fff8720: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0e7fff8730: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0e7fff8740: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0e7fff8750: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==2509==ABORTING
However, after pulling your branch, I fail to run the doctests at all:
shep@GAMING:~/rust/jetscii$ RUSTFLAGS=-Zsanitizer=address cargo test
Compiling jetscii v0.5.2 (/home/shep/rust/jetscii)
Finished test [unoptimized + debuginfo] target(s) in 1.08s
Running unittests src/lib.rs (target/debug/deps/jetscii-a76e6f5b4ca7b67e)
running 17 tests
test simd::test::byte_substring_has_false_positive ... ok
test simd::test::byte_substring_is_not_found ... ok
test simd::test::byte_substring_is_found ... ok
test simd::test::byte_substring_needle_is_longer_than_16_bytes ... ok
test simd::test::can_search_in_null_bytes ... ok
test simd::test::can_search_for_null_bytes ... ok
test simd::test::does_not_access_memory_after_haystack_when_haystack_is_multiple_of_16_bytes_and_no_match ... ok
test simd::test::misalignment_does_not_cause_a_false_positive_before_start ... ok
test simd::test::space_is_found ... ok
test simd::test::space_not_found ... ok
test simd::test::works_at_page_boundary ... ok
test simd::test::works_on_nonaligned_beginnings ... ok
test simd::test::xml_delim_3_is_found ... ok
test simd::test::xml_delim_5_is_found ... ok
test simd::test::works_as_find_does_for_various_memory_offsets ... ok
test simd::test::works_as_find_does_for_up_to_and_including_16_bytes ... ok
test simd::test::works_as_find_does_for_byte_substrings ... ok
test result: ok. 17 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.11s
Doc-tests jetscii
running 6 tests
test src/lib.rs - (line 76) ... ok
test src/lib.rs - (line 62) ... ok
test src/lib.rs - (line 47) ... FAILED
test src/lib.rs - (line 37) ... FAILED
test src/lib.rs - (line 24) ... FAILED
test src/lib.rs - (line 11) ... FAILED
failures:
---- src/lib.rs - (line 47) stdout ----
error: linking with `cc` failed: exit status: 1
|
= note: "cc" "-m64" "/tmp/rustcZzyQc4/symbols.o" "/tmp/rustdoctestQlpeC0/rust_out.rust_out.a83be413-cgu.0.rcgu.o" "/tmp/rustdoctestQlpeC0/rust_out.3gcfx1k3cfjrd912.rcgu.o" "-Wl,--as-needed" "-L" "/home/shep/rust/jetscii/target/debug/deps" "-
L" "/home/shep/rust/jetscii/target/debug/deps" "-L" "/home/shep/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/home/shep/rust/jetscii/target/debug/deps/libjetscii-0a19caaf5d7d801d
.rlib" "-Wl,--start-group" "/home/shep/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-91db243dd05c003b.rlib" "/home/shep/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_6
4-unknown-linux-gnu/lib/libpanic_unwind-72269a4525d4f5cf.rlib" "/home/shep/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-28d8f1c01a28b12d.rlib" "/home/shep/.rustup/toolchains/nightly-x86
_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-5b78018a9f8ae4bc.rlib" "/home/shep/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-f4160de9657f17b2.rlib" "/home
/shep/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-1cd8b958acdf2395.rlib" "/home/shep/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librust
c_demangle-a4c4a7e7edfa8aea.rlib" "/home/shep/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-061c02acc74ada37.rlib" "/home/shep/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib
/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-2aed706f056a5dfb.rlib" "/home/shep/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-1e1f90ff4bfdca6f.rlib" "/home/shep/.rustup/toolch
ains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-2d16c932daf0ad41.rlib" "/home/shep/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_allo
c-8f15fae89f489a33.rlib" "/home/shep/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-81f3d85dace75e64.rlib" "/home/shep/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_
64-unknown-linux-gnu/lib/libcfg_if-e071db8735f10456.rlib" "/home/shep/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-6db7e05a8de4df10.rlib" "/home/shep/.rustup/toolchains/nightly-x86_64-unk
nown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-7c03f666869e802a.rlib" "/home/shep/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-2a6a2797f7a73818.rlib"
"/home/shep/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-0e3656b1fda5fd7b.rlib" "-Wl,--end-group" "/home/shep/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unkno
wn-linux-gnu/lib/libcompiler_builtins-b09abe545ed38eb1.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-L" "/home/shep/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu
/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/tmp/rustdoctestQlpeC0/rust_out" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro,-znow" "-nodefaultlibs"
= note: /usr/bin/ld: /home/shep/rust/jetscii/target/debug/deps/libjetscii-0a19caaf5d7d801d.rlib(jetscii-0a19caaf5d7d801d.11vo16b4hra8239y.rcgu.o): in function `core::slice::<impl [T]>::as_mut_ptr':
/rustc/46b8c23f3eb5e4d0e0aa27eb3f20d5b8fc3ed51f/library/core/src/slice/mod.rs:506: undefined reference to `__asan_option_detect_stack_use_after_return'
/usr/bin/ld: /rustc/46b8c23f3eb5e4d0e0aa27eb3f20d5b8fc3ed51f/library/core/src/slice/mod.rs:506: undefined reference to `__asan_stack_malloc_0'
/usr/bin/ld: /rustc/46b8c23f3eb5e4d0e0aa27eb3f20d5b8fc3ed51f/library/core/src/slice/mod.rs:506: undefined reference to `__asan_report_store8'
/usr/bin/ld: /rustc/46b8c23f3eb5e4d0e0aa27eb3f20d5b8fc3ed51f/library/core/src/slice/mod.rs:506: undefined reference to `__asan_report_store8'
/usr/bin/ld: /home/shep/rust/jetscii/target/debug/deps/libjetscii-0a19caaf5d7d801d.rlib(jetscii-0a19caaf5d7d801d.11vo16b4hra8239y.rcgu.o): in function `core::slice::<impl [T]>::copy_from_slice':
/rustc/46b8c23f3eb5e4d0e0aa27eb3f20d5b8fc3ed51f/library/core/src/slice/mod.rs:3214: undefined reference to `__asan_option_detect_stack_use_after_return'
/usr/bin/ld: /rustc/46b8c23f3eb5e4d0e0aa27eb3f20d5b8fc3ed51f/library/core/src/slice/mod.rs:3214: undefined reference to `__asan_stack_malloc_1'
/usr/bin/ld: /rustc/46b8c23f3eb5e4d0e0aa27eb3f20d5b8fc3ed51f/library/core/src/slice/mod.rs:3214: undefined reference to `__asan_report_store8'
/usr/bin/ld: /rustc/46b8c23f3eb5e4d0e0aa27eb3f20d5b8fc3ed51f/library/core/src/slice/mod.rs:3214: undefined reference to `__asan_report_store8'
...
/usr/bin/ld: /rustc/46b8c23f3eb5e4d0e0aa27eb3f20d5b8fc3ed51f/library/core/src/slice/index.rs:378: undefined reference to `__asan_report_store8'
/usr/bin/ld: /rustc/46b8c23f3eb5e4d0e0aa27eb3f20d5b8fc3ed51f/library/core/src/slice/index.rs:378: undefined reference to `__asan_report_load8'
/usr/bin/ld: /rustc/46b8c23f3eb5e4d0e0aa27eb3f20d5b8fc3ed51f/library/core/src/slice/index.rs:378: undefined reference to `__asan_report_load8'
/usr/bin/ld: /home/shep/rust/jetscii/target/debug/deps/libjetscii-0a19caaf5d7d801d.rlib(jetscii-0a19caaf5d7d801d.2k24pa8ki81vk4mu.rcgu.o): in function `<core::ops::range::RangeFrom<usize> as core::slice::index::SliceIndex<[T]>>::inde
x':
/rustc/46b8c23f3eb5e4d0e0aa27eb3f20d5b8fc3ed51f/library/core/src/slice/index.rs:388: undefined reference to `__asan_option_detect_stack_use_after_return'
/usr/bin/ld: /rustc/46b8c23f3eb5e4d0e0aa27eb3f20d5b8fc3ed51f/library/core/src/slice/index.rs:388: undefined reference to `__asan_stack_malloc_0'
/usr/bin/ld: /rustc/46b8c23f3eb5e4d0e0aa27eb3f20d5b8fc3ed51f/library/core/src/slice/index.rs:388: undefined reference to `__asan_report_store8'
/usr/bin/ld: /rustc/46b8c23f3eb5e4d0e0aa27eb3f20d5b8fc3ed51f/library/core/src/slice/index.rs:388: undefined reference to `__asan_report_store8'
/usr/bin/ld: /rustc/46b8c23f3eb5e4d0e0aa27eb3f20d5b8fc3ed51f/library/core/src/slice/index.rs:390: undefined reference to `__asan_handle_no_return'
collect2: error: ld returned 1 exit status
= help: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
= note: use the `-l` flag to specify native libraries to link
= note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorustc-link-libkindname)
error: aborting due to previous error
Couldn't compile the test.
failures:
src/lib.rs - (line 11)
src/lib.rs - (line 24)
src/lib.rs - (line 37)
src/lib.rs - (line 47)
test result: FAILED. 2 passed; 4 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.16s
Oh, I need to set RUSTDOCFLAGS as well, of course.
Yeah sorry, when I'm testing crates I do this:
RUSTDOCFLAGS=-Zsanitizer=address ASAN_OPTIONS="detect_leaks=0 detect_stack_use_after_return=1" RUSTFLAGS="-Zsanitizer=address" cargo +nightly test -Zbuild-std --target=x86_64-unknown-linux-gnu --no-fail-fast -- --test-threads=1
and sometimes I get overzealous with minimizing that ball of options
Alright, I ran this in CI and got a failure, so I've added it to this PR to prevent any regressions. Thanks!
Performance changes look to be mostly noise
$ cargo benchcmp before after
name before ns/iter after ns/iter diff ns/iter diff % speedup
bench::space_ascii_chars 356,259 (14716 MB/s) 353,817 (14818 MB/s) -2,442 -0.69% x 1.01
bench::space_stdlib_find_char 312,285 (16788 MB/s) 310,645 (16877 MB/s) -1,640 -0.53% x 1.01
bench::space_stdlib_find_char_set 2,287,519 (2291 MB/s) 2,283,963 (2295 MB/s) -3,556 -0.16% x 1.00
bench::space_stdlib_find_closure 2,330,487 (2249 MB/s) 2,314,242 (2265 MB/s) -16,245 -0.70% x 1.01
bench::space_stdlib_find_string 2,130,275 (2461 MB/s) 2,143,435 (2446 MB/s) 13,160 0.62% x 0.99
bench::space_stdlib_iterator_position 1,168,010 (4488 MB/s) 1,138,967 (4603 MB/s) -29,043 -2.49% x 1.03
bench::substring_stdlib_find 461,680 (11356 MB/s) 461,587 (11358 MB/s) -93 -0.02% x 1.00
bench::substring_with_created_searcher 358,382 (14629 MB/s) 358,360 (14630 MB/s) -22 -0.01% x 1.00
bench::xml_delim_3_ascii_chars 358,341 (14630 MB/s) 358,085 (14641 MB/s) -256 -0.07% x 1.00
bench::xml_delim_3_stdlib_find_char_closure 2,594,423 (2020 MB/s) 2,592,599 (2022 MB/s) -1,824 -0.07% x 1.00
bench::xml_delim_3_stdlib_find_char_set 2,721,806 (1926 MB/s) 2,706,089 (1937 MB/s) -15,717 -0.58% x 1.01
bench::xml_delim_3_stdlib_iterator_position 2,269,648 (2309 MB/s) 2,270,007 (2309 MB/s) 359 0.02% x 1.00
bench::xml_delim_5_ascii_chars 359,318 (14591 MB/s) 358,681 (14617 MB/s) -637 -0.18% x 1.00
bench::xml_delim_5_stdlib_find_char_closure 2,596,877 (2018 MB/s) 2,591,359 (2023 MB/s) -5,518 -0.21% x 1.00
bench::xml_delim_5_stdlib_find_char_set 3,432,148 (1527 MB/s) 3,428,845 (1529 MB/s) -3,303 -0.10% x 1.00
bench::xml_delim_5_stdlib_iterator_position 2,270,511 (2309 MB/s) 2,268,070 (2311 MB/s) -2,441 -0.11% x 1.00
Thanks! Released in 0.5.3 ❤️
The previous implementation only did reads in batches of 16 bytes, which could overflow immediately in the misalignment branch, and if there was any haystack left after the main chunk loop, the last read would overflow there as well.
Found by running tests with RUSTFLAGS=-Zsanitizer=address