Closed emusgrave closed 5 months ago
stephenberry
commented
6 months ago Thanks for sharing this issue. The string writing was recently optimized, so I can understand that certain conditions may have been overlooked.
I intend Glaze to pass address sanitizers and not use any undefined behavior, so this will definitely be a priority to fix. However, I'm going to be out of pocket for a few weeks, so it will be probably a month before I can solve this.
stephenberry
commented
5 months ago What OS and compiler are you using? I added your example to the JSON unit tests in #726, but I couldn't get the clang address sanitizer to complain on my Arm Mac. See if you can test again with the main branch and see if the problem still exists.
CramBL
commented
5 months ago I have also been stuck on an older version (1.9.1) of Glaze for the same reason. Here's an example of the heap buffer overflow from a Catch2 that failed because ASAN caught something:
I wanted to make a good bug report but I simply don't have time for it right now, I haven't worked on that project for a while, but I think I should chime in on this issue because it has persisted through many versions of glaze according to my CI, and it would be a shame if it wasn't fixed because glaze is a great project that I've l loved using up until that issue arose.
Links to GitHub action jobs that failed from ASAN + glaze in catch2 tests in case they could be of any use:
Note that the same CI has builds for macos and windows that do not fail. It only fails on my linux builds.
Glaze 2.0.1
ubuntu 22.04, llvm-17.0.6, Release ubuntu 22.04, llvm-17.0.6, Debug ubuntu 22.04, gcc-13.1, Release ubuntu 22.04, gcc-13.1, Debug
Glaze 1.9.9
ubuntu 22.04, llvm-17.0.6, Release ubuntu 22.04, llvm-17.0.6, Debug ubuntu 22.04, gcc-13.1, Release ubuntu 22.04, gcc-13.1, Debug
Glaze 1.9.8.1
ubuntu 22.04, llvm-17.0.6, Release ubuntu 22.04, llvm-17.0.6, Debug ubuntu 22.04, gcc-13.1, Release ubuntu 22.04, gcc-13.1, Debug
stephenberry
commented
5 months ago @CramBL, thanks for the additional information. I will prioritize fixing this. It's very helpful to know that you're seeing the issue on linux, as I should be able to locate it and debug it now.
stephenberry
commented
5 months ago @emusgrave and @CramBL
It is looking like this is a bug in GCC's address santizer. The more recent string serialization/parsing in Glaze utilizes SWAR (SIMD Within A Register). This allows us to look at and copy eight bytes at a time rather than just one.
To achieve this, we need to ensure that the string we are reading/copying has a capacity that is a multiple of 8. The code looks like:
value.reserve(round_up_to_multiple<8>(n));We never actually use these bytes for parsing/serializing. Instead, we just make sure they are allocated so that we can work on 8 byte chunks. When this reservation does not exist, Clang appropriately gives an address sanitizer error. But, the Clang address sanitizer is satisfied when we appropriately make this allocation. The standard library requires that calling reserve allocates the new capacity or greater: cppreference string.reserve().
So, there is nothing illegal or dangerous with what Glaze is doing. The address sanitizer should be satisfied because we are not reading beyond the string's allocated memory. However, GCC complains. It probably uses the size of the string to determine the boundary limits on the address sanitizer, rather than the capacity.
I'm going to continue doing more testing to make sure my findings are true.
Interestingly, I can't get @emusgrave issue to fail the address sanitizer on compiler explorer (which I'm pretty sure is using linux based servers). Here's a link to a simple example
stephenberry
commented
5 months ago I'll also note that GCC does properly raise address sanitizer errors on compiler explorer, as shown here
Note that if you add buffer.reserve(64); then the address sanitizer error goes away as expected.
So, if someone could get compiler explorer to give an address sanitizer error with Glaze, it would be a big help to me. It should be with string reading/writing, but I can't figure out how to trigger it.
emusgrave
commented
5 months ago I'm actually using Windows, so my issue is happening with VS 2022 17.8.6, compiler version 19.38. Unfortunately Compiler Explorer doesn't have glaze available when using the msvc compiler.
I'll re-run my testing and see if I can narrow down the version of glaze where this asan error started.
emusgrave
commented
5 months ago I can confirm that the error starts to appear in v1.9.9 when the string manipulation was updated with #642.
Open question, as I'm not well-versed in address sanitizer... would Compiler Explorer show the error if it's using a copy of glaze that wasn't built with -fsanitize=address? I assume it will since it's happening in the json/write.hpp header.
I jumped through hoops with vcpkg to make sure all of the 3rd party libraries I am using were compiled with the sanitize flag.
I will start up a Linux box and run the same small test program locally and see whether I am seeing the problem there.
emusgrave
commented
5 months ago Ok, after building on Linux I was able to confirm that the previous test code isn't causing the issue. However, I was able to create another test that causes it. The code just loops and adds characters to the input string until it fails. On my local machine the printf's were working and it threw on the 26th character. On godbolt the printf doesn't seem to come out?
std::string baseJson = "\"X";
int maxLength = 1024;
printf("%lu: %s\n", baseJson.size(), baseJson.data());
for (int i = baseJson.length(); i < maxLength; ++i)
{
baseJson += "X";
std::string rawJson = baseJson;
auto res = glz::read_json<std::string>(rawJson.append("\""));
if (res.has_value())
{
auto parsed = glz::write_json(res.value());
printf("parsed ok: (%lu) %s\n", parsed.size(), parsed.data());
}
}Nice! I assume Godbolt only displays stderr when the exit code is non-zero. Running without sanitizer produces the expected output.
stephenberry
commented
5 months ago Thanks guys! I'll dig into this more.
CramBL
commented
5 months ago -fsanitize=memory also triggers it on @emusgrave's godbolt link
==1==WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x564831cb9fe6 in void glz::detail::serialize_string<8ul, char, char, unsigned long>(char const*, char*, unsigned long&) /opt/compiler-explorer/libs/glaze/trunk/include/glaze/json/write.hpp:222:68
#1 0x564831cb9fe6 in void glz::detail::to_json<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>::op<glz::opts{10u, false, true, true, true, false, (char)32, (unsigned char)3, false, true, false, false, false, 0u, false, false, false, false, false, false, true, false, false, false, false, true}, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>&, glz::context&, unsigned long&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>&, glz::context&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>&, unsigned long&) /opt/compiler-explorer/libs/glaze/trunk/include/glaze/json/write.hpp:350:25
#2 0x564831cb9fe6 in void glz::detail::write<10u>::op<glz::opts{10u, false, true, true, true, false, (char)32, (unsigned char)3, false, true, false, false, false, 0u, false, false, false, false, false, false, true, false, false, false, false, true}, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>&, glz::context&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>&, unsigned long&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>&, glz::context&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>&, unsigned long&) /opt/compiler-explorer/libs/glaze/trunk/include/glaze/json/write.hpp:41:16
#3 0x564831cb9fe6 in void glz::write<glz::opts{10u, false, true, true, true, false, (char)32, (unsigned char)3, false, true, false, false, false, 0u, false, false, false, false, false, false, true, false, false, false, false, true}, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, glz::context&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>&, glz::context&) /opt/compiler-explorer/libs/glaze/trunk/include/glaze/core/write.hpp:41:7
#4 0x564831cb90ea in void glz::write<glz::opts{10u, false, true, true, true, false, (char)32, (unsigned char)3, false, true, false, false, false, 0u, false, false, false, false, false, false, true, false, false, false, false, true}, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>&) /opt/compiler-explorer/libs/glaze/trunk/include/glaze/core/write.hpp:51:7
#5 0x564831cb90ea in auto glz::write_json<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>&) /opt/compiler-explorer/libs/glaze/trunk/include/glaze/json/write.hpp:1239:7
#6 0x564831cb8696 in main /app/example.cpp:15:27
#7 0x7fe597a29d8f (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: c289da5071a3399de893d2af81d6a30c62646e1e)
#8 0x7fe597a29e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: c289da5071a3399de893d2af81d6a30c62646e1e)
#9 0x564831c28344 in _start (/app/output.s+0x32344)
SUMMARY: MemorySanitizer: use-of-uninitialized-value /opt/compiler-explorer/libs/glaze/trunk/include/glaze/json/write.hpp:222:68 in void glz::detail::serialize_string<8ul, char, char, unsigned long>(char const*, char*, unsigned long&)@emusgrave and @CramBL There was an issue with writing, with where the null character sits in memory not being considered. This has been fixed in #758
However, there is another danger. That is small string optimization (SSO). Even if we reserve enough memory, the string may not be contained in the heap memory, but instead on the small chunk of stack memory. However, it seems that alignment solves this problem and the stack memory has sufficient space for SWAR. Hence, the address sanitizer doesn't raise issues until we reach the heap at 16 characters. SSO supports up to 15 characters in a string.
It still seems like MSVC and GCC are incorrectly raising address sanitizer issues when proper memory is reserved. If I reserve memory beyond what should be needed I can get MSVC to stop raising ASAN issues, but I don't want to allocate more memory than needed.
I'm still working through this, but I should have a solution soon.
stephenberry
commented
5 months ago Yeah, even when the capacity is 31 and we are only accessing through index 23, MSVC raises address sanitizer issues because we are loading data beyond our 16 byte string value.
I would prefer to pass address sanitizers than have maximum performance. There is a slightly slower approach using SWAR that was used before, and I might just revert back to that approach.
stephenberry
commented
5 months ago @emusgrave and @CramBL, I've merged in #759, which should fix address sanitizers complaining about writing strings. Let me know if you run into any more address sanitizer problems. I'll be releasing this soon in a version bump.
CramBL
commented
5 months ago It seems that the issue persists: https://gcc.godbolt.org/z/1Mrzvcdr5
Updated glaze on my own project and I'm also still seeing the issue in CI on Ubuntu only.
stephenberry
commented
5 months ago @CramBL, I think it just took a bit for compiler explorer to update the Glaze trunk (it should pull nightly). Your compiler explorer link is passing for me.
But, I am intrigued that you have an issue with your CI. Could you verify again that you are using the latest version of Glaze?
CramBL
commented
5 months ago Yea Godbolt link is good now.
My own project's CI is such a mess now after I left it for a few months.
Building in a container with Ubuntu 22.04, LLVM 17.0.6 and testing with UB, thread, and address sanitizer passes with no issues. Running through GitHub actions on 16 different platforms has half of them failing to even compile and a few (Ubuntu 22.04, GCC 13.2, LTO) triggers the address sanitizer with output like this:
18: ==12660==ERROR: AddressSanitizer: unknown-crash on address 0x60400000196b at pc 0x5645152c7293 bp 0x7ffc55765870 sp 0x7ffc55765860
18: WRITE of size 8 at 0x60400000196b thread T0
18: #0 0x5645152c7292 in void glz::detail::from_json<scryfall::Card>::op<glz::opts{10u, false, true, true, true, false, (char)32, (unsigned char)3, false, true, false, false, false, 0u, false, false, false, false, false, false, true, false, false, true, false, true}, glz::string_literal<1ul>{}, scryfall::Card&, glz::context&, char const*&, char const*&>(scryfall::Card&, glz::context&, char const*&, char const*&) [clone .lto_priv.0] (/home/runner/work/mtgo-collection-manager/mtgo-collection-manager/mtgoparser/build/test/Release/test_full_collection_parse+0x4d6292) (BuildId: 0fc462fdb26e01979a5a9f0c944d1e79c4809c3e)
18: #1 0x5645151e8cc8 in scryfall::ReadJsonVector(std::filesystem::__cxx11::path const&) (/home/runner/work/mtgo-collection-manager/mtgo-collection-manager/mtgoparser/build/test/Release/test_full_collection_parse+0x3f7cc8) (BuildId: 0fc462fdb26e01979a5a9f0c944d1e79c4809c3e)
18: #2 0x564515259656 in CATCH2_INTERNAL_TEST_9() [clone .lto_priv.0] (/home/runner/work/mtgo-collection-manager/mtgo-collection-manager/mtgoparser/build/test/Release/test_full_collection_parse+0x468656) (BuildId: 0fc462fdb26e01979a5a9f0c944d1e79c4809c3e)
18: #3 0x5645153d0dc4 in Catch::RunContext::invokeActiveTestCase() (/home/runner/work/mtgo-collection-manager/mtgo-collection-manager/mtgoparser/build/test/Release/test_full_collection_parse+0x5dfdc4) (BuildId: 0fc462fdb26e01979a5a9f0c944d1e79c4809c3e)
18: #4 0x5645151a0a19 in main (/home/runner/work/mtgo-collection-manager/mtgo-collection-manager/mtgoparser/build/test/Release/test_full_collection_parse+0x3afa19) (BuildId: 0fc462fdb26e01979a5a9f0c944d1e79c4809c3e)
18: #5 0x7fe0a3629d8f (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: c289da5071a3399de893d2af81d6a30c62646e1e)
18: #6 0x7fe0a3629e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: c289da5071a3399de893d2af81d6a30c62646e1e)
18: #7 0x5645151ade64 in _start (/home/runner/work/mtgo-collection-manager/mtgo-collection-manager/mtgoparser/build/test/Release/test_full_collection_parse+0x3bce64) (BuildId: 0fc462fdb26e01979a5a9f0c944d1e79c4809c3e)
18:
18: 0x604000001971 is located 0 bytes after 33-byte region [0x604000001950,0x604000001971)
18: allocated by thread T0 here:
7: #0 0x558e07e44725 in void glz::detail::from_json<goatbots::CardDefinition>::op<glz::opts{10u, false, true, true, true, false, (char)32, (unsigned char)3, false, true, false, false, false, 0u, false, false, false, false, false, false, true, false, false, false, false, true}, glz::string_literal<1ul>{}, goatbots::CardDefinition&, glz::context&, char const*&, char const*&>(goatbots::CardDefinition&, glz::context&, char const*&, char const*&) [clone .lto_priv.0] (/home/runner/work/mtgo-collection-manager/mtgo-collection-manager/mtgoparser/build/test/Release/test_json_parse+0x353725) (BuildId: d2706cb04a1220785cc016955bd3bfcd99a251c8)
7: #1 0x558e07de43b5 in boost::outcome_v2::basic_result<boost::unordered::unordered_flat_map<unsigned int, goatbots::CardDefinition, boost::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<std::pair<unsigned int const, goatbots::CardDefinition> > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::conditional<false, boost::outcome_v2::policy::terminate, std::conditional<false, boost::outcome_v2::policy::error_code_throw_as_system_error<boost::unordered::unordered_flat_map<unsigned int, goatbots::CardDefinition, boost::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<std::pair<unsigned int const, goatbots::CardDefinition> > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, void>, std::conditional<true, boost::outcome_v2::policy::exception_ptr_rethrow<boost::unordered::unordered_flat_map<unsigned int, goatbots::CardDefinition, boost::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<std::pair<unsigned int const, goatbots::CardDefinition> > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, void>, boost::outcome_v2::policy::fail_to_compile_observers>::type>::type>::type> goatbots::ReadJsonMap<boost::unordered::unordered_flat_map<unsigned int, goatbots::CardDefinition, boost::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<std::pair<unsigned int const, goatbots::CardDefinition> > > >(std::filesystem::__cxx11::path const&) (/home/runner/work/mtgo-collection-manager/mtgo-collection-manager/mtgoparser/build/test/Release/test_json_parse+0x2f33b5) (BuildId: d2706cb04a1220785cc016955bd3bfcd99a251c8)
7: #2 0x558e07dc3c2e in CATCH2_INTERNAL_TEST_16() [clone .lto_priv.0] (/home/runner/work/mtgo-collection-manager/mtgo-collection-manager/mtgoparser/build/test/Release/test_json_parse+0x2d2c2e) (BuildId: d2706cb04a1220785cc016955bd3bfcd99a251c8)
7: #3 0x558e07ed8884 in Catch::RunContext::invokeActiveTestCase() (/home/runner/work/mtgo-collection-manager/mtgo-collection-manager/mtgoparser/build/test/Release/test_json_parse+0x3e7884) (BuildId: d2706cb04a1220785cc016955bd3bfcd99a251c8)
7: #4 0x558e07d6a24b in main (/home/runner/work/mtgo-collection-manager/mtgo-collection-manager/mtgoparser/build/test/Release/test_json_parse+0x27924b) (BuildId: d2706cb04a1220785cc016955bd3bfcd99a251c8)
7: #5 0x7f4015829d8f (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: c289da5071a3399de893d2af81d6a30c62646e1e)
7: #6 0x7f4015829e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: c289da5071a3399de893d2af81d6a30c62646e1e)
7: #7 0x558e07d75df4 in _start (/home/runner/work/mtgo-collection-manager/mtgo-collection-manager/mtgoparser/build/test/Release/test_json_parse+0x284df4) (BuildId: d2706cb04a1220785cc016955bd3bfcd99a251c8)
7:
7: 0x6040000006b1 is located 0 bytes after 33-byte region [0x604000000690,0x6040000006b1)
7: allocated by thread T0 here:
7: #0 0x7f40168dfba8 in operator new(unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:95
7: #1 0x558e07db298a in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_mutate(unsigned long, unsigned long, char const*, unsigned long) (/home/runner/work/mtgo-collection-manager/mtgo-collection-manager/mtgoparser/build/test/Release/test_json_parse+0x2c198a) (BuildId: d2706cb04a1220785cc016955bd3bfcd99a251c8)
7:
18: #0 0x7fe0a46dfba8 in operator new(unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:95
18: #1 0x5645152380ca in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_mutate(unsigned long, unsigned long, char const*, unsigned long) (/home/runner/work/mtgo-collection-manager/mtgo-collection-manager/mtgoparser/build/test/Release/test_full_collection_parse+0x4470ca) (BuildId: 0fc462fdb26e01979a5a9f0c944d1e79c4809c3e)
18:
7: SUMMARY: AddressSanitizer: unknown-crash (/home/runner/work/mtgo-collection-manager/mtgo-collection-manager/mtgoparser/build/test/Release/test_json_parse+0x353725) (BuildId: d2706cb04a1220785cc016955bd3bfcd99a251c8) in void glz::detail::from_json<goatbots::CardDefinition>::op<glz::opts{10u, false, true, true, true, false, (char)32, (unsigned char)3, false, true, false, false, false, 0u, false, false, false, false, false, false, true, false, false, false, false, true}, glz::string_literal<1ul>{}, goatbots::CardDefinition&, glz::context&, char const*&, char const*&>(goatbots::CardDefinition&, glz::context&, char const*&, char const*&) [clone .lto_priv.0]
18: SUMMARY: AddressSanitizer: unknown-crash (/home/runner/work/mtgo-collection-manager/mtgo-collection-manager/mtgoparser/build/test/Release/test_full_collection_parse+0x4d6292) (BuildId: 0fc462fdb26e01979a5a9f0c944d1e79c4809c3e) in void glz::detail::from_json<scryfall::Card>::op<glz::opts{10u, false, true, true, true, false, (char)32, (unsigned char)3, false, true, false, false, false, 0u, false, false, false, false, false, false, true, false, false, true, false, true}, glz::string_literal<1ul>{}, scryfall::Card&, glz::context&, char const*&, char const*&>(scryfall::Card&, glz::context&, char const*&, char const*&) [clone .lto_priv.0]
18: Shadow bytes around the buggy address:
18: 0x604000001680: fa fa 00 00 00 00 06 fa fa fa 00 00 00 00 00 04
18: 0x604000001700: fa fa 00 00 00 00 06 fa fa fa 00 00 00 00 00 03
18: 0x604000001780: fa fa 00 00 00 00 00 fa fa fa 00 00 00 00 01 fa
18: 0x604000001800: fa fa 00 00 00 00 03 fa fa fa 00 00 00 00 01 fa
18: 0x604000001880: fa fa 00 00 00 00 00 fa fa fa 00 00 00 00 00 02
18: =>0x604000001900: fa fa 00 00 00 00 00 03 fa fa 00 00 00[00]01 fa
18: 0x604000001980: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
18: 0x604000001a00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
18: 0x604000001a80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
18: 0x604000001b00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
18: 0x604000001b80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
18: Shadow byte legend (one shadow byte represents 8 application bytes):
18: Addressable: 00
18: Partially addressable: 01 02 03 04 05 06 07
18: Heap left redzone: fa
18: Freed heap region: fd
18: Stack left redzone: f1
18: Stack mid redzone: f2
18: Stack right redzone: f3
18: Stack after return: f5
18: Stack use after scope: f8
18: Global redzone: f9
18: Global init order: f6
18: Poisoned by user: f7
18: Container overflow: fc
18: Array cookie: ac
18: Intra object redzone: bb
18: ASan internal: fe
18: Left alloca redzone: ca
18: Right alloca redzone: cb
7: Shadow bytes around the buggy address:
7: 0x604000000400: fa fa 00 00 00 00 04 fa fa fa 00 00 00 00 00 01
7: 0x604000000480: fa fa 00 00 00 00 03 fa fa fa 00 00 00 00 01 fa
7: 0x604000000500: fa fa 00 00 00 00 00 01 fa fa 00 00 00 00 01 fa
7: 0x604000000580: fa fa 00 00 00 00 01 fa fa fa 00 00 00 00 01 fa
7: 0x604000000600: fa fa 00 00 00 00 01 fa fa fa 00 00 00 00 01 fa
7: =>0x604000000680: fa fa 00 00 00[00]01 fa fa fa fa fa fa fa fa fa
7: 0x604000000700: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
7: 0x604000000780: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
7: 0x604000000800: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
7: 0x604000000880: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
7: 0x604000000900: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
7: Shadow byte legend (one shadow byte represents 8 application bytes):
7: Addressable: 00
7: Partially addressable: 01 02 03 04 05 06 07
7: Heap left redzone: fa
7: Freed heap region: fd
7: Stack left redzone: f1
7: Stack mid redzone: f2
7: Stack right redzone: f3
7: Stack after return: f5
7: Stack use after scope: f8
7: Global redzone: f9
7: Global init order: f6
7: Poisoned by user: f7
7: Container overflow: fc
7: Array cookie: ac
7: Intra object redzone: bb
7: ASan internal: fe
7: Left alloca redzone: ca
7: Right alloca redzone: cb
7: ==12638==ABORTING
18: ==12660==ABORTINGI'm not sure why this happens, and I'm unlikely to investigate it more as this project was already on ice for me, I only briefly revived it to try to contribute to this issue.
stephenberry
commented
5 months ago Thanks for the additional info. This was just fixed with a recent string reading address sanitizer fix (#773). I'm going to be making address sanitizer checks a part of the unit testing and GitHub Actions. So, hopefully we can avoid these in the future.
I'm using ASAN and have stumbled on a heap-buffer-overflow when calling write_json.
Here is a minimal code example:
It succeeds for shorter strings, but once you hit a length of 16 the problems start. I haven't had a chance to really dig into glaze's code, and perhaps some of it is just runtime dependent, but I also saw it work fine with string lengths of 23, 30, 31 characters.
The exception occurs at the following line:
std::memcpy(&swar, in, Bytes);For the moment I rolled back to an older working version. Let me know if you need more info or if I can be of any help in fixing this. I know you've done a lot of work in this library for performance, so I assume there are reasons for skipping certain safety checks (and inherent danger thereof) in the buffer manipulation that is happening in the
serialize_stringfunction.