Fixed MemoryError when decoding large definite strings

[agronholm]

Relates to #198.

[coveralls]

coverage: 93.169% (-0.07%) from 93.237% when pulling cbc78c5e704c39e0151a0e26853879000f20bbaf on fix-memoryerror into 0d54000bc17f677ac99cb82cc417589610d789d7 on master.

[mschwager]

I fuzzed this branch a bit, and it produced the following crash:

Output

``` ==1==ERROR: AddressSanitizer: heap-use-after-free on address 0x51600068f190 at pc 0xffff9530ccb8 bp 0xffffc175bb00 sp 0xffffc175baf8 READ of size 8 at 0x51600068f190 thread T0 #0 0xffff9530ccb4 in Py_SIZE /usr/include/python3.11/object.h:142:20 #1 0xffff9530ccb4 in PyBytes_GET_SIZE /usr/include/python3.11/cpython/bytesobject.h:45:12 #2 0xffff9530ccb4 in fp_read_object /app/cbor2/source/decoder.c:367:47 #3 0xffff95308d8c in decode_definite_bytestring /app/cbor2/source/decoder.c:554:21 #4 0xffff95308d8c in decode_bytestring /app/cbor2/source/decoder.c:625:15 #5 0xffff95306834 in decode /app/cbor2/source/decoder.c:1736:27 #6 0xffff9530b1d4 in decode_map /app/cbor2/source/decoder.c:914:33 #7 0xffff953068b4 in decode /app/cbor2/source/decoder.c:1739:27 #8 0xffff9530bea0 in decode_semantic /app/cbor2/source/decoder.c:988:29 #9 0xffff953068d4 in decode /app/cbor2/source/decoder.c:1740:27 #10 0xffff9530b460 in decode_map /app/cbor2/source/decoder.c:894:27 #11 0xffff953068b4 in decode /app/cbor2/source/decoder.c:1739:27 #12 0xffff9530a634 in decode_definite_array /app/cbor2/source/decoder.c:818:28 #13 0xffff9530a634 in decode_array /app/cbor2/source/decoder.c:875:16 #14 0xffff95306814 in decode /app/cbor2/source/decoder.c:1738:27 #15 0xffff95312550 in CBORDecoder_decode_stringref_ns /app/cbor2/source/decoder.c:1458:15 #16 0xffff9530be40 in decode_semantic /app/cbor2/source/decoder.c:977:31 #17 0xffff953068d4 in decode /app/cbor2/source/decoder.c:1740:27 #18 0xffff9530b47c in decode_map /app/cbor2/source/decoder.c:899:33 #19 0xffff953068b4 in decode /app/cbor2/source/decoder.c:1739:27 #20 0xffff9530b47c in decode_map /app/cbor2/source/decoder.c:899:33 #21 0xffff953068b4 in decode /app/cbor2/source/decoder.c:1739:27 #22 0xffff953107b8 in CBORDecoder_decode_bigfloat /app/cbor2/source/decoder.c:1246:13 #23 0xffff9530bff8 in decode_semantic /app/cbor2/source/decoder.c:969:31 #24 0xffff953068d4 in decode /app/cbor2/source/decoder.c:1740:27 #25 0xffff953358f0 in CBOR2_load /app/cbor2/source/module.c:318:19 #26 0xffff953358f0 in CBOR2_loads /app/cbor2/source/module.c:367:19 #27 0x4c9d58 (/usr/bin/python3.11+0x4c9d58) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #28 0x494544 in _PyObject_MakeTpCall (/usr/bin/python3.11+0x494544) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #29 0x4aa238 in _PyEval_EvalFrameDefault (/usr/bin/python3.11+0x4aa238) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #30 0x4e2ce8 in _PyFunction_Vectorcall (/usr/bin/python3.11+0x4e2ce8) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #31 0xffff955db6bc in pybind11::detail::simple_collector<(pybind11::return_value_policy)1>::call(_object*) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1502:47 #32 0xffff955db6bc in pybind11::object pybind11::detail::object_api::operator()<(pybind11::return_value_policy)1, pybind11::bytes>(pybind11::bytes&&) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1672:95 #33 0xffff955db6bc in pybind11::detail::type_caster, void>::load(pybind11::handle, bool)::func_wrapper::operator()(pybind11::bytes) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/functional.h:109:82 #34 0xffff955db6bc in void std::__invoke_impl, void>::load(pybind11::handle, bool)::func_wrapper&, pybind11::bytes>(std::__invoke_other, pybind11::detail::type_caster, void>::load(pybind11::handle, bool)::func_wrapper&, pybind11::bytes&&) /usr/include/c++/12/bits/invoke.h:61:36 #35 0xffff955db6bc in _ZSt10__invoke_rIvRZN8pybind116detail11type_casterISt8functionIFvNS0_5bytesEEEvE4loadENS0_6handleEbE12func_wrapperJS4_EENSt9enable_ifIXsrSt6__and_IJSt7is_voidIT_ESt14__is_invocableIT0_JDpT1_EEEE5valueESE_E4typeEOSH_DpOSI_ /usr/include/c++/12/bits/invoke.h:154:33 #36 0xffff955db6bc in std::_Function_handler, void>::load(pybind11::handle, bool)::func_wrapper>::_M_invoke(std::_Any_data const&, pybind11::bytes&&) /usr/include/c++/12/bits/std_function.h:290:30 #37 0xffff955c270c in std::function::operator()(pybind11::bytes) const /usr/include/c++/12/bits/std_function.h:591:9 #38 0xffff955c270c in void pybind11::detail::argument_loader::call_impl&, 0ul, pybind11::detail::void_type>(std::function&, std::integer_sequence, pybind11::detail::void_type&&) && /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1480:37 #39 0xffff955c270c in _ZNO8pybind116detail15argument_loaderIJNS_5bytesEEE4callIvNS0_9void_typeERSt8functionIFvS2_EEEENSt9enable_ifIXsrSt7is_voidIT_E5valueES5_E4typeEOT1_ /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1454:65 #40 0xffff955c270c in void pybind11::cpp_function::initialize&, void, pybind11::bytes, pybind11::return_value_policy>(std::function&, void (*)(pybind11::bytes), pybind11::return_value_policy const&)::'lambda1'(pybind11::detail::function_call&)::operator()(pybind11::detail::function_call&) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:254:75 #41 0xffff955c270c in void pybind11::cpp_function::initialize&, void, pybind11::bytes, pybind11::return_value_policy>(std::function&, void (*)(pybind11::bytes), pybind11::return_value_policy const&)::'lambda1'(pybind11::detail::function_call&)::_FUN(pybind11::detail::function_call&) /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:224:21 #42 0xffff955d1ee4 in pybind11::cpp_function::dispatcher(_object*, _object*, _object*) /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:946:35 #43 0x4c9d58 (/usr/bin/python3.11+0x4c9d58) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #44 0x5987d4 in PyObject_CallObject (/usr/bin/python3.11+0x5987d4) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #45 0xffff9509c0dc in pybind11::detail::simple_collector<(pybind11::return_value_policy)1>::call(_object*) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1502:47 #46 0xffff9509c0dc in pybind11::object pybind11::detail::object_api::operator()<(pybind11::return_value_policy)1, pybind11::bytes>(pybind11::bytes&&) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1672:95 #47 0xffff9509c0dc in pybind11::detail::type_caster, void>::load(pybind11::handle, bool)::func_wrapper::operator()(pybind11::bytes) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/functional.h:109:82 #48 0xffff9509c0dc in void std::__invoke_impl, void>::load(pybind11::handle, bool)::func_wrapper&, pybind11::bytes>(std::__invoke_other, pybind11::detail::type_caster, void>::load(pybind11::handle, bool)::func_wrapper&, pybind11::bytes&&) /usr/include/c++/12/bits/invoke.h:61:36 #49 0xffff9509c0dc in _ZSt10__invoke_rIvRZN8pybind116detail11type_casterISt8functionIFvNS0_5bytesEEEvE4loadENS0_6handleEbE12func_wrapperJS4_EENSt9enable_ifIXsrSt6__and_IJSt7is_voidIT_ESt14__is_invocableIT0_JDpT1_EEEE5valueESE_E4typeEOSH_DpOSI_ /usr/include/c++/12/bits/invoke.h:154:33 #50 0xffff9509c0dc in std::_Function_handler, void>::load(pybind11::handle, bool)::func_wrapper>::_M_invoke(std::_Any_data const&, pybind11::bytes&&) /usr/include/c++/12/bits/std_function.h:290:30 #51 0xffff95089d38 in std::function::operator()(pybind11::bytes) const /usr/include/c++/12/bits/std_function.h:591:9 #52 0xffff95089d38 in atheris::TestOneInput(unsigned char const*, unsigned long) /tmp/pip-install-ssq6l7v4/atheris_afc2a48c09d548c399b1f66614c10d64/src/native/core.cc:146:26 #53 0xffff98985308 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /home/tcwg-buildslave/workspace/tcwg-llvm-release/tcwg-jade-03/final/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:614:13 #54 0xffff98984bec in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) /home/tcwg-buildslave/workspace/tcwg-llvm-release/tcwg-jade-03/final/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:516:7 #55 0xffff989862d0 in fuzzer::Fuzzer::MutateAndTestOne() /home/tcwg-buildslave/workspace/tcwg-llvm-release/tcwg-jade-03/final/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:760:19 #56 0xffff9898712c in fuzzer::Fuzzer::Loop(std::__Fuzzer::vector>&) /home/tcwg-buildslave/workspace/tcwg-llvm-release/tcwg-jade-03/final/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:905:5 #57 0xffff98976d00 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /home/tcwg-buildslave/workspace/tcwg-llvm-release/tcwg-jade-03/final/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:911:6 #58 0xffff9508ac28 in atheris::start_fuzzing(std::vector, std::allocator>, std::allocator, std::allocator>>> const&, std::function const&) /tmp/pip-install-ssq6l7v4/atheris_afc2a48c09d548c399b1f66614c10d64/src/native/core.cc:226:15 #59 0xffff9509b9f8 in void pybind11::detail::argument_loader, std::allocator>, std::allocator, std::allocator>>> const&, std::function const&>::call_impl, std::allocator>, std::allocator, std::allocator>>> const&, std::function const&), 0ul, 1ul, pybind11::detail::void_type>(void (*&)(std::vector, std::allocator>, std::allocator, std::allocator>>> const&, std::function const&), std::integer_sequence, pybind11::detail::void_type&&) && /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1480:37 #60 0xffff9509b9f8 in _ZNO8pybind116detail15argument_loaderIJRKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS8_EERKSt8functionIFvNS_5bytesEEEEE4callIvNS0_9void_typeERPFvSC_SI_EEENSt9enable_ifIXsrSt7is_voidIT_E5valueESL_E4typeEOT1_ /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1454:65 #61 0xffff9509b9f8 in void pybind11::cpp_function::initialize, std::allocator>, std::allocator, std::allocator>>> const&, std::function const&), void, std::vector, std::allocator>, std::allocator, std::allocator>>> const&, std::function const&, pybind11::name, pybind11::scope, pybind11::sibling>(void (*&)(std::vector, std::allocator>, std::allocator, std::allocator>>> const&, std::function const&), void (*)(std::vector, std::allocator>, std::allocator, std::allocator>>> const&, std::function const&), pybind11::name const&, pybind11::scope const&, pybind11::sibling const&)::'lambda1'(pybind11::detail::function_call&)::operator()(pybind11::detail::function_call&) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:254:75 #62 0xffff9509b9f8 in void pybind11::cpp_function::initialize, std::allocator>, std::allocator, std::allocator>>> const&, std::function const&), void, std::vector, std::allocator>, std::allocator, std::allocator>>> const&, std::function const&, pybind11::name, pybind11::scope, pybind11::sibling>(void (*&)(std::vector, std::allocator>, std::allocator, std::allocator>>> const&, std::function const&), void (*)(std::vector, std::allocator>, std::allocator, std::allocator>>> const&, std::function const&), pybind11::name const&, pybind11::scope const&, pybind11::sibling const&)::'lambda1'(pybind11::detail::function_call&)::_FUN(pybind11::detail::function_call&) /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:224:21 #63 0xffff950988b8 in pybind11::cpp_function::dispatcher(_object*, _object*, _object*) /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:946:35 #64 0x4c9d58 (/usr/bin/python3.11+0x4c9d58) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #65 0x5987d4 in PyObject_CallObject (/usr/bin/python3.11+0x5987d4) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #66 0xffff955bbdac in pybind11::detail::simple_collector<(pybind11::return_value_policy)1>::call(_object*) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1502:47 #67 0xffff955bbdac in pybind11::object pybind11::detail::object_api>::operator()<(pybind11::return_value_policy)1, std::vector, std::allocator>, std::allocator, std::allocator>>>&, std::function&>(std::vector, std::allocator>, std::allocator, std::allocator>>>&, std::function&) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1672:95 #68 0xffff955bbdac in atheris::Fuzz() /tmp/pip-install-ssq6l7v4/atheris_afc2a48c09d548c399b1f66614c10d64/src/native/atheris.cc:249:29 #69 0xffff955c1540 in void pybind11::detail::argument_loader<>::call_impl(void (*&)(), std::integer_sequence, pybind11::detail::void_type&&) && /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1480:37 #70 0xffff955c1540 in _ZNO8pybind116detail15argument_loaderIJEE4callIvNS0_9void_typeERPFvvEEENSt9enable_ifIXsrSt7is_voidIT_E5valueES4_E4typeEOT1_ /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1454:65 #71 0xffff955c1540 in void pybind11::cpp_function::initialize(void (*&)(), void (*)(), pybind11::name const&, pybind11::scope const&, pybind11::sibling const&)::'lambda1'(pybind11::detail::function_call&)::operator()(pybind11::detail::function_call&) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:254:75 #72 0xffff955c1540 in void pybind11::cpp_function::initialize(void (*&)(), void (*)(), pybind11::name const&, pybind11::scope const&, pybind11::sibling const&)::'lambda1'(pybind11::detail::function_call&)::_FUN(pybind11::detail::function_call&) /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:224:21 #73 0xffff955d1ee4 in pybind11::cpp_function::dispatcher(_object*, _object*, _object*) /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:946:35 #74 0x4c9d58 (/usr/bin/python3.11+0x4c9d58) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #75 0x494544 in _PyObject_MakeTpCall (/usr/bin/python3.11+0x494544) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #76 0x4aa238 in _PyEval_EvalFrameDefault (/usr/bin/python3.11+0x4aa238) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #77 0x4a0b5c in PyEval_EvalCode (/usr/bin/python3.11+0x4a0b5c) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #78 0x5fafa4 (/usr/bin/python3.11+0x5fafa4) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #79 0x5f7bcc (/usr/bin/python3.11+0x5f7bcc) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #80 0x60875c (/usr/bin/python3.11+0x60875c) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #81 0x608304 in _PyRun_SimpleFileObject (/usr/bin/python3.11+0x608304) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #82 0x60806c in _PyRun_AnyFileObject (/usr/bin/python3.11+0x60806c) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #83 0x606318 in Py_RunMain (/usr/bin/python3.11+0x606318) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #84 0x5d0150 in Py_BytesMain (/usr/bin/python3.11+0x5d0150) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #85 0xffff9866777c (/lib/aarch64-linux-gnu/libc.so.6+0x2777c) (BuildId: 122e8b69a986ce5b1fde3a0fa5d5c4fd522c701f) #86 0xffff98667854 in __libc_start_main (/lib/aarch64-linux-gnu/libc.so.6+0x27854) (BuildId: 122e8b69a986ce5b1fde3a0fa5d5c4fd522c701f) #87 0x5cffec in _start (/usr/bin/python3.11+0x5cffec) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) 0x51600068f190 is located 16 bytes inside of 518-byte region [0x51600068f180,0x51600068f386) freed by thread T0 here: #0 0xffff98a99a9c in free /home/tcwg-buildslave/workspace/tcwg-llvm-release/tcwg-jade-03/final/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:52:3 #1 0x4c00f4 (/usr/bin/python3.11+0x4c00f4) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #2 0xffff9530cac4 in Py_DECREF /usr/include/python3.11/object.h:538:9 #3 0xffff9530cac4 in fp_read_object /app/cbor2/source/decoder.c:363:17 #4 0xffff95308d8c in decode_definite_bytestring /app/cbor2/source/decoder.c:554:21 #5 0xffff95308d8c in decode_bytestring /app/cbor2/source/decoder.c:625:15 #6 0xffff95306834 in decode /app/cbor2/source/decoder.c:1736:27 #7 0xffff9530b1d4 in decode_map /app/cbor2/source/decoder.c:914:33 #8 0xffff953068b4 in decode /app/cbor2/source/decoder.c:1739:27 #9 0xffff9530bea0 in decode_semantic /app/cbor2/source/decoder.c:988:29 #10 0xffff953068d4 in decode /app/cbor2/source/decoder.c:1740:27 #11 0xffff9530b460 in decode_map /app/cbor2/source/decoder.c:894:27 #12 0xffff953068b4 in decode /app/cbor2/source/decoder.c:1739:27 #13 0xffff9530a634 in decode_definite_array /app/cbor2/source/decoder.c:818:28 #14 0xffff9530a634 in decode_array /app/cbor2/source/decoder.c:875:16 #15 0xffff95306814 in decode /app/cbor2/source/decoder.c:1738:27 #16 0xffff95312550 in CBORDecoder_decode_stringref_ns /app/cbor2/source/decoder.c:1458:15 #17 0xffff9530be40 in decode_semantic /app/cbor2/source/decoder.c:977:31 #18 0xffff953068d4 in decode /app/cbor2/source/decoder.c:1740:27 #19 0xffff9530b47c in decode_map /app/cbor2/source/decoder.c:899:33 #20 0xffff953068b4 in decode /app/cbor2/source/decoder.c:1739:27 #21 0xffff9530b47c in decode_map /app/cbor2/source/decoder.c:899:33 #22 0xffff953068b4 in decode /app/cbor2/source/decoder.c:1739:27 #23 0xffff953107b8 in CBORDecoder_decode_bigfloat /app/cbor2/source/decoder.c:1246:13 #24 0xffff9530bff8 in decode_semantic /app/cbor2/source/decoder.c:969:31 #25 0xffff953068d4 in decode /app/cbor2/source/decoder.c:1740:27 #26 0xffff953358f0 in CBOR2_load /app/cbor2/source/module.c:318:19 #27 0xffff953358f0 in CBOR2_loads /app/cbor2/source/module.c:367:19 #28 0x4c9d58 (/usr/bin/python3.11+0x4c9d58) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #29 0x494544 in _PyObject_MakeTpCall (/usr/bin/python3.11+0x494544) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #30 0x4aa238 in _PyEval_EvalFrameDefault (/usr/bin/python3.11+0x4aa238) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #31 0x4e2ce8 in _PyFunction_Vectorcall (/usr/bin/python3.11+0x4e2ce8) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #32 0xffff955db6bc in pybind11::detail::simple_collector<(pybind11::return_value_policy)1>::call(_object*) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1502:47 #33 0xffff955db6bc in pybind11::object pybind11::detail::object_api::operator()<(pybind11::return_value_policy)1, pybind11::bytes>(pybind11::bytes&&) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1672:95 #34 0xffff955db6bc in pybind11::detail::type_caster, void>::load(pybind11::handle, bool)::func_wrapper::operator()(pybind11::bytes) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/functional.h:109:82 #35 0xffff955db6bc in void std::__invoke_impl, void>::load(pybind11::handle, bool)::func_wrapper&, pybind11::bytes>(std::__invoke_other, pybind11::detail::type_caster, void>::load(pybind11::handle, bool)::func_wrapper&, pybind11::bytes&&) /usr/include/c++/12/bits/invoke.h:61:36 #36 0xffff955db6bc in _ZSt10__invoke_rIvRZN8pybind116detail11type_casterISt8functionIFvNS0_5bytesEEEvE4loadENS0_6handleEbE12func_wrapperJS4_EENSt9enable_ifIXsrSt6__and_IJSt7is_voidIT_ESt14__is_invocableIT0_JDpT1_EEEE5valueESE_E4typeEOSH_DpOSI_ /usr/include/c++/12/bits/invoke.h:154:33 #37 0xffff955db6bc in std::_Function_handler, void>::load(pybind11::handle, bool)::func_wrapper>::_M_invoke(std::_Any_data const&, pybind11::bytes&&) /usr/include/c++/12/bits/std_function.h:290:30 #38 0xffff955c270c in std::function::operator()(pybind11::bytes) const /usr/include/c++/12/bits/std_function.h:591:9 #39 0xffff955c270c in void pybind11::detail::argument_loader::call_impl&, 0ul, pybind11::detail::void_type>(std::function&, std::integer_sequence, pybind11::detail::void_type&&) && /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1480:37 #40 0xffff955c270c in _ZNO8pybind116detail15argument_loaderIJNS_5bytesEEE4callIvNS0_9void_typeERSt8functionIFvS2_EEEENSt9enable_ifIXsrSt7is_voidIT_E5valueES5_E4typeEOT1_ /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1454:65 #41 0xffff955c270c in void pybind11::cpp_function::initialize&, void, pybind11::bytes, pybind11::return_value_policy>(std::function&, void (*)(pybind11::bytes), pybind11::return_value_policy const&)::'lambda1'(pybind11::detail::function_call&)::operator()(pybind11::detail::function_call&) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:254:75 #42 0xffff955c270c in void pybind11::cpp_function::initialize&, void, pybind11::bytes, pybind11::return_value_policy>(std::function&, void (*)(pybind11::bytes), pybind11::return_value_policy const&)::'lambda1'(pybind11::detail::function_call&)::_FUN(pybind11::detail::function_call&) /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:224:21 previously allocated by thread T0 here: #0 0xffff98a99d30 in malloc /home/tcwg-buildslave/workspace/tcwg-llvm-release/tcwg-jade-03/final/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:69:3 #1 0x5a25d4 (/usr/bin/python3.11+0x5a25d4) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #2 0x4d10ec (/usr/bin/python3.11+0x4d10ec) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #3 0x4c800c (/usr/bin/python3.11+0x4c800c) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #4 0x4c7988 in PyObject_CallFunctionObjArgs (/usr/bin/python3.11+0x4c7988) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #5 0xffff9530c920 in fp_read_object /app/cbor2/source/decoder.c:356:15 #6 0xffff95308d8c in decode_definite_bytestring /app/cbor2/source/decoder.c:554:21 #7 0xffff95308d8c in decode_bytestring /app/cbor2/source/decoder.c:625:15 #8 0xffff95306834 in decode /app/cbor2/source/decoder.c:1736:27 #9 0xffff9530b1d4 in decode_map /app/cbor2/source/decoder.c:914:33 #10 0xffff953068b4 in decode /app/cbor2/source/decoder.c:1739:27 #11 0xffff9530bea0 in decode_semantic /app/cbor2/source/decoder.c:988:29 #12 0xffff953068d4 in decode /app/cbor2/source/decoder.c:1740:27 #13 0xffff9530b460 in decode_map /app/cbor2/source/decoder.c:894:27 #14 0xffff953068b4 in decode /app/cbor2/source/decoder.c:1739:27 #15 0xffff9530a634 in decode_definite_array /app/cbor2/source/decoder.c:818:28 #16 0xffff9530a634 in decode_array /app/cbor2/source/decoder.c:875:16 #17 0xffff95306814 in decode /app/cbor2/source/decoder.c:1738:27 #18 0xffff95312550 in CBORDecoder_decode_stringref_ns /app/cbor2/source/decoder.c:1458:15 #19 0xffff9530be40 in decode_semantic /app/cbor2/source/decoder.c:977:31 #20 0xffff953068d4 in decode /app/cbor2/source/decoder.c:1740:27 #21 0xffff9530b47c in decode_map /app/cbor2/source/decoder.c:899:33 #22 0xffff953068b4 in decode /app/cbor2/source/decoder.c:1739:27 #23 0xffff9530b47c in decode_map /app/cbor2/source/decoder.c:899:33 #24 0xffff953068b4 in decode /app/cbor2/source/decoder.c:1739:27 #25 0xffff953107b8 in CBORDecoder_decode_bigfloat /app/cbor2/source/decoder.c:1246:13 #26 0xffff9530bff8 in decode_semantic /app/cbor2/source/decoder.c:969:31 #27 0xffff953068d4 in decode /app/cbor2/source/decoder.c:1740:27 #28 0xffff953358f0 in CBOR2_load /app/cbor2/source/module.c:318:19 #29 0xffff953358f0 in CBOR2_loads /app/cbor2/source/module.c:367:19 #30 0x4c9d58 (/usr/bin/python3.11+0x4c9d58) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #31 0x494544 in _PyObject_MakeTpCall (/usr/bin/python3.11+0x494544) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) #32 0x4aa238 in _PyEval_EvalFrameDefault (/usr/bin/python3.11+0x4aa238) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711) SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/python3.11/object.h:142:20 in Py_SIZE Shadow bytes around the buggy address: 0x51600068ef00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x51600068ef80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x51600068f000: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x51600068f080: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x51600068f100: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa =>0x51600068f180: fd fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd 0x51600068f200: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x51600068f280: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x51600068f300: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x51600068f380: fd fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x51600068f400: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==1==ABORTING MS: 4 InsertRepeatedBytes-InsertRepeatedBytes-InsertByte-CMP- DE: "~\377\377\377\377\377\377\336"-; base unit: b4d12705e8bb56c5481ecec9ebfda713fc9676b1 artifact_prefix='/tmp/output/'; Test unit written to /tmp/output/crash-54a4c03551e10f514542bdf35c00142e617b1ff1 ```

I've attached the crash here: crash-54a4c03551e10f514542bdf35c00142e617b1ff1.txt

[mschwager]

Would you be interested in having your project be a part of OSS-Fuzz? That could help with automatically finding crashes and bugs for you. I could take care of adding it as a new project - all I'd need is a primary contact email address.

[agronholm]

Sounds okay. Is it comparable with Hypothesis?

[agronholm]

And my primary contact address is alex.gronholm@nextday.fi.

[mschwager]

And my primary contact address is alex.gronholm@nextday.fi.

Hmm, the adding new projects docs say that a Google account is required. Is that email address connected to a Google account by chance?

Fuzzing testing is similar in some ways to Hypothesis. OSS-Fuzz is a project by Google to provide free compute cycles to fuzz OSS software. For more information on fuzzing, I'd recommend starting here.

[agronholm]

Yes, that's associated with a Google account.

[agronholm]

I've attached the crash here: crash-54a4c03551e10f514542bdf35c00142e617b1ff1.txt

What do I do with this if I want to reproduce the crash?

[mschwager]

I've attached the crash here: crash-54a4c03551e10f514542bdf35c00142e617b1ff1.txt

What do I do with this if I want to reproduce the crash?

I tested it out like this:

$ python -m cbor2.tool -p crash-54a4c03551e10f514542bdf35c00142e617b1ff1.txt 
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "lib/python3.11/site-packages/cbor2/tool.py", line 225, in <module>
    main()
  File "lib/python3.11/site-packages/cbor2/tool.py", line 208, in main
    objs = (load(infile, tag_hook=my_hook),)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
MemoryError

[agronholm]

Alright, I was able to reproduce the crash here. I think I need to start fuzzing the code before I push anything to master.

[agronholm]

Looks like this is just another case of allocating a huge amount of memory (4629771061636907009) bytes, but this triggers a MemoryError with the Python implementation too.

[agronholm]

Turns out that the problem wasn't triggered with a BytesIO instance, but it is triggered with a real open file.

[agronholm]

All other Python CBOR implementations I tested with also raised a MemoryError when trying to decode input like that.

[agronholm]

I have a fix that avoids MemoryError in both implementations, but it introduced a new bug on the C side that I'm still tracking down.

[agronholm]

Alright, what's left now is to fuzz this branch before I merge.

[agronholm]

I'm not getting any MemoryError anymore with fuzzing. Would you like to verify?

[mschwager]

I'm not getting any MemoryError anymore with fuzzing. Would you like to verify?

I found another crash with the following file: crash-c528afcec87be909de91322a14693702fd1d44a0.txt

I think I'm correctly fuzzing this branch, but I'm not sure. Are you able to reproduce the crash?

[agronholm]

Yeah, I can reproduce it. Looking into it now.

[agronholm]

No, wait, I forgot to recompile after switching branches. I'm getting this instead now: _cbor2.CBORDecodeEOF: premature end of stream (expected to read 65536 bytes, got 509 instead). This is the exception we should be getting, yes?

[agronholm]

I'm not seeing anything bad in this branch after the commit I just pushed a little while ago, so I'm merging it.

[mschwager]

Looks like the OSS-Fuzz PR went through: https://github.com/google/oss-fuzz/pull/11444 🎉