search

python / cpython

The Python programming language
https://www.python.org
Other
63.89k stars 30.58k forks source link

Reading undefined value in pickle module w/memory sanitizer enabled #116550

Open wrongnull opened 9 months ago

wrongnull commented 9 months ago

Bug report

Bug description:

cpython configured in linux with:

CC=clang CXX=clang++ CXXFLAGS=$"-fsanitize=memory" CFLAGS="-fsanitize=memory" LDFLAGS="-fsanitize=memory" ./configure --with-pydebug

then ran the test

./python -m test -v test_pickle

and got the following stack trace from clang memory sanitizer:

test_persistence (test.test_pickle.CPersPicklerTests.test_persistence) ... ==15598==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x7f09ee2e57ee in load_int /home/wrongnull/projects/cpython/./Modules/_pickle.c:5094:19
    #1 0x7f09ee2e57ee in load /home/wrongnull/projects/cpython/./Modules/_pickle.c:6778:9
    #2 0x55dfab5a3d4c in method_vectorcall_FASTCALL_KEYWORDS_METHOD /home/wrongnull/projects/cpython/Objects/descrobject.c:380:24
    #3 0x55dfab586695 in _PyObject_VectorcallTstate /home/wrongnull/projects/cpython/./Include/internal/pycore_call.h:168:11
    #4 0x55dfab586695 in PyObject_Vectorcall /home/wrongnull/projects/cpython/Objects/call.c:327:12
    #5 0x55dfab86414f in _PyEval_EvalFrameDefault /home/wrongnull/projects/cpython/Python/generated_cases.c.h:817:23
    #6 0x55dfab58c629 in _PyObject_VectorcallTstate /home/wrongnull/projects/cpython/./Include/internal/pycore_call.h:168:11
    #7 0x55dfab58c629 in method_vectorcall /home/wrongnull/projects/cpython/Objects/classobject.c:92:18
    #8 0x55dfab860f66 in _PyEval_EvalFrameDefault /home/wrongnull/projects/cpython/Python/generated_cases.c.h:1252:26
    #9 0x55dfab585255 in _PyObject_VectorcallDictTstate /home/wrongnull/projects/cpython/Objects/call.c:135:15
    #10 0x55dfab587497 in _PyObject_Call_Prepend /home/wrongnull/projects/cpython/Objects/call.c:504:24
    #11 0x55dfab6d7911 in slot_tp_call /home/wrongnull/projects/cpython/Objects/typeobject.c:9111:15
    #12 0x55dfab5855a1 in _PyObject_MakeTpCall /home/wrongnull/projects/cpython/Objects/call.c:242:18
    #13 0x55dfab86414f in _PyEval_EvalFrameDefault /home/wrongnull/projects/cpython/Python/generated_cases.c.h:817:23
    #14 0x55dfab58c629 in _PyObject_VectorcallTstate /home/wrongnull/projects/cpython/./Include/internal/pycore_call.h:168:11
    #15 0x55dfab58c629 in method_vectorcall /home/wrongnull/projects/cpython/Objects/classobject.c:92:18
    #16 0x55dfab860f66 in _PyEval_EvalFrameDefault /home/wrongnull/projects/cpython/Python/generated_cases.c.h:1252:26
    #17 0x55dfab585255 in _PyObject_VectorcallDictTstate /home/wrongnull/projects/cpython/Objects/call.c:135:15
    #18 0x55dfab587497 in _PyObject_Call_Prepend /home/wrongnull/projects/cpython/Objects/call.c:504:24
    #19 0x55dfab6d7911 in slot_tp_call /home/wrongnull/projects/cpython/Objects/typeobject.c:9111:15
    #20 0x55dfab5855a1 in _PyObject_MakeTpCall /home/wrongnull/projects/cpython/Objects/call.c:242:18
    #21 0x55dfab86414f in _PyEval_EvalFrameDefault /home/wrongnull/projects/cpython/Python/generated_cases.c.h:817:23
    #22 0x55dfab58c629 in _PyObject_VectorcallTstate /home/wrongnull/projects/cpython/./Include/internal/pycore_call.h:168:11
    #23 0x55dfab58c629 in method_vectorcall /home/wrongnull/projects/cpython/Objects/classobject.c:92:18
    #24 0x55dfab860f66 in _PyEval_EvalFrameDefault /home/wrongnull/projects/cpython/Python/generated_cases.c.h:1252:26
    #25 0x55dfab585255 in _PyObject_VectorcallDictTstate /home/wrongnull/projects/cpython/Objects/call.c:135:15
    #26 0x55dfab587497 in _PyObject_Call_Prepend /home/wrongnull/projects/cpython/Objects/call.c:504:24
    #27 0x55dfab6d7911 in slot_tp_call /home/wrongnull/projects/cpython/Objects/typeobject.c:9111:15
    #28 0x55dfab5855a1 in _PyObject_MakeTpCall /home/wrongnull/projects/cpython/Objects/call.c:242:18
    #29 0x55dfab86414f in _PyEval_EvalFrameDefault /home/wrongnull/projects/cpython/Python/generated_cases.c.h:817:23
    #30 0x55dfab82b3f6 in _PyEval_EvalFrame /home/wrongnull/projects/cpython/./Include/internal/pycore_ceval.h:114:16
    #31 0x55dfab82b3f6 in _PyEval_Vector /home/wrongnull/projects/cpython/Python/ceval.c:1820:12
    #32 0x55dfab82b3f6 in PyEval_EvalCode /home/wrongnull/projects/cpython/Python/ceval.c:599:21
    #33 0x55dfab82481a in builtin_exec_impl /home/wrongnull/projects/cpython/Python/bltinmodule.c:1132:17
    #34 0x55dfab82481a in builtin_exec /home/wrongnull/projects/cpython/Python/clinic/bltinmodule.c.h:521:20
    #35 0x55dfab64ea35 in cfunction_vectorcall_FASTCALL_KEYWORDS /home/wrongnull/projects/cpython/Objects/methodobject.c:441:24
    #36 0x55dfab586695 in _PyObject_VectorcallTstate /home/wrongnull/projects/cpython/./Include/internal/pycore_call.h:168:11
    #37 0x55dfab586695 in PyObject_Vectorcall /home/wrongnull/projects/cpython/Objects/call.c:327:12
    #38 0x55dfab86414f in _PyEval_EvalFrameDefault /home/wrongnull/projects/cpython/Python/generated_cases.c.h:817:23
    #39 0x55dfab9c1e63 in pymain_run_module /home/wrongnull/projects/cpython/Modules/main.c:297:14
    #40 0x55dfab9c09f3 in pymain_run_python /home/wrongnull/projects/cpython/Modules/main.c:622:21
    #41 0x55dfab9c09f3 in Py_RunMain /home/wrongnull/projects/cpython/Modules/main.c:707:5
    #42 0x55dfab9c1c0c in pymain_main /home/wrongnull/projects/cpython/Modules/main.c:737:12
    #43 0x55dfab9c1cfe in Py_BytesMain /home/wrongnull/projects/cpython/Modules/main.c:761:12
    #44 0x7f09ef9060cf in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #45 0x7f09ef906188 in __libc_start_main csu/../csu/libc-start.c:360:3
    #46 0x55dfab441f24 in _start (/home/wrongnull/projects/cpython/python+0x9cf24) (BuildId: 38426e8be805b3772ff93442d6e53b59067ed95b)

SUMMARY: MemorySanitizer: use-of-uninitialized-value /home/wrongnull/projects/cpython/./Modules/_pickle.c:5094:19 in load_int
Exiting

In particular I'm not clear with the purpose of the following line https://github.com/python/cpython/blob/1e68c4b87633b17da1b602b86f5d23bbe106398f/Modules/_pickle.c#L5094

CPython versions tested on:

3.13

Operating systems tested on:

Linux

gaogaotiantian commented 9 months ago

Which version of clang are you using?

wrongnull commented 9 months ago

Which version of clang are you using?

This is reproduced in versions 16, 17 and 18 of clang

gaogaotiantian commented 9 months ago

Ah, I gave up. I can't even build Python with MSAN. Have similar issues as https://github.com/openssl/openssl/issues/17784 and from https://github.com/python/cpython/issues/79395#issuecomment-1689052679 it seems like the MSAN support is not that good for CPython. My guess is errno is somehow not set in strtol, but as I can't repro the issue, I can't move forward on this.

wrongnull commented 9 months ago

Ah, I gave up. I can't even build Python with MSAN. Have similar issues as openssl/openssl#17784 and from #79395 (comment) it seems like the MSAN support is not that good for CPython. My guess is errno is somehow not set in strtol, but as I can't repro the issue, I can't move forward on this.

TBH I have some issues with MSAN build of interpreter as well. I got the following stacktrace at the last step of building:

./python -E -S -m sysconfig --generate-posix-vars ;\
if test $? -ne 0 ; then \
        echo "generate-posix-vars failed" ; \
        rm -f ./pybuilddir.txt ; \
        exit 1 ; \
fi
./python -E -c 'import sys ; from sysconfig import get_platform ; print("%s-%d.%d" % (get_platform(), *sys.version_info[:2]))' >platform
Uninitialized bytes in __interceptor_fopen64 at offset 0 inside [0x70200000b400, 25)
==50784==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x7fcd53df6716 in BIO_new_file (/lib/x86_64-linux-gnu/libcrypto.so.3+0xe4716) (BuildId: b6484831f6d9d00d02367f3b7756115c57b53d2a)
    #1 0x7fcd53e2ee0c  (/lib/x86_64-linux-gnu/libcrypto.so.3+0x11ce0c) (BuildId: b6484831f6d9d00d02367f3b7756115c57b53d2a)
    #2 0x7fcd53e302b1 in CONF_modules_load_file_ex (/lib/x86_64-linux-gnu/libcrypto.so.3+0x11e2b1) (BuildId: b6484831f6d9d00d02367f3b7756115c57b53d2a)
    #3 0x7fcd53ed38ff  (/lib/x86_64-linux-gnu/libcrypto.so.3+0x1c18ff) (BuildId: b6484831f6d9d00d02367f3b7756115c57b53d2a)
    #4 0x7fcd56245cbe in __pthread_once_slow nptl/pthread_once.c:116:7
    #5 0x7fcd53ee1d9c in CRYPTO_THREAD_run_once (/lib/x86_64-linux-gnu/libcrypto.so.3+0x1cfd9c) (BuildId: b6484831f6d9d00d02367f3b7756115c57b53d2a)
    #6 0x7fcd53ed44f0 in OPENSSL_init_crypto (/lib/x86_64-linux-gnu/libcrypto.so.3+0x1c24f0) (BuildId: b6484831f6d9d00d02367f3b7756115c57b53d2a)
    #7 0x7fcd53ee8f8b in OBJ_obj2nid (/lib/x86_64-linux-gnu/libcrypto.so.3+0x1d6f8b) (BuildId: b6484831f6d9d00d02367f3b7756115c57b53d2a)
    #8 0x7fcd53dd1c76  (/lib/x86_64-linux-gnu/libcrypto.so.3+0xbfc76) (BuildId: b6484831f6d9d00d02367f3b7756115c57b53d2a)
    #9 0x7fcd53dd1f03 in d2i_ASN1_OBJECT (/lib/x86_64-linux-gnu/libcrypto.so.3+0xbff03) (BuildId: b6484831f6d9d00d02367f3b7756115c57b53d2a)
    #10 0x7fcd53ee92e0 in OBJ_txt2obj (/lib/x86_64-linux-gnu/libcrypto.so.3+0x1d72e0) (BuildId: b6484831f6d9d00d02367f3b7756115c57b53d2a)
    #11 0x7fcd542cdb4f in _ssl_txt2obj_impl /home/wrongnull/projects/cpython/./Modules/_ssl.c:5608:11
    #12 0x7fcd542cdb4f in _ssl_txt2obj /home/wrongnull/projects/cpython/./Modules/clinic/_ssl.c.h:1473:20
    #13 0x563c4b2c12c4 in cfunction_vectorcall_FASTCALL_KEYWORDS /home/wrongnull/projects/cpython/Objects/methodobject.c:441:24
    #14 0x563c4b1df16b in _PyObject_VectorcallTstate /home/wrongnull/projects/cpython/./Include/internal/pycore_call.h:168:11
    #15 0x563c4b504997 in _PyEval_EvalFrameDefault /home/wrongnull/projects/cpython/Python/generated_cases.c.h:1397:19
    #16 0x563c4b1e947b in _PyObject_VectorcallTstate /home/wrongnull/projects/cpython/./Include/internal/pycore_call.h:168:11
    #17 0x563c4b1e7471 in method_vectorcall /home/wrongnull/projects/cpython/Objects/classobject.c:92:18
    #18 0x563c4b1df16b in _PyObject_VectorcallTstate /home/wrongnull/projects/cpython/./Include/internal/pycore_call.h:168:11
    #19 0x563c4b1e470a in object_vacall /home/wrongnull/projects/cpython/Objects/call.c:819:14
    #20 0x563c4b1e4c96 in PyObject_CallFunctionObjArgs /home/wrongnull/projects/cpython/Objects/call.c:926:14
    #21 0x563c4b354ac9 in type_new_set_names /home/wrongnull/projects/cpython/Objects/typeobject.c:10328:25
    #22 0x563c4b354ac9 in type_new_impl /home/wrongnull/projects/cpython/Objects/typeobject.c:3982:9
    #23 0x563c4b354ac9 in type_new /home/wrongnull/projects/cpython/Objects/typeobject.c:4107:12
    #24 0x563c4b373faf in tp_new_wrapper /home/wrongnull/projects/cpython/Objects/typeobject.c:8647:11
    #25 0x563c4b2c2fbf in cfunction_call /home/wrongnull/projects/cpython/Objects/methodobject.c:540:18
    #26 0x563c4b1e1438 in _PyObject_Call /home/wrongnull/projects/cpython/Objects/call.c:361:18
    #27 0x563c4b4d355b in _PyEval_EvalFrameDefault /home/wrongnull/projects/cpython/Python/generated_cases.c.h:1252:26
    #28 0x563c4b1df547 in _PyObject_VectorcallDictTstate /home/wrongnull/projects/cpython/Objects/call.c:135:15
    #29 0x563c4b1e1d9c in _PyObject_Call_Prepend /home/wrongnull/projects/cpython/Objects/call.c:504:24
    #30 0x563c4b36965a in slot_tp_new /home/wrongnull/projects/cpython/Objects/typeobject.c:9385:14
    #31 0x563c4b34fc2d in type_call /home/wrongnull/projects/cpython/Objects/typeobject.c:1777:11
    #32 0x563c4b1dfa36 in _PyObject_MakeTpCall /home/wrongnull/projects/cpython/Objects/call.c:242:18
    #33 0x563c4b4af906 in builtin___build_class__ /home/wrongnull/projects/cpython/Python/bltinmodule.c:210:15
    #34 0x563c4b2c12c4 in cfunction_vectorcall_FASTCALL_KEYWORDS /home/wrongnull/projects/cpython/Objects/methodobject.c:441:24
    #35 0x563c4b1df16b in _PyObject_VectorcallTstate /home/wrongnull/projects/cpython/./Include/internal/pycore_call.h:168:11
    #36 0x563c4b4f82b2 in _PyEval_EvalFrameDefault /home/wrongnull/projects/cpython/Python/generated_cases.c.h:817:23
    #37 0x563c4b4bb9ab in _PyEval_EvalFrame /home/wrongnull/projects/cpython/./Include/internal/pycore_ceval.h:114:16
    #38 0x563c4b4bb9ab in _PyEval_Vector /home/wrongnull/projects/cpython/Python/ceval.c:1820:12
    #39 0x563c4b4bb9ab in PyEval_EvalCode /home/wrongnull/projects/cpython/Python/ceval.c:599:21
    #40 0x563c4b4b2d68 in builtin_exec_impl /home/wrongnull/projects/cpython/Python/bltinmodule.c:1132:17
    #41 0x563c4b4b2d68 in builtin_exec /home/wrongnull/projects/cpython/Python/clinic/bltinmodule.c.h:521:20
    #42 0x563c4b2c12c4 in cfunction_vectorcall_FASTCALL_KEYWORDS /home/wrongnull/projects/cpython/Objects/methodobject.c:441:24
    #43 0x563c4b4d355b in _PyEval_EvalFrameDefault /home/wrongnull/projects/cpython/Python/generated_cases.c.h:1252:26
    #44 0x563c4b1df16b in _PyObject_VectorcallTstate /home/wrongnull/projects/cpython/./Include/internal/pycore_call.h:168:11
    #45 0x563c4b1e470a in object_vacall /home/wrongnull/projects/cpython/Objects/call.c:819:14
    #46 0x563c4b1e4234 in PyObject_CallMethodObjArgs /home/wrongnull/projects/cpython/Objects/call.c:880:24
    #47 0x563c4b5cfd4a in import_find_and_load /home/wrongnull/projects/cpython/Python/import.c:2758:11
    #48 0x563c4b5cfd4a in PyImport_ImportModuleLevelObject /home/wrongnull/projects/cpython/Python/import.c:2838:15
    #49 0x563c4b4cee5b in import_name /home/wrongnull/projects/cpython/Python/ceval.c:2631:16
    #50 0x563c4b4cee5b in _PyEval_EvalFrameDefault /home/wrongnull/projects/cpython/Python/generated_cases.c.h:2957:19
    #51 0x563c4b4bb9ab in _PyEval_EvalFrame /home/wrongnull/projects/cpython/./Include/internal/pycore_ceval.h:114:16
    #52 0x563c4b4bb9ab in _PyEval_Vector /home/wrongnull/projects/cpython/Python/ceval.c:1820:12
    #53 0x563c4b4bb9ab in PyEval_EvalCode /home/wrongnull/projects/cpython/Python/ceval.c:599:21
    #54 0x563c4b4b2d68 in builtin_exec_impl /home/wrongnull/projects/cpython/Python/bltinmodule.c:1132:17
    #55 0x563c4b4b2d68 in builtin_exec /home/wrongnull/projects/cpython/Python/clinic/bltinmodule.c.h:521:20
    #56 0x563c4b2c12c4 in cfunction_vectorcall_FASTCALL_KEYWORDS /home/wrongnull/projects/cpython/Objects/methodobject.c:441:24
    #57 0x563c4b4d355b in _PyEval_EvalFrameDefault /home/wrongnull/projects/cpython/Python/generated_cases.c.h:1252:26
    #58 0x563c4b1df16b in _PyObject_VectorcallTstate /home/wrongnull/projects/cpython/./Include/internal/pycore_call.h:168:11
    #59 0x563c4b1e470a in object_vacall /home/wrongnull/projects/cpython/Objects/call.c:819:14
    #60 0x563c4b1e4234 in PyObject_CallMethodObjArgs /home/wrongnull/projects/cpython/Objects/call.c:880:24
    #61 0x563c4b5cfd4a in import_find_and_load /home/wrongnull/projects/cpython/Python/import.c:2758:11
    #62 0x563c4b5cfd4a in PyImport_ImportModuleLevelObject /home/wrongnull/projects/cpython/Python/import.c:2838:15
    #63 0x563c4b4cee5b in import_name /home/wrongnull/projects/cpython/Python/ceval.c:2631:16
    #64 0x563c4b4cee5b in _PyEval_EvalFrameDefault /home/wrongnull/projects/cpython/Python/generated_cases.c.h:2957:19
    #65 0x563c4b4bb9ab in _PyEval_EvalFrame /home/wrongnull/projects/cpython/./Include/internal/pycore_ceval.h:114:16
    #66 0x563c4b4bb9ab in _PyEval_Vector /home/wrongnull/projects/cpython/Python/ceval.c:1820:12
    #67 0x563c4b4bb9ab in PyEval_EvalCode /home/wrongnull/projects/cpython/Python/ceval.c:599:21
    #68 0x563c4b4b2d68 in builtin_exec_impl /home/wrongnull/projects/cpython/Python/bltinmodule.c:1132:17
    #69 0x563c4b4b2d68 in builtin_exec /home/wrongnull/projects/cpython/Python/clinic/bltinmodule.c.h:521:20
    #70 0x563c4b2c12c4 in cfunction_vectorcall_FASTCALL_KEYWORDS /home/wrongnull/projects/cpython/Objects/methodobject.c:441:24
    #71 0x563c4b4d355b in _PyEval_EvalFrameDefault /home/wrongnull/projects/cpython/Python/generated_cases.c.h:1252:26
    #72 0x563c4b1df16b in _PyObject_VectorcallTstate /home/wrongnull/projects/cpython/./Include/internal/pycore_call.h:168:11
    #73 0x563c4b1e470a in object_vacall /home/wrongnull/projects/cpython/Objects/call.c:819:14
    #74 0x563c4b1e4234 in PyObject_CallMethodObjArgs /home/wrongnull/projects/cpython/Objects/call.c:880:24
    #75 0x563c4b5cfd4a in import_find_and_load /home/wrongnull/projects/cpython/Python/import.c:2758:11
    #76 0x563c4b5cfd4a in PyImport_ImportModuleLevelObject /home/wrongnull/projects/cpython/Python/import.c:2838:15
    #77 0x563c4b4b053a in builtin___import___impl /home/wrongnull/projects/cpython/Python/bltinmodule.c:277:12
    #78 0x563c4b4b053a in builtin___import__ /home/wrongnull/projects/cpython/Python/clinic/bltinmodule.c.h:107:20
    #79 0x563c4b2c12c4 in cfunction_vectorcall_FASTCALL_KEYWORDS /home/wrongnull/projects/cpython/Objects/methodobject.c:441:24
    #80 0x563c4b1e21f7 in _PyObject_VectorcallTstate /home/wrongnull/projects/cpython/./Include/internal/pycore_call.h:168:11
    #81 0x563c4b1e21f7 in _PyObject_CallFunctionVa /home/wrongnull/projects/cpython/Objects/call.c:552:18
    #82 0x563c4b1e1fd8 in PyObject_CallFunction /home/wrongnull/projects/cpython/Objects/call.c:574:14
    #83 0x563c4b5cedbd in PyImport_Import /home/wrongnull/projects/cpython/Python/import.c:3023:9
    #84 0x563c4b5ceade in PyImport_ImportModule /home/wrongnull/projects/cpython/Python/import.c:2466:14
    #85 0x7fcd54ba247a in module_init /home/wrongnull/projects/cpython/./Modules/_asynciomodule.c:3631:26
    #86 0x7fcd54ba247a in module_exec /home/wrongnull/projects/cpython/./Modules/_asynciomodule.c:3758:9
    #87 0x563c4b2c6266 in PyModule_ExecDef /home/wrongnull/projects/cpython/Objects/moduleobject.c:442:23
    #88 0x563c4b5d642b in exec_builtin_or_dynamic /home/wrongnull/projects/cpython/Python/import.c:784:12
    #89 0x563c4b5d642b in _imp_exec_dynamic_impl /home/wrongnull/projects/cpython/Python/import.c:3774:12
    #90 0x563c4b5d642b in _imp_exec_dynamic /home/wrongnull/projects/cpython/Python/clinic/import.c.h:513:21
    #91 0x563c4b2c1991 in cfunction_vectorcall_O /home/wrongnull/projects/cpython/Objects/methodobject.c:512:24
    #92 0x563c4b4d355b in _PyEval_EvalFrameDefault /home/wrongnull/projects/cpython/Python/generated_cases.c.h:1252:26
    #93 0x563c4b4bb9ab in _PyEval_EvalFrame /home/wrongnull/projects/cpython/./Include/internal/pycore_ceval.h:114:16
    #94 0x563c4b4bb9ab in _PyEval_Vector /home/wrongnull/projects/cpython/Python/ceval.c:1820:12
    #95 0x563c4b4bb9ab in PyEval_EvalCode /home/wrongnull/projects/cpython/Python/ceval.c:599:21
    #96 0x563c4b665a91 in run_eval_code_obj /home/wrongnull/projects/cpython/Python/pythonrun.c:1291:9
    #97 0x563c4b664d96 in run_mod /home/wrongnull/projects/cpython/Python/pythonrun.c:1376:19
    #98 0x563c4b65f4f3 in pyrun_file /home/wrongnull/projects/cpython/Python/pythonrun.c:1212:15
    #99 0x563c4b65f4f3 in _PyRun_SimpleFileObject /home/wrongnull/projects/cpython/Python/pythonrun.c:461:13
    #100 0x563c4b65ec5e in _PyRun_AnyFileObject /home/wrongnull/projects/cpython/Python/pythonrun.c:77:15
    #101 0x563c4b6b7e23 in pymain_run_file_obj /home/wrongnull/projects/cpython/Modules/main.c:357:15
    #102 0x563c4b6b7e23 in pymain_run_file /home/wrongnull/projects/cpython/Modules/main.c:376:15
    #103 0x563c4b6b7e23 in pymain_run_python /home/wrongnull/projects/cpython/Modules/main.c:628:21
    #104 0x563c4b6b7e23 in Py_RunMain /home/wrongnull/projects/cpython/Modules/main.c:707:5
    #105 0x563c4b6b89e0 in pymain_main /home/wrongnull/projects/cpython/Modules/main.c:737:12
    #106 0x563c4b6b8aca in Py_BytesMain /home/wrongnull/projects/cpython/Modules/main.c:761:12
    #107 0x7fcd561d10cf in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #108 0x7fcd561d1188 in __libc_start_main csu/../csu/libc-start.c:360:3
    #109 0x563c4afeaf44 in _start (/home/wrongnull/projects/cpython/python+0x9df44) (BuildId: 808378f45460e7a3a98609cd122b229585b0f0f7)

SUMMARY: MemorySanitizer: use-of-uninitialized-value (/lib/x86_64-linux-gnu/libcrypto.so.3+0xe4716) (BuildId: b6484831f6d9d00d02367f3b7756115c57b53d2a) in BIO_new_file
Exiting
make: *** [Makefile:1471: checksharedmods] Error 1

However python executable itself has been compiled fine

wrongnull@DESKTOP-22AKPRT:~/projects/cpython$ ./python
Python 3.13.0a4+ (heads/main:7cee276d55, Mar 10 2024, 01:59:47) [Clang 16.0.6 (15)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
Legoclones commented 2 weeks ago

In particular I'm not clear with the purpose of the following line

https://github.com/python/cpython/blob/1e68c4b87633b17da1b602b86f5d23bbe106398f/Modules/_pickle.c#L5094

This is a great question and I haven't seen it answered yet so I'll take a shot myself. According to strtol() documentation, errno is set to ERANGE if the string has so many digits the value cannot be represented by a long, and either LONG_MAX or LONG_MIN is returned. This checks if errno is set, and if so, then the value is too great to be represented by long and we need to directly go from string to PyLong using PyLong_FromString().

What I don't understand is why the (*endptr != '\n' && *endptr != '\0') condition exists. endptr is set to the first invalid value encountered in the string. Assuming only valid characters are present in the string, endptr should be a newline because afaik the _Unpickler_Readline() function will always include the newline at the end of the line read. Assuming invalid characters exist (like 999z), it will then use the same logic as described above for when there's too many digits. I would think the code would either continue and ignore the invalid characters on the end, or throw an error if endptr is not a newline.

Perhaps the original coder 10+ years ago said something like "endptr should be a newline but if for some reason it's also a null byte we're probably fine, if it's anything else maybe it's just something weird with strtol so we'll try to run it through PyLong_FromString() and if that also fails (which it probably will), we'll just catch all invalid numbers there". So perhaps just to simplify catching invalid numbers?

I also have to note that the only edge case that I can think of where a number from strtol would have endptr not set to a newline AND PyLong_FromString() can successfully decode the number would be a number with trailing non-newline whitespace.

As for why this line was triggered by an uninitialized value error? Not sure, as errno was just set to 0 and endptr will always be set to the first non-decimal number, which should be a newline or user-specified non-decimal number. In either case, endptr should not be pointing to uninitialized memory. I would guess it's just the sanitizer being extra cautious.