search

MarshalX / python-libipld

🏎️ Fast Python library to work with IPLD: DAG-CBOR, CID, CAR, multibase
https://pypi.org/project/libipld/
MIT License
17 stars 2 forks source link

Use stable pyo3 v0.21.1 release #43

Closed MarshalX closed 2 months ago

MarshalX commented 7 months ago

who broke my project for the past 2 weeks 😢 let's compare the last successful run with today`s one:

diff:

codspeed-hq[bot] commented 7 months ago

CodSpeed Performance Report

Merging #43 will not alter performance

Comparing stable-pyo3 (3a8dcd7) with main (04ba564)

Summary

✅ 192 untouched benchmarks

MarshalX commented 7 months ago

@davidhewitt fyi. some random segfault after upgrading to rust 1.17.1. idk is it on my side or pyo3`s. i think it could be related to https://blog.rust-lang.org/2024/03/30/i128-layout-update.html. for now, i will stay on rust 1.16.0. didn't expect it from rust...

davidhewitt commented 7 months ago

That's a concern. Are you able to identify what code is segfaulting, does it reproduce locally? We should understand if it's a PyO3 or Rust problem, and either way it's probably going to be classed as a regression from stable-to-stable worth a bugfix (unless it really is related to the i128 layout changes).

MarshalX commented 7 months ago

@davidhewitt can't reproduce locally but probably because of my mac... (stable-aarch64-apple-darwin updated - rustc 1.77.1)

from blog post:

There are also no changes to architectures other than x86-32 and x86-64.

gonna try on remote machine

MarshalX commented 7 months ago

@davidhewitt I can't reproduce on stable-x86_64-unknown-linux-gnu updated - rustc 1.77.1 😢 i've tried 1:1 flow from CI. with PGO data gathering. exit code is always 0; tests are passed. i have no ideas...

davidhewitt commented 7 months ago

:(

Maybe can SSH into the runner and inspect the core dumps? See https://github.com/PyO3/pyo3/discussions/2781#discussioncomment-4228785 for ideas on how to do this.

MarshalX commented 7 months ago

@davidhewitt im sorry silly question what the difference between pip install -v -e . without installed rust toolchains and with? i mean unit test workflow passes and codspeed does not. one calls just pip install -v -e . the other one uses dtolnay/rust-toolchain and then calls pip install -v -e .

upd. ig unit tests passes baceuse github runner has pre-installed rust https://github.com/actions/runner-images/blob/main/images/ubuntu/scripts/build/install-rust.sh

davidhewitt commented 7 months ago

Yep preinstalled rust toolchain which is presumably older than 1.77 still.

MarshalX commented 7 months ago

@davidhewitt i ve had some success with CI debugging. here is what i got:

  1. segfault happens with builds that were built with env var RUSTFLAGS="-Cprofile-generate=profdata"
  2. When i pass debug=true in Cargo.toml segfault doesn't happen!
  3. with opt-level=2 segfault doesn't happen
  4. image
Text version ```shell Reading symbols from /opt/hostedtoolcache/Python/3.12.2/x64/bin/python3.12... [New LWP 12989] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/opt/hostedtoolcache/Python/3.12.2/x64/bin/python /opt/hostedtoolcache/Python/3'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007effeaeedba5 in __llvm_write_binary_ids () from /opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/libipld/libipld.cpython-312-x86_64-linux-gnu.so (gdb) bt #0 0x00007effeaeedba5 in __llvm_write_binary_ids () from /opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/libipld/libipld.cpython-312-x86_64-linux-gnu.so #1 0x00007effeaeee54f in lprofWriteDataImpl () from /opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/libipld/libipld.cpython-312-x86_64-linux-gnu.so #2 0x00007effeaeee42c in lprofWriteData () from /opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/libipld/libipld.cpython-312-x86_64-linux-gnu.so #3 0x00007effeaeec799 in writeFile () from /opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/libipld/libipld.cpython-312-x86_64-linux-gnu.so #4 0x00007effeaeec512 in __llvm_profile_write_file () from /opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/libipld/libipld.cpython-312-x86_64-linux-gnu.so #5 0x00007effecc45495 in __run_exit_handlers (status=0, listp=0x7effece1a838 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at ./stdlib/exit.c:113 #6 0x00007effecc45610 in __GI_exit (status=) at ./stdlib/exit.c:143 #7 0x00007effed3299ab in Py_Exit (sts=0) at Python/pylifecycle.c:3060 #8 0x00007effed32bcab in handle_system_exit () at Python/pythonrun.c:756 #9 0x00007effed32baf8 in _PyErr_PrintEx (tstate=0x7effed6ab648 <_PyRuntime+459656>, set_sys_last_vars=1) at Python/pythonrun.c:765 #10 0x00007effed1b5402 in PyErr_PrintEx (set_sys_last_vars=1) at Python/pythonrun.c:845 #11 PyErr_Print () at Python/pythonrun.c:851 #12 _PyRun_SimpleFileObject (fp=, filename=, closeit=1, flags=0x7ffeed8807f0) at Python/pythonrun.c:439 #13 0x00007effed32b6c7 in _PyRun_AnyFileObject (fp=fp@entry=0x55cf58114490, filename=filename@entry=0x7effecb2f870, closeit=closeit@entry=1, flags=flags@entry=0x7ffeed8807f0) at Python/pythonrun.c:78 #14 0x00007effed3343e8 in pymain_run_file_obj (skip_source_first_line=, filename=0x7effecb2f870, program_name=0x7effecb2f810) at Modules/main.c:360 #15 pymain_run_file (config=0x7effed64e228 <_PyRuntime+77672>) at Modules/main.c:379 #16 pymain_run_python (exitcode=0x7ffeed8807c4) at Modules/main.c:629 #17 Py_RunMain () at Modules/main.c:709 #18 0x00007effed333f4d in Py_BytesMain (argc=, argv=) at Modules/main.c:763 #19 0x00007effecc29d90 in __libc_start_call_main (main=main@entry=0x55cf56838060
, argc=argc@entry=3, argv=argv@entry=0x7ffeed880a48) at ../sysdeps/nptl/libc_start_call_main.h:58 #20 0x00007effecc29e40 in __libc_start_main_impl (main=0x55cf56838060
, argc=3, argv=0x7ffeed880a48, init=, fini=, rtld_fini=, stack_end=0x7ffeed880a38) at ../csu/libc-start.c:392 #21 0x000055cf56838095 in _start () ```
Dump summary ```shell PID: 12989 (pytest) UID: 1001 (runner) GID: 127 (docker) Signal: 11 (SEGV) Timestamp: Mon 2024-04-01 20:45:59 UTC (2min 35s ago) Command Line: /opt/hostedtoolcache/Python/3.12.2/x64/bin/python /opt/hostedtoolcache/Python/3.12.2/x64/bin/pytest . Executable: /opt/hostedtoolcache/Python/3.12.2/x64/bin/python3.12 Control Group: /system.slice/runner-provisioner.service Unit: runner-provisioner.service Slice: system.slice Boot ID: 5b47de7d31be4ba3ac7a1d4bfccc60c9 Machine ID: 87c9cb22384e417dbad323c8a7783269 Hostname: fv-az979-578 Storage: /var/lib/systemd/coredump/core.pytest.1001.5b47de7d31be4ba3ac7a1d4bfccc60c9.12989.1712004359000000.zst (present) Disk Size: 6.9M Message: Process 12989 (pytest) of user 1001 dumped core. Found module linux-vdso.so.1 with build-id: 151d9b890f2a92eb30984bdf8c8d264337b7b456 Found module libgcc_s.so.1 with build-id: 12424bb285e85ee967d17e4f87e4106889688eb0 Found module libipld.cpython-312-x86_64-linux-gnu.so with build-id: 43dc019f23e4f2fe07b7bbaf058486b239c29ac2 Found module libssl.so.3 with build-id: 9e2ad2d446d1e0e442154f9df9a3daf6a04bd645 Found module _ssl.cpython-312-x86_64-linux-gnu.so with build-id: 450f823f91c0465be9a47c253c91afe5ae848168 Found module _statistics.cpython-312-x86_64-linux-gnu.so with build-id: 3df44b2f27d3c7422a609431b5a7344bd7c3f886 Found module _lsprof.cpython-312-x86_64-linux-gnu.so with build-id: ca6e3d25651783d61b6f7c76a04f163ac5be2fb4 Found module _queue.cpython-312-x86_64-linux-gnu.so with build-id: e981ca495370f5ac867b7d5f8a49047034edad6c Found module _blake2.cpython-312-x86_64-linux-gnu.so with build-id: f308a79989edd16bff485378362a7b422e46fe60 Found module libcrypto.so.3 with build-id: ca84e22a798dabf117de600cb13469a59f775d2a Found module _hashlib.cpython-312-x86_64-linux-gnu.so with build-id: 8984f3ca2381a3cbb369f3c7aeb8a9b3122bf6ba Found module pyexpat.cpython-312-x86_64-linux-gnu.so with build-id: bdd1aac422932880036e73e05e1d39655b586095 Found module _elementtree.cpython-312-x86_64-linux-gnu.so with build-id: 0cf6d36b147ecd083fdedfe5d223be5597b9b30a Found module _posixsubprocess.cpython-312-x86_64-linux-gnu.so with build-id: c0fb0a4e59feaae1956db1a3c7d903e9110e45b8 Found module fcntl.cpython-312-x86_64-linux-gnu.so with build-id: b651a042c8c2c3cdf32921f7dbbbd534184d1941 Found module _decimal.cpython-312-x86_64-linux-gnu.so with build-id: 2c274387bfdebc25573c1ac2c460a76edadad83e Found module _heapq.cpython-312-x86_64-linux-gnu.so with build-id: d88205451b958a9c655ed5ef36b9aff8085b9505 Found module _json.cpython-312-x86_64-linux-gnu.so with build-id: f85a045dd58f1354cb1d2d787511bb54f65cfbe7 Found module _datetime.cpython-312-x86_64-linux-gnu.so with build-id: 9b9e29e510192e8bf237f2b1c0322a3bc78b0b1b Found module array.cpython-312-x86_64-linux-gnu.so with build-id: d5be42f2fde84d729406e16b01800aedca0bb38c Found module select.cpython-312-x86_64-linux-gnu.so with build-id: d9cb78bacb0d6ea54bbf2e4e614a1a122f0c8ee2 Found module _socket.cpython-312-x86_64-linux-gnu.so with build-id: ee6916c3154f6c453c8a1419cebdfa0c01c9b65d Found module _sha2.cpython-312-x86_64-linux-gnu.so with build-id: 502fc65c7626e053c47d67169d58ad116885ef9f Found module _random.cpython-312-x86_64-linux-gnu.so with build-id: 3b62fa7ba651f2ee03148ea4c73a898bcfcbcf24 Found module _struct.cpython-312-x86_64-linux-gnu.so with build-id: c64dcced7af8a3aa224d54cde5c4b9f07084637b Found module binascii.cpython-312-x86_64-linux-gnu.so with build-id: 2ea43e93b90075ab910aa5efe465f7248068dd9e Found module _csv.cpython-312-x86_64-linux-gnu.so with build-id: c9a54841ae7c5c8bdcb62b46b308ed3bc88605b6 Found module libuuid.so.1 with build-id: 64c0d0cb22fa2bdeca075a0c0418ba5ff314b220 Found module _uuid.cpython-312-x86_64-linux-gnu.so with build-id: cc667838deb0fa7cd4ca5c23173c377411360452 Found module unicodedata.cpython-312-x86_64-linux-gnu.so with build-id: dcdfe492981da31e3f725d4e08848754edfe88bd Found module liblzma.so.5 with build-id: b85da6c48eb60a646615392559483b93617ef265 Found module _lzma.cpython-312-x86_64-linux-gnu.so with build-id: 5c4273615de20517bea244ca4440675b311804a2 Found module libbz2.so.1.0 with build-id: e56b62c27bcc7ace8f9be36b255bd7b31bfde405 Found module _bz2.cpython-312-x86_64-linux-gnu.so with build-id: cbb7f493ef63c0e1b17e306b860badaa2919056c Found module libz.so.1 with build-id: 30840b79ac329ecbf1dec0bb60180eed256d319f Found module zlib.cpython-312-x86_64-linux-gnu.so with build-id: 764c9f059514c4616d487786a646236f5a101de4 Found module _bisect.cpython-312-x86_64-linux-gnu.so with build-id: c70b6223bdefdfcd8bb4fb6b1fe4ef4ac06fb44f Found module math.cpython-312-x86_64-linux-gnu.so with build-id: bb6fc1ebbfb4ffc9e3e37fd2eb696bc29fe3a6f7 Found module _opcode.cpython-312-x86_64-linux-gnu.so with build-id: b2b598c52d5048d7d5bfd6ef276047e65d430b4e Found module ld-linux-x86-64.so.2 with build-id: 15921ea631d9f36502d20459c43e5c85b7d6ab76 Found module libm.so.6 with build-id: a88ef0199bd5e742ebd0c359edf5cb2be0ec08b5 Found module libc.so.6 with build-id: c289da5071a3399de893d2af81d6a30c62646e1e Found module libpython3.12.so.1.0 with build-id: 6c2f4f5468d6052d6cfbbbc81c360981a2248326 Found module python3.12 with build-id: f4f4bbd3f484b21b9ba4ea26f7c7139b77fc2f12 Stack trace of thread 12989: #0 0x00007effeaeedba5 __llvm_write_binary_ids (libipld.cpython-312-x86_64-linux-gnu.so + 0xedba5) #1 0x00007effeaeee54f lprofWriteDataImpl (libipld.cpython-312-x86_64-linux-gnu.so + 0xee54f) #2 0x00007effeaeee42c lprofWriteData (libipld.cpython-312-x86_64-linux-gnu.so + 0xee42c) #3 0x00007effeaeec799 writeFile (libipld.cpython-312-x86_64-linux-gnu.so + 0xec799) #4 0x00007effeaeec512 __llvm_profile_write_file (libipld.cpython-312-x86_64-linux-gnu.so + 0xec512) #5 0x00007effecc45495 __run_exit_handlers (libc.so.6 + 0x45495) #6 0x00007effecc45610 __GI_exit (libc.so.6 + 0x45610) #7 0x00007effed3299ab Py_Exit (libpython3.12.so.1.0 + 0x3299ab) #8 0x00007effed32bcab handle_system_exit (libpython3.12.so.1.0 + 0x32bcab) #9 0x00007effed32baf8 _PyErr_PrintEx (libpython3.12.so.1.0 + 0x32baf8) #10 0x00007effed1b5402 PyErr_PrintEx (libpython3.12.so.1.0 + 0x1b5402) #11 0x00007effed32b6c7 _PyRun_AnyFileObject (libpython3.12.so.1.0 + 0x32b6c7) #12 0x00007effed3343e8 pymain_run_file_obj (libpython3.12.so.1.0 + 0x3343e8) #13 0x00007effed333f4d Py_BytesMain (libpython3.12.so.1.0 + 0x333f4d) #14 0x00007effecc29d90 __libc_start_call_main (libc.so.6 + 0x29d90) #15 0x00007effecc29e40 __libc_start_main_impl (libc.so.6 + 0x29e40) #16 0x000055cf56838095 _start (python3.12 + 0x1095) ```

Upd. found mention of segfault near of __llvm_write_binary_ids here: https://github.com/pydantic/pydantic-core/issues/451#issuecomment-1482788263 Upd2. they updated their forked llvm for rustc v1.17. and this PR introduces changes that now give segfault (ig) https://reviews.llvm.org/D146976 (commit in rust`s fork)

davidhewitt commented 7 months ago

I think probably the best course of action here is to report this to the rust-lang/rust main repo, hopefully someone working on the compiler has ideas.

MarshalX commented 7 months ago

@davidhewitt I started to prepare issue and go to CI SSH once again to gather proper info about rustc version and... it passes! i just rerun failed workflow from yesterday https://github.com/MarshalX/python-libipld/pull/44. well... some temporary magic from GHA runners? or maybe they notices some crashes/issues and fixed already

MarshalX commented 7 months ago

but in this PR still fails. i can't take it anymore 😢 i will just sit and wait for some fix from GitHub