Panic in python > 0.18.2 and <= 0.20.0 around tokio runtime

[dispalt]

Environment

Delta-rs version: 0.19.x and 0.20.0

Binding:

Environment:

• Cloud provider: dlt
• OS: docker.io/python:3.12 (so debian 12), also fedora, locally, fails
• Other: using through dagster and dlt although its happening locally with just dlt

---

Bug

What happened: It seems to be panicking on startup, this is the full stacktrace for 0.20.0

It looks like something that was happening here, so maybe dagster is forking? https://github.com/delta-io/delta-rs/issues/2744

it seems that the default method is to spawn

thread '<unnamed>' panicked at python/src/utils.rs:27:18:
Failed to record PID for tokio runtime.: 17
stack backtrace:
   0:     0x7f9fcffb1425 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h1b9dad2a88e955ff
   1:     0x7f9fcffdf71b - core::fmt::write::h4b5a1270214bc4a7
   2:     0x7f9fcffad96f - std::io::Write::write_fmt::hd04af345a50c312d
   3:     0x7f9fcffb2721 - std::panicking::default_hook::{{closure}}::h96ab15e9936be7ed
   4:     0x7f9fcffb23fc - std::panicking::default_hook::h3cacb9c27561ad33
   5:     0x7f9fcffb2d81 - std::panicking::rust_panic_with_hook::hfe205f6954b2c97b
   6:     0x7f9fcffb2be7 - std::panicking::begin_panic_handler::{{closure}}::h6cb44b3a50f28c44
   7:     0x7f9fcffb18e9 - std::sys::backtrace::__rust_end_short_backtrace::hf1c1f2a92799bb0e
   8:     0x7f9fcffb2874 - rust_begin_unwind
   9:     0x7f9fcc3c66f3 - core::panicking::panic_fmt::h3d8fc78294164da7
  10:     0x7f9fcc3c6bc6 - core::result::unwrap_failed::hfa79a499befff387
  11:     0x7f9fcc7671af - deltalake::utils::rt::hc867454adffc6e35
  12:     0x7f9fcc74338f - pyo3::marker::Python::allow_threads::hadc6b1bd3a240fc1
  13:     0x7f9fcc71209c - deltalake::RawDeltaTable::__pymethod___new____::hae794cb633174403
  14:     0x7f9fcc6f2d9c - pyo3::impl_::trampoline::trampoline::h5354ddcc56d7d643
  15:     0x7f9fcc711b11 - deltalake::<impl pyo3::impl_::pyclass::PyMethods<deltalake::RawDeltaTable> for pyo3::impl_::pyclass::PyClassImplCollector<deltalake::RawDeltaTable>>::py_methods::ITEMS::trampoline::had418f7cb47bb9e7
  16:     0x7fa006c3a0fa - type_call
                               at /usr/src/python/Objects/typeobject.c:1665:11
  17:     0x7fa006c3a03c - _PyObject_MakeTpCall
                               at /usr/src/python/Objects/call.c:240:18
  18:     0x7fa006c4307f - _PyEval_EvalFrameDefault
                               at /usr/src/python/Python/bytecodes.c:2715:19
  19:     0x7fa006c3b37d - _PyObject_FastCallDictTstate
                               at /usr/src/python/Objects/call.c:144:15
  20:     0x7fa006c5e6fc - _PyObject_Call_Prepend
                               at /usr/src/python/Objects/call.c:508:24
  21:     0x7fa006c5e6fc - slot_tp_init
                               at /usr/src/python/Objects/typeobject.c:9023:15
  22:     0x7fa006c3a136 - type_call
                               at /usr/src/python/Objects/typeobject.c:1677:19
  23:     0x7fa006c3a03c - _PyObject_MakeTpCall
                               at /usr/src/python/Objects/call.c:240:18
  24:     0x7fa006c4307f - _PyEval_EvalFrameDefault
                               at /usr/src/python/Python/bytecodes.c:2715:19
  25:     0x7fa006c6f389 - _PyFunction_Vectorcall
                               at /usr/src/python/Objects/call.c:419:16
  26:     0x7fa006c6f389 - _PyObject_VectorcallTstate
                               at /usr/src/python/./Include/internal/pycore_call.h:92:11
  27:     0x7fa006c6f389 - method_vectorcall
                               at /usr/src/python/Objects/classobject.c:69:20
  28:     0x7fa006bcac70 - thread_run
                               at /usr/src/python/./Modules/_threadmodule.c:1114
  29:     0x7fa006b8833c - pythread_wrapper
                               at /usr/src/python/Python/thread_pthread.h:237
  30:     0x7fa0068a8144 - <unknown>
  31:     0x7fa0069287dc - <unknown>
  32:                0x0 - <unknown>

What you expected to happen:

How to reproduce it:

In my case the first dlt load succeeds and most of the time any update to the data fails.

More details:

[ion-elgreco]

This is the PR that added this change, https://github.com/delta-io/delta-rs/pull/2765

Not clear why it fails setting the PID though, would have to see the underlying error

[dispalt]

This is the PR that added this change, #2765

Not clear why it fails setting the PID though, would have to see the underlying error

Any ideas how I would do that?

[ion-elgreco]

You will have to change the code to not use expect but panic in another way with the original error msg