I got a segfault on program exit when using hdf5 in a multithread (rayon) environment. Coredump:

coredumpctl gdb
           PID: 333137 (python)
           UID: 1000 (hendrik)
           GID: 1000 (hendrik)
        Signal: 11 (SEGV)
     Timestamp: Thu 2021-06-10 15:10:29 CEST (9s ago)
  Command Line: python -c from tests.test_pyclc import *; test_dataloader('~/data/hdf5')
    Executable: /usr/bin/python3.9
 Control Group: /user.slice/user-1000.slice/user@1000.service/app.slice/app-Alacritty.slice/app-Alacritty-307265.scope
          Unit: user@1000.service
     User Unit: app-Alacritty-307265.scope
         Slice: user-1000.slice
     Owner UID: 1000 (hendrik)
       Boot ID: f551cdaff1f44507b231985077fdc4ae
    Machine ID: fef964380b2449f7aa6d9beaddc5719b
      Hostname: T480s
       Storage: /var/lib/systemd/coredump/core.python.1000.f551cdaff1f44507b231985077fdc4ae.333137.1623330629000000.zst (present)
     Disk Size: 288.3M
       Message: Process 333137 (python) of user 1000 dumped core.

                Stack trace of thread 333168:
                #0  0x00007f28c411959d H5SL_release_common (libhdf5.so.103 + 0x20759d)
                #1  0x00007f28c4125ee5 H5SL_free (libhdf5.so.103 + 0x213ee5)
                #2  0x00007f28c3fb8503 H5D__chunk_io_term (libhdf5.so.103 + 0xa6503)
                #3  0x00007f28c3fd9e70 H5D__read (libhdf5.so.103 + 0xc7e70)
                #4  0x00007f28c3fda42a H5Dread (libhdf5.so.103 + 0xc842a)
                #5  0x00007f28c43472c6 n/a (/home/hendrik/.virtualenvs/clcrs/lib/python3.9/site-packages/pyclc.cpython-39-x86_64-linux-gnu.so + 0xb22c6)

Here is the full backtrace:

Thanks for reporting - wonder if you could come up with a minimal example that crashes?

(looks like a double free, perhaps?)

What version of hdf5-c did you use? If its reproducible using the static build you could include the address sanitizer, see in the github actions run addr_san in #139 for this.

@aldanor Yes I will try to provide a minimal example. A first guess would be due to calling rust via pyo3 wrapper (and only in release mode).

@mulimoen Do you mean libhdf5? I use the fedora stable version:

$ dnf info hdf5-devel
Last metadata expiration check: 0:00:20 ago on Thu 10 Jun 2021 10:18:49 PM CEST.
Installed Packages
Name         : hdf5-devel
Version      : 1.10.6
Release      : 5.fc34

I think this bug is most likely due to PyO3 not cleaning up correctly. The current process hierarchy is: Python --> PyO3 RustWrapper --> Rust Dataloader -> Submission thread -> Rayon parallel iterator (i.e. a rayon thread pool), which reads from hdf5. Rust dataloader holds hdf5 handles and passes them as immutable references to the rayon thread pool.

So I guess, PyO3 drops the Rust wrapper, but somehow the rayon thread pool lives on (maybe also the submission thread which holds the threadpool). This may result in a cleanup of the hdf5 handles while the thread pool still tries to access them. Stopping the Submission thread and thus also the rayon thread pool explicitly in python via atexit.register(), solves the issue.

holds hdf5 handles and passes them as immutable references to the rayon thread pool

Is this a &Handle or a Handle? The latter should be easier to use correctly, and should prevent the object from being doubly closed. Each Handle, whether it be a File, a Dataset or something else is rather like an Arc on the libhdf5 side of things.

Sorry for the late reply. My data structure is as follows:

struct Dataset {
    hdf5_handles: Vec<hdf5::File>,
}
struct DataLoader {
  ds: Arc<Dataset>,
}
impl DataLoader {
// ...
    fn start_worker(&mut self) -> std::thread::JoinHandle<()> {
        let ds = Arc::clone(self.ds);
        let thread_handle = std::thread::spawn(move || {
            0..10.par_bridge().for_each( || {  // par_bridge (rayon) iterates in parallel over range and results in parallel hdf5 access
                let data = ds.read_from_hdf5();
                // Send via some channel; closing the channel will stop this parallel loop
            });
        });
        thread_handle
    }
    fn drop(&mut self) {
        // Close the data output channel so that the rayon parallel iter stops
        // And join the worker thread afterwards
    }
}

When calling via pyo3, I think drop() is not called and the thread lives on until the segfault happens. Using pythons atexit() I can make sure that rayon threadpool and worker thread are joined, which resolves this segfault issue. I guess this issue is on the PyO3 side not on HDF5.

This is an ugly issue, I can replicate with (no python):

use hdf5::*;

fn main() {
    let file = File::create("file.h5").unwrap();
    let ds = file.new_dataset::<i32>().create("ds").unwrap();

    let h = std::thread::spawn(move || {
        std::mem::drop(ds);
    });
    // std::mem::drop(file);
    // h.join().unwrap();
}

I initially thought this was related to how we handle objects (see #139), but this made no difference. The fault happens when hdf5 is calling atexit (stdlib.h), invalidating all handles, but rust tries to clean up the (now gone) handle in the child thread on program exit. Thus the double free.

To fix this we can use H5dont_atexit on library initialisation. This usually guards against unclosed objects, but in Rust we guarantee closing objects with Drop.