sgkit-dev / bio2zarr

Convert bioinformatics file formats to Zarr
Apache License 2.0
27 stars 7 forks source link

Intermittent backtraces when running many dexplode-partitions #148

Closed jeromekelleher closed 6 months ago

jeromekelleher commented 6 months ago

When running dexplode-partition over 13517 partitions of chr2 in parallel using xargs, I see quite a few of these:

Exception ignored in: <module 'threading' from '/usr/lib/python3.9/threading.py'>
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 1415, in _shutdown
    atexit_call()
  File "/usr/lib/python3.9/concurrent/futures/process.py", line 93, in _python_exit
    thread_wakeup.wakeup()
  File "/usr/lib/python3.9/concurrent/futures/process.py", line 79, in wakeup
    self._writer.send_bytes(b"")
  File "/usr/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/usr/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes
    self._send(header + buf)
  File "/usr/lib/python3.9/multiprocessing/connection.py", line 373, in _send
    n = write(self._handle, buf)
OSError: [Errno 9] Bad file descriptor

Don't know what's going on here, will update as I get more info

benjeffery commented 6 months ago

I've not seen this FWIW

jeromekelleher commented 6 months ago

Closed in #164