search

PPPLDeepLearning / plasma-python

PPPL deep learning disruption prediction package
http://tigress-web.princeton.edu/~alexeys/docs-web/html/
79 stars 43 forks source link

Mulitprocessing errors in pre-processing? #82

Open rkube opened 2 years ago

rkube commented 2 years ago

Hi, I'm trying to pre-process the FRNN dataset on traverse and get these errors when running guarantee_preprocessed.py. Does anyone know what the cause of this may be?

Traceback (most recent call last):
  File "/home/rkube/.conda/envs/frnn/lib/python3.8/site-packages/multiprocess-0.70.12.2-py3.8.egg/multiprocess/process.py", line 315, in _bootstrap
    self.run()
  File "/home/rkube/.conda/envs/frnn/lib/python3.8/site-packages/multiprocess-0.70.12.2-py3.8.egg/multiprocess/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rkube/.conda/envs/frnn/lib/python3.8/site-packages/multiprocess-0.70.12.2-py3.8.egg/multiprocess/pool.py", line 136, in worker
    put((job, i, (False, wrapped)))
  File "/home/rkube/.conda/envs/frnn/lib/python3.8/site-packages/multiprocess-0.70.12.2-py3.8.egg/multiprocess/queues.py", line 371, in put
    self._writer.send_bytes(obj)
  File "/home/rkube/.conda/envs/frnn/lib/python3.8/site-packages/multiprocess-0.70.12.2-py3.8.egg/multiprocess/connection.py", line 203, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/rkube/.conda/envs/frnn/lib/python3.8/site-packages/multiprocess-0.70.12.2-py3.8.egg/multiprocess/connection.py", line 414, in _send_bytes
    self._send(header + buf)
  File "/home/rkube/.conda/envs/frnn/lib/python3.8/site-packages/multiprocess-0.70.12.2-py3.8.egg/multiprocess/connection.py", line 371, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Signal Electron temperature profile, shot 150509 contains no data [omit]
...saved shot 150509
Process ForkPoolWorker-75:
Traceback (most recent call last):
  File "/home/rkube/.conda/envs/frnn/lib/python3.8/site-packages/multiprocess-0.70.12.2-py3.8.egg/multiprocess/pool.py", line 131, in worker
    put((job, i, result))
  File "/home/rkube/.conda/envs/frnn/lib/python3.8/site-packages/multiprocess-0.70.12.2-py3.8.egg/multiprocess/queues.py", line 371, in put
    self._writer.send_bytes(obj)
  File "/home/rkube/.conda/envs/frnn/lib/python3.8/site-packages/multiprocess-0.70.12.2-py3.8.egg/multiprocess/connection.py", line 203, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/rkube/.conda/envs/frnn/lib/python3.8/site-packages/multiprocess-0.70.12.2-py3.8.egg/multiprocess/connection.py", line 414, in _send_bytes
    self._send(header + buf)
  File "/home/rkube/.conda/envs/frnn/lib/python3.8/site-packages/multiprocess-0.70.12.2-py3.8.egg/multiprocess/connection.py", line 371, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rkube/.conda/envs/frnn/lib/python3.8/site-packages/multiprocess-0.70.12.2-py3.8.egg/multiprocess/process.py", line 315, in _bootstrap
    self.run()
  File "/home/rkube/.conda/envs/frnn/lib/python3.8/site-packages/multiprocess-0.70.12.2-py3.8.egg/multiprocess/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rkube/.conda/envs/frnn/lib/python3.8/site-packages/multiprocess-0.70.12.2-py3.8.egg/multiprocess/pool.py", line 136, in worker
    put((job, i, (False, wrapped)))
  File "/home/rkube/.conda/envs/frnn/lib/python3.8/site-packages/multiprocess-0.70.12.2-py3.8.egg/multiprocess/queues.py", line 371, in put
    self._writer.send_bytes(obj)
  File "/home/rkube/.conda/envs/frnn/lib/python3.8/site-packages/multiprocess-0.70.12.2-py3.8.egg/multiprocess/connection.py", line 203, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/rkube/.conda/envs/frnn/lib/python3.8/site-packages/multiprocess-0.70.12.2-py3.8.egg/multiprocess/connection.py", line 414, in _send_bytes
    self._send(header + buf)
  File "/home/rkube/.conda/envs/frnn/lib/python3.8/site-packages/multiprocess-0.70.12.2-py3.8.egg/multiprocess/connection.py", line 371, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
...saved shot 147205
Signal Electron temperature profile, shot 139545 contains no data [omit]
Process ForkPoolWorker-71:
rkube commented 2 years ago

I've replaced pathos with https://docs.python.org/3/library/multiprocessing.html and I'm getting the same errors.