search

njcuk9999 / apero-drs

A PipelinE to Reduce Observations - The DRS for SPIRou (CFHT)
MIT License
12 stars 0 forks source link

[APERO] Problem with pickling on Macs #702

Closed njcuk9999 closed 1 year ago

njcuk9999 commented 1 year ago

Similar problem as seem previously.

When using > 1 core Mac will complain about pickling.

Error from @clairem789

15:10:12.419-!!|PROC| E[01-010-00001]: Unhandled error has occurred: Error <class '_pickle.PicklingError'>
15:10:12.420-!!|PROC|
15:10:12.420-!!|PROC| Traceback (most recent call last):
15:10:12.421-!!|PROC|  File "/Users/moutou/apero-drs/apero/core/utils/drs_startup.py", line 432, in run
15:10:12.421-!!|PROC|   llmain = func(recipe, params)
15:10:12.421-!!|PROC|  File "/Users/moutou/apero-drs/tools/bin/apero_processing.py", line 208, in __main__
15:10:12.422-!!|PROC|   raise e
15:10:12.422-!!|PROC|  File "/Users/moutou/apero-drs/tools/bin/apero_processing.py", line 166, in __main__
15:10:12.422-!!|PROC|   rlist = drs_processing.generate_run_list(params, findexdbm, runtable,
15:10:12.422-!!|PROC|  File "/Users/moutou/apero-drs/apero/tools/module/processing/drs_processing.py", line 862, in generate_run_list
15:10:12.423-!!|PROC|   return generate_ids(params, findexdbm, runtable, skiptable, rlist)
15:10:12.423-!!|PROC|  File "/Users/moutou/apero-drs/apero/tools/module/processing/drs_processing.py", line 1281, in generate_ids
15:10:12.423-!!|PROC|   process.start()
15:10:12.423-!!|PROC|  File "/Users/moutou/miniconda3/envs/ap284/lib/python3.9/multiprocessing/process.py", line 121, in start
15:10:12.424-!!|PROC|   self._popen = self._Popen(self)
15:10:12.424-!!|PROC|  File "/Users/moutou/miniconda3/envs/ap284/lib/python3.9/multiprocessing/context.py", line 224, in _Popen
15:10:12.424-!!|PROC|   return _default_context.get_context().Process._Popen(process_obj)
15:10:12.424-!!|PROC|  File "/Users/moutou/miniconda3/envs/ap284/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
15:10:12.425-!!|PROC|   return Popen(process_obj)
15:10:12.425-!!|PROC|  File "/Users/moutou/miniconda3/envs/ap284/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
15:10:12.425-!!|PROC|   super().__init__(process_obj)
15:10:12.425-!!|PROC|  File "/Users/moutou/miniconda3/envs/ap284/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
15:10:12.426-!!|PROC|   self._launch(process_obj)
15:10:12.426-!!|PROC|  File "/Users/moutou/miniconda3/envs/ap284/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch
15:10:12.426-!!|PROC|   reduction.dump(process_obj, fp)
15:10:12.426-!!|PROC|  File "/Users/moutou/miniconda3/envs/ap284/lib/python3.9/multiprocessing/reduction.py", line 60, in dump
15:10:12.427-!!|PROC|   ForkingPickler(file, protocol).dump(obj)
15:10:12.427-!!|PROC| _pickle.PicklingError: Can't pickle <class 'pseudo_const.PseudoConstants'>: attribute lookup PseudoConstants on pseudo_const failed
15:10:12.427-!!|PROC|
15:10:12.448-**|PROC| ***************************************************************************
15:10:12.455-@!|PROC| W[40-003-00005]: Recipe apero_processing has NOT been successfully completed
15:10:12.462-**|PROC| ***************************************************************************
njcuk9999 commented 1 year ago

So I narrowed this down to the new multiprocessing in generate_ids (designed to make the validation and skipping N times faster).

This works on Linux machines it is very hard to test without a Mac.

I'm working in branch v0.7.285-live

So far I've tried adding a new way to multiprocess (with pathos) however on linux it just freezes when calling the multiprocessing function. (commit b76bc500ce2dba84789df1d6debae433fb7fd6b2)

Note pathos is a fork of multiprocess which uses dill instead of pickle - dill is meant to be able to serialize more than pickle (more of a hack if this works)

I've also tried removing "mod" as an argument to generate_id (commit 63f75f85c1ff17152c258c1e0100fa9bc08fcda7).

I've also tried removing "inrecipelist" and "indexdb" objects (instead sending in the recipe name and reloading the recipe class, and reloading the indexdb inside generate_id function (commit b76bc500ce2dba84789df1d6debae433fb7fd6b2)

Currently I don't know why pathos is not working.