search

Neuraxio / Neuraxle

The world's cleanest AutoML library ✨ - Do hyperparameter tuning with the right pipeline abstractions to write clean deep learning production pipelines. Let your pipeline steps have hyperparameter spaces. Design steps in your pipeline like components. Compatible with Scikit-Learn, TensorFlow, and most other libraries, frameworks and MLOps environments.
https://www.neuraxle.org/
Apache License 2.0
608 stars 62 forks source link

Bug: Multiprocessing error "cannot pickle 'weakref' object" under Python 3.9 and Mac OS #510

Closed guillaume-chevalier closed 3 years ago

guillaume-chevalier commented 3 years ago

Describe the bug Multithreading error when using processes instead of just threads.

To Reproduce

 TypeError
cannot pickle 'weakref' object
  File "/Users/guic/Documents/Neuraxle/neuraxle/distributed/streaming.py", line 197, in start
    p.start()
  File "/Users/guic/Documents/Neuraxle/neuraxle/distributed/streaming.py", line 519, in transform_data_container
    step.start(context)
  File "/Users/guic/Documents/Neuraxle/neuraxle/base.py", line 1287, in handle_transform
    data_container = self.transform_data_container(data_container, context)
  File "/Users/guic/Documents/Neuraxle/neuraxle/base.py", line 3537, in transform
    data_container = self.handle_transform(data_container, context)
  File "/Users/guic/Documents/Neuraxle/testing/test_streaming.py", line 323, in test_queued_pipeline_with_savers (Current frame)
    outputs = p.transform(list(range(100)))

Expected behavior No error

Suggested Fix Investigate on Python 3.8 to 3.9 changes, and if it can be caused by Mac OS.

Additional context Never tried on Mac OS Python 3.9 before. Coming from 3.8.