Closed tobiasraabe closed 9 months ago
Loky uses cloudpickle under the hood to make it possible to call parallel execution on interactively or locally defined functions / class methods.
So investigating the root cause would probably involve in setting a break point somewhere appropriate in cloudpickle to find out what's wrong here.
This example contains an error and does not reproduce the problem I am facing. I will come back if I manage to replicate it. Thanks for your time 🙏.
Hi!
I have a package called pytask, which behaves almost like pytest in that it dynamically collects tasks (task and not test functions) from modules and executes them.
Then, there is also pytask-parallel (similar to pytest-xdist) that provides multiple backends to parallelize the execution of tasks, and one is loky. When executing tasks in parallel, I get an error with loky.
Here is a minimal example with a
task_example.py
which defines the task function and inmain.py
you find the runner that dynamically imports the function and executes it with loky.If you run
python main.py
, you see the following output.Interestingly, this error does not occur if you comment out the line with
sys.modules[module_name] = mod
.But, this line is necessary since you otherwise see errors with
dataclasses
: https://github.com/pytask-dev/pytask/pull/373.Using another backend like
concurrent.futures.ProcessPoolExecutor
does not lead to this error.I hope you have more insights into why this error is happening. If you need more info, I am happy to give it to you.
Thanks for looking into this issue! 🙏