Closed ma-sadeghi closed 4 months ago
moved from https://github.com/MilesCranmer/PySR/pull/535:
@ma-sadeghi: @MilesCranmer Was there any trick to make Julia multiprocessing work with JuliaCall? I used to use PyJulia, and it worked fine calling a Julia function that's using Julia's multiprocessing, but I wanted to switch over to JuliaCall, but I can no longer call that function (JuliaPy/PythonCall.jl/issues/454).
I've tried a workaround, which is to use Python multiprocessing to call the serial version of that function in Julia, and that also turned out not to work either (JuliaPy/PythonCall.jl/issues/455).
I'd appreciate any pointers/tips. Thanks!
It doesn’t seem to be an issue for me due to the way SymbolicRegression.jl uses multiprocessing — since it is basically calling addprocs from within the Julia code. None of the stuff it is putting on Julia workers is actually accessible from Python.
The issue is a serialization issue. The closest fix I can see is to manually figure out how to serialize the arguments.
@mkitti Thanks. One question: serialization in the Python side or Julia? I'm a bit confused: I can successfully call the serial version of the same function from Python, so the output object type seems to be able to travel from Julia to Python. The parallel version returns a Vector of that object, is the output being a Vector causing the issue?
I do not mean "serialization" in terms of "serial" vs "parallel", I mean saving all the arguments to disk and reloading them. That's what you're doing when you use pmap
. That's why pickle is involved.
Yeah, I understand. Forgive my ignorance on how JuliaCall (or other language interop tools) works internally. What I meant was that pmap is called inside the Julia function, so the serialization is done by Julia, why is it that when the Vector is returned, JuliaCall can't transfer it back? I'm clearly missing how the interop is done, I was just curious which part I'm not getting
The error occurs with Python serialization, pickle. Attempt to serialize the arguments yourself with pickle rather than trying to pass them.
@mkitti Thanks for the pointers, they helped me pin down the issue. The issue was the numpy arrays (measurements
and frequencies
) that were being passed as input. I just added:
measurements = Array(measurements)
frequencies = Array(frequencies)
to the Julia function and it seems to have made the serializer happy.
Just curious, is this expected behaviour for numpy arrays? Or is it a bug?
Sounds kind of buggy, but I'm not sure. Now that you know this, try to create the simplest minimum working example possible. It might be useful to post this as a new issue.
There you go: #459 cc: @mkitti
This is now fixed on main - the underlying issue being that PyArray
was not serializable. Though unless you really do need a PyArray
I'd recommend just converting it to Array
.
Affects: JuliaCall
Describe the bug I'm trying to call a Julia function (from Python) that returns a vector of objects generated via
pmap
on multiple workers. (Not sure if thepmap
is even relevant, just in case)Reproduce the bug The error gets thrown as soon as I call this Julia function:
https://github.com/MaximeVH/EquivalentCircuits.jl/blob/875522d8c7da5a774b20d497cb045177c97017cf/src/CircuitEvolution.jl#L358-L399
Error message
Your system Please provide detailed information about your system: