search

SciML / diffeqpy

Solving differential equations in Python using DifferentialEquations.jl and the SciML Scientific Machine Learning organization
MIT License
508 stars 39 forks source link

Error when importing diffeqpy parallely in large number of sub-processes. #98

Closed sibyjackgrove closed 8 months ago

sibyjackgrove commented 2 years ago

I have a program that spawns a large number of sub-processes and imports diffeqpy within each sub-process using: from diffeqpy import ode It works fine when the number of sub-processes is less than 75. But above that I am getting the following error: "exception: access violation writing 0x000001EC9127C000"

I am using the following: diffeqpy: 1.2.0 pyjulia: 0.5.7 Julia 1.5.3, 1.6, 1.7

I have a feeling this could be a pyjulia problem. But I just wanted to check here to see if anyone else has encountered it.

ChrisRackauckas commented 2 years ago

Sounds like a pyjulia problem. I'm not sure we can solve it at this repo.

sibyjackgrove commented 2 years ago

@ChrisRackauckas I tried an experiment where I only imported the Main function using pyjulia with many subprocesses. In this case, I was able to start more than 75 sub-processes without encountering the error. I checked the memory footprints of the sub-processes. It seems to importing diffeqpy almost double the memory footprint of the Mainfunction. Could the higher memory footprint when using diffeqpybe the cause of the problem?

ChrisRackauckas commented 2 years ago

Is it dying because you run out of memory? That would be a good way to diagnose that 😅

sibyjackgrove commented 2 years ago

@ChrisRackauckas No, the workstation I am using has 128 GB of memory. And only about 50% memory utilization is there at the time of the error.

ChrisRackauckas commented 2 years ago

@tkf might have an idea.

tkf commented 2 years ago

Maybe you are using fork? See: https://github.com/JuliaPy/pyjulia/issues/448#issuecomment-994965548

JacobNeumann88 commented 2 years ago

Has this issue been resolved? I am trying to use the diffeqpy package in a general fitting software (PyBNF) but without the dask compatibility I am getting nowhere. If I try to use julia prior to submitting the jobs I get a error trying to pickle pycall objects. If I import julia in the function, it is extremely slow and prone to crashing.

sibyjackgrove commented 2 years ago

@JacobNeumann88 No my issue is not resolved. So in my application, I am limited to using less than 50 instances of diffeqpy. I have not used it with Dask.

If I import julia in the function, it is extremely slow and prone to crashing. I am importing diffeqpy within each sub-process. Depending upon the number of sub-processes it takes anywhere from 10 seconds to 20 seconds. And if I have more than 50 sub-processes my program crashes (not immediately)

ChrisRackauckas commented 8 months ago

Was a PyJulia thing, now gone with PythonCall.