search

SciML / diffeqpy

Solving differential equations in Python using DifferentialEquations.jl and the SciML Scientific Machine Learning organization
MIT License
531 stars 39 forks source link

consider using a separate python package to manage Julia dependency #99

Closed jlapeyre closed 11 months ago

jlapeyre commented 2 years ago

Here is a quick example: https://github.com/jlapeyre/diffeq_julia

This handles installing Julia and Julia packages. It allows optionally compiling a system image particular to the diffeq project.

ChrisRackauckas commented 2 years ago

Yes, those are major issues I've been hoping to solve!!! Automated installation and https://github.com/SciML/diffeqpy/issues/56. I would absolutely love to see a PR on this. I think it will greatly improve the library. Would you be willing to do the PR?

jlapeyre commented 2 years ago

I can probably find time over the next few weeks to make a PR that's a little more polished than the proof of concept that I posted above. But, I'd probably need help on the fine points of python packaging, e.g. setup.cfg, making sure CI works, etc.

ChrisRackauckas commented 2 years ago

Yeah, no worries. I will find the time, and hopefully @tkf could help.

sibyjackgrove commented 2 years ago

I just tested this and found this issue. https://github.com/jlapeyre/diffeq_julia/issues/1 Just thought I'll mention it in case someone else is finding similar issues.

jlapeyre commented 2 years ago

I just tested this and found this issue. jlapeyre/diffeq_julia#1

A couple of problems turned up in that issue, both because PyCall can't be used with more than one libpython. It's a major thorn that has received a lot of attention.

I made some changes to julia_project, which is the python package that manages a Julia dependency in a python package. It detects when this problem arises upon importing the python module, eg import diffeq_julia. It then offers to perform one of the two most common solutions. It also offers some tools in case you know of the problem in advance.

The main goal of julia_project is to manage the Julia dependency without the Python end user learning or knowing anything at all about Julia.

jlapeyre commented 2 years ago

@sibyjackgrove helped to debug https://github.com/jlapeyre/diffeq_julia .

The package depends on https://github.com/jlapeyre/julia_project, which factors out managing a Julia project within a python package.

The latter depends in turn on https://github.com/jlapeyre/find_julia, which handles finding a path to a Julia exectuable, and if not found installing Julia via jill.py.

The directory with the system-image-compilation scripts is here: https://github.com/jlapeyre/diffeq_julia/tree/main/diffeq_julia/sys_image

compile_exercise_script.jl is currently a dummy example. If it were replaced by something to exercise the Julia differential equations packages, then latency would be reduced. However, as it stands, compilation reduces startup time significantly.

EDIT: I think this should perhaps get a bit more testing in the wild before a PR is made to diffeqpy. The tests in julia_project and find_julia are minimal at best. Testing properly would require a lot of mocking of the state of your system. i.e. if and where julia is installed. If PyCall is installed, how it is built, etc. All the debugging was around these issues.

jlapeyre commented 2 years ago

I added a compile_exercise_script.jl to the code that builds the system image. It runs a few examples that should be compiled into the system image. This seems to have at best a small effect on the latency of running the python examples in the diffeqpy README.

ChrisRackauckas commented 11 months ago

This is now handled by PythonCall.