search

JuliaPy / pyjuliapkg

Manage your Julia dependencies from Python
MIT License
49 stars 13 forks source link

Add option to merge Julia dependencies found in `juliapkg.json` files with existing `Project.toml` #9

Closed dingraha closed 2 years ago

dingraha commented 2 years ago

This PR changes the behavior of JuliaPkg when the environment variable PYTHON_JULIAPKG_MERGE_PROJECT is non-empty. When that's true, JuliaPkg will attempt to merge the Julia dependencies found in the juliapkg.json files into the Project.toml file instead of overwriting it. It finds compatible ranges of versions fo each Julia package using JuliaPkg's Compat class. I added a description of this option to the README.md.

This change is motivated by the following use case for JuliaPkg: I have a Julia package (call it FooBase.jl) that I want to add as a dependency to a Python package (call it pyfoo). Then I'd like to use pyfoo and FooBase.jl in a Julia package Foo.jl. To do that, I need to make sure the version of FooBase.jl that pyfoo uses is compatible/the same as the one I'll use in Foo.jl. The easiest solution I could think of is to make JuliaPkg use the same Project.toml as whatever I'm using in Julia through the PYTHON_JULIAPKG_PROJECT variable, and then let the Julia package manager figure out the compatible versions.

cjdoris commented 2 years ago

Interesting idea, I need to mull it over. In there any particular reason in your use case why you can't just repeat the FooBase dependency in both juliapkg.json for pyfoo and Project.toml for Foo.jl?

dingraha commented 2 years ago

In there any particular reason in your use case why you can't just repeat the FooBase dependency in both juliapkg.json for pyfoo and Project.toml for Foo.jl?

OK, hmm... I assumed that wouldn't work. From the JuliaPkg readme https://github.com/cjdoris/pyjuliapkg#where-are-julia-packages-installed it seemed in that case that the Julia packages that JuliaPkg finds in the juliapkg.json files would be installed in a seperate Julia environment (maybe {env}/julia_env or ~/.julia/environments/pyjuliapkg). But I'm testing it out now and it does seem to be working fine: I can't find a {env}/julia_env or ~/.julia/environments/pyjuliapkg directory anywhere that JuliaPkg is using, and my simple tests are working fine.

So, how do CondaPkg and JuliaPkg work in this case, where I have added a Julia dependency (FooBase.jl) to a Python package pyfoo using JuliaPkg, and then make pyfoo a dependency of a Julia package Foo.jl using CondaPkg? It seems like JuliaPkg is able to use the Foo.jl package environment when installing dependencies, which is exactly what I want. Is that true?

cjdoris commented 2 years ago

I'm a little confused, so let's say you have three directories FooBase, pyfoo and Foo. The first and last are Julia packages and the middle one is a Python package.

In pyfoo/src/foo/juliapkg.json you add FooBase as a dependency. In Foo/CondaPkg.toml you add pyfoo and in Foo/Project.toml you add FooBase.

Now if you install pyfoo somewhere it should install FooBase for you into whatever Python environment you are in.

And separately if you use your Foo package, it should install both pyfoo and FooBase for you into whatever Julia project you are in (e.g. you could be in the Foo project itself).

You shouldn't need to use PYTHON_JULIAPKG_PROJECT at all for this to work.

It's admittedly a little inelegant that you need to repeat the FooBase dependency in two places, but this is because JuliaPkg and CondaPkg are totally separate: dependencies don't get recursively passed from Julia to Python to Julia when you have a situation like yours.

dingraha commented 2 years ago

OK, great. I tested what you suggested and it seems to work just like you said. So no need for this PR, which is wonderful.

For posterity, the thing I was worried about was what would happen when I added methods to functions defined FooBase.jl from Foo.jl, and then called them from pyfoo. I wanted some way of making sure that the FooBase.jl that Foo.jl was using was the same as the one pyfoo uses. But it all seems to work!

If you or anyone else is interested, I created examples of what I'm talking about here: https://github.com/dingraha/FooBase.jl, https://github.com/dingraha/pyfoo, https://github.com/dingraha/Foo.jl.

Thanks again for this package—PythonCall, JuliaCall etc. are great.

cjdoris commented 2 years ago

I'm glad it works. I just looked at those 3 repos and they are exactly as I'd expect them to be.