search

avik-pal / Wandb.jl

Unofficial Julia bindings for logging experiments to wandb.ai
https://avik-pal.github.io/Wandb.jl/stable/
MIT License
80 stars 10 forks source link

Allow user to use `PyCall.jl` #28

Closed MilesCranmer closed 5 months ago

MilesCranmer commented 6 months ago

@avik-pal Most packages still use PyCall.jl instead of PythonCall.jl so it would be preferred if there was an option for the user to choose which Python backend they want – PyCall.jl or PythonCall.jl.

For example, PyPlot.jl and ScikitLearn.jl still uses PyCall.jl. I'm not sure what sorts of issues this causes if Julia is connected to libpython in two ways, simultaneously, but it sounds a bit risky?

Even more preferable would be to avoid Python at all (like how TensorBoardLogger.jl is written in pure Julia) but it seems Wandb doesn't have much support for that yet, unfortunately.

avik-pal commented 6 months ago

Most packages still use PyCall.jl instead of PythonCall.jl so it would be preferred if there was an option for the user to choose which Python backend they want – PyCall.jl or PythonCall.jl.

Do you know of any package doing something similar? My preferred solution would be to direct users to put a compat on v0.4 if they prefer using PyCall.jl. And I can backport any fixes as and when needed.

Even more preferable would be to avoid Python at all (like how TensorBoardLogger.jl is written in pure Julia) but it seems Wandb doesn't have much support for that yet, unfortunately.

Yeah, it doesn't seem like they do. I know https://github.com/JuliaAI/MLFlowClient.jl is a pure julia but non-wandb alternative.

MilesCranmer commented 6 months ago

Do you know of any package doing something similar? My preferred solution would be to direct users to put a compat on v0.4 if they prefer using PyCall.jl. And I can backport any fixes as and when needed.

No, actually is the first package I've seen using PythonCall.jl at all. Most use PyCall.jl still (though I think the trend is indeed to move to PythonCall). I'm just worried about having both active at the same time, especially the direction I am headed, it would be:

PySR
<-> PyJulia & PyCall.jl
<-> SymbolicRegression.jl
   -> Wandb.jl -> PythonCall.jl -> wandb
  <-> PyPlot.jl <-> PyCall.jl <-> matplotlib

Which seems a bit scary. But maybe this isn't unusual and they all open different processes anyways?

avik-pal commented 6 months ago

Yeah I am not confident about both running simultaneously either. What do you feel about the solution with a compat on v0.4?

MilesCranmer commented 6 months ago

The tricky part is that I probably wouldn't depend on Wandb (or other loggers) myself; I would just expose an interface to them with AbstractLogger.

How similar are the PyCall and PythonCall interfaces? I wonder if it's workable to have something like Turing.setadbackend but for Python interfaces: https://turinglang.org/dev/docs/using-turing/autodiff

Maybe in the future we can refactor it into an interface package so someone can just choose the backend for all Python packages...

MilesCranmer commented 5 months ago

https://juliapy.github.io/PythonCall.jl/stable/pycall/#Tips

To force PythonCall to use the same Python interpreter as PyCall, set the environment variable JULIA_PYTHONCALL_EXE to "@PyCall". Note that this will opt out of automatic dependency management using CondaPkg.

avik-pal commented 5 months ago

Seems like a viable path to go down. If you can create a draft PR, I can review it. I am open to even cut out a breaking release if that helps in wider adoption

MilesCranmer commented 5 months ago

I switched to PythonCall.jl recently and am convinced it's the way to go. So no need, but thanks anyways!