Make pyjulia easy to install

[madeleineudell]

I'd like to make it easy for Python denizens to use my Julia packages, so am writing a thin Python wrapper using pyjulia. But to make my wrapper easily installable, I'd need pyjulia to be easily installable, too. Would it be possible to allow installation via pip/conda/etc?

This would move us in the direction of having pyjulia be as easy to use from Python as PyCall is from Julia.

Bonus points if it auto-installs Julia upon not finding a valid Julia installation.

[ChrisRackauckas]

Bonus points if it auto-installs Julia upon not finding a valid Julia installation.

Up for this. @tkf what would it take to achieve this? I'm curious how possible it is.

[tkf]

In principle, we can "just" write a simple installer by porting the existing shell script https://github.com/JuliaCI/install-julia to Python. But dealing with integrity check and security concerns (like checking PGP signatures) is a very daunting task. It essentially means that PyJulia becomes a package manager which I'd like to avoid. But maybe adding unsafe=True option to julia.install() function https://pyjulia.readthedocs.io/en/latest/installation.html#step-3-install-julia-packages-required-by-pyjulia that auto-installs everything is an OK compromise.

[ChrisRackauckas]

@christopher-dG let's discuss how this vednering of Julia might be possible. Maybe just detect the architecture and install the generic binary? Maybe something from binary builder can be helpful. @staticfloat might be helpful as well.

[christopher-dG]

So I think the best option is to enhance the pyjulia.install() function as @tkf mentioned rather than putting Julia installation to install or load time. We could use Julia_jll, or just download the tarballs from julialang.org.

I guess we should do nothing if there is already a Julia v1+ installed, and install to somewhere in the pyjulia installation otherwise.

[christopher-dG]

I wrote some ugly code that downloads, verifies (hash + GPG), and extracts Julia, obviously it needs a lot less stuff hard-coded and probably some Python 2 compat work but this is mostly just to show that the verification is not actually that difficult.

def install_julia():
    # Check if Julia is already installed
    if which("julia") is not None:
        return "julia"
    dest = os.path.expanduser("~/.local/share/pyjulia/installs/julia-1.4.2")
    os.makedirs(os.path.dirname(dest), exist_ok=True)
    julia_bin = os.path.join(dest, "bin", "julia")
    if os.path.isdir(dest):
        return julia_bin
    # Download the tarball
    url = "https://julialang-s3.julialang.org/bin/linux/x64/1.4/julia-1.4.2-linux-x86_64.tar.gz"
    tar = download(url)
    # Find the expected hash of the tarball
    hashes = download("https://julialang-s3.julialang.org/bin/checksums/julia-1.4.2.sha256")
    with open(hashes) as f:
        m = re.search(r"(.*?)\s+julia-1\.4\.2-linux-x86_64.tar.gz", f.read())
    if not m:
        raise RuntimeError("Hash verification failed (could not find hash)")
    expected_sha = m[1]
    # Compute the hash of the tarball and compare
    observed_sha = hashlib.sha256()
    with open(tar, "rb") as f:
        observed_sha.update(f.read())
    if expected_sha != observed_sha.hexdigest():
        raise RuntimeError("Hash verification failed (hashes did not match)")
    # Verify the GPG signature
    if which("gpg") is None:
        print("no gpg found, skipping signature verification", file=sys.stderr)
    else:
        # Import the public key
        key = download("https://julialang.org/assets/juliareleases.asc")
        subprocess.check_call(["gpg", "--import", key])
        # Verify the signature
        sig = download(url + ".asc")
        subprocess.check_call(["gpg", "--verify", sig, tar])
    # Extract the archive
    with tarfile.open(tar) as f:
        f.extractall(os.path.dirname(dest))
    return julia_bin

def download(url):
    resp = urlopen(url)
    body = resp.read()
    _, path = tempfile.mkstemp(prefix="pyjulia_")
    with open(path, "wb") as f:
        f.write(body)
    return path

[stevengj]

See also #392

[tkf]

FYI, there is a cross-platform installer written in Python JILL.py https://github.com/johnnychen94/jill.py so one option is to use it.

But the hard part is how to manage the set of installed Julia binaries. I'd prefer something like official cross-platform "juliaup" https://github.com/JuliaLang/julia/issues/36672 to handle it. Ideally, we can distribute it as a part of wheel if it is reasonably small and manylinux-compatible. Or, at least, we should decide the common directory structure that can be shared with other tools (like juliaup and JILL.py). Maybe something like ~/.julia/opt/julia-1.4.2/ (so that ~/.julia/opt/julia-1.4.2/bin/julia is the binary).

[tkf]

This discussion comes up in multiple places (e.g., https://github.com/Non-Contradiction/JuliaCall/pull/135) so I'd like to dump my thoughts on this.

Frankly, I am against each X-Julia bridge library for language X to implement its own Julia installer because:

1. There would be multiple copies of julia (> 300 MiB each).
2. Each language has to implement and maintain non-trivial code (proxy, retry, mirror, integrity check, UI/API for managing installed Julia, atomic data store update for multi-process support, etc.)

I think a better solution is to distribute a CLI juliaup or something to manage multiple julia installations. I prefer this approach because it'd help people who only use Julia as well. This CLI can even have a UI for managing system images (a la jlm). I've been thinking to implement it in Rust or Go or something (so that single-binary distribution is very easy) and maybe even try to upstream to https://github.com/JuliaLang/ at some point.

[ChrisRackauckas]

The number of people trying to call into Julia from R and Python is a few orders of magnitude higher than anything else, so I think that it getting something in Python (since R is almost complete) is much more important than a general solution. But yes a general solution that all bridge libraries could use would be nice.

[christopher-dG]

I'm also in favour of eventually having one canonical installer/version-manager, similar to how Rust has the rustup that everyone uses and is officially recommended by the Rust devs. But it just takes time for any such tool to gain traction and become widely used.

[ChrisRackauckas]

I started setting up diffeqpy to auto-install Julia via jill.py in https://github.com/SciML/diffeqpy/pull/85, but I really think this should be something that's part of pyjulia instead.

Pinging @johnnychen94 who might be interested in this.

[johnnychen94]

I wrote jill.py in Python because that's the only language I'm familiar with when I started the project. It's a CLI tool so can be convenient to use. The only downside of jill.py I see from now is that you need to include python as the dependency. It is not a large dependency but is definitely not a minimal solution.

FWIW, jill.py requires python>=3.6

[ChrisRackauckas]

Python as a dependency is no problem for pyjulia 😄

[willow-ahrens]

Would this issue be any easier to solve using https://github.com/JuliaLang/juliaup?

[ChrisRackauckas]

PythonCall/JuliaCall solves this, so diffeqpy already moved at least

[mkitti]

juliacall is not in conda. I think it should be called "pyjuliacall" there to distinguish from "r-juliacall", etc.

[ChrisRackauckas]

That probably is a much easier thing to do than reviving a defunct repo 😅

[willow-ahrens]

Does JuliaCall install Julia? I can't seem to find the line that does.

[ChrisRackauckas]

That's done through jill, just look at how diffeqpy does it with the two. https://github.com/SciML/diffeqpy/blob/master/diffeqpy/__init__.py#L1-L47 that's all you need and Julia is auto-installed with the right packages.

pip install diffeqpy
from diffeqpy import de

that is then all a user has to do.

[willow-ahrens]

Thanks for the help! After digging in to this, JuliaCall DOES install Julia through pyjuliapkg if you let it. I tested this behavior in a docker environment. pyjuliapkg is nice enough to respect juliaup if you use it, so SciML may want to consider letting juliacall do its thing, rather than pre-emptively using jill.py to install.