respect pyproject.toml

[Roger-luo]

I'm wondering if we can instead of creating a juliapkg.json, but putting a field inside pyproject.toml like

[julia.dependencies]
julia = "1.8"
Example = "0.5"
PythonCall = "0.8"

this would making packaging a lot easier because pyproject.toml will be saved in a package wheel.

[cjdoris]

Can you reliably find all the pyproject.toml files of all installed packages, regardless of how they were installed (from wheels or source)?

[MilesCranmer]

+1 for this, I also think as a longterm goal this would feel much more in-line with best practices to be able to specify this info there. Although @Roger-luo I think the syntax should be modified (see below)

@cjdoris the the role of pyproject.toml is just as a build configuration file, so it actually does not included in the installed package. So it would require a bit of additional config; see below. (While it is the "correct" option, I guess I'm also not sure the extra effort is worth it though...)

I think what would make sense is to have something like what setuptools_scm does:

[build-system]
requires = ["setuptools", "juliapkg_setup"]
build-backend = "setuptools.build_meta"

[project]
name = "myproject"
version = "0.1.0"
dependencies = [
  "juliapkg",
  "juliacall",
  # other python dependencies
]

[tool.juliapkg]
julia_requires = "~1.6.7, ~1.7, ~1.8, ~1.9, =1.10.0, ^1.10.3"
load_file = "src/julia_deps.py"  # Is generated during the build step

[tool.juliapkg.packages]
SymbolicRegression = { uuid = "8254be44-1295-4e6a-a16d-46603ac705cb", version = "=0.24.4" }
Serialization = { uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b", version = "1" }

which would generate src/julia_deps.py at build time which would contain the following:

import juliapkg

juliapkg.require_julia("~1.6.7, ~1.7, ~1.8, ~1.9, =1.10.0, ^1.10.3")
juliapkg.add("SymbolicRegression", "8254be44-1295-4e6a-a16d-46603ac705cb", version="=0.24.4")
juliapkg.add("Serialization", "9e88b42a-f829-5b0c-bbe9-9e923198166b", version="1")

Then, the package developer would add the following to the top of their src/__init__.py file:

import .julia_deps
...

This is the same way that version information is usually passed from pyproject.toml to the installed Python library using https://github.com/pypa/setuptools_scm. For example in my PySR package you can see I import this file version.py that doesn't actually exist yet:

# This file is created by setuptools_scm during the build process:
from .version import __version__

I think we could do something similar for pyjuliapkg, since we would want to transfer build information from the pyproject.toml to the Python runtime.

[cjdoris]

This is a neat idea. I've wondered about doing the analogue over in CondaPkg too - namely putting the dependencies into Project.toml instead of CondaPkg.toml. The situation is easier there because CondaPkg.toml is automatically gets bundled into the package it's in.

Instead of generating a julia_deps.py file (you're not supposed to call juliapkg.add every time you start julia, and anyway it would have no effect if juliapkg had already resolved first), I think it should just generate a juliapkg.json file. This way we can keep the existing functionality for finding dependencies.

Feel free to develop this (it would be a new package, which is convenient) - but it's not a thing I'll have time to work on.

[cjdoris]

One potential down-side - would it tie you into using a particular build system? There are quite a few Python build systems nowadays and I wouldn't want to force people into using only one. Taking setuptools_scm as the prototype - does that support build systems other than setuptools?

[MilesCranmer]

Good points.

Actually after reading more about it, while my suggestion was technically the "best practices" way of doing it, it just makes it so so so much more complicated for both us and the users.

I think the better option is to simply package pyproject.toml with Python and read it at import (just like juliapkg.json). For this, apparently all we would need is for users to specify a MANIFEST.in file in the root of their repo with:

include pyproject.toml

Then we could just have juliapkg read from that.

[MilesCranmer]

Thinking about this more, I do wonder if the Julia installation should happen during pip install rather than import. Then this sort of thing would already work with the original syntax @Roger-luo posted above (which I think is much nicer).

This is also how Cython packages get built — they compile during the pip install rather than at import time.

[Dale-Black]

I personally think adding Julia at pip install would be a beneficial change. Its the exact same amount of time in total, but waiting a while for a pip install feels a lot more normal than waiting for an import call to finish installing Julia.