Switch to hatch to handle panel.js build and bundling steps

[philippjfr]

We are currently relying on setting SETUPTOOLS_ENABLE_FEATURES: "legacy-editable" to allow our setup.py to execute the CustomDevelopCommand. We should instead migrate the contents of our setup.py to the pyproject.toml file and then use hatch to configure any additional build steps.

[maximlt]

Isn't this never going to be supported by setuptools?

[hoxbro]

I would take heavy inspiration from how Bokeh does it.

For the migration to pyproject.toml, I 100% agree.

[philippjfr]

I'm also okay with not using hatch. I've used it for various Jupyter based projects and it's worked well but I'm not tied to it. The main thing is to support building without using legacy modes.

[hoxbro]

The main thing is to support building without using legacy modes.

As someone who had to spend a half-day to find out I needed to enable legacy-mode I wholeheartedly agree.

[ofek]

Hey I'm the maintainer of Hatchling, please let me know if you have any questions!

Looking at the setup.py of Bokeh, I think those build steps can be easily ported using a custom build hook. Here are some examples:

• https://github.com/jupyter/nbconvert/blob/e355aecd1cd27d130859611af6c233012bb8b927/hatch_build.py#L78
• https://github.com/facelessuser/Rummage/blob/4ef419382af739e46f4479cdafc576d5844093ce/hatch_build.py#L63

[maximlt]

Just some context for you @ofek as indeed moving to Hatchling is something that we could be interested in but we've got some special requirements that make any transition not as straightforward as it should be. Primarily we use conda pretty much everywhere on our CI. Our dependency requirements, be it for testing or docs building purposes, are pretty extensive and tap into the PyData ecosystem, so much that I'm pretty sure conda makes our life a lot easier. For a practical example, our ecosystem touches the Python geo-stack (GeoViews) that is known to be difficult to install. With conda we can quite reliably deal with these complex cases and test our packages on the 3 main platforms.

Years ago this team built a custom developer tool aimed at making it easy to develop and maintain Python libraries: pyctdev. We use it a lot on our CI, less locally as its UX isn't great. Yet it has some cool features so feel free to have a look at it and see what kind of problems its creators were trying to solve at the time and why we're still using it. For instance:

• singular API to work with both pip and conda (we don't use much its pip support so not sure how reliable it is or complete)
• ability to collect the build, install and extras requirements from a setup.py, to install them in a conda environment. In practice you can run doit develop_install -o tests -o docs -c conda-forge to install in the activated environment the runtime deps plus the tests and docs extras deps from the conda-forge channel. That command also runs pip install -e . --no-deps --no-build-isolation.
• ability to build packages for PyPI and for conda, and test the built artifacts

I've seen that the Python packaging/testing tooling has gained some conda support, I know pdm and nox have some built-in support, there are the tox-conda and hatch-conda plugins, so it's definitely getting better but I'm not sure their support is good enough for us to move our entire ecosystem to one of these. So we stick to our old broken tool :)

[ofek]

That is very interesting, thanks for the information!

The hatch-conda plugin you mention is just for environment management actually rather than anything to do with the build system. As far as I understand Conda already supports the new packaging standards (PEP 517/PEP 660) and the format for metadata declaration (PEP 621/PEP 631). Hatchling is already available and is used to build other Conda packages: https://github.com/conda-forge/hatchling-feedstock

So, I think this should Just Work ™️

[maximlt]

The hatch-conda plugin you mention is just for environment management actually rather than anything to do with the build system.

Indeed, and that's anyway unrelated to this issue, but I still wanted to mention it as this is something we are also looking into.

As far as I understand Conda already supports the new packaging standards (PEP 517/PEP 660) and the format for metadata declaration (PEP 621/PEP 631). Hatchling is already available and is used to build other Conda packages: https://github.com/conda-forge/hatchling-feedstock

So, I think this should Just Work ™️

I'd hope so too! I really don't know though if conda-build can deal with dynamic fields like version in our case. I haven't yet dug into this.