search

mwouts / jupytext

Jupyter Notebooks as Markdown Documents, Julia, Python or R scripts
https://jupytext.readthedocs.io
MIT License
6.59k stars 386 forks source link

PEP621 #1076

Closed LecrisUT closed 11 months ago

LecrisUT commented 1 year ago

Looking at the setup.py, it seems quite straightforward to upgrade to a single PEP621 pyproject.toml file. The one thing that I am not sure of is jupyter_packaging. Is there a way to build it without setup.py?

mwouts commented 1 year ago

Hey @LecrisUT , yes indeed while trying to do #1077 I found out that the current setup.py is a bit outdated. It is indeed possible to build Jupyter extensions with just a pyproject.toml file, however I'd like to keep the option to not build the extension depending on an environment variable (because the extension has many requirements). At the moment I am not very familiar with the pyproject.toml file so I'll probably start by porting other simpler projects, and then come back to this one.

LecrisUT commented 1 year ago

t is indeed possible to build Jupyter extensions with just a pyproject.toml file, however I'd like to keep the option to not build the extension depending on an environment variable (because the extension has many requirements).

That is really tricky and non-standard. It's also incompatible with CI tools, e.g. in steup-python, it uses setup.py/pyproject.toml, etc. to setup pypi cache. If there are two setup environments there can be contamination from one to another.

How about splitting it into jupytest-core that is purely Python and this one would be just a thin wrapper around it. But also there are cases where you can alter the dependencies and maybe even the installer based on the os environment. Maybe jupyter's build system offers something similar?

In Fedora packaging it is often that the pip will install in a package subfolder to make it python self-consistent, but when packaging it is moved to the system's locations. Maybe these can give some design ideas.

mahendrapaipuri commented 11 months ago

What is the objective of controlling the building of extension via env variable? I assume you must be distributing the transpiled JS with the python package. So, unless users are doing a developmental install, they do not have to build the extension themselves.

I will be happy to port the extension to JLab 4 and pyproject.toml if you guys need a hand.

mwouts commented 11 months ago

Hey @mahendrapaipuri , the environment variable was introduced to reduce the build time of the package in the contexts when the Jupyter Lab extension was not required. I guess that now with Jupyter Lab 4 we always want the extension to be bundled with the Python package, so yes we can remove it.

I will be happy to port the extension to JLab 4 and pyproject.toml if you guys need a hand.

Oh yes that would be much appreciated! The JLab extension is the part that I find the most difficult to maintain. A few months back I had a look at the JLab extension cookie cutters, and I could see that the build tools have changed significantly, but I could not really identify how to make the update, so definitively I'd take any help on this!

BTW, we currently have an extension that is compatible with both Jupyter Lab 3 and Jupyter Lab 4. If you need to make it JLab 4 only I would be fine with that option too.

mahendrapaipuri commented 11 months ago

the environment variable was introduced to reduce the build time of the package in the contexts when the Jupyter Lab extension was not required. I guess that now with Jupyter Lab 4 we always want the extension to be bundled with the Python package, so yes we can remove it.

Alright, I see your point. If I understand correctly, the frontend extension is not critical for jupytext as it has its own CLI. Splitting the code base into two repos can be a potential solution: jupytext and jupytext-extension. Of course it increases maintenance burden and it will be a breaking change. Maybe it is an idea for you to think into the future.

Oh yes that would be much appreciated! The JLab extension is the part that I find the most difficult to maintain. A few months back I had a look at the JLab extension cookie cutters, and I could see that the build tools have changed significantly, but I could not really identify how to make the update, so definitively I'd take any help on this!

Yes, JupyterLab extensions are using hatch build system now. Ok, I will try to work on it and see if we can support both JLab 3 and 4.

Cheers!

LecrisUT commented 11 months ago

It does not have to be two repos. Modern design can have them in subdirectories. But also a common approach is to include all files in the wheel, but have a specific optional-dependencies with the extensions dependencies, and a simple try...catch to check if the dependencies are included.

I don't know if I can help and research the packaging there, but if you can make a PR, I can catch up and help with a review.

mahendrapaipuri commented 11 months ago

Ok. I will try to put up a draft PR and we can discuss over it. Cheers!

mwouts commented 11 months ago

Awesome, thanks! I am also in favor of just one repository.

For information this repository contains the Jupytext Python API and CLI, a contents manager, a short Jupyter Lab server extension (which only role is to activate the contents manager), and the Jupyter Lab front end extension (so two Jlab extensions). There is also a former Jupyter Notebook extension but I guess that one can be discarded now.