search

mesonbuild / meson

The Meson Build System
http://mesonbuild.com
Apache License 2.0
5.35k stars 1.53k forks source link

Expose a PEP 517 backend #7863

Closed FFY00 closed 11 months ago

FFY00 commented 3 years ago

Python packaging is shifting away from setuptools as the only real build backend option. PEP 517 introduces a standard interface to insteract with build backends, PEP 518 introduces a standard way to specify which backend the project uses.

I think it would be interesting to see if meson could expose a Python build system backend, so that it could be used in Python packaging.

Would this be something you might consider?

jpakkane commented 3 years ago

I have not looked at the details, but in theory yes.

dcbaker commented 3 years ago

Id like to do it. The one trick is that we'd probably need a toml parser, which isn't in the standard library (yet).

lazka commented 3 years ago

There is https://gitlab.com/thiblahute/mesonpep517

FFY00 commented 3 years ago

Id like to do it. The one trick is that we'd probably need a toml parser, which isn't in the standard library (yet).

Great, I would also like to help out.

The toml parser would be the easy part, we can pull toml as an optional dependency.

PEP 517 mandates two hooks, build_sdist and build_wheel.

build_sdist build a source distribution. There isn't a standard for this yet, so we essentially just have to produce a tarball with the source code, which is pretty easy.

build_wheel is the one that requires work. It builds a Python wheel file, which is described in PEP 427.

A quick overview, a Python wheel is a zip file and its contents look something like this:

python_module1
python_module2
my_package.dist-info
my_package.data

my_package.dist-info is the metadata directory and my_package.data the data directory. The data directory can contain headers, scripts and general data, each in its own subdirectory.

Before starting working on this, everyone should read the whole PEP.

There is gitlab.com/thiblahute/mesonpep517

I am aware, but it does not give me much confidence.

mesonpep517-releases

lazka commented 3 years ago

I am aware, but it does not give me much confidence.

Why is that?

FFY00 commented 3 years ago

Why is that?

Did you see the picture I attached? That does not seem like a reasonable release model, IMO.

The author is also inactive, at least in the recent months. Here's an issue I'd really like to see addressed. https://gitlab.com/thiblahute/mesonpep517/-/issues/2

There are some other issues, like the fact that it uses an unmaintained toml parser library.

I have considered using it in one of the projects I maintain, but given the userbase of my project I am really not confident at all to do it.

tp-m commented 3 years ago

cc @thiblahute

thiblahute commented 3 years ago

@FFY00 Well I released often in my early testing process as it made the process simpler.. I do not think that means anything in how good the code/approach is or isn't....

I didn't remove intreehook requirement as it doesn't cause any issue for me and thus I do not have enough incentive to move away from it until now... patches are very much welcome!

I think we could/should move mesonpep517 in the meson three at some point but having it external also does the job . Now not reusing that work in the process for the reasons you invoke here doesn't make much sense to me :)

dcbaker commented 3 years ago

I can't remember which one, but I read somewhere the Brett Cannon had one of the toml modules in mind to recommend for core after toml hits 1.0. I think it would be worth figuring out where upstream is leaning and using that one

FFY00 commented 3 years ago

@FFY00 Well I released often in my early testing process as it made the process simpler.. I do not think that means anything in how good the code/approach is or isn't....

Sure, but I'd use alpha, beta or pre-releases in those cases. The version numbering also struck me as very odd.

I didn't remove intreehook requirement as it doesn't cause any issue for me and thus I do not have enough incentive to move away from it until now... patches are very much welcome!

Great :grin: Glad to know you are not inactive and you indeed looked at the issue.

I think we could/should move mesonpep517 in the meson three at some point but having it external also does the job . Now not reusing that work in the process for the reasons you invoke here doesn't make much sense to me :)

Yeah, I think we could use mesonpep517 in the process.

I can't remember which one, but I read somewhere the Brett Cannon had one of the toml modules in mind to recommend for core after toml hits 1.0. I think it would be worth figuring out where upstream is leaning and using that one

There are essentially two toml parsers, toml and pytoml. pytoml has been deprecated in favor of toml. Some projects still use pytoml but toml is definitely the recomended one. You can have a look at PyPA projects, you will see that's what they are using.

If anything makes into the standard library it will very likely be toml. And I am pretty sure this is the toml parser Brett was refeering to.

thiblahute commented 3 years ago

@FFY00 Well I released often in my early testing process as it made the process simpler.. I do not think that means anything in how good the code/approach is or isn't....

Sure, but I'd user alpha, beta or pre-releases in those cases. The version numbering also struck me a very odd.

Well, it is semantically correct, but yes it is the result of development cycles rather than really releasing.

I now ported mesonpep517 to toml and backend-path, it was totally trivial.

FRidh commented 3 years ago

Related issue https://github.com/mesonbuild/meson/issues/7695.

FFY00 commented 2 years ago

FYI, I have started working on this.

thiblahute commented 2 years ago

FYI, I have started working on this.

You know https://gitlab.com/thiblahute/mesonpep517 is pretty active and implements most of the spec and related ones too? Scipy is in the process of moving to using it. What are you working on? I think we should move mesonpep517 to the mesonbuild org on github I wonder what is wrong with just making it official for you.

eli-schwartz commented 2 years ago

Some thoughts on what this would probably entail:

Id like to do it. The one trick is that we'd probably need a toml parser, which isn't in the standard library (yet).

FFY00 commented 2 years ago

You know gitlab.com/thiblahute/mesonpep517 is pretty active and implements most of the spec and related ones too? Scipy is in the process of moving to using it. What are you working on? I think we should move mesonpep517 to the mesonbuild org on github I wonder what is wrong with just making it official for you.

mesonpep517 makes several decisions I am not comfortable with, the main one being introspecting intro-installed.json which does not provide any context on files. My reasoning for wanting the python build backend in meson itself is so that we can develop it side by side as meson is not really designed to be used the way mesonpep517 is using it. Currently, mesonpep517 is resorting to fragile heuristics like looking at file extensions to figure out if the binary distributions are supposed to be pure or not (https://gitlab.com/thiblahute/mesonpep517/-/blob/master/mesonpep517/buildapi.py#L407), which demonstrates the interface issue. We could fix this in meson by making it export more useful information about the installed files, but relying on this interaction makes progress slow and the development harder, as it does not only creates a gap between the development team but it also restricts us to only be able to use interfaces meson is comfortable making public.

Overall, I think mesonpep517 is a great PoC but not suitable for production use. We can, of course, improve it, but I think getting it to the state I would be comfortable using it on big projects would be much easier if the development was tightly integrated with meson itself. Anyway, as I said in the PR, this is not my call, the meson maintainers will have to decide on it.

xclaesse commented 2 years ago

Currently, mesonpep517 is resorting to fragile heuristics like looking at file extensions to figure out if the binary distributions are supposed to be pure or not

I'm not sure about the whole story here, but it feels like something we can fix by using install tags: https://github.com/mesonbuild/meson/pull/9110

FFY00 commented 2 years ago

Sure, although as far as I understand it would not be optimal. My solution was to export more detailed information about the installed files.

9d80045 (#9128)

thiblahute commented 2 years ago

meson is not really designed to be used the way mesonpep517 is using it

I do not see why you would say that, it has been developed making sure to provide introspection data so building tools based on it was easy. This is exactly that power that we are leveraging here.

Currently, mesonpep517 is resorting to fragile heuristics like looking at file extensions to figure out if the binary distributions are supposed to be pure or not (https://gitlab.com/thiblahute/mesonpep517/-/blob/master/mesonpep517/buildapi.py#L407), which demonstrates the interface issue

That only demonstrates that I used a quick solution when implementing because something was missing at that point. This is not the case anymore and we can now use intro-targets.json determine that in a cleaner way.

In any case, you are welcome to reuse anything you want from mesonpep517 I made sure to use the same license as meson itself because of that. I agree integrating that component in meson could make it somehow simpler (though I have no proof of that at this point).

I also think that for this particular component avoiding external dependencies doesn't make sense at all as the main advantage of implementing pep517 is that projects built with meson can very easily leverage the Python package ecosystem (not saying for the rest of meson, but for that particular component).

xclaesse commented 2 years ago

I also think that for this particular component avoiding external dependencies doesn't make sense at all as the main advantage of implementing pep517 is that projects built with meson can very easily leverage the Python package ecosystem (not saying for the rest of meson, but for that particular component).

I personally wouldn't be shocked if some methods in the python module requires external python module, especially for something as specific as this. IMHO as long as the core of meson rely only on the stdlib we should be good. But I don't know if other Meson dev share that opinion.

tristan957 commented 2 years ago

As someone who has worked with and on mesonpep517, I definitely feel that moving into at least the mesonbuild org here on GitHub is a step in the right direction. mesonpep517 is also super far along down the pep517 spec and starting to get pep621 support. I am not sure I see advantage in merging mesonpep517 into meson proper except the ability to use internal APIs. @FFY00 maybe you could shed some light on why you think this behavior needs to exist in Meson? I definitely would hate to schism here if something could be resolved.

nanonyme commented 2 years ago

I would in fact suggest working on bootstrapping story here. Examples

Neither of these two cases require setuptools nor the soon (3.12) to be removed distutils. From that point of view if this project grew native PEP517 support, it might make sense to have it be a subproject (within same repo) or another project completely so that it could be used as the primary build system for meson.

eli-schwartz commented 2 years ago

As I said above:

an MVP should let meson build itself using meson, and install itself using meson too -- all without any dependencies

That is, officially supporting pep517 in meson should mean that meson can achieve the same goals in installing python software, EITHER by building a wheel and letting pip or python-installer install that wheel, OR by running ninja install. Both should produce the same files, including the dist-info, so ninja install would install a proper python package and not just copy over some .py files.

FFY00 commented 2 years ago

That is, officially supporting pep517 in meson should mean that meson can achieve the same goals in installing python software, EITHER by building a wheel and letting pip or python-installer install that wheel, OR by running ninja install. Both should produce the same files, including the dist-info, so ninja install would install a proper python package and not just copy over some .py files.

I just wanted to point out that Meson being able to bootstrap itself doesn't actually have to depend on it having a public backend, it can use a private in-tree backend that only implements the necessary to generate distributions for itself.

That said, I do think it having a public backend would make sense. But, as you point out, there are some challenges, and the decision is not up to me. I am just waiting to see what you decide.

thiblahute commented 2 years ago

I just wanted to point out that Meson being able to bootstrap itself doesn't actually have to depend on it having a public backend, it can use a private in-tree backend that only implements the necessary to generate distributions for itself.

We can use an external backend implementation, I am planning on writing a meson patch to use mesonpep517.

nanonyme commented 2 years ago

Just clarifying: meson can't bootstrap itself right now, right? Since there's no meson.build so it can't build itself.

eli-schwartz commented 2 years ago

We can use an external backend implementation, I am planning on writing a meson patch to use mesonpep517.

It currently does use an external backend called "setuptools". If we are going to move backends I'd like the move to represent a growth in functionality, not a reduction.

For example, with setuptools, meson can be installed using a "legacy" route as python setup.py install, mesonpep517 does of course require pep517. And in general the entire bootstrap setup of python modules on Linux distros is not (yet) prepared for pep517, but is very well prepared for setuptools.

Also, setuptools is actually a very good python build backend, both with and without pep517. :)

So I see no reason to give up setuptools unless we can actually solve the bootstrap problem once and for all, completely, 100%.

I just wanted to point out that Meson being able to bootstrap itself doesn't actually have to depend on it having a public backend, it can use a private in-tree backend that only implements the necessary to generate distributions for itself.

Bootstrapping does NOT mean "generate distributions". It means "install distributions". Otherwise it is not bootstrapping, it is just... generating stuff which your actual bootstrap chain needs to process, but only after the full bootstrap chain has run.

This is why neither flit nor poetry are bootstrappable, but setuptools is (and while it does vendor its own dependencies for convenience, its dependencies traditionally use stdlib distutils rather than setuptools to build, for precisely this reason).

FFY00 commented 2 years ago

It currently does use an external backend called "setuptools". If we are going to move backends I'd like the move to represent a growth in functionality, not a reduction.

For example, with setuptools, meson can be installed using a "legacy" route as python setup.py install, mesonpep517 does of course require pep517. And in general the entire bootstrap setup of python modules on Linux distros is not (yet) prepared for pep517, but is very well prepared for setuptools.

Also, setuptools is actually a very good python build backend, both with and without pep517. :)

So I see no reason to give up setuptools unless we can actually solve the bootstrap problem once and for all, completely, 100%.

I don't really follow your point here, it would grow functionality. By using a backend like mesonpep517, Meson would be able to bootstrap itself -- you could run Meson from source to install itself.

nanonyme commented 2 years ago

@eli-schwartz this is inaccurate. I wrote a PoC wheel installer with no external dependencies which cannot handle dependencies etc and was perfectly able to bootstrap fully functional flit using it. It was not super-hard, biggest issue is having some thing that generates entrypoints. Yes, setuptools can be bootstrapped without pep517 by directly invoking setup.py. But the chains are so complex that wheel problems supporting pep517 itself due to cyclic dependencies.

FFY00 commented 2 years ago

FWIW, I have been working on https://github.com/FFY00/mesonpy, which deals properly with project shared libraries that are required by the Python module. It also packages executables (as wheel scripts) and has a fairly decent test coverage.

Only Linux is currently tested, but I will be working on fixing and validating support for Windows and MacOS soon.

nanonyme commented 1 year ago

Just FWIW: the chains look like this in modern world:

Python->flit_core->pypa-installer (invoking flit_core+using pypa-installer sources to install the wheel) Python->setuptools->wheel (bdist_wheel with some hackery to use wheel form source root+pypa-installer)->meson (bdist_wheel+pypa-installer)

So arguably it would offer a bit simpler bootstrapping path if Meson was self-bootstrappable instead of using setuptools. Not necessary that much but some. Even using flit_core instead of setuptools would help some. (this is from distro point of view; end-user would just run pip and get things working out of the box no matter which PEP517 backend is used) Note that using setuptools install codepath (without building wheel) is deprecated.

eli-schwartz commented 1 year ago

I don't really follow your point here, it would grow functionality. By using a backend like mesonpep517, Meson would be able to bootstrap itself -- you could run Meson from source to install itself.

You go from needing 2 packages (setuptools and meson) to needing 13 packages -- wheel, toml, setuptools, packaging, pyparsing, mesonpep517, meson itself, a build frontend such as build, tomli, tomli_w, pep517, flit_core, and an installer frontend such as installer, and after all that you still have a dependency cycle between mesonpep517 and meson.

Switching from mesonpep517 pushes that sideways to mesonpy, wheel, setuptools, packaging, pyparsing, pyproject-metadata, build, pep517, tomli, tomli_w, flit_core, installer, and meson itself, which is still 13 packages, still has the dependency cycle, and may require a 14th one in the form of a wheel providing ninja.

You would not be able to run meson from source to install itself, because you need ten other python packages to collaborate in order to create and install the dist-info. This is problematic for Meson, because Meson tries very hard to be dependency-free, not because we don't like other projects providing useful functionality, but because... Meson needs to be as easy as possible to build and install at the lowest levels of a distro bootstrap. The fact that we use setuptools at all is a sacrifice because there's basically no choice, we have to use something.

My ultimate goal is to go from 1 to 0.

Even using flit_core instead of setuptools would help some. (this is from distro point of view; end-user would just run pip and get things working out of the box no matter which PEP517 backend is used) Note that using setuptools install codepath (without building wheel) is deprecated.

Unfortunately, using flit_core is actively a regression.

The fact that setuptools setup.py install (yes, I know) is deprecated does not mean it stopped existing. Currently using this deprecated route is a working 1-dependency bootstrap chain. Switching to flit would mean, once again, requiring wheel, setuptools, packaging, pyparsing, build, tomli, tomli_w, pep517, flit_core, installer. Which is 10 packages, a small reduction in complexity compared to using a third-party meson pep517 backend that does wheels. But a lot more complex than setup.py install which is 1 package.

...

As I said in a comment above, my ultimate goal is that, with 0 dependencies:

I want to move off of a deprecated setup.py install to something that is maximally bootstrappable, an exotic need that unfortunately Meson does have.

In the meantime, people who want to install Meson with pip and don't care about low-level OS bootstrapping, can simply pip install, using the modern approach of building a wheel, and the fact that some people are using deprecated routes is not a problem.

The only people using a deprecated route are the people who have to, for which neither flit, nor mesonpep517, nor meson-python are options.

nanonyme commented 1 year ago

No, flit_core does not need all those things you mentioned. The pep517 backend everyone is using is flit_core. Creating wheel is a trivial thing and flit_core does it without wheel package. A command-line frontend various people are using for development is flit. It takes two-step bootstrap from Python to have fully functional wheel installing system with flit_core. You can trust me on this, I have implemented it for a meta-distro. It is absolutely awesome to be self-bootstrappable.

eli-schwartz commented 1 year ago

The backend requires a frontend.

Flit (not core) is indeed an alternative frontend to build, but...

... that introduces its own dependencies in addition to flit_core, namely: docutils, requests, urllib3, idna, charset_normalizer, and of course all those use setuptools+wheel which means you still need the build frontend.

eli-schwartz commented 1 year ago

It is absolutely awesome to be self-bootstrappable.

Self-bootstrappable is indeed awesome, but "it only requires 2 dependencies" is not "self", and I'm still not clear on what frontend you use to invoke flit_core.

nanonyme commented 1 year ago

It is absolutely awesome to be self-bootstrappable.

Self-bootstrappable is indeed awesome, but "it only requires 2 dependencies" is not "self", and I'm still not clear on what frontend you use to invoke flit_core.

I meant self-bootstrappable as being able to install Meson using Meson from source root or some other simple bootstrapping mechanism.

nanonyme commented 1 year ago

There is no frontend other than python3 -flit_core.wheel needed for a distro when pep517 backend is known to be flit_core, really. After that you get a wheel you can unpack either directly or by using the recent pypa/installer.

eli-schwartz commented 1 year ago

There is no frontend other than python3 -flit_core.wheel needed

Oh huh, TIL. Last I checked, flit_core didn't actually provide a CLI. Now it does. That's actually pretty neat.

https://github.com/eli-schwartz/meson/commit/83a2e6692c6c2b75c532048cc7c230beb43194ea is still the long-term direction I'd like to go in, though.

nanonyme commented 1 year ago

Yeah, things have been changing a lot during last half a year or so. We will most likely gradually see pep517, packaging etc switch to flit_core now that bootstrapping chain exists.

mattip commented 11 months ago

I think this can be closed? mesonpy as a backend works.

tristan957 commented 11 months ago

@FFY00 is the intention of mesonpy to eventually merge it into this repo? If not, I agree that this should be closed

rgommers commented 11 months ago

I believe that (a) meson-python is not going to be merged as a whole into this repo and (b) @eli-schwartz has plans to eventually include a PEP 517 backend directly into meson which may take lessons from meson-python (but have zero dependencies and fewer bells and whistles).

Probably good to close this issue indeed.

nanonyme commented 11 months ago

Is there a second issue about that venture by @eli-schwartz ? It would be good to link from here so people can follow up if it ends up being furthered or abandoned.