build system abstraction PEP

[rbtcollins]

[rgommers]

You may want to edit the title of this PR to reflect its content.

[rgommers]

This version looks quite good and clear to me.

[dstufft]

https://mail.python.org/pipermail/distutils-sig/2016-January/028152.html

[takluyver]

Is there still interest in pushing this forwards?

[rbtcollins]

certainly there is interest in pushing this forward. I don't know that we have to support all the possible interactions of ways of putting a Python layout together: at the heart of it, we're never going to be called for a different interpreter - pip always uses the interpreter of the resulting thing to invoke builds.

[takluyver]

I don't know that we have to support all the possible interactions of ways of putting a Python layout together: at the heart of it, we're never going to be called for a different interpreter - pip always uses the interpreter of the resulting thing to invoke builds.

Was this meant as a reply to this comment of mine? If so: pip itself will always be running in the target Python interpreter, but that need not be true of the build tool it's invoking for pip install -e. In the case of flit, for instance, it requires Python 3, so if you want to install into a Python 2.7 environment, flit will necessarily be running with another interpreter, and it needs to know where to install to.

We can also imagine build tools that need not run inside a Python interpreter at all.

Maybe development installs (-e) are too complex a topic, and we should leave them out of this proposal and write a separate proposal to standardise that interface.

[njsmith]

Wait, this py3 thing is a problem :-(

Even without editable mode, there's a problem with pip install pkgname (assuming no wheel is available, which I recognize is unlikely for flit-using packages), or pip install path/ (much more likely, since this will be the standard incantation that we teach users for installing arbitrary packages from e.g. git checkouts, instead of setup.py install). These commands have to be run using the target python, simply because that's the only python we know about or that is guaranteed to exist -- random users don't necessarily have a python 3 interpreter available, and pip install python3 isn't really a thing.

In a sense I guess this is not too different from how packages may also require that you provide C compiler and so forth. But in that case we still assume that the logic that provides glue between pip and the C compiler has to be pip installable, because what other choice do we have? I suppose you could make a python 2 compatible flit shim that just finds a copy of python3 and uses it to flit (or print a sensible error if this fails).

(Or I guess technically there's nothing stopping us from making wheels that contain a python 3 interpreter inside them, except good sense and/or a certain ingrained conformity to traditional norms.)

[takluyver]

...I semi-seriously like the last suggestion. Provide 'Python 2' wheels for flit that bundle their own copy of a Python 3 interpreter. I'm probably not actually going to bother to do that, because messing around with all the different builds sounds tedious, but it's amusing to know that it's possible.

[rgommers]

But in that case we still assume that the logic that provides glue between pip and the C compiler has to be pip installable, because what other choice do we have?

That it's not pip-installable, which is perfectly reasonable.

I think you're asking for major problems by trying to have non-wheel cross-talk between different Python installs. I'd suggest to stick to a simple rule: either the build tool uses the same interpreter, or it's managed by the user.

In the case of flit, for instance, it requires Python 3, so if you want to install into a Python 2.7 environment, flit will necessarily be running with another interpreter, and it needs to know where to install to.

flit doesn't need to install anything, that's what pip is for. Even for a develop command, flit should build the project in-place, and pip should take care of installing the egg_info or similar file to make that in-place build visible right?

[rbtcollins]

So python3 and python2 and flit. when pip is installing something in a python 2 environment there is no guarantee python3 is available - and pip won't know how to install flit in that situation - it would need to be treated like e.g. 'make' or 'autoconf'. If flit wants to be used for building python2 compatible things, and be like that, it certainly can. That doesn't affect the spec here though - just use the already defined variable PYTHON to pass the PYTHON that pip is actually using down for however flit's cross-python stuff is defined, and you should be good to go.

[takluyver]

@rgommers :

Even for a develop command, flit should build the project in-place, and pip should take care of installing the egg_info or similar file to make that in-place build visible right?

Possibly, but that is not what this proposal currently says. It says that pip invokes the build tool to do a development install however the build tool wants.

I also don't like the 'egg link' method that setuptools/pip uses, so flit will retain the ability to do development installs using symlinks, even if pip doesn't invoke it.

[rgommers]

Even for a develop command, flit should build the project in-place, and pip should take care of installing the egg_info or similar file to make that in-place build visible right?

Possibly, but that is not what this proposal currently says. It says that pip invokes the build tool to do a development install however the build tool wants.

Ah, I didn't see that. That is then completely out of line with the intention of the rest of this PEP. The build tool should stick to building, not installing. It will also give problems: if the build tool inserts some egg_info thingy or symlink or ... that pip doesn't know about, and then the user does a normal install, there are two installed versions and it's not clear which one is first on sys.path. @rbtcollins can this be changed?

I also don't like the 'egg link' method that setuptools/pip uses, so flit will retain the ability to do development installs using symlinks, even if pip doesn't invoke it.

Totally agree that egg_info, .pth files, etc. all smell.

[takluyver]

I'm inclined to say that we should drop development installs from this proposal for now, and come back to them separately. It feels like they're a pretty big topic on their own.

[rbtcollins]

re: editable installs. They exist, pip supports them. We can either break the workflow of any users of a project that adopts this thing, or we can keep supporting them and in future do the hard work to standardise them. I prefer not to break users.

[rgommers]

re: editable installs. They exist, pip supports them. We can either break the workflow of any users of a project that adopts this thing, or we can keep supporting them and in future do the hard work to standardise them. I prefer not to break users.

Nothing needs to be broken for users. My point is that the text now says that the build tool should install, while it should imho be saying that the build tool should build and the install tool should install that build, i.e. make it visible via an egg-info file or symlink.

I think that that's what you already meant, so I'm simply asking for a textual clarification. If that's not what you meant, then I don't understand the reason why you're letting the build tool take responsibility for installing something.

[dstufft]

I'm going to merge this PR not to accept it, but so that it's managed inside of the repository and I'll give @rbtcollins commit bit. I'm also going to assign it a number.

[dstufft]

Ok. This is now PEP 516.

[takluyver]

re: editable installs. They exist, pip supports them. We can either break the workflow of any users of a project that adopts this thing, or we can keep supporting them and in future do the hard work to standardise them. I prefer not to break users.

It's certainly nice if we can keep it working for users, but defining how it works seems to be a major difficulty with this PEP, and I'd rather it went through without a provision for development installs than got stuck trying to deal with that.

@rgommers:

My point is that the text now says that the build tool should install, while it should imho be saying that the build tool should build and the install tool should install that build, i.e. make it visible via an egg-info file or symlink.

If we do this, then we need to define what the build tool should build. In the normal install case, it builds a wheel, but that doesn't work for a development install. Probably the logical thing for it to build is a dist-info directory, but that's still somewhat ambiguous - e.g. dist-info inside a wheel is a bit different from the installed dist-info defined in PEP 376. Also, for instance, there's not much point in recording hashes of all the files in a development install, because the point is that they may change without reinstallation.

[rgommers]

If we do this, then we need to define what the build tool should build.

Only where, not what. The install tool doesn't have to, and must not, care what is built by the build tool.

In the normal install case, it builds a wheel, but that doesn't work for a development install. Probably the logical thing for it to build is a dist-info directory, but that's still somewhat ambiguous

No, it builds only whatever is needed to make the package work completely in the location it's building the development build. So usually nothing but compiled extensions and generated Python files (version.py for example). dist-info isn't needed for anything here.

e.g. dist-info inside a wheel is a bit different from the installed dist-info defined in PEP 376. Also, for instance, there's not much point in recording hashes of all the files in a development install, because the point is that they may change without reinstallation.

Hashes are not needed for anything, and what pip needs is already covered by the PEP, this text:

Note that the metadata generated by the metadata command, and the metadata
present in a generated wheel must be identical.

[njsmith]

@rgommers: editable installs need metadata so that pip and other tools can figure out that the install exists, can know what version it is (so that they know it can satisfy dependencies), and to support uninstall. All of these are a bit broken right now, but just removing the metadata entirely would be a regression. Also, editable installs may include scripts, which need to get installed onto PATH somehow. Also, in a perfect world they should do things like respect the platlib / purelib distinction, and support all the various file install locations that wheels do -- this doesn't actually happen right now, but if we sit down to do editable installs "right" then it probably needs to be sorted.

I started out with a similar opinion as you, but @rbtcollins convinced me that the swamp is much wider and deeper than that :-(

[rgommers]

editable installs need metadata so that pip and other tools can figure out that the install exists, can know what version it is (so that they know it can satisfy dependencies), and to support uninstall.

Clearly yes, but that metadata is what is output by the metadata command in this PEP and needs to be put there by the installer right? I've just tried pip install -e . --user to verify what it does, which isn't that complex. A file packagename.egg-link goes into site-packages, and a package-name.egg-info/ is put in .. pip uninstall package then only removes packagename.egg-link.

All of these are a bit broken right now, but just removing the metadata entirely would be a regression.

No one suggests removing the metadata. It can be retrieved by the install tool with the metadata and put in the right place.

editable installs may include scripts, which need to get installed onto PATH somehow.

Yes, that's probably the one thing that's necessarily a bit complicated.

Also, in a perfect world they should do things like respect the platlib / purelib distinction

Why? Is there a usecase other than in-place builds that I'm missing? The pip docs explain exactly nothing about what editable installs are for, so I'm just going with the only usecase I'm aware of.

I started out with a similar opinion as you, but @rbtcollins convinced me that the swamp is much wider and deeper than that :-(

If there's a discussion somewhere that I missed, can you provide a link? I'd really like to understand the actual problem here, because it seems quite problematic to me to have a PEP that attempts to cleanly split build from install, and then has a develop command that let's the build tool do an install.