Scope of this project and API design

[pradyunsg]

This project is born out of https://discuss.python.org/t/3869/. Let me know here if you participated there and want push access.

As a first step, we should figure out what exactly do we want this project to be. :)

[FFY00]

Getting off topic. This has been extensively discussed, for my use-case pip is not an option.

[pfmoore]

OK. I still feel that this project could be Python 3 only, with a backport maintained by people who need Python 2, but I'm not a project maintainer, so it's not my call. I was hoping for a bit more explanation of the decision (I'll re-read this thread, but I didn't spot much about Python 2 on my first reading) but again, not my call.

[sinoroc]

Are there enough projects that are both Python 2 and PEP 517 to make it worth the extra work? If I am not mistaken, it is the combination that would be critical here.

[FFY00]

Are there enough projects that are both Python 2 and PEP 517 to make it worth the extra work? If I am not mistaken, it is the combination that would be critical here.

PEP517 is only part of the issue, the removal of setup.py install will force us to install packages via wheels. We need an installer.

OK. I still feel that this project could be Python 3 only, with a backport maintained by people who need Python 2, but I'm not a project maintainer, so it's not my call. I was hoping for a bit more explanation of the decision (I'll re-read this thread, but I didn't spot much about Python 2 on my first reading) but again, not my call.

I would not be opposed to this, but backporting is something we will need to do. I think I would still prefer to have at least one Python 2 release instead of backporting.

[brettcannon]

I think I would still prefer to have at least one Python 2 release instead of backporting.

I don't know how much work is left, but if the folks doing the work are okay with getting an initial version out supporting Python 2 and then immediately switching the project's policy to Python 3 only then that should satisfy everyone, correct?

[gaborbernat]

I don't know how much work is left, but if the folks doing the work are okay with getting an initial version out supporting Python 2 and then immediately switching the project's policy to Python 3 only then that should satisfy everyone, correct?

Well for adopting it in virtualenv we'd need some potential bug fixes in there...

[pfmoore]

As a consumer, that works for me. (I arrived here as a result of doing some code review where I was distracted by Python 2 compatibility details, but that's a different perspective).

[brettcannon]

@gaborbernat I'm sure it can be kept around as a branch for any bugfixes people wanted to send. I think the key worry coming from @pfmoore is maintenance overhead.

[sinoroc]

One thing that annoys me about existing wheel installing tools is they all expect to run “in-environment” i.e. introspect what itself is in and behave accordingly. But this does not need to be the case since installing a wheel is simply extracting and moving files around. Explicitely specifying each path would enable the user to install a wheel to any environment as long as they know its layout, which is a huge win for out-of-environment tools like Poetry and Pipenv.

I'm curious about this kind of approach, although I don't know concretely how feasible it is. Probably adds a lot of complexity. I believe I have seen code in virtualenv that does this though, is that right? And that could (at least partially) solve the issue of having to write Python 2 compatible code.

[pfmoore]

I'm curious about this kind of approach

I would use this all the time. Examples:

1. Pip's --target mechanism, for installing to an arbitrary location.
2. Installing into a Python instance other than sys.executable - one preliminary query to ask the target interpreter where things like site-packages are, and then pass all the directories to the installer.
3. Preparing an install to be copied to a target environment that doesn't want build/install tools on it (a docker container, a low-resource embedded system, ...)

With a reasonably flexible mechanism, pip could use this to avoid the need to have pip installed in every environment where you want to install stuff. So, you could (for example) build a virtual environment via something like

python -m venv --no-pip xxx
# Note the following uses the system pip to install into the venv
python -m pip --target-interpreter xxx\scripts\python.exe foo bar baz

Personally, I'd consider "install to an arbitrary location" as the #1 most important feature of a wheel installer API.

[flying-sheep]

Hia, what’s the state of this? I’m pretty sure this (or something doing what this is supposed to do) is the future for distro tooling:

1. python -m build "$srcdir"
2. python -m installer --prefix="$pkgdir/usr" dist/...-1.0.whl

[FFY00]

No work on the CLI yet but I think the API is very close.

[pradyunsg]

Alrighty. The implementation is done, and the documentation site still needs a bunch of work. Here's a brief on how stuff ended up being designed:

There's an installer.install function, which takes a "source", a "destination" and additional generated-at-install-time metadata files. Both the WheelSource and WheelDestination are protocols and correspond to what installer.install needs.

The WheelSource object can be implemented without a dependency on this package. Nearly all of the logic for how-to-actually-place-files lies in the WheelDestination objects, which can be implemented more easily using utilities provided by this package.

---

Everything should have a docstring though, so it should be (mostly) possible to understand what does what based on reading those.

I'll cut a release sometime over the coming week. I expect that we will have to iterate on the API, or provide more utilities to make it easier to do certain tasks, but I do feel like this is mostly done.

I'll close this issue when the initial PyPI release goes out. Please feel free to file new issues for providing feedback on various aspects of the API.

[takluyver]

Thanks @pradyunsg! I might be able to help with docs at some point.

I can't see in the code the WheelFile class which the docs landing page mentions - is that still to come? Or is the idea that callers will implement that to use installer? (I'm :crossed_fingers: that won't be necessary :slightly_smiling_face: )

[takluyver]

Also, should it validate the RECORD file in the wheel while installing, or is that up to the caller?

[pradyunsg]

is that still to come?

Yea, that's the only "main" piece left to write here. I forgot about that. 😅

It should be relatively straightforward to write though, and some part of me is hoping that wheel's WheelFile will become compatible at some point in the future.

[pradyunsg]

Also, should it validate the RECORD file in the wheel while installing, or is that up to the caller?

Since the caller has access to the entire wheel file (i.e. the WheelSource object) already, they can perform any validation that they might want to on top of it. I'm unsure about how we'd want to support this though -- my current plan is "add a utility function that does various validations, that caller will call before installer.install", and destination objects can perform this validation too "during the install".

Thoughts welcome on how to handle this though!

[pfmoore]

Is it still the plan to drop Python 2 support once the initial release is made?

[pradyunsg]

Is it still the plan to drop Python 2 support once the initial release is made?

42 -- June this year.

[pradyunsg]

https://pypi.org/project/installer/0.2.0/ -- it's on PyPI now. https://installer.rtfd.io/ -- for the documentation.

Thanks everyone! I'll close this issue now. Please feel free to file new issues for specific things that might need changing! ^>^