new 'pre' syntax in Pipfile

[gsemet]

Placeholder ticket for the proposition we talk with Kenneth.

Add the following syntax in the Pipfile:

...
mypkg = { version = "*", pre="true"}
...

This would allow to allow prerelease resolution for one package only.

Change the pipenv install mypkg --pre behavior. It will set the setting described above only for this very package. Today it turns on a general 'pre' setting for the whole Pipfile, leading other packages to also resolve to prerelease.

Change the pipenv update and pipenv lock behavior. They should take into account the presence of the pre=true setting for some package.

Here are some impact I think might happen

• if a version of package is describe at >=1.0 in one dependency, and another package define the pre=true while a new version 2.0.0.b1 is available on pypi, pipenv install should take it
• I don't see any syntax to turn the general pre setting in the Pipfile. Maybe we can document users need to do a pipenv update --pre or pipenv lock --pre to turn the general 'pre' settings on.

[gsemet]

I am assigning myself to this task, but I might take some time to enter into the code

[techalchemy]

@frostming how does this proposal interact with the changes you made here recently?

[frostming]

The change in my PR is: when installing wildcard versions from CLI, Pipfile won't be modified.

With the proposal here, when we do pipenv install mypkg --pre, we are willing to replace the entry, right? The simpliest way is to add an extra arg pre to add_package_to_pipfile.

I am glad to help when it comes true.

[Gr1N]

Hello! Any updates or estimates on this?

[remingtonc]

If there's no progress on this I'd like to make it happen, otherwise I will revert to virtualenv/pip.

@frostming With your proposal, how would that end up looking in the Pipfile? I'm not familiar with how requirements are really parsed, briefly browsed some code. Completely ballparking, please correct me. Would it have to follow syntax akin to: EDIT: Per initial issue comment:

...
mypkg = { version = "*", pre="true"}
...

[gsemet]

I was on it, but there have been some change in the resolver I need to redo it completly. I would be glad to get some help on it

[jxltom]

I found that pin to specific prerelease version works for even allow_prereleases or --pre is provided

[ekhaydarov]

@gsemet where are you with making this a reality? Would be happy to help push this along

ive solved it in a weird way, definitely should make it available like @gsemet has suggested. However, you can fix a single package as a prerelease without wrecking the rest of the pipfile by going the editable route and directly specifying the release tag rather than the package in pypa i.e.

marshmallow = {editable = true,ref = "3.0.0rc4",git = "https://github.com/marshmallow-code/marshmallow.git"}

[ajenkins-cargometrics]

@gsemet where are you with making this a reality? Would be happy to help push this along

ive solved it in a weird way, definitely should make it available like @gsemet has suggested. However, you can fix a single package as a prerelease without wrecking the rest of the pipfile by going the editable route and directly specifying the release tag rather than the package in pypa i.e.

marshmallow = {editable = true,ref = "3.0.0rc4",git = "https://github.com/marshmallow-code/marshmallow.git"}

Thanks for the workaround. I will note that this also works without editable = true, as in:

marshmallow = {ref = "3.0.0rc4",git = "https://github.com/marshmallow-code/marshmallow.git"}

[gsemet]

@ekhaydarov Did nothing really, sorry :( had a baby, and has been lost in the resolver once :( not have many will to dig into it for this damned resolver for the moment :'(

[con-f-use]

Okay so what I'm about to outline might either be a bug or a stronger incentive for this feature request...

Assume we have this Pipfile:

[dev-packages]
my_package = { path = ".", editable=true }

And the package in . builds a pre-release version, then the whole affair ends in:

[pipenv.exceptions.ResolutionFailure]:       ResolutionFailure: ERROR: ERROR: Could not find a version that matches my_package 

Which is either a bug, because if people specify path="." , they expect that whatever is in . gets installed regardless of its pre-release/alpha/beta status (especially if it's in the [dev-packages] section) or it is a strong incentive to resolve this issue, because if not resolved, there is no other way to develop your alpha/beta/pre-release version with pipenv. One could even argue that if a package is in the [dev-packages] section, users would expect that alpha, beta and pre-release versions get installed even without explicitly allowing pre-releases versions for those.

You tell me if, I should rather open another issue to have pre-releases be considered if path= is specified in a dependency and then another for them to be considered for all packages under [dev-packages].

[gsemet]

Hi. I do not have any issue using pipenv install -e . or -e deps/mysubmodule, even with preversion. Don't know if it comes from the -e.

But I actually stopped using this feature for current module, too complex to maintain. Just have Pipfile[.lock] declare your dependencies, and have a Makefile with a target doing:

install-dev:
    pipenv install --dev --ignore-pipfile --deploy
    pipenv run pip install -e .

Bonus:

update:
    pipenv update --clear

Full cookiecutter provided here.

[con-f-use]

Ha! Thanks, you helped me realize that my problem is something slightly different.

Locking failes because the project in path='./' itself has a requirement that's only satisfyable by dev versions. So it's one level below.

[DrPyser]

If I take the road of specifying versions explicitly, is there a way to specify versions loosely without allowing pre-releases when allow_prereleases = true?

[DrPyser]

By "loosely" I mean something like ==x.* or >=x.y, i.e. a partially fixed version.

[gsemet]

yes only for non-prerelease:

pipenv install 'mydeps>=1.2,<1.3'

I agree that ==1.2 should means every bugfix version (1.2.1, 1.2.2, 1.2.3,...), but we don't want to have all the mess npm have with this.

[DrPyser]

I could reformulate: is there a way to turn the problem around and use allow_prereleases = true(because I need prerelease versions of some packages) but explicitly disallow prerelease versions for other packages while still using loose(partially pinned/bounded) version specifications?

But then I'm realizing that this wouldn't solve the problem for subdependencies.

Seems to me this issue is pretty critical. It doesn't actually make sense to allow prerelease versions for all your dependencies, since the usability of prerelease versions depends on each project. Some projects have very stable prereleases while others don't, and using their prerelease versions will break your app.

Isn't there a workable fix which could at least allow a pinned version to be a prerelease, without requiring the global allow_prerelease = true?

[DrPyser]

@gsemet So ==1.2.* would include prereleases, but >=1.2,<1.3 would not?

[gsemet]

if you use --pre >=1.2,<1.3 will find preversion. Most of the time you do not want to take prerelease, and if really, you need to freeze the version to use. --pre is global, meaning it will find prerelease for every dependencies. The pre syntax in pipfile would allow to set one or two deps to find prerelease, and only them. But we do have it.

[con-f-use]

Just wanted to mention #3651 here, because I proposed a per-package level option of either requires=[...] or pip_options=[...] to solve that issue, just like pre discussed here. At some point the maintainers could think about making a bunch of other Pipenv and pip options available per-package.

[DrPyser]

Sorry, I thought you couldn't pin prerelease versions without using allow_prereleases = true. The problem is less critical than I thought, then.

[j-walker23]

It is really painful we can't do this for dev packages.

[ncoghlan]

Just noting that pipenv isn't currently abiding by one of the usability recommendations that was included in the PEP 440 specification. Specifically: "By default, dependency resolutions tools SHOULD ... accept remotely available pre-releases for version specifiers where there is no final or post release that satisfies the version specifier"

[dragon788]

This bit me recently when trying to use Pipenv to set up and manage some load tests using locustio. It requires a newer version of gevent and wouldn't successfully let me pipenv install locustio until I added the --pre argument.

[andrey-semakin]

It constantly bits me, because black doesn't have stable releases yet.

pipenv + black = 💥

[ghost]

@andrey-semakin FYI, black = "==19.10b0" works. Not ideal of course, but pipenv will install it without issue.

[ghost]

FYI, black = "==19.10b0" works. Not ideal of course, but pipenv will install it without issue.

Just thought I'd note here that this won't work if you have a different dependency which requires black, but doesn't pin it to a specific version, e.g. pytest-black or flake8-black. More in-depth description of that problem over at #3928 and peterjc/flake8-black#18.

So for this Pipfile, there is no workaround which doesn't involve globally enabling prereleases, which is a big no-no for many users.

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[packages]

[dev-packages]
black = "==19.10b0"
pytest-black = "==0.3.8"

[j-walker23]

@tharradine ha! I wasted so, so, so, so many hours on this exact scenario bc of the successful workarounds mentioned. I finally gave up, and force all my devs to manually pip install the black stuff, or their forced commit hooks will never work : ).

And I always get complaints when their env is updated and now missing this manual pip dependency. It's freaking never ending.

I just sucked it up bc I knew I'd never get any help here because of the package lol. It's https://github.com/pausan/cblack.

Black for 2 indent python!! haha, not going to be any love there.

Glad you posted some legit third party packages!

[wlievens]

Is there really no effective workaround or progress on this? I'd like to be able to use pre-releases for internally-generated artifacts during development, but still depend on release versions of public libraries. I'm not sure pipenv is suited for this (trivial and common) workflow while this issue persists.

[Diaoul]

Just switch to some other tool like poetry, there is little hope this will get solved soon given the history.

[wlievens]

Yeah in fact I was just in progress of trying out poetry.

[kristang]

Is this issue still valid for the latest 2020 releases?

I might be misunderstanding the issue, or lucky with my dependencies.

I am pinning a rc-specific version of a package in my pipfile, and allow_prereleases = true.

[packages]
<PACKAGE_NAME> = "==0.12.80rc53"

[requires]
python_version = "3.8"

[pipenv]
allow_prereleases = true

In this case, is the only package with a pre-release version.

Is the issue that one wishes to allow for "*" and including the latest pre-release in that version identifier, on a per-package level, and not entire pipenv environment?

[obi1kenobi]

I believe pipenv will always allow you to pin pre-releases explicitly with an == bound, as your example shows — and it does not require allow_prereleases = true. That option says "prereleases are okay for any package" so if you had a second package in your example file, and that package had a higher-numbered pre-release available, pipenv would pull it in.

This is limiting in an interesting way that unfortunately comes up often (e.g. with the black autoformatter package, which is still considered in beta):

• You can set black = "==19.10b0" but if e.g. black publishes a 20.08b0 that is not going to be pulled in until you change the == bound.
• You can't say black>=19.10b0 + "pre-releases are okay but just for this package".

The pre syntax, if my understanding is correct, intends to allow the "pre-releases are okay but just for this package" use case.

[kristang]

Currently, I need to include allow_prereleases = true to get my dependencies to resolve, with a single package fixed as an rc.

In this case I have a rc version of , with another package having a requirement on as well:

Could not find a version that matches `<PACKAGE_NAME> = "==0.12.80rc53",>=0.1.59`

Then I get a list of "Tried" where it actually lists "==0.12.80rc53", but it still says it cannot resolve the dependencies.

[SheepReaper]

@kristang are you certain about your Pipfile syntax?

[packages]
some-package = "==0.12.80rc53"

it doesn't make sense to have a compound conditional if you have an exact version requirement I had the problem with black, and can verify that allow_prereleases is not required when using the above syntax

[kristang]

@bryan5989 yeah I made some error somewhere. I think I confused myself with a number of conflating issues and versioning breaking from an internal Python feed.

I have since gotten things to work as expected.

[1Mark]

Any progress on this?

[thehesiod]

I believe there should be a test-case where the Pipfile has this enabled for PKG-A set to a pre-release version range (VR1), and another PKG-B includes PKG-A as a dep w/ a non pre-release version w/in the range of VR1. Right now this is marked as Could not find a version that matches... since it can't combine any pre-release with release versions.

[astutejoe]

This would be great actually

[matteius]

We rely on the pip resolver, which specifies the flag globally, and so it would not be possible to support a per-package pre-release specification without pip supporting it first, is my understanding.

[joshsleeper]

@matteius thanks for the explanation and all your hard work!

1. would it be worth getting such a feature request filed in https://github.com/pypa/pip then and allow this issue to stay open for the decent number of us that are very interested in this in pipenv?
   • doing so would help avoid (or at least funnel) duplicate feature requests while still giving folks a place to watch for progress
2. if you're still set on actually closing the issue, maybe worth changing it from "closed as completed" to "closed as not planned" (configured via the little dropdown on the close issue button), that way there's no confusion about whether or not it was implemented?
   • https://docs.github.com/en/issues/tracking-your-work-with-issues/closing-an-issue

[matteius]

@joshsleeper I think if you or someone can take point on opening an issue with pip and linking it here, we can keep this open until such a determination is made by pip team -- if the pip team is not interested in supporting this on a per-package level, then I don't think we can support it in pipenv without extensive patching of the pip resolver, something which we are trying to avoid.

[joshsleeper]

oh for sure, 0% interest in you all crafting a bespoke replacement resolver just for this edge case 😅 in the meantime, thanks for reopening in the hopes that the pip team shows interest in such a feature!

[ncoghlan]

For individual packages, https://peps.python.org/pep-0440/#handling-of-pre-releases already requires filters like "> last.final.release" to accept pre-releases when no matching final release is available, which is the specifier you're going to need anyway if relying on a feature or fix that's only available in an upcoming release.

Is there a use case for accepting all pre-releases of a package in all situations, rather than only needing to allow their use when the newest final release doesn't meet a project's needs?

[matteius]

@ncoghlan Thanks for the reference, I had not read that prior.

By default, dependency resolution tools SHOULD: accept already installed pre-releases for all version specifiers accept remotely available pre-releases for version specifiers where there is no final or post release that satisfies the version specifier exclude all other pre-releases from consideration

I would say that in the case of pipenv, the resolution likely would not happen due to the package already being installed, so the happy path behavior is we accept remotely available pre-releases for version specifiers where there is no final or post release that satisfies the version specifier. Also the current pip resolver does accept or deny prereleases for all version specifiers, so it meets the should criteria.

The reason this ticket exists is the statement:

Dependency resolution tools MAY also allow the above behaviour to be controlled on a per-distribution basis.

The tool may allow that, but pip resolver does not currently, and there may be some mild use cases for it.

Trying to think of one, but I think it more described a behavior of the old pip resolver, that for a long while black was only pre-released, and so people wanted the ability to install the latest version of black, and I think there were issues with resolving it unless pre-releases were allowed, but other packages had more problematic pre-release versions. I could generalize this to say, you might trust a pre-release of a particular package developer more than another and want to resolve the latest releases of only those items, and not something else in your dependency tree that maybe the pre-releases are too experimental.

I think it just comes down to supporting more user preferences -- there are other things that ideally could be specified on a package level, such as what index it is resolved/installed from, without requiring multiple invocations of pip, but that is out of scope of this issue report, but just thinking out loud about things I would want from the next great iteration of the pip and its amazing resolver.