Implicit post release requirements are stripped which causes errors if there is no corresponding non-post-release pypi package

[techalchemy]

When I install a package with the intent to upgrade in the future (>=), if the package has a dash in its version, Pipenv strips the dash when generating the lockfile which results in failed installation.

~/git $ mkdir tmp
~/git $ cd tmp

 ~/g/tmp $ pipenv install --python=python3.6 "ibm-db-sa-py3>=0.3.1-1"
Creating a Pipfile for this project...
Creating a virtualenv for this project...
Using python3.6 to create virtualenv...
⠋Already using interpreter /usr/bin/python3.6
Using base prefix '/usr'
New python executable in /home/hawk/.local/share/virtualenvs/tmp-f_tdiakk/bin/python3.6
Also creating executable in /home/hawk/.local/share/virtualenvs/tmp-f_tdiakk/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /home/hawk/.local/share/virtualenvs/tmp-f_tdiakk
Installing ibm-db-sa-py3>=0.3.1-1...
Collecting ibm-db-sa-py3>=0.3.1-1
Collecting sqlalchemy>=0.7.3 (from ibm-db-sa-py3>=0.3.1-1)
Installing collected packages: sqlalchemy, ibm-db-sa-py3
Successfully installed ibm-db-sa-py3-0.3.1.post1 sqlalchemy-1.1.13

Adding ibm-db-sa-py3>=0.3.1-1 to Pipfile's [packages]...

 ~/g/tmp $ pipenv lock
Locking [dev-packages] dependencies...
⠹Locking [packages] dependencies...
⠦Updated Pipfile.lock!

 ~/g/tmp $ grep ibm Pipfile
ibm-db-sa-py3 = ">=0.3.1-1"

 ~/g/tmp $ grep -a1 ibm Pipfile.lock
        "ibm-db-sa-py3": {
            "version": "==0.3.1"

 ~/g/tmp $ pipenv uninstall --all
Un-installing all packages from virtualenv...
Found 2 installed package(s), purging...
Uninstalling ibm-db-sa-py3-0.3.1.post1:
  Successfully uninstalled ibm-db-sa-py3-0.3.1.post1
Uninstalling SQLAlchemy-1.1.13:
  Successfully uninstalled SQLAlchemy-1.1.13

Environment now purged and fresh!

 ~/g/tmp $ pipenv install
No package provided, installing all dependencies.
Pipfile found at /home/hawk/git/tmp/Pipfile. Considering this to be the project home.
Installing dependencies from Pipfile.lock...
An error occured while installing! 0/2 - 00:00:00
Collecting ibm-db-sa-py3==0.3.1

  Could not find a version that satisfies the requirement ibm-db-sa-py3==0.3.1 (from versions: 0.3.0, 0.3.0.post1, 0.3.1.post1)
No matching distribution found for ibm-db-sa-py3==0.3.1

[================================] 0/2 - 00:00:00

[kennethreitz]

Let's take TDD here: write a failing test in a branch, then fix it. Can you do that @techalchemy?

[nateprewitt]

Before we go any farther here, there is a lengthy issue about this but it seems to have been closed. This is an issue with the pypi endpoint. Some packages do post-releases and use the proper notation with a dash, others do a post release but don't work with the dash notation. We removed the dash notation because it seems the latter was more common but that's clearly not a workable solution.

We'll need some form of a retry solution for this case I think. I'll add the appropriate issue number when I've got a moment to find it.

[kennethreitz]

To be clear — this IBM's fault for using such a dumb versioning scheme.

[nateprewitt]

@kennethreitz, this is actually a valid versioning scheme and there's even a pep explicitly defining its use. The problem is the implementation isn't consistent.

[kennethreitz]

Grrrrr

[kennethreitz]

Semver or Calver all the things

[techalchemy]

if we have good examples of each case it might not be that difficult, but might be ugly

[nateprewitt]

Ok, #270 was the original issue and #314 was the patch that is causing us to hit this undesired behaviour. I paged Donald on why this was happening but never got a response. We need a way to try the full version and then strip the post release version and try again.

[techalchemy]

@nateprewitt I believe this is an issue in this specific case because there is no non-post-release version of the package 'ibm-db-sa-py3' -- note there is only 0.3.1.post1, 0.3.0.post1, and 0.3.0, but not 0.3.1

It's possible that pypi would correctly resolve 0.3.1 to 0.3.1.post1 if there were a release package available for 0.3.1. As you mentioned this could be as simple as a fallback check for the post-release itself

[kennethreitz]

Closing, as this seems to be a pypi issue. I'd ask in #pypa on irc.

[techalchemy]

I think it is and it isnt, but I think we can test-resolve the package with pip-tools -- it has functionality to determine the correct version to 'pin' given a requirement. So to solve the issue I originally posted I think it's as simple as asking pypi for the correct pin or at least checking whether a version can be resolved before pinning it in the lockfile

As a sidenote, I wonder if its really correct to drop explicit post release references in requirements files or pipfiles when they are not pinned... in my case above, we are explicitly looking for the post release or newer.

[nateprewitt]

I think there's more we can do here. I'm thinking #314 was a mistake or at the very least it's incomplete. I think we can do better than we currently are so I'd like to keep this open while we look at alternatives.