pip installs version 0.9.0 on Python 2.7

[tleonhardt]

Last night I put out the 0.8.6 release which works on either Python 2.7 or 3.4+ and contains various bug fixes.

This morning I put out the 0.9.0 release which only supports Python 3.4+. I noticed that when using pip with a Python 2.7 release that pip now attempts to install cmd2 version 0.9.0, which it should not.

We thought we had properly modified setup.py so that version 0.9.0 would only attempt to install on Python 3.4+. Obviously, something is wrong. We need to either fix setup.py or find a way to modify the configuration on PyPI so that on Python 2.7, version 0.8.x installs, but on Python 3.4+ version 0.9.x installs.

[tleonhardt]

@kotfu @kmvanbrunt This should be regarded as a high-priority issue and we should try to fix it soon because it will affect anyone trying to install the latest version of cmd2 via pip on Python 2.7.

I'm not sure how to fix it and am about to hop on a cross-country flight. But if either of you have any insight, it would be great.

[kotfu]

I’ll have a look.

[kotfu]

Tried following instructions from https://packaging.python.org/guides/dropping-older-python-versions/. Tried it with rc releases. Didn't work. Tried it with 0.9.0.1 and 0.8.6.1 releases. Didn't work.

Tried building wheels and pushing 0.9.1 and 0.8.7 releases. Now pip2 (i.e. pip running in a python 2.7.15 environment) tries to install 0.9.0 as the most recent version supported by python 2. It appears that somehow we got python requires package metadata messed up on our versions, and pypi thinks 0.9.0 is the latest one with support for python 2. I don't know how to fix this.

[LipingMao]

@kotfu Is it possible to delete the 0.9.0 relased package? refer [1]. This problem break the CI job of kolla after cmd2 released to 0.9.0 .

[1]https://stackoverflow.com/questions/20403387/how-to-remove-a-package-from-pypi [2]https://review.openstack.org/#/q/project:openstack/kolla

[tleonhardt]

I just deleted the 0.9.0 released package. I will delete more releases if necessary

[kotfu]

Provisional testing indicates this may have solved the issue. I'll continue testing.

Thanks @LipingMao for the suggestion! 🥇 👏 🎉

[tleonhardt]

I also deleted the 0.9.1rc1 release because it looked like there might have been a chance that it could have tried to install this for Python 2 if someone used the --pre flag with pip.

[LipingMao]

@kotfu @tleonhardt thanks for your quick help!

looks it has been removed from https://pypi.org/simple/cmd2/ . Kolla CI still failed because it using mirror pypi repo[1], the 0.9.0 is still there, I would try to check in mirror side.

[1]http://mirror.regionone.limestone.openstack.org:8080/pypi/packages/53/e5/5ec46c74d13b488dedc93d5ff951f459bd898f4315c7564fb67f30d7aec9/

[tleonhardt]

@LipingMao Do you have any advice on how to get that PyPI mirror updated?

[LipingMao]

@tleonhardt I have no idea, let's wait for some time to see if it is updated. I would also try to ask others in irc channel to see if they know mirror will auto updated.

[apevec]

If removed releases are not deleted in the mirror, openstack-infra need to check their pypi mirroring, from PEP-0381: "If a package is deleted on the central server, they MUST delete the package and all associated files."

[fungi]

The aforementioned mirror is not an official PyPI mirror, and is only used by the OpenStack community's CI system, so this is not an appropriate venue to reach its operators.

However as one of its operators I feel compelled to point out that it uses https://pypi.org/project/bandersnatch/ to pull updates from PyPI, but per its logs bandersnatch did not see a deletion of cmd2-0.9.0-py2.py3-none-any.whl presumably because only the release was removed and not its files. I can still similarly download https://files.pythonhosted.org/packages/53/e5/5ec46c74d13b488dedc93d5ff951f459bd898f4315c7564fb67f30d7aec9/cmd2-0.9.0-py2.py3-none-any.whl (supporting that theory). This should be fine, as the release removal has taken it out of the index so that pip will no longer find it there: http://mirror.regionone.limestone.openstack.org:8080/pypi/simple/cmd2/

[mclapinski]

@fungi Are you sure this is fine? I think I'm still getting failures in OpenStack CI related to that. Maybe the maintainers of cmd2 could remove the files so bandersnatch will notice it? I know they don't have to do it but maybe it would be a nice gesture for the community. Obviously unless it's not possible; I don't actually know how PyPI works.

[fungi]

It is not indexed in our PyPI mirror, but still appears separately in the wheelhouse cache for our CI (which is additive and hand-maintained where removals are concerned). There is work underway to address that, but please direct discussion to OpenStack mailing lists and IRC channels. Let's not pollute this bug report further.

[tleonhardt]

I tried finding some way in PyPI to delete the old files, but I can't find it. I deleted the problematic releases and now that they are deleted I can't find a way in PyPI's UI to deleted the remaining files. If someone can tell me how to do it, I would be happy to do so.

[fungi]

Normally you'd be able to use the management interface to delete those files from the release prior to deleting the release itself, so something like https://pypi.org/manage/project/cmd2/release/0.9.0/ (I haven't tested if that works for releases which have already been deleted, but I wouldn't be surprised to discover that it doesn't).

[tleonhardt]

The PyPI management interface has links of the form: https://pypi.org/manage/project/cmd2/release/0.9.0/

But it won't let me manage a release which has been deleted.

I had assumed that deleting a release would automatically delete associated files. Maybe you have to delete the files prior to deleting the release?

[fungi]

That's what I assume as well. In the case of our CI system what we're seeing with our PyPI mirrors now (which presumably others are experiencing with their bandersnatch-maintained mirrors as well) is not that 0.9.0 packages (sdist or wheel) are hanging around anywhere. It's that Python 2.7 environments lacking a pin for cmd2<0.9 are now retrieving 0.9.1 as an sdist and then breaking once they attempt to install it with an error like:

cmd2 requires Python '>=3.4' but the running Python is 2.7.6

I expect the proper solution for downstream consumers is to declare PEP 508 environment markers along the lines of:

cmd2;python_version>="3.4"
cmd2<0.9;python_version=="2.7"

[tleonhardt]

@fungi I agree that what is probably the best solution moving forward is to have projects which use Python 2.7 environments pin to cmd2<0.9.

[tleonhardt]

@kotfu I reached out and got a response from a Python core developer who said:

""" Just do a new release with a post1 release version, e.g. 0.9.0.post1 and that will override the previous release for everyone who tries to download. But otherwise I don't think you can delete the whole files to avoid a leftpad fiasco in Python-land. """

If you want to go ahead and try that post release trick, I think it would be a good idea to prevent any possibility of anyone getting the messed up 0.9.0 release. Or I can try it later.

[tleonhardt]

I published the 0.9.0.post1 release in attempts at overriding any previous 0.9.0 files.

[mmalchuk]

didn't work either

$ pip2 install cmd2 Collecting cmd2 Downloading >https://files.pythonhosted.org/packages/07/4a/f8c16175fa806b9c81f60688411b09a84a23decf774f2cdd5657e43dfe9b/cmd2-0.9.0.post1.tar.gz (136kB) 100% |████████████████████████████████| 143kB 1.8MB/s cmd2 requires Python '>=3.4' but the running Python is 2.7.15

[queria]

Seems that 0.9.0.post1 broke few things again, I have two occasions

 Collecting cmd2>=0.6.7 (from cliff!=2.9.0,>=2.8.0->osc-lib>=1.8.0->python-ironicclient>=1.14.0->shade==1.25.0)
  Downloading https://files.pythonhosted.org/packages/07/4a/f8c16175fa806b9c81f60688411b09a84a23decf774f2cdd5657e43dfe9b/cmd2-0.9.0.post1.tar.gz (136kB)
    Complete output from command python setup.py egg_info:
    /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
      warnings.warn(msg)
    error in cmd2 setup command: Invalid environment marker: python_version<'3.5'

 #  (with pip 0.9.1 used)

or also

Collecting cmd2>=0.6.7 (from cliff==1.15.0->-r rhos-qe-jenkins/requirements.txt (line 15))
   Downloading https://files.pythonhosted.org/packages/07/4a/f8c16175fa806b9c81f60688411b09a84a23decf774f2cdd5657e43dfe9b/cmd2-0.9.0.post1.tar.gz (136kB)
cmd2 requires Python '>=3.4' but the running Python is 2.7.5

[tleonhardt]

@queria Thanks for the heads up regarding it being broken again! So much for the good intentions on my part trying to "fix things more".

I deleted the 0.9.0.post1 release, including the files this time.

Hopefully things should be back to the way they were.

[tleonhardt]

@kotfu So was the actual fix for this issue in the following commit where you removed support for building universal wheels? https://github.com/python-cmd2/cmd2/commit/47908e7e67682d0f28ee7bf7680c8192dd8ad8a7

[tleonhardt]

@LipingMao Do you believe that this Issue can be closed or do you believe further work should be done on our end?

[kotfu]

@tleonhardt I'm not exactly sure. I'm guessing it was not the removal of the universal wheel for our python 3.x only versions. I think it was the addition of python-requires >=2.7 to the python2 branch, and publishing that to pypi, and then publishing another release of the master branch with python-requires >=3.4.

[tleonhardt]

I'm going to close this Issue because I think it has been effectively mitigated as best we can.

If there are on-going problems related to this, then we can re-open it.