Error with `import pip` in pip 9.0.2

[Miserlou]

Some users are now experiencing an import error when calling import pip in a function with pip 9.0.2. Downgrading to 9.0.1 fixes the issue.

Trace: https://user-images.githubusercontent.com/2273226/37558391-5e41fc94-2a24-11e8-9fdc-5884445e829a.png

More details here: https://github.com/Miserlou/Zappa/issues/1446

[dstufft]

Can anyone verify if https://github.com/pypa/pip/pull/5100 solves this for them?

[dstufft]

Scratch that, I think #5100 is wrong, could you verify https://github.com/pypa/pip/pull/5101 instead please.

[thet]

@dstufft I can confirm, the fix in #5101 solves the problem for me.

[msarahan]

Thanks @dstufft for your time in addressing this. I will work with the Anaconda teams to communicate this issue and help them move away from importing pip.

[vdog]

For the record in this thread, #5101 solved my issue as well.

[RemiCardona]

Ditto, #5101 allows the import to work inside our tool. Caveat emptor: I haven't tested the patched pip nor our tool extensively.

[dstufft]

This should be fixed in 9.0.3.

[diginc]

Thanks for the fast resolution, from someone who never wrote import pip in their life but was a consumer of a package that did. After reading through this thread sounds like lots of apps have imported pip, unknowingly against best practices, and lots of dev and users are potentially effected by v9.0.2 and v10.

I second/third/Nth a proper deprecation warning being added to make things smoother. maybe even a pypi.python.org post?

[Miserlou]

Hooray! Thanks for this!

I'd also really love a deprecation warning, and, more importantly - proper instruction on how to programmatically inspect python environments in a pip10 world! Clearly there is a need for that, given that over 700,000 applications were affected by this bug.

[dstufft]

pip list --format=json?

[drunkwcodes]

Firstly thank you all contributing to pip people for it totally covers all of my use cases with a few user-friendly options and arguments.

Due to this "surprisingly widely adopted and useful undefined behavior," do we need to make a piplib as libgit2 for git? Please note that libgit2 is not sharing any code with git, and is maintained by other group from git's. And it is good for git ecosystem. Maybe piplib will cover some interesting use cases which we didn't expect.

Here is a similar story: https://public-inbox.org/git/1267957655.3759.29.camel@mattotaupa/T/

[sruggier]

@drunkwcodes I suspect that what you propose is already implemented in the existing packages that the pip documentation mentions, packaging, setuptools, and distlib. As far as I can tell from this thread, there's no problem with a gap in functionality, but some people have problems with tools that try to import and inspect every module, and treat import failures as fatal errors.

It seems to me that such tools could work around this problem by wrapping import statements in a try/except block, but that also seems like a questionable precedent to set. Given the release of pip 9.0.3, though, I think it's probably not worth the time to debate the import issue unless the problem happens again in pip 10. Anyway, as long as the maintainers don't go out of their way to make import pip fail, or summarily reject patches that fix such failures, there would be common ground to stand on.

[drunkwcodes]

@sruggier The packages metioned are all good, and WheelFile.install() also needs more promotion. But the one-stop service pip.main() provided is irreplaceable with all of the above. It's still worth trying.

The most important is that I hope these needs are held by another project, and pip10 will arrive smoothly without extra guarantees. The deprecation and minimizing the code base are the whole points for a major release.

Concrete implementation details for a permanent infrastructure "software" is ridiculous. You can't judge maintainers by the documented abuse case and hold back the wheel of pip.

If you insist to use pip as a lib, a lot of assumptions will need to be reconsidered.

[pfmoore]

@drunkwcodes Just to be clear, pip.main() is the easiest usage to replace - you simply need to use subprocess.run([sys.executable, '-m', 'pip', <rest of args>]).

Also, the reason WheelFile.install() isn't promoted is that the wheel project have also stated that they do not provide a user-visible API - we originally mentioned wheel in the pip docs, but removed it at their request. The wheel PEP is pretty clear about how you install a wheel, though - it's not hard to implement in a 3rd party module.

[honnibal]

I appreciate the work you all do on pip, and know it's not easy. But for the record:

I'm a little disappointed that people haven't already moved away from import pip. The people hit by this issue will still need to prepare for pip 10

spaCy has moved away from import pip. But some people are still using spaCy 1, which did import pip --- and for obvious reasons, didn't pin pip down to a patch release. If the change had come in at v10, our old versions wouldn't have been affected. We've just issued a hotfix to address this.

[kalefranz]

proper instruction on how to programmatically inspect python environments in a pip10 world

What is PyPA's position on distlib? Pip is obviously using it in some capacity, but so too is it using packaging, which distlib purports to deprecate.

If there isn't an official position, then at least current thoughts and opinions from pip's core maintainers would be much appreciated. If there are relevant discussions on this topic elsewhere, a simple pointer to other discussions is good enough for me.

[pfmoore]

I'm not aware that distlib deprecates packaging. It mentions "distutils2/packaging" but distutils2 was something different.

My personal view is that both distlib and packaging are perfectly reasonable things to use. You should evaluate them yourself to confirm they meet your needs, obviously, just like any other dependency you rely on.

[kalefranz]

Maybe deprecates is too strong of word then. The source of my current understanding:

https://distlib.readthedocs.io/en/latest/overview.html#distlib-evolved-out-of-packaging

[pfmoore]

Ah, that's a different "packaging" - it was the proposed stdlib module that was intended to replace distutils. It's completely different from the PyPI packaging project. It might be worth asking the distlib project to clarify that distinction a little better.

[tuukkamustonen]

It might be worth asking the distlib project to clarify that distinction a little better.

Already done that :) See: https://bitbucket.org/pypa/distlib/issues/100/clarify-project-positioning-and-status

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.