Closed shadycuz closed 2 years ago
I am having the same issue. Not sure how to fix.
@trishankatdatadog I had more time to investigate and while I haven't found any fixes I do think I have started narrowing down the problem space. I still can't rule out poetry for sure but it seems like it's a pip
or python issue from a change to the underlying Github actions servers. You can see a link to my issue with them above ^.
@shadycuz let me know what you find, because I gave up, would rather focus on my code right now
I'm having the same problem.
This is my log
nox > Command pip install --constraint=/tmp/tmp2khsb66t xdoctest failed with exit code 1:
Collecting xdoctest
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
xdoctest from https://files.pythonhosted.org/packages/36/59/e70a32f7d5e2c4933bda64991ce5254f553c31ddd918f7b9a7e732cdd1e2/xdoctest-0.15.0-py2.py3-none-any.whl#sha256=695ea04303a48cbb319709270d43f7bae7f3de3701aec73f09d90a216499992e
nox > Session xdoctest-3.9.1 failed.
I do have my hashes in requirements.txt
xdoctest==0.15.0 \
--hash=sha256:695ea04303a48cbb319709270d43f7bae7f3de3701aec73f09d90a216499992e \
--hash=sha256:7f0a184d403b69b166ebec1aadb13c98c96c59101e974ae2e4db4c3a803ec371
1. Why is it filed as a poetry issue?
2. Read https://github.com/pypa/pip/issues/8792 (and probably some other pip tickets as well)
Looks like you are all using the same pattern, but I am not sure I understand it and whether or not it makes sense. Would you mind walking me through it with a minimal reproducible example?
My gut feeling is that it boils down to pip install Alpha Bravo==1.2.3 --hash=xyz:abc
, and pip is very clear that you can't have requirements both with hash and without hash in the same installation command. On the other hand you seem to mean that it used to work, and/or does still work in some cases...
Would you mind walking me through it with a minimal reproducible example?
This is the smallest reproducible example I can find right now. Does this help?
I'd prefer you walk me through the thinking process that lead you to using this pattern.
requirements.txt
then use this as a constraint file to install development dependencies?I have my answers for most of these questions, but they probably don't match yours. Point is: maybe there is a better alternative.
@Corfucinas
I do have my hashes in
requirements.txt
xdoctest==0.15.0 \ --hash=sha256:695ea04303a48cbb319709270d43f7bae7f3de3701aec73f09d90a216499992e \ --hash=sha256:7f0a184d403b69b166ebec1aadb13c98c96c59101e974ae2e4db4c3a803ec371
Thanks for posting this. I was running my builds in CI/CD and did not have access to the contents of the exported file. It's great that we can confirm the hashes are present.
I'd prefer you walk me through the thinking process that lead you to using this pattern.
All good questions. It very largely came from hypermodern Python. I'll dig deeper and see what broke.
What's strange is that the nox setup works locally on my machine but not remotely on GitHub.
As for hashes in requirements.txt, I can confirm that when I rerun the offending command on my machine, I see the hashes.
@sinoroc
Why is it filed as a poetry issue?
Read pypa/pip#8792 (and probably some other pip tickets as well)
I filed this as a poetry
issue because I use poetry
to manage my dependencies and not pip
. I understand it's most likely an issue with something other than poetry
since I did not change the version of poetry
when this problem occured, but since this issue affects other users of poetry
it was probably a good place to create it in the end.
I'd prefer you walk me through the thinking process that lead you to using this pattern.
@trishankatdatadog and I both got the pattern from the excellent blog series Hypermodern Python. I'm not sure about @Corfucinas
- Why do you do what you do?
- Why do you export into
requirements.txt
then use this as a constraint file to install development dependencies?- What are the other things you have tried and did not work?
- Why don't you install all development dependencies?
- And so on
To be honest I do it because the blog told me so. I didn't spend much time thinking about it. I know the blog has a companion repo. The companion repo seems to have evolved beyond what was instructed in the blog posts. It seems they found better ways to do things. I unfortunately didn't realize this until after following the now outdated blog posts. I can say that up until this issue I was extremely happy with my new development workflow compared to just using pip
and virtualenv
in the past.
I have my answers for most of these questions, but they probably don't match yours. Point is: maybe there is a better alternative.
If you have a fix I would be eager to test it, even if the fix is just changing how we install our development dependencies for nox
.
This workflow still works fine locally. I'm guessing its something else that changed not related to poetry
. I have confirmed that the github actions did change the underlying image used for it's runners. I have issue with them as well.
@Corfucinas @trishankatdatadog
Seems like this was already discussed and solved in the Hypermodern repo.
@sinoroc Yes you linked directly to the issue =) with pip.
I feel good enough about closing this now. The issue is with the change to how pip
handles the constraints flag. pip
maintainers have confirmed it's not a bug but a design choice based on the new pip
resolver.
Workarounds:
poetry export
command
--without-hashes,
pip --contstraints
flag.python -m pip install --upgrade pip==20.2.4
virtualenv
that depends on pip, make sure you pin it.
python -m pip install --upgrade virtualenv==20.0.26
Or use a env var VIRTUALENV_PIP=20.2.4
@shadycuz Be careful with pinning too eagerly. I guess you all know that by now. :) Additionally I do not believe pinning virtualenv has decisive influence on the pip version used.
I do not know if the message went through to you all, so I will repeat it here. From what I understood, the issue is triggered by having in the same pip install
command requirements both with hashes and without hashes. So things like pip install Something --constraint export.txt
will always fail if export.txt
contains hashes, since Something
has no hash on the command line. So indeed you can:
Something
and install everything with pip install --requirement export.txt
Both have their pros and cons. Altogether the integration between poetry and tox (nox) could be improved. I started a plug-in to attempt to fix this integration (tox-poetry-dev-dependencies), but it is only half way there.
Anyway, looks like for now the best compromise for your use cases might be to export without hashes.
Some things to look for in the future:
Wanted to quickly chime in that nox-poetry
has just been updated to address this: https://github.com/cjolowicz/nox-poetry/releases/tag/v0.7.0
This happened to me because of environment markers, I was running pip on a version of Python that didn't match the tool.poetry.dependencies.python
set in pyproject.toml
. Therefore some dependencies were not specified for that Python version in requirements.txt
, and pip would fail (it would implicitly look for a version of that lib to satisfy dependencies of other requirements, then fail with this obtuse message about hashes).
E.g. I have python = "^3.7"
and my requirements.txt contains:
advocate==1.0.0
urllib3==1.26.4; python_version >= "3.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version < "4" and python_version >= "3.7"
When running that through pip on Python 3.6, no version of urllib3
is pinned, but advocate
depends on urllib3
. Pip would grab some version of urllib3
that satisfies advocate
's dependencies and then complain that urllib3
does not specify hashes (the requirement it grabbed automatically has no hashes, even though the requirements.txt hash hashes).
Collecting urllib3<2.0,>=1.22
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
urllib3<2.0,>=1.22 from https://files.pythonhosted.org/packages/5f/64/43575537846896abac0b15c3e5ac678d787a4021e906703f1766bfb8ea11/urllib3-1.26.6-py2.py3-none-any.whl#sha256=39fb8672126159acb139a7718dd10806104dec1e2f0f6c88aab05d17df10c8d4 (from advocate==1.0.0->-r requirements.txt (line 1))
Hopefully that helps somebody else.
I feel good enough about closing this now. The issue is with the change to how
pip
handles the constraints flag.pip
maintainers have confirmed it's not a bug but a design choice based on the newpip
resolver.Workarounds:
- For Hypermodern python users pass this flag in your
poetry export
command--without-hashes,
Perhaps a better solution to disabling hashes would be to disable the pip resolver. It doesn't make sense to me that pip's resolver would struggle with this when poetry has already done the dependency resolution. That's what is exported to the requirements file. To do this in CI, do something like the following:
pip install --no-deps -r requirements.txt
This can be useful in some cases where you are using a combination of internal packages and PyPI packages and want to maintain the hashes for security reasons.
The issue is with the change to how
pip
handles the constraints flag.pip
maintainers have confirmed it's not a bug but a design choice based on the newpip
resolver.
~it looks like it's fixed here https://github.com/pypa/pip/pull/8839 ?~
ah no it's not
Hash-checking mode requires that all requirements are specified as a == match on a version and may not work well in combination with constraints (see pypa/pip#9020 and pypa/pip#8792)
pip
maintainers have confirmed it's not a bug but a design choice based on the newpip
resolver.
however that doesn't seem to be the case: @pfmoore
I think the issue is in hash-checking mode, which shouldn't be checking for explicit
==
requirements https://github.com/pypa/pip/issues/9020#issuecomment-713103977
A better link for the pip-side of things -- we consider this a bug and that it needs someone to pick it up and file a PR for it: https://github.com/pypa/pip/issues/9243
Still a problem on the poetry side. I'm not pinning anything, just trying to dockerise python, and the export command fails, despite none of https://github.com/python-poetry/poetry/issues/3472#issuecomment-744356551 being relevant for me: I'm not downgrading pip because of a poetry issue — this gotcha has to go instead. (And vice versa; shame on pip for breaking previous versions in a minor/patch upgrade!)
I'm not sure how to interpret those errors, but I have the following:
Error:
ERROR:
In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
python-poetry/poetry#28 104.9 google-api-core[grpc]<3.0.0dev,>=1.22.2
from https://files.pythonhosted.org/packages/20/ca/db3b51439a29e29d94df81bef558ecdf7ca481e925cc99e90c126a6d3f7a/google_api_core-2.5.0-py2.py3-none-any.whl#sha256=7d030edbd3a0e994d796e62716022752684e863a6df9864b6ca82a1616c2a5a6
(from google-cloud-appengine-logging==1.1.0->-r /requirements.txt (line 460))
And line 460 of the requirements.txt is the following:
google-cloud-appengine-logging==1.1.0; python_version >= "3.6" and python_version < "4.0" \
--hash=sha256:91fe9b0833f6e1a46293dcc0e483716372c9ff4a95ebe51276c5f0092cb9defd \
--hash=sha256:438bdd8e4497368764d86f14b0eae295fc8a41f5a5b83fce89ae5a71067882d2
Attempted solutions:
NB, I also tried @jaron-l's solution (--no-deps
) and while it removed the error, later down the line my airflow webserver could not boot properly (502 gateway). My research has led me to believe that a common cause of airflow webserver 502 is due to improper dependency installation.
EDIT:
Fix proposed by @shadycuz (fix pip version to pip==20.2.4
) solved the issue on my side.
This being said, if anyone knows how to resolve the version fix manually via some poetry magic (except --without-hashes
), I'm all ears.
https://github.com/pypa/pip/issues/9243 just got fixed by https://github.com/pypa/pip/pull/10962 thanks so much @q0w !
https://github.com/pypa/pip/issues/9243 was reopened so I think this ticket should be reopened again. I ran into this same issue while trying to automatize one pipeline today.
Reopening as the original fix has been reverted but is being tracked here https://github.com/pypa/pip/issues/11019
What is this issue saying that poetry is doing wrong or should do differently? It's a long and confusing thread.
Is python-poetry/poetry#5537 relevant? That fixed an issue where poetry export
for a package with extras resulted in the same error message in this issues title (when trying to pip install the exported requirements.txt)
To be fair @dimbleby , I dont know and I never knew or if I did know, I dont anymore ;)
It's been referenced a lot and has had a lot of activity. I know that when pip changed something. Poetry went from working to not working. Then pip was fixed and it started working again. I think this issue was just to point people to a workaround and possible action from poetry maintainers if they needed to do something to poetry internals.
I haven't had the problem reoccur but it's also a very complicated problem. With the next release of pip
. I'm guessing my original problem comes back. So I reopened the issue.
If contributors think this issue is no longer useful, we can close it again. That way people experiencing issues could create new issues with less chatter and hopefully, someone can describe their problem in a much more elegant way than I ever could.
Special shoutout to Poetry and its contributors, who have made creating and maintaining python applications an enjoyable experience. I really feel bad for all the projects still using setup.py and requirements.txt files.
I agree that lots of people seem to think that Something is Wrong, but no-one seems to have identified any relevant way in which poetry could be improved. In that context I don't see that an open issue here is doing any good - what should the maintainers to do with it? Indeed, if there's not a poetry problem: why should poetry maintainers do anything with it at all?
(So far as being a pointer to a workaround goes: a closed issue works just as well as an open issue.)
If you or anyone else can make the case that your error is attributable to poetry then please do: that would be very helpful. Otherwise I'd suggest that this is better closed.
The problem with a closed issue is that it signals that the problem doesn't exist or was fixed. This current problem doesn't exist currently, but it will for sure soon.
While I agree their doesn't seem to be much the poetry maintainers can do if it's a third-party issue, it is still relevant to the users of poetry who will encounter this issue and by default, the search on a repo does not look at closed issues.
But I think its time to close this, as its long and convoluted and I dont remember any of the details of how to reproduce.
I can reproduce this issue, but it's only related to the poetry export requirements.txt file and the specific use in nox (which was worked around by disabling security)
This isn't something that can be fixed in peotry, but it could be something that peotry can provide a workaround for eg a new command like poetry contrained-venv-install ./venv/ flake8 flake8-black flake8-bugbear
that uses the poetry lock file to update the specified venv with a subset of deps
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
-vvv
option).Issue
My builds were working fine locally and remotely. Here is an example of the first 12 or so lines...
I added a package with
poetry add cfn-flip
My builds still work locally but fail in github actions:
Since my builds still worked locally I thought it might be an issue with PyPi, so I waited a day, but they are still failing. Up above I have linked my PR so you should be able to see the changes to the toml and the lock file. I'm very unsure what to try next except maybe deleting the lock file and having poetry recreate it again.
Locally and in the CI/CD I'm running the same Nox and Poetry versions. I think this might be related to python-poetry/poetry-plugin-export#38 and possibly python-poetry/poetry-plugin-export#145 since both of those are about
poetry export
.