Closed mgab closed 4 years ago
Hello @mgab,
Thanks for the perfectly described issue! It's candy to my eyes ☺️ Regarding the issue, it's not a bug and this is how pip-compile currently works with constraints files. Please see this section and related comment https://github.com/jazzband/pip-tools/pull/1037#issuecomment-578470265.
@atugushev: In this section, I think it would be very reasonable for users to expect pytz==2019.2
to be added to dev-requirements.txt
, via this lineage: django-debug-toolbar==2.0 -> django==2.1.12 -> pytz==2019.2
.
This example fails if used with --generate-hashes
:
# requirements.in
django<2.2
# requirements.txt
#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile --generate-hashes requirements.in
#
django==2.1.15 \
--hash=sha256:48522428f4a285cf265af969f4744c5ebb027c7f41958ba48b639ace2068ffe7 \
--hash=sha256:a794f7a2f4b7c928eecfbc4ebad03712ff27fb545abe269bf01aa8500781eb1c
pytz==2019.3 \
--hash=sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d \
--hash=sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be \
# via django
# dev-requirements.in
-c requirements.txt
django-debug-toolbar
# dev-requirements.txt
#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile --generate-hashes dev-requirements.in
#
django-debug-toolbar==2.1 \
--hash=sha256:24c157bc6c0e1648e0a6587511ecb1b007a00a354ce716950bff2de12693e7a8 \
--hash=sha256:77cfba1d6e91b9bc3d36dc7dc74a9bb80be351948db5f880f2562a0cbf20b6c5
django==2.1.15 \
--hash=sha256:48522428f4a285cf265af969f4744c5ebb027c7f41958ba48b639ace2068ffe7 \
--hash=sha256:a794f7a2f4b7c928eecfbc4ebad03712ff27fb545abe269bf01aa8500781eb1c \
# via django-debug-toolbar
sqlparse==0.3.0 \
--hash=sha256:40afe6b8d4b1117e7dff5504d7a8ce07d9a1b15aeeade8a2d10f130a834f8177 \
--hash=sha256:7c3dca29c022744e95b547e867cee89f4fce4373f3549ccd8797d8eb52cdb873 \
# via django-debug-toolbar
Failure when installing dev-requirements.txt
:
$ pip install -r dev-requirements.txt
Collecting django-debug-toolbar==2.1
Using cached django_debug_toolbar-2.1-py3-none-any.whl (198 kB)
Collecting django==2.1.15
Using cached Django-2.1.15-py3-none-any.whl (7.3 MB)
Collecting sqlparse==0.3.0
Using cached sqlparse-0.3.0-py2.py3-none-any.whl (39 kB)
Collecting pytz
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
pytz from https://files.pythonhosted.org/packages/e7/f9/f0b53f88060247251bf481fa6ea62cd0d25bf1b11a87888e53ce5b7c8ad2/pytz-2019.3-py2.py3-none-any.whl#sha256=1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d (from django==2.1.15->-r dev-requirements.txt (line 10))
@jeevb
In this case pip install -r requirements.txt -r dev-requirements.txt
must be used. It would be nice to have it mentioned in README too.
@atugushev
If a constraints file is NOT specified in dev-requirements.in
, the resulting dev-requirements.txt
works on its own:
# dev-requirements.in
django-debug-toolbar
Notice how the below file is different from the one generated with -c requirements.txt
:
# dev-requirements.txt
#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile --generate-hashes dev-requirements.in
#
django-debug-toolbar==2.1 \
--hash=sha256:24c157bc6c0e1648e0a6587511ecb1b007a00a354ce716950bff2de12693e7a8 \
--hash=sha256:77cfba1d6e91b9bc3d36dc7dc74a9bb80be351948db5f880f2562a0cbf20b6c5
django==2.1.15 \
--hash=sha256:48522428f4a285cf265af969f4744c5ebb027c7f41958ba48b639ace2068ffe7 \
--hash=sha256:a794f7a2f4b7c928eecfbc4ebad03712ff27fb545abe269bf01aa8500781eb1c \
# via django-debug-toolbar
pytz==2019.3 \
--hash=sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d \
--hash=sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be \
# via django
sqlparse==0.3.0 \
--hash=sha256:40afe6b8d4b1117e7dff5504d7a8ce07d9a1b15aeeade8a2d10f130a834f8177 \
--hash=sha256:7c3dca29c022744e95b547e867cee89f4fce4373f3549ccd8797d8eb52cdb873 \
# via django-debug-toolbar
This works:
$ pip install -r dev-requirements.txt
Collecting django-debug-toolbar==2.1
Using cached django_debug_toolbar-2.1-py3-none-any.whl (198 kB)
Collecting django==2.1.15
Using cached Django-2.1.15-py3-none-any.whl (7.3 MB)
Collecting pytz==2019.3
Using cached pytz-2019.3-py2.py3-none-any.whl (509 kB)
Collecting sqlparse==0.3.0
Using cached sqlparse-0.3.0-py2.py3-none-any.whl (39 kB)
Installing collected packages: sqlparse, pytz, django, django-debug-toolbar
Successfully installed django-2.1.15 django-debug-toolbar-2.1 pytz-2019.3 sqlparse-0.3.0
Is it expected that adding -c requirements.txt
to dev-requirements.in
breaks the install of the dev-requirements.txt
file on its own?
@jeevb
Is it expected that adding
-c requirements.txt
todev-requirements.in
breaks the install of the individualdev-requirements.txt
file?
In this workflow, yes it is. If you use -r requirements.txt
, then you'd be able to install dev-requirements.txt
file on its own.
In this case
pip install -r requirements.txt -r dev-requirements.txt
must be used. It would be nice to have it mentioned in README too.
FTR, addressed in a tracking issue #1043.
@atugushev Thanks for the clarification!
@mgab
FYI, this bug should be fixed by #1037.
Thanks @atugushev and @jeevb! And sorry for the late response, I wasn't able to come back to you earlier!
Reading the following discussion in the #1037 pull request I see that you managed to understand what was the problem I was trying to point at despite my silence 😄. Thanks for the perseverance and fast reaction! And yes, it seems to solve the bug!
pip-tools v4.4.1
is released.
First of all, thanks a lot for developing and maintaining the package! It is a great tool!
Background
Regarding the issue, when using layered requirements (eg
test.txt
being a superset ofmain.txt
), two referencing modes are available-r
that defines requirements, and-c
that defines constraints.Including both
-r main.in
and-c main.txt
intest.in
should ensure thattest.txt
is a superset ofmain.txt
. It should produce the same results as including just-r main.txt
except by the fact that it keeps the comments indicating where indirect dependencies came from.Issue description
However, using both
-r main.in
and-c main.txt
at the same time drops some dependencies frommain.txt
that are not propagated totest.txt
.When installing
pip install -r test.txt
, the missing packages are installed anyway (they were dependencies, indeed). And in fact, the--verbose
flag of pip-tools shows that the missing packages are taken into account until the very end of the compilation process but are not included in the output.On the other hand, using only
-r main.in
does not miss any package (but does not ensure that the versions are the same).Environment Versions
Steps to replicate
To ensure reproducibility I ran this in a docker, but I was able to reproduce it elsewhere:
Install pip-tools
Create two simple layered requirements with
Compile
main.in
requirements with: -> [captured output]Compile
test.in
requirements with: -> [captured output]Expected result
Every package listed in
main.txt
and its version should appear also intest.txt
.test.txt
could include additional packages iftest.in
specified additional dependencies.This is the content of
main.txt
and the expected content oftest.txt
:Actual result
Instead,
test.txt
has less packages thanmain.txt
. The actual content oftest.txt
is:This happened with other library combinations, have not figured out the pattern yet.
As mentioned,
pip install -r test.txt
will also installsix
, and--verbose
shows that pip-tools is consideringsix
until the very end of the compilation process.