Using a combination of `--generate-hashes`, `-c constraints.txt`, `--resolver=backtracking` and `--strip-extras` doesn't currently work

[pawelad]

Hi,

First of all, thanks for the package and all the hard work you put into it.

I tried looking through the repo for a similar bug, but I haven't found anything. At the same time, I don't think I'm doing anything too much out of the ordinary - I'm using:

• --generate-hashes, for obvious security reasons
• -c constraints.txt at the top of the dev requirements files, as that's the recommended layered approach
• --resolver=backtracking, for better package resolving (and it will be the default at some point)
• --strip-extras, because otherwise using --resolver=backtracking failed with Constraints cannot have extras

The above scenario unfortunately produces requirement files that cannot be installed because of this:

ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
PyJWT[crypto]<3.0.0,>=1.5.2 from https://files.pythonhosted.org/packages/40/46/505f0dd53c14096f01922bf93a7abb4e40e29a06f858abbaa791e6954324/PyJWT-2.6.0-py3-none-any.whl (from drf-jwt==1.19.2->-r /home/circleci/project/requirements/prod.txt (line 269))

As I understand it, --strip-extras, which is needed for --resolver=backtracking to work with-c constraints.txt, is currently incompatible with --generate-hashes, because it needs all dependencies to have hashes and (correctly?) differences between PyJWT[crypto] and PyJWT.

My current workaround will probably involve dropping --resolver=backtracking and --strip-extras for now, but I wanted to write up this issue while I'm debugging all this.

Related issues: #398, #1092, #1300

Environment Versions

1. macOS 12.6.1
2. Python 3.8.13
3. pip 22.3.1
4. pip-compile, version 6.10.0

Steps to replicate

Given requirements files:

$ cat main.in
django==3.1.14
djangorestframework==3.12.4
drf-jwt==1.19.2
PyJWT[crypto]==2.1.0

$ cat dev.in
-c main.txt

black

$ python3 -m piptools compile --generate-hashes --strip-extras --resolver=backtracking main.in
#
# This file is autogenerated by pip-compile with python 3.8
# To update, run:
#
#    pip-compile --generate-hashes --resolver=backtracking --strip-extras main.in
#
asgiref==3.5.2 \
    --hash=sha256:1d2880b792ae8757289136f1db2b7b99100ce959b2aa57fd69dab783d05afac4 \
    --hash=sha256:4a29362a6acebe09bf1d6640db38c1dc3d9217c68e6f9f6204d72667fc19a424
    # via django
cffi==1.15.1 \
    --hash=sha256:00a9ed42e88df81ffae7a8ab6d9356b371399b91dbdf0c3cb1e84c03a13aceb5 \
    --hash=sha256:03425bdae262c76aad70202debd780501fabeaca237cdfddc008987c0e0f59ef \
    --hash=sha256:04ed324bda3cda42b9b695d51bb7d54b680b9719cfab04227cdd1e04e5de3104 \
    --hash=sha256:0e2642fe3142e4cc4af0799748233ad6da94c62a8bec3a6648bf8ee68b1c7426 \
    --hash=sha256:173379135477dc8cac4bc58f45db08ab45d228b3363adb7af79436135d028405 \
    --hash=sha256:198caafb44239b60e252492445da556afafc7d1e3ab7a1fb3f0584ef6d742375 \
    --hash=sha256:1e74c6b51a9ed6589199c787bf5f9875612ca4a8a0785fb2d4a84429badaf22a \
    --hash=sha256:2012c72d854c2d03e45d06ae57f40d78e5770d252f195b93f581acf3ba44496e \
    --hash=sha256:21157295583fe8943475029ed5abdcf71eb3911894724e360acff1d61c1d54bc \
    --hash=sha256:2470043b93ff09bf8fb1d46d1cb756ce6132c54826661a32d4e4d132e1977adf \
    --hash=sha256:285d29981935eb726a4399badae8f0ffdff4f5050eaa6d0cfc3f64b857b77185 \
    --hash=sha256:30d78fbc8ebf9c92c9b7823ee18eb92f2e6ef79b45ac84db507f52fbe3ec4497 \
    --hash=sha256:320dab6e7cb2eacdf0e658569d2575c4dad258c0fcc794f46215e1e39f90f2c3 \
    --hash=sha256:33ab79603146aace82c2427da5ca6e58f2b3f2fb5da893ceac0c42218a40be35 \
    --hash=sha256:3548db281cd7d2561c9ad9984681c95f7b0e38881201e157833a2342c30d5e8c \
    --hash=sha256:3799aecf2e17cf585d977b780ce79ff0dc9b78d799fc694221ce814c2c19db83 \
    --hash=sha256:39d39875251ca8f612b6f33e6b1195af86d1b3e60086068be9cc053aa4376e21 \
    --hash=sha256:3b926aa83d1edb5aa5b427b4053dc420ec295a08e40911296b9eb1b6170f6cca \
    --hash=sha256:3bcde07039e586f91b45c88f8583ea7cf7a0770df3a1649627bf598332cb6984 \
    --hash=sha256:3d08afd128ddaa624a48cf2b859afef385b720bb4b43df214f85616922e6a5ac \
    --hash=sha256:3eb6971dcff08619f8d91607cfc726518b6fa2a9eba42856be181c6d0d9515fd \
    --hash=sha256:40f4774f5a9d4f5e344f31a32b5096977b5d48560c5592e2f3d2c4374bd543ee \
    --hash=sha256:4289fc34b2f5316fbb762d75362931e351941fa95fa18789191b33fc4cf9504a \
    --hash=sha256:470c103ae716238bbe698d67ad020e1db9d9dba34fa5a899b5e21577e6d52ed2 \
    --hash=sha256:4f2c9f67e9821cad2e5f480bc8d83b8742896f1242dba247911072d4fa94c192 \
    --hash=sha256:50a74364d85fd319352182ef59c5c790484a336f6db772c1a9231f1c3ed0cbd7 \
    --hash=sha256:54a2db7b78338edd780e7ef7f9f6c442500fb0d41a5a4ea24fff1c929d5af585 \
    --hash=sha256:5635bd9cb9731e6d4a1132a498dd34f764034a8ce60cef4f5319c0541159392f \
    --hash=sha256:59c0b02d0a6c384d453fece7566d1c7e6b7bae4fc5874ef2ef46d56776d61c9e \
    --hash=sha256:5d598b938678ebf3c67377cdd45e09d431369c3b1a5b331058c338e201f12b27 \
    --hash=sha256:5df2768244d19ab7f60546d0c7c63ce1581f7af8b5de3eb3004b9b6fc8a9f84b \
    --hash=sha256:5ef34d190326c3b1f822a5b7a45f6c4535e2f47ed06fec77d3d799c450b2651e \
    --hash=sha256:6975a3fac6bc83c4a65c9f9fcab9e47019a11d3d2cf7f3c0d03431bf145a941e \
    --hash=sha256:6c9a799e985904922a4d207a94eae35c78ebae90e128f0c4e521ce339396be9d \
    --hash=sha256:70df4e3b545a17496c9b3f41f5115e69a4f2e77e94e1d2a8e1070bc0c38c8a3c \
    --hash=sha256:7473e861101c9e72452f9bf8acb984947aa1661a7704553a9f6e4baa5ba64415 \
    --hash=sha256:8102eaf27e1e448db915d08afa8b41d6c7ca7a04b7d73af6514df10a3e74bd82 \
    --hash=sha256:87c450779d0914f2861b8526e035c5e6da0a3199d8f1add1a665e1cbc6fc6d02 \
    --hash=sha256:8b7ee99e510d7b66cdb6c593f21c043c248537a32e0bedf02e01e9553a172314 \
    --hash=sha256:91fc98adde3d7881af9b59ed0294046f3806221863722ba7d8d120c575314325 \
    --hash=sha256:94411f22c3985acaec6f83c6df553f2dbe17b698cc7f8ae751ff2237d96b9e3c \
    --hash=sha256:98d85c6a2bef81588d9227dde12db8a7f47f639f4a17c9ae08e773aa9c697bf3 \
    --hash=sha256:9ad5db27f9cabae298d151c85cf2bad1d359a1b9c686a275df03385758e2f914 \
    --hash=sha256:a0b71b1b8fbf2b96e41c4d990244165e2c9be83d54962a9a1d118fd8657d2045 \
    --hash=sha256:a0f100c8912c114ff53e1202d0078b425bee3649ae34d7b070e9697f93c5d52d \
    --hash=sha256:a591fe9e525846e4d154205572a029f653ada1a78b93697f3b5a8f1f2bc055b9 \
    --hash=sha256:a5c84c68147988265e60416b57fc83425a78058853509c1b0629c180094904a5 \
    --hash=sha256:a66d3508133af6e8548451b25058d5812812ec3798c886bf38ed24a98216fab2 \
    --hash=sha256:a8c4917bd7ad33e8eb21e9a5bbba979b49d9a97acb3a803092cbc1133e20343c \
    --hash=sha256:b3bbeb01c2b273cca1e1e0c5df57f12dce9a4dd331b4fa1635b8bec26350bde3 \
    --hash=sha256:cba9d6b9a7d64d4bd46167096fc9d2f835e25d7e4c121fb2ddfc6528fb0413b2 \
    --hash=sha256:cc4d65aeeaa04136a12677d3dd0b1c0c94dc43abac5860ab33cceb42b801c1e8 \
    --hash=sha256:ce4bcc037df4fc5e3d184794f27bdaab018943698f4ca31630bc7f84a7b69c6d \
    --hash=sha256:cec7d9412a9102bdc577382c3929b337320c4c4c4849f2c5cdd14d7368c5562d \
    --hash=sha256:d400bfb9a37b1351253cb402671cea7e89bdecc294e8016a707f6d1d8ac934f9 \
    --hash=sha256:d61f4695e6c866a23a21acab0509af1cdfd2c013cf256bbf5b6b5e2695827162 \
    --hash=sha256:db0fbb9c62743ce59a9ff687eb5f4afbe77e5e8403d6697f7446e5f609976f76 \
    --hash=sha256:dd86c085fae2efd48ac91dd7ccffcfc0571387fe1193d33b6394db7ef31fe2a4 \
    --hash=sha256:e00b098126fd45523dd056d2efba6c5a63b71ffe9f2bbe1a4fe1716e1d0c331e \
    --hash=sha256:e229a521186c75c8ad9490854fd8bbdd9a0c9aa3a524326b55be83b54d4e0ad9 \
    --hash=sha256:e263d77ee3dd201c3a142934a086a4450861778baaeeb45db4591ef65550b0a6 \
    --hash=sha256:ed9cb427ba5504c1dc15ede7d516b84757c3e3d7868ccc85121d9310d27eed0b \
    --hash=sha256:fa6693661a4c91757f4412306191b6dc88c1703f780c8234035eac011922bc01 \
    --hash=sha256:fcd131dd944808b5bdb38e6f5b53013c5aa4f334c5cad0c72742f6eba4b73db0
    # via cryptography
cryptography==3.4.8 \
    --hash=sha256:0a7dcbcd3f1913f664aca35d47c1331fce738d44ec34b7be8b9d332151b0b01e \
    --hash=sha256:1eb7bb0df6f6f583dd8e054689def236255161ebbcf62b226454ab9ec663746b \
    --hash=sha256:21ca464b3a4b8d8e86ba0ee5045e103a1fcfac3b39319727bc0fc58c09c6aff7 \
    --hash=sha256:34dae04a0dce5730d8eb7894eab617d8a70d0c97da76b905de9efb7128ad7085 \
    --hash=sha256:3520667fda779eb788ea00080124875be18f2d8f0848ec00733c0ec3bb8219fc \
    --hash=sha256:3c4129fc3fdc0fa8e40861b5ac0c673315b3c902bbdc05fc176764815b43dd1d \
    --hash=sha256:3fa3a7ccf96e826affdf1a0a9432be74dc73423125c8f96a909e3835a5ef194a \
    --hash=sha256:5b0fbfae7ff7febdb74b574055c7466da334a5371f253732d7e2e7525d570498 \
    --hash=sha256:695104a9223a7239d155d7627ad912953b540929ef97ae0c34c7b8bf30857e89 \
    --hash=sha256:8695456444f277af73a4877db9fc979849cd3ee74c198d04fc0776ebc3db52b9 \
    --hash=sha256:94cc5ed4ceaefcbe5bf38c8fba6a21fc1d365bb8fb826ea1688e3370b2e24a1c \
    --hash=sha256:94fff993ee9bc1b2440d3b7243d488c6a3d9724cc2b09cdb297f6a886d040ef7 \
    --hash=sha256:9965c46c674ba8cc572bc09a03f4c649292ee73e1b683adb1ce81e82e9a6a0fb \
    --hash=sha256:a00cf305f07b26c351d8d4e1af84ad7501eca8a342dedf24a7acb0e7b7406e14 \
    --hash=sha256:a305600e7a6b7b855cd798e00278161b681ad6e9b7eca94c721d5f588ab212af \
    --hash=sha256:cd65b60cfe004790c795cc35f272e41a3df4631e2fb6b35aa7ac6ef2859d554e \
    --hash=sha256:d2a6e5ef66503da51d2110edf6c403dc6b494cc0082f85db12f54e9c5d4c3ec5 \
    --hash=sha256:d9ec0e67a14f9d1d48dd87a2531009a9b251c02ea42851c060b25c782516ff06 \
    --hash=sha256:f44d141b8c4ea5eb4dbc9b3ad992d45580c1d22bf5e24363f2fbf50c2d7ae8a7
    # via pyjwt
django==3.1.14 \
    --hash=sha256:0fabc786489af16ad87a8c170ba9d42bfd23f7b699bd5ef05675864e8d012859 \
    --hash=sha256:72a4a5a136a214c39cf016ccdd6b69e2aa08c7479c66d93f3a9b5e4bb9d8a347
    # via
    #   -r main.in
    #   djangorestframework
    #   drf-jwt
djangorestframework==3.12.4 \
    --hash=sha256:6d1d59f623a5ad0509fe0d6bfe93cbdfe17b8116ebc8eda86d45f6e16e819aaf \
    --hash=sha256:f747949a8ddac876e879190df194b925c177cdeb725a099db1460872f7c0a7f2
    # via
    #   -r main.in
    #   drf-jwt
drf-jwt==1.19.2 \
    --hash=sha256:63c3d4ed61a1013958cd63416e2d5c84467d8ae3e6e1be44b1fb58743dbd1582 \
    --hash=sha256:660bc66f992065cef59832adcbbdf871847e9738671c19e5121971e773768235
    # via -r main.in
pycparser==2.21 \
    --hash=sha256:8ee45429555515e1f6b185e78100aea234072576aa43ab53aefcae078162fca9 \
    --hash=sha256:e644fdec12f7872f86c58ff790da456218b10f863970249516d60a5eaca77206
    # via cffi
pyjwt==2.1.0 \
    --hash=sha256:934d73fbba91b0483d3857d1aff50e96b2a892384ee2c17417ed3203f173fca1 \
    --hash=sha256:fba44e7898bbca160a2b2b501f492824fc8382485d3a6f11ba5d0c1937ce6130
    # via
    #   -r main.in
    #   drf-jwt
pytz==2022.6 \
    --hash=sha256:222439474e9c98fced559f1709d89e6c9cbf8d79c794ff3eb9f8800064291427 \
    --hash=sha256:e89512406b793ca39f5971bc999cc538ce125c0e51c27941bef4568b460095e2
    # via django
sqlparse==0.4.3 \
    --hash=sha256:0323c0ec29cd52bceabc1b4d9d579e311f3e4961b98d174201d5622a23b85e34 \
    --hash=sha256:69ca804846bb114d2ec380e4360a8a340db83f0ccf3afceeb1404df028f57268
    # via django

$ python3 -m piptools compile --generate-hashes --strip-extras --resolver=backtracking dev.in
#
# This file is autogenerated by pip-compile with python 3.8
# To update, run:
#
#    pip-compile --generate-hashes --resolver=backtracking --strip-extras dev.in
#
black==22.10.0 \
    --hash=sha256:14ff67aec0a47c424bc99b71005202045dc09270da44a27848d534600ac64fc7 \
    --hash=sha256:197df8509263b0b8614e1df1756b1dd41be6738eed2ba9e9769f3880c2b9d7b6 \
    --hash=sha256:1e464456d24e23d11fced2bc8c47ef66d471f845c7b7a42f3bd77bf3d1789650 \
    --hash=sha256:2039230db3c6c639bd84efe3292ec7b06e9214a2992cd9beb293d639c6402edb \
    --hash=sha256:21199526696b8f09c3997e2b4db8d0b108d801a348414264d2eb8eb2532e540d \
    --hash=sha256:2644b5d63633702bc2c5f3754b1b475378fbbfb481f62319388235d0cd104c2d \
    --hash=sha256:432247333090c8c5366e69627ccb363bc58514ae3e63f7fc75c54b1ea80fa7de \
    --hash=sha256:444ebfb4e441254e87bad00c661fe32df9969b2bf224373a448d8aca2132b395 \
    --hash=sha256:5b9b29da4f564ba8787c119f37d174f2b69cdfdf9015b7d8c5c16121ddc054ae \
    --hash=sha256:5cc42ca67989e9c3cf859e84c2bf014f6633db63d1cbdf8fdb666dcd9e77e3fa \
    --hash=sha256:5d8f74030e67087b219b032aa33a919fae8806d49c867846bfacde57f43972ef \
    --hash=sha256:72ef3925f30e12a184889aac03d77d031056860ccae8a1e519f6cbb742736383 \
    --hash=sha256:819dc789f4498ecc91438a7de64427c73b45035e2e3680c92e18795a839ebb66 \
    --hash=sha256:915ace4ff03fdfff953962fa672d44be269deb2eaf88499a0f8805221bc68c87 \
    --hash=sha256:9311e99228ae10023300ecac05be5a296f60d2fd10fff31cf5c1fa4ca4b1988d \
    --hash=sha256:974308c58d057a651d182208a484ce80a26dac0caef2895836a92dd6ebd725e0 \
    --hash=sha256:b8b49776299fece66bffaafe357d929ca9451450f5466e997a7285ab0fe28e3b \
    --hash=sha256:c957b2b4ea88587b46cf49d1dc17681c1e672864fd7af32fc1e9664d572b3458 \
    --hash=sha256:e41a86c6c650bcecc6633ee3180d80a025db041a8e2398dcc059b3afa8382cd4 \
    --hash=sha256:f513588da599943e0cde4e32cc9879e825d58720d6557062d1098c5ad80080e1 \
    --hash=sha256:fba8a281e570adafb79f7755ac8721b6cf1bbf691186a287e990c7929c7692ff
    # via -r dev.in
click==8.1.3 \
    --hash=sha256:7682dc8afb30297001674575ea00d1814d808d6a36af415a82bd481d37ba7b8e \
    --hash=sha256:bb4d8133cb15a609f44e8213d9b391b0809795062913b383c62be0ee95b1db48
    # via black
mypy-extensions==0.4.3 \
    --hash=sha256:090fedd75945a69ae91ce1303b5824f428daf5a028d2f6ab8a299250a846f15d \
    --hash=sha256:2d82818f5bb3e369420cb3c4060a7970edba416647068eb4c5343488a6c604a8
    # via black
pathspec==0.10.2 \
    --hash=sha256:88c2606f2c1e818b978540f73ecc908e13999c6c3a383daf3705652ae79807a5 \
    --hash=sha256:8f6bf73e5758fd365ef5d58ce09ac7c27d2833a8d7da51712eac6e27e35141b0
    # via black
platformdirs==2.5.4 \
    --hash=sha256:1006647646d80f16130f052404c6b901e80ee4ed6bef6792e1f238a8969106f7 \
    --hash=sha256:af0276409f9a02373d540bf8480021a048711d572745aef4b7842dad245eba10
    # via black
tomli==2.0.1 \
    --hash=sha256:939de3e7a6161af0c887ef91b7d41a53e7c5a1ca976325f429cb46ea9bc30ecc \
    --hash=sha256:de526c12914f0c550d15924c62d72abc48d6fe7364aa87328337a31007fe8a4f
    # via black
typing-extensions==4.4.0 \
    --hash=sha256:1511434bb92bf8dd198c12b1cc812e800d4181cfcb867674e0f8279cc93087aa \
    --hash=sha256:16fa4864408f655d35ec496218b85f79b3437c829e93320c7c9215ccfd92489e
    # via black

Expected result

I'd like to be able to use generated requirements files to install the packages.

Actual result

I can't install the packages, neither with pip-sync or with pip install -r:

$ python -m piptools sync main.txt dev.txt
Collecting asgiref==3.5.2
  Using cached asgiref-3.5.2-py3-none-any.whl (22 kB)
Collecting black==22.10.0
  Using cached black-22.10.0-cp38-cp38-macosx_11_0_arm64.whl (1.2 MB)
Collecting cffi==1.15.1
  Using cached cffi-1.15.1.tar.gz (508 kB)
  Preparing metadata (setup.py) ... done
Collecting cryptography==3.4.8
  Using cached cryptography-3.4.8-cp36-abi3-macosx_11_0_arm64.whl (1.9 MB)
Collecting django==3.1.14
  Using cached Django-3.1.14-py3-none-any.whl (7.8 MB)
Collecting djangorestframework==3.12.4
  Using cached djangorestframework-3.12.4-py3-none-any.whl (957 kB)
Collecting drf-jwt==1.19.2
  Using cached drf_jwt-1.19.2-py2.py3-none-any.whl (21 kB)
Collecting mypy-extensions==0.4.3
  Using cached mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)
Collecting pathspec==0.10.2
  Using cached pathspec-0.10.2-py3-none-any.whl (28 kB)
Collecting platformdirs==2.5.4
  Using cached platformdirs-2.5.4-py3-none-any.whl (14 kB)
Collecting pycparser==2.21
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting pyjwt==2.1.0
  Using cached PyJWT-2.1.0-py3-none-any.whl (16 kB)
Collecting pytz==2022.6
  Using cached pytz-2022.6-py2.py3-none-any.whl (498 kB)
Collecting sqlparse==0.4.3
  Using cached sqlparse-0.4.3-py3-none-any.whl (42 kB)
Collecting typing-extensions==4.4.0
  Using cached typing_extensions-4.4.0-py3-none-any.whl (26 kB)
Requirement already satisfied: tomli>=1.1.0 in /Users/pawelad/.pyenv/versions/3.8.13/envs/tmp/lib/python3.8/site-packages (from black==22.10.0->-r /var/folders/zd/bh5ny2dj5hv8x_5vf92tnfz40000gn/T/tmp5g4e645s (line 4)) (2.0.1)
Requirement already satisfied: click>=8.0.0 in /Users/pawelad/.pyenv/versions/3.8.13/envs/tmp/lib/python3.8/site-packages (from black==22.10.0->-r /var/folders/zd/bh5ny2dj5hv8x_5vf92tnfz40000gn/T/tmp5g4e645s (line 4)) (8.1.3)
Collecting PyJWT[crypto]<3.0.0,>=1.5.2
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    PyJWT[crypto]<3.0.0,>=1.5.2 from https://files.pythonhosted.org/packages/40/46/505f0dd53c14096f01922bf93a7abb4e40e29a06f858abbaa791e6954324/PyJWT-2.6.0-py3-none-any.whl (from drf-jwt==1.19.2->-r /var/folders/zd/bh5ny2dj5hv8x_5vf92tnfz40000gn/T/tmp5g4e645s (line 117))
Traceback (most recent call last):
  File "/Users/pawelad/.pyenv/versions/3.8.13/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/pawelad/.pyenv/versions/3.8.13/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/pawelad/.pyenv/versions/tmp/lib/python3.8/site-packages/piptools/__main__.py", line 19, in <module>
    cli()
  File "/Users/pawelad/.pyenv/versions/tmp/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/pawelad/.pyenv/versions/tmp/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/pawelad/.pyenv/versions/tmp/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/pawelad/.pyenv/versions/tmp/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/pawelad/.pyenv/versions/tmp/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/pawelad/.pyenv/versions/tmp/lib/python3.8/site-packages/piptools/scripts/sync.py", line 177, in cli
    sync.sync(
  File "/Users/pawelad/.pyenv/versions/tmp/lib/python3.8/site-packages/piptools/sync.py", line 240, in sync
    run(  # nosec
  File "/Users/pawelad/.pyenv/versions/3.8.13/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/pawelad/.pyenv/versions/tmp/bin/python', '-m', 'pip', 'install', '-r', '/var/folders/zd/bh5ny2dj5hv8x_5vf92tnfz40000gn/T/tmp5g4e645s']' returned non-zero exit status 1.

$ pip install -r main.txt
Collecting asgiref==3.5.2
  Using cached asgiref-3.5.2-py3-none-any.whl (22 kB)
Collecting cffi==1.15.1
  Using cached cffi-1.15.1.tar.gz (508 kB)
  Preparing metadata (setup.py) ... done
Collecting cryptography==3.4.8
  Using cached cryptography-3.4.8-cp36-abi3-macosx_11_0_arm64.whl (1.9 MB)
Collecting django==3.1.14
  Using cached Django-3.1.14-py3-none-any.whl (7.8 MB)
Collecting djangorestframework==3.12.4
  Using cached djangorestframework-3.12.4-py3-none-any.whl (957 kB)
Collecting drf-jwt==1.19.2
  Using cached drf_jwt-1.19.2-py2.py3-none-any.whl (21 kB)
Collecting pycparser==2.21
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting pyjwt==2.1.0
  Using cached PyJWT-2.1.0-py3-none-any.whl (16 kB)
Collecting pytz==2022.6
  Using cached pytz-2022.6-py2.py3-none-any.whl (498 kB)
Collecting sqlparse==0.4.3
  Using cached sqlparse-0.4.3-py3-none-any.whl (42 kB)
Collecting PyJWT[crypto]<3.0.0,>=1.5.2
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    PyJWT[crypto]<3.0.0,>=1.5.2 from https://files.pythonhosted.org/packages/40/46/505f0dd53c14096f01922bf93a7abb4e40e29a06f858abbaa791e6954324/PyJWT-2.6.0-py3-none-any.whl (from drf-jwt==1.19.2->-r main.txt (line 111))

[pawelad]

Ah, I just found the upstream issue in pip - https://github.com/pypa/pip/issues/9644

I guess there's nothing we can do on pip-tools side? I wonder what should be the workaround? I went back to using the old resolver (and dropped --skip-extras), but if I understand it correctly, this will become a breaking bug when pip-tools 7 is released and the new resolver will become the default?

[AndydeCleyre]

In your reproductions it looks like needed dependencies are not included in the compiled txts: backports.zoneinfo is absent from main.txt, though apparently required by django, and typing-extensions is absent from dev.txt, though apparently required by black.

This may be a bug, or it may be a case of mismatched environments between compilation and installation. Looking at black's pyproject.toml, I see that typing_extensions is required only when Python is < 3.10. Similarly for django, backports.zoneinfo is only required when Python is < 3.9.

So probably you are running pip-compile in an environment with Python > 3.10, but pip install in one with Python < 3.9. This is not currently expected to succeed, when the requirements differ across those environments.

See also:

• 1326

• 826

[pawelad]

You're 100% right about mixing Python 3.8 and 3.10 (my project Python version vs. the temporary venv I created to reproduce), but I'm almost certain that the bigger problem still stands (as 'confirmed' by https://github.com/pypa/pip/issues/9644).

I'll update the description in the coming days.

[pawelad]

@AndydeCleyre I updated issue description with pins from my production environment and making sure I'm doing everything on Python 3.8.13, pip 22.3.1 and pip-tools 6.10.0. Let me know if you still can't reproduce it.

Like I said, I believe it's a known bug in pip's new resolver (https://github.com/pypa/pip/issues/9644) with known workarounds (https://github.com/pypa/pip/issues/9644#issuecomment-813432613) and lack of resources to fix (https://github.com/pypa/pip/issues/9644#issuecomment-1022002270)

I guess there's nothing to do on pip-tools side, but having it open here for visibility might have some worth, especially since this will become a much more pressing issue when the new resolver is gonna be enabled by default (I'd recommend postponing that until this bug is fixed).

FWIW, my current workaround is to stop using the new resolver until this gets fixed.

[snmishra]

My current workaround is to use --no-deps with pip install -r. Since pip-compile has specified all dependencies, --no-deps seems quite reasonable.

[AndydeCleyre]

Seen again as #1865 (closed as dupe of this one). The title here could probably use an update (note that --strip-extras and -c` are not necessary to reproduce).