Ignore hashes for version control repositories

a666 commented 4 years ago

What's the problem this feature will solve?

Right now we are having the following situation.

Having a requirements.in like:

# requirements.in
django
git+ssh://git@private.server/private_app.git@0.3#egg=private_app

and doing:

$ pip-compile --generate-hashes -o requirements.txt requirements.in && pip-sync requirements.txt

gives us the error:

ERROR: Can't verify hashes for these requirements because we don't have a way to hash version control repositories:
    private_app from git+ssh://****@private.server/private_app.git@0.3#egg=private_app 
    (from -r /tmp/tmpii51x619 (line 217))

Replacing our repo with -e git+ssh://git@private.server/private_app.git@0.3#egg=private_app changes the error to

ERROR: The editable requirement posgrado_catalogs from 
git+ssh://****@private.server/private_app.git@0.3#egg=private_app
(from -r /tmp/tmp317kk_qv (line 172)) cannot be installed when requiring hashes, 
because there is no single file to hash.

Describe the solution you'd like

It would be ideal for version control repositories to skip hashing (since access and integrity is handled differently) while keeping it for every other package.

Alternative Solutions

pipenv (ugh) seems to do it.

atugushev commented 4 years ago

Hello @a666,

Thanks for the issue! I believe this should be implemented on pip side. See related issues:

However, there are possible solutions:

using a hashable URL like https://github.com/jazzband/pip-tools/archive/SOMECOMMIT.zip.
split requirements files for hashable and non-hashable packages.

a666 commented 4 years ago

Thank you @atugushev for the quick answer.

I forgot to precise that we are using your first solution for the time being.

cjerdonek commented 3 years ago

Hi @atugushev, what did you mean by this?

split requirements files for hashable and non-hashable packages.

Did you mean split the requirements.in file? Or did you mean split the requirements.txt file output by pip-compile (e.g. into hashable and non-hashable requirements)? (The latter could presumably be done by parsing the output, though it's not so elegant.)

By the way, it seems like if #333 were implemented, that would be another possible solution. If one knows certain VCS requirements are included in a requirements.in file, then the names of those packages could also be passed in via #333's --exclude option to exclude those lines from the output.

atugushev commented 3 years ago

Did you mean split the requirements.in file?

Hello @cjerdonek! That's exactly what I mean. Yeah, --exclude seems like a good solution.

cjerdonek commented 3 years ago

Thanks, @atugushev! However, won't splitting the requirements.in file into VCS and non-VCS requirements mean that the dependencies of the VCS requirements (which can be hashable) won't get the benefit of the hashes, because those dependencies would be part of the requirements.txt file corresponding to the VCS requirements.in? Or am I missing part of your suggestion?

atugushev commented 3 years ago

@cjerdonek, yes, you are right.

cjerdonek commented 3 years ago

Okay, thanks a lot for confirming.

sergeyklay commented 3 years ago

Hello,

Not sure if this is the right place, but I'd like to show you my case so that you have more information about the issues that arise. Most of the code has been omitted to show only the gist:

tox.ini

[tox]
minversion = 3.22
envlist = py{37,38,39,310}

[testenv]
extras = testing
deps =
    -rrequirements.txt
commands =
    coverage erase
    coverage run -m pytest {posargs}

requirements.in

django

# ...

Command to compile requirements.txt file

pip-compile --generate-hashes --output-file=requirements.txt requirements.in

requirements.txt

# line line 11
asgiref==3.3.4 \
    --hash=sha256:92906c611ce6c967347bbfea733f13d6313901d54dcca88195eaeb52b2a8e8ee \
    --hash=sha256:d1216dfbdfb63826470995d31caed36225dcaf34f182e0fa257a4dd9e86f1b78
    # via django

# ...

Relevant part of setup.py

EXTRAS_REQUIRE = {
    'testing': [
        'pytest>=6.2.0',
        'pytest-cov>=2.11.1',
        'pytest-django>=4.2.0', 
        'factory-boy>=3.2.0',
        'faker>=8.1.0',
    ],
}

# ...

if __name__ == '__main__':
    setup(
        # ...

        extras_require=EXTRAS_REQUIRE,
    )

Command to install deps and run tests

tox

Output from GitHub Actions

Collecting typing-extensions ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not: typing-extensions from https://files.pythonhosted.org/packages/2e/35/6c4fff5ab443b57116cb1aad46421fb719bed2825664e8fe77d66d99bcbc/typing_extensions-3.10.0.0-py3-none-any.whl#sha256=779383f6086d90c99ae41cf0ff39aac8a7937a9283ce0a414e5dd782f4c94a84 (from asgiref==3.3.4->-r requirements.txt (line 11))

=================================== log end ====================================

ERROR: could not install deps [-rrequirements.txt]; v = InvocationError('/home/runner/work/branch/branch/.tox/py37/bin/python -m pip install -rrequirements.txt', 1) ___ summary ____

ERROR: py37: could not install deps [-rrequirements.txt]; v = InvocationError('/home/runner/work/branch/branch/.tox/py37/bin/python -m pip install -rrequirements.txt', 1)

This issue occurs only for Python 3.7. Possible due this part of setup.cfg from asgiref repo:

[options]
python_requires = >=3.6
packages = find:
include_package_data = true
install_requires =
    typing_extensions; python_version < "3.8"
zip_safe = false

Possible related issues:

https://github.com/cjolowicz/cookiecutter-hypermodern-python/issues/583

Let me know if I can provide more information.

nstylo commented 2 years ago

I'd also like to see this feature. Currently we have some local dependencies checked into VCS (with git lfs) and I'd like to ignore those dependencies when running pip-compile --generate-hashes

GabrielDumbrava commented 2 years ago

Thank you @atugushev for the quick answer.

I forgot to precise that we are using your first solution for the time being.

Hi! Can you please give me an example of your requirements.in file that works for you? I can't make it work. Here's mine:

requirements.in

pillow
-e git+https://github.com/hvlads/django-ckeditor-5/archive/073b40bbf506d084f9b20788e6a17187001b65ae.zip#egg=django-ckeditor-5

I get an error when pip-compile is trying to clone:

$pip-compile --generate-hashes requirements.in

ERROR:pip.subprocessor:[present-diagnostic] git clone --filter=blob:none --quiet https://github.com/hvlads/django-ckeditor-5/archive/073b40bbf506d084f9b20788e6a17187001b65ae.zip /[SOME_PATH]/src/django-ckeditor-5 exited with 128
Traceback (most recent call last):

If I ran the clone myself:

$git clone --filter=blob:none --quiet https://github.com/hvlads/django-ckeditor-5/archive/073b40bbf506d084f9b20788e6a17187001b65ae.zip

remote: Not Found
fatal: repository 'https://github.com/hvlads/django-ckeditor-5/archive/073b40bbf506d084f9b20788e6a17187001b65ae.zip/' not found

Getting the archive with wget works:

$ wget https://github.com/hvlads/django-ckeditor-5/archive/073b40bbf506d084f9b20788e6a17187001b65ae.zip
--2022-06-06 10:55:09--  https://github.com/hvlads/django-ckeditor-5/archive/073b40bbf506d084f9b20788e6a17187001b65ae.zip
Resolving github.com (github.com)... 140.82.121.4
Connecting to github.com (github.com)|140.82.121.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/hvlads/django-ckeditor-5/zip/073b40bbf506d084f9b20788e6a17187001b65ae [following]
--2022-06-06 10:55:10--  https://codeload.github.com/hvlads/django-ckeditor-5/zip/073b40bbf506d084f9b20788e6a17187001b65ae
Resolving codeload.github.com (codeload.github.com)... 140.82.121.9
Connecting to codeload.github.com (codeload.github.com)|140.82.121.9|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/zip]
Saving to: ‘073b40bbf506d084f9b20788e6a17187001b65ae.zip’

073b40bbf506d084f9b20788e6a17187001b65ae.zip                                      [  <=>                                                                                                                                                                                             ]   1,63M  2,46MB/s    in 0,7s    

2022-06-06 10:55:11 (2,46 MB/s) - ‘073b40bbf506d084f9b20788e6a17187001b65ae.zip’ saved [1708665]

If I replace the url with the redirected one https://codeload.github.com/hvlads/django-ckeditor-5/zip/073b40bbf506d084f9b20788e6a17187001b65ae I get the same result.

AndydeCleyre commented 2 years ago

@GabrielDumbrava

This link isn't actually a git protocol thing, and not really editable, so try this change:

- -e git+https://github.com/hvlads/djan...
+ https://github.com/hvlads/djan...

jedie commented 1 year ago

I have a similar error like above https://github.com/jazzband/pip-tools/issues/994#issuecomment-855204293

ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not: typing-extensions from https://files.pythonhosted.org/packages/2e/35/6c4fff5ab443b57116cb1aad46421fb719bed2825664e8fe77d66d99bcbc/typing_extensions-3.10.0.0-py3-none-any.whl#sha256=779383f6086d90c99ae41cf0ff39aac8a7937a9283ce0a414e5dd782f4c94a84 (from asgiref==3.3.4->-r requirements.txt (line 11))

In my case, it's also GitHub Actions and happens only with Python 3.9 (with 3.10 and 3.11 everything is fine). Looks like:

Collecting typing-extensions>=3.10.0.0
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    typing-extensions>=3.10.0.0 from https://files.pythonhosted.org/packages/0b/8e/f1a0a5a76cfef77e1eb6004cb49e5f8d72634da638420b9ea492ce8305e8/typing_extensions-4.4.0-py3-none-any.whl (from black==22.10.0->-r /tmp/tmphawi3o6_ (line 7))

Complete CI run: https://github.com/jedie/cookiecutter_templates/actions/runs/3509450818/jobs/5878554915

requirements.in file is: https://github.com/jedie/cookiecutter_templates/blob/3610e920badf4053112cd1bef28cf59f00e7010f/managetemplates/requirements.in
requirements.txt file is: https://github.com/jedie/cookiecutter_templates/blob/3610e920badf4053112cd1bef28cf59f00e7010f/managetemplates/requirements.txt

black defines typing_extensions in pyproject.toml / dependencies with:

"typing_extensions>=3.10.0.0; python_version < '3.10'",

See: https://github.com/psf/black/blob/27932494bcefac03497dd92dcf0c59a04c10d757/pyproject.toml#L65-L73

atugushev commented 1 year ago

It seems like hash-checking mode for VCS URLs would be supported soon:

https://github.com/pypa/pip/pull/11968

After the PR is merged, some work would be required on the pip-tools side to hide the warning.

jazzband / pip-tools