Open jamesbraza opened 1 year ago
For open source users, likely this will be empty (no index URL). Regardless of the default being
--no-emit-index-url
or not, these users will not be affected, as just PyPI is used.
I'm not sure what kind of end user group you mean exactly with open source users?
EDIT:
Understood. Thanks for reporting, sounds like a reasonable request!
Yeah sorry for being unclear there, I just meant open source users likely don't use package mangers beside PyPI, so they don't have an extra index URL.
Here is a slightly more backwards compatible idea:
pip-compile
continues with the current default of emitting index URLspip-compile
to detect if a secret is present inside an extra index URL--allow-secrets-in-index-url
is passed, pip-compile
will throw a nonzero exit codeThat way, users who don't have secrets in the index URL can still rely on index URL being emitted by default.
That way, users who don't have secrets in the index URL can still rely on index URL being emitted by default.
I think I like the former suggestion better. Users may wonder why some index URLs are emitted, but others not.
- Unless a flag
--allow-secrets-in-index-url
is passed,pip-compile
will throw a nonzero exit code
Also, this is a bit long for a command line flag.
Also I would add that putting the index-url in the requirements is unrecognized when referring to the requirements in project.toml.
Traceback (most recent call last):
File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/_vendor/packaging/requirements.py", line 35, in __init__
parsed = _parse_requirement(requirement_string)
File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/_vendor/packaging/_parser.py", line 64, in parse_requirement
return _parse_requirement(Tokenizer(source, rules=DEFAULT_RULES))
File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/_vendor/packaging/_parser.py", line 73, in _parse_requirement
name_token = tokenizer.expect(
File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/_vendor/packaging/_tokenizer.py", line 140, in expect
raise self.raise_syntax_error(f"Expected {expected}")
File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/_vendor/packaging/_tokenizer.py", line 165, in raise_syntax_error
raise ParserSyntaxError(
setuptools.extern.packaging._tokenizer.ParserSyntaxError: Expected package name at the start of dependency specifier
--index-url https://repo-url-here
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/jonathan/.pyenv/versions/3.8.16/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
main()
File "/Users/jonathan/.pyenv/versions/3.8.16/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/Users/jonathan/.pyenv/versions/3.8.16/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=['wheel'])
File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires
self.run_setup()
File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 341, in run_setup
exec(code, locals())
File "<string>", line 1, in <module>
File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 103, in setup
return distutils.core.setup(**attrs)
File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 159, in setup
dist.parse_config_files()
File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 653, in parse_config_files
pyprojecttoml.apply_configuration(self, filename, ignore_option_errors)
File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/config/pyprojecttoml.py", line 67, in apply_configuration
return _apply(dist, config, filepath)
File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/config/_apply_pyprojecttoml.py", line 60, in apply
dist._finalize_requires()
File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 390, in _finalize_requires
self._normalize_requires()
File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 405, in _normalize_requires
self.install_requires = list(map(str, _reqs.parse(install_requires)))
File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/_vendor/packaging/requirements.py", line 37, in __init__
raise InvalidRequirement(str(e)) from e
setuptools.extern.packaging.requirements.InvalidRequirement: Expected package name at the start of dependency specifier
--index-url https://repo-url-here
^
FTR this request would pair nicely with https://github.com/jazzband/pip-tools/issues/2051.
Also I would add that putting the index-url in the requirements is unrecognized when referring to the requirements in project.toml.
Requirements files are pip-specific and contain PEP 508 requirement specifiers and pip CLI options. pyproject.toml
deps have an interoperability standard that is targeting many tools across the ecosystem, not just pip so it's expected that pip's args wouldn't be compatible with that.
Motivation
Currently,
pip-compile
by default will emit any index URLs used.For open source users, likely this will be empty (no index URL). Regardless of the default being
--no-emit-index-url
or not, these users will not be affected, as just PyPI is used.For non-open source users, this may not be empty. For these users, forgetting to include
--no-emit-index-url
represents a security risk, because if this opt-in arg is forgotten, a company secret will be leaked.This actually happened to me tonight when I accidentally leaked my company's internal package manager's index URL. Luckily, I realized it within 15 minutes, and refreshed my token. Now I am getting an email from GitGuardian saying their service detected a string secret was leaked.
For all of my company's
pip-compile
use cases, we append--no-emit-index-url
. Thus for engineers at my company, a useful default would be--no-emit-index-url
.Request
I think the default of
pip-compile
should be secure (and not bias towards leaking secrets).This entails:
--no-emit-index-url
the new default--emit-index-url
or--enable-emit-index-url
as an opt-in flag