Request: making `--no-emit-index-url` the default of `pip-compile`

[jamesbraza]

Motivation

Currently, pip-compile by default will emit any index URLs used.

For open source users, likely this will be empty (no index URL). Regardless of the default being --no-emit-index-url or not, these users will not be affected, as just PyPI is used.

For non-open source users, this may not be empty. For these users, forgetting to include --no-emit-index-url represents a security risk, because if this opt-in arg is forgotten, a company secret will be leaked.

This actually happened to me tonight when I accidentally leaked my company's internal package manager's index URL. Luckily, I realized it within 15 minutes, and refreshed my token. Now I am getting an email from GitGuardian saying their service detected a string secret was leaked.

For all of my company's pip-compile use cases, we append --no-emit-index-url. Thus for engineers at my company, a useful default would be --no-emit-index-url.

Request

I think the default of pip-compile should be secure (and not bias towards leaking secrets).

This entails:

1. Making --no-emit-index-url the new default
2. Adding --emit-index-url or --enable-emit-index-url as an opt-in flag

[chrysle]

For open source users, likely this will be empty (no index URL). Regardless of the default being --no-emit-index-url or not, these users will not be affected, as just PyPI is used.

I'm not sure what kind of end user group you mean exactly with open source users?

EDIT:

Understood. Thanks for reporting, sounds like a reasonable request!

[jamesbraza]

Yeah sorry for being unclear there, I just meant open source users likely don't use package mangers beside PyPI, so they don't have an extra index URL.

Here is a slightly more backwards compatible idea:

1. pip-compile continues with the current default of emitting index URLs
2. Enable pip-compile to detect if a secret is present inside an extra index URL
3. Unless a flag --allow-secrets-in-index-url is passed, pip-compile will throw a nonzero exit code

That way, users who don't have secrets in the index URL can still rely on index URL being emitted by default.

[chrysle]

That way, users who don't have secrets in the index URL can still rely on index URL being emitted by default.

I think I like the former suggestion better. Users may wonder why some index URLs are emitted, but others not.

3. Unless a flag --allow-secrets-in-index-url is passed, pip-compile will throw a nonzero exit code

Also, this is a bit long for a command line flag.

[jonathan-s]

Also I would add that putting the index-url in the requirements is unrecognized when referring to the requirements in project.toml.

      Traceback (most recent call last):
        File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/_vendor/packaging/requirements.py", line 35, in __init__
          parsed = _parse_requirement(requirement_string)
        File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/_vendor/packaging/_parser.py", line 64, in parse_requirement
          return _parse_requirement(Tokenizer(source, rules=DEFAULT_RULES))
        File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/_vendor/packaging/_parser.py", line 73, in _parse_requirement
          name_token = tokenizer.expect(
        File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/_vendor/packaging/_tokenizer.py", line 140, in expect
          raise self.raise_syntax_error(f"Expected {expected}")
        File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/_vendor/packaging/_tokenizer.py", line 165, in raise_syntax_error
          raise ParserSyntaxError(
      setuptools.extern.packaging._tokenizer.ParserSyntaxError: Expected package name at the start of dependency specifier
          --index-url https://repo-url-here
          ^

      The above exception was the direct cause of the following exception:

      Traceback (most recent call last):
        File "/Users/jonathan/.pyenv/versions/3.8.16/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Users/jonathan/.pyenv/versions/3.8.16/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/Users/jonathan/.pyenv/versions/3.8.16/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
        File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires
          self.run_setup()
        File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 341, in run_setup
          exec(code, locals())
        File "<string>", line 1, in <module>
        File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 103, in setup
          return distutils.core.setup(**attrs)
        File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 159, in setup
          dist.parse_config_files()
        File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 653, in parse_config_files
          pyprojecttoml.apply_configuration(self, filename, ignore_option_errors)
        File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/config/pyprojecttoml.py", line 67, in apply_configuration
          return _apply(dist, config, filepath)
        File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/config/_apply_pyprojecttoml.py", line 60, in apply
          dist._finalize_requires()
        File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 390, in _finalize_requires
          self._normalize_requires()
        File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 405, in _normalize_requires
          self.install_requires = list(map(str, _reqs.parse(install_requires)))
        File "/private/var/folders/tv/8px6tt4151d9nf8pdl8ydjsh0000gn/T/pip-build-env-r5ujckb4/overlay/lib/python3.8/site-packages/setuptools/_vendor/packaging/requirements.py", line 37, in __init__
          raise InvalidRequirement(str(e)) from e
      setuptools.extern.packaging.requirements.InvalidRequirement: Expected package name at the start of dependency specifier
          --index-url https://repo-url-here
          ^

[webknjaz]

FTR this request would pair nicely with https://github.com/jazzband/pip-tools/issues/2051.

[webknjaz]

Also I would add that putting the index-url in the requirements is unrecognized when referring to the requirements in project.toml.

Requirements files are pip-specific and contain PEP 508 requirement specifiers and pip CLI options. pyproject.toml deps have an interoperability standard that is targeting many tools across the ecosystem, not just pip so it's expected that pip's args wouldn't be compatible with that.