Accept extra parameters provided to pip inside requirement files

[mbacicc]

I have some django packages which depend on custom packages that we self host on our internal pypiserver. To make it work we add this first line to requirements.txt:

--extra-index-url http://xxx.yyy:8080/ --trusted-host xxx.yyy
<packages_list>

This line generates an exception in bandersnatch:

2021-09-22 09:52:33,508 INFO: Selected storage backend: filesystem (configuration.py:128)

2021-09-22 09:52:33,509 INFO: Selected compare method: hash (configuration.py:174)

2021-09-22 09:52:33,610 INFO: considering /requirements/xxx-requirements.txt (allowlist_name.py:111)

Traceback (most recent call last):

  File "/usr/local/lib/python3.9/site-packages/packaging/requirements.py", line 102, in __init__

    req = REQUIREMENT.parseString(requirement_string)

  File "/usr/local/lib/python3.9/site-packages/pyparsing.py", line 1955, in parseString

    raise exc

  File "/usr/local/lib/python3.9/site-packages/pyparsing.py", line 3250, in parseImpl

    raise ParseException(instring, loc, self.errmsg, self)

pyparsing.ParseException: Expected W:(abcd...), found '-'  (at char 0), (line:1, col:1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main

    return _run_code(code, main_globals, None,

  File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code

    exec(code, run_globals)

  File "/usr/local/lib/python3.9/site-packages/bandersnatch/main.py", line 223, in <module>

    exit(main())

  File "/usr/local/lib/python3.9/site-packages/bandersnatch/main.py", line 219, in main

    return asyncio.run(async_main(args, config))

  File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run

    return loop.run_until_complete(main)

  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete

    return future.result()

  File "/usr/local/lib/python3.9/site-packages/bandersnatch/main.py", line 184, in async_main

    return await bandersnatch.mirror.mirror(config)

  File "/usr/local/lib/python3.9/site-packages/bandersnatch/mirror.py", line 1019, in mirror

    mirror = BandersnatchMirror(

  File "/usr/local/lib/python3.9/site-packages/bandersnatch/mirror.py", line 201, in __init__

    super().__init__(master=master, workers=workers)

  File "/usr/local/lib/python3.9/site-packages/bandersnatch/mirror.py", line 43, in __init__

    self.filters = LoadedFilters(load_all=True)

  File "/usr/local/lib/python3.9/site-packages/bandersnatch/filter.py", line 151, in __init__

    self._load_filters(self.ENTRYPOINT_GROUPS)

  File "/usr/local/lib/python3.9/site-packages/bandersnatch/filter.py", line 180, in _load_filters

    plugin_instance = plugin_class()

  File "/usr/local/lib/python3.9/site-packages/bandersnatch/filter.py", line 55, in __init__

    self.initialize_plugin()

  File "/usr/local/lib/python3.9/site-packages/bandersnatch_filter_plugins/allowlist_name.py", line 30, in initialize_plugin

    self.allowlist_package_names = self._determine_unfiltered_package_names()

  File "/usr/local/lib/python3.9/site-packages/bandersnatch_filter_plugins/allowlist_name.py", line 150, in _determine_unfiltered_package_names

    filtered_requirements |= _parse_package_lines(req_fh.readlines())

  File "/usr/local/lib/python3.9/site-packages/bandersnatch_filter_plugins/allowlist_name.py", line 127, in _parse_package_lines

    requirement = Requirement(package_line.strip())

  File "/usr/local/lib/python3.9/site-packages/packaging/requirements.py", line 104, in __init__

    raise InvalidRequirement(

packaging.requirements.InvalidRequirement: Parse error at "'--extra-'": Expected W:(abcd...)```

[cooperlees]

Howdy - Thanks for the request. Seems fine to add this support. PR would be accepted.

Is there a requirements parsing spec or library we can copy / include into bandersnatch to handle all these edge cases? Would love to re-invent as little as possible here.

Naturally, please just add some tests showing this working etc.

[mbacicc]

Actually I'm not so sure that my way of using the requirements file is correct, neither I know if it is a supported way of doing that. I do this to speed up tests in our Gitlab CI/CD to take packages from our local bandersnatch and pypiserver package servers, avoiding the download of common packages from internet on every commit push.

At the moment I did a trick to remove the extra parameters with a python script which then uploads the edited requirements file to the docker volume of bandersnatch, but I can try to investigate on how to solve this at a bandersnatch level.

[cooperlees]

Ahh ok. If this is not a standard then I’d be hesitant to accept it in our parsing code …

[mbacicc]

I actually took this from here: Stackoverflow - Using pip with two --extra-index-url arguments that both point to the same domain But I'm not sure if it's a wanted feature by pip.

[cooperlees]

Since it's not a standard, I don't think we want to support this. I also think you realized you don't need it. Please reopen if I misunderstood anything.

[mbacicc]

Last time we considered this I had less time and experience. I dropped this idea and i parsed the requirements file before sending it to bandersnatch, remove the lines starting with -.

It is actually possible to add pip options inside the requirement files as stated in the Requirements file specifications on pypa/pip docs.

I will try to do a PR where bandersnatch at least survives when it encounters such options, dropping the lines. Maybe in the future we can see if some of these can be useful in the mirroring process.