handling requirements that specify a url and not a version

[shubhbapna]

Not all requirements need to specify a version and can instead specify a url from which the package can be downloaded: https://packaging.python.org/en/latest/specifications/version-specifiers/#direct-references

Currently, fromager is not handling this directly (right now any such requirement creates a specifier set that is empty and so the resolver will simply fail to resolve the requirement). We have the following options:

• explicitly deny the use of URLs in requirements and throw an error saying that instead of failing to resolve
• allow use of URLs which should bypass the resolution process:
  • should we allow all URLs? Should we enforce specific standards as mentioned in the python packaging document?

[tiran]

Requirements with URL specifiers are not allowed on pypi.org. No official package can have a requirement with an URL. Warehouse blocks uploads with requirements that have an URL.

URL requirements have their use case. For example they can be used to install a package directly from a git commit, tag, or branch:

>>> from packaging.requirements import Requirement
>>> req = Requirement("instructlab @ git+https://github.com/instructlab/instructlab@5a70892b706373c7bc1784b4b9569bd7abb139df")
>>> req.url
'git+https://github.com/instructlab/instructlab@5a70892b706373c7bc1784b4b9569bd7abb139df'

$ pip download --no-deps "git+https://github.com/instructlab/instructlab@5a70892b706373c7bc1784b4b9569bd7abb139df"
Collecting git+https://github.com/instructlab/instructlab@5a70892b706373c7bc1784b4b9569bd7abb139df
  Cloning https://github.com/instructlab/instructlab (to revision 5a70892b706373c7bc1784b4b9569bd7abb139df) to /home/cheimes/tmp/pip-req-build-shysuyxt
  Running command git clone --filter=blob:none --quiet https://github.com/instructlab/instructlab /home/cheimes/tmp/pip-req-build-shysuyxt
  Running command git rev-parse -q --verify 'sha^5a70892b706373c7bc1784b4b9569bd7abb139df'
  Running command git fetch -q https://github.com/instructlab/instructlab 5a70892b706373c7bc1784b4b9569bd7abb139df
  Running command git checkout -q 5a70892b706373c7bc1784b4b9569bd7abb139df
  Resolved https://github.com/instructlab/instructlab to commit 5a70892b706373c7bc1784b4b9569bd7abb139df
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Saved ./instructlab-0.20.0a3.dev36.zip
Successfully downloaded instructlab

The syntax works best with hooks like setuptools-scm, which calculate the version from git tags and git history.

[dhellmann]

I can see git URLs being useful for testing builds of packages before they are released. OTOH, most of the top-level packages we're building downstream are themselves pretty straightforward, so it's really the updates to the requirements list that we care about.

Maybe we can think through what a workflow might be and describe which commands are used and what will have to change to support it?