Support for URL requirements

[jonbuffington]

Thanks for creating and sharing your rules!

Do you plan to support URL requirements?

E.g., Using a URL requirement:

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile requirements.in
#
...
https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.2.0/en_core_web_sm-2.2.0.tar.gz
...

And calling pip_import results in:

ERROR: no such package '@py//': pip_import failed:  (Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/private/var/tmp/_bazel_jcb/6b822ab9db83c06cf3104c3b5c927ad9/external/com_github_ali5h_rules_pip/tools/piptool.par/__main__.py", line 184, in <module>
  File "/private/var/tmp/_bazel_jcb/6b822ab9db83c06cf3104c3b5c927ad9/external/com_github_ali5h_rules_pip/tools/piptool.par/__main__.py", line 141, in main
  File "/private/var/tmp/_bazel_jcb/6b822ab9db83c06cf3104c3b5c927ad9/external/com_github_ali5h_rules_pip/tools/piptool.par/__main__.py", line 48, in as_tuple
TypeError: Expected a pinned InstallRequirement, got https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.2.0/en_core_web_sm-2.2.0.tar.gz (from -r /private/var/tmp/_bazel_jcb/6b822ab9db83c06cf3104c3b5c927ad9/external/py/requirements.txt (line 22))

Unfortunately, the spacy language models can only be installed via a custom script or URL requirement. https://spacy.io/usage/models#models-download

[jonbuffington]

If I change the requirement from:

https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.2.0/en_core_web_sm-2.2.0.tar.gz

to:

https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.2.0/en_core_web_sm-2.2.0.tar.gz#egg=en_core_web_sm==2.2.0,

the is_pinned_requirement() function returns true and the following is written to requirements.bzl:

  if "pypi__3_7__en_core_web_sm_2_2_0" not in native.existing_rules():
    whl_library(
        name = "pypi__3_7__en_core_web_sm_2_2_0",
        pkg = "en-core-web-sm",
        requirements_repo = "@py",
        python_version = "3.7",
        extras = [],
        pip_args = pip_args,
    )

but pypi__3_7__en_core_web_sm_2_2_0 is not created.

[ali5h]

will take a look

[ali5h]

it is not being created because it is probably not used anywhere in your bazel rules, add it as a dependency to a rules and it will be fetched and created

[ali5h]

i tested this, as you said version must be specified

[jonbuffington]

Thanks for looking at the issue. Unfortunately, the import is not created even though I have a requirement in a few BUILD files. My current work around is:

requirements-model.txt:

# Per https://spacy.io/usage/models#models-download
https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.2.0/en_core_web_sm-2.2.0.tar.gz#egg=en_core_web_sm==2.2.0

WORKSPACE:

# vvv Remove once https://github.com/ali5h/rules_pip/issues/3 is resolved. vvv
load("@rules_python//python:pip.bzl", _tmp_pip_import = "pip_import")
_tmp_pip_import(
    name = "py_model",
    python_interpreter = "python3.7",
    requirements = "//py:requirements-model.txt",
)
load("@py_model//:requirements.bzl", _tmp_pip_install = "pip_install")
_tmp_pip_install()
# ^^^ Remove once https://github.com/ali5h/rules_pip/issues/3 is resolved. ^^^

BUILD

...
load("@rules_python//python:defs.bzl", "py_library")
load("@py//:requirements.bzl", "requirement")
load("@py_model//:requirements.bzl", requirement_model = "requirement")
...
py_library(
    name = "shared",
    srcs = [":srcs"],
    deps = [
        requirement("google-cloud-storage"),
        requirement("python-dateutil"),
        requirement("scrapy"),
        requirement("spacy"),
        requirement_model("en_core_web_sm"),
    ],
    visibility = ["//visibility:public"],
)

[ali5h]

did you run pip-compile on you requirements files? I will remove that soon, but until then you need to do that

[jonbuffington]

Yes. We are using pip-tools to generate the requirements*.txt files.

Currently, the pip_install phase is were the issue appears to exist. The rules silently fail to create pypi__3_7__en_core_web_sm_2_2_0. Our test targets fail with a model import error if the above work around is not used.

[ali5h]

please check this, seems to work for me https://github.com/ali5h/rules_pip/compare/http-package

[jonbuffington]

Your changes work well! The one subtle change was that I needed to use:

requirement("en-core-web-sm")

instead of rules_python's:

requirement("en_core_web_sm"),

Note the use of dashes instead of underscores in the requirement string.

Thanks much for your help!

[jonbuffington]

@ali5h I am attempting to update to v3.0.0 and it looks like there is a regression. I am unable to use a URL requirement again. If my requirements.txt file contains:

https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.3.1/en_core_web_sm-2.3.1.tar.gz#egg=en_core_web_sm==2.3.1

Running bazel outputs the following error: …

Traceback (most recent call last):
  File "/home/jcb/.cache/bazel/_bazel_jcb/e69664926aacd6cf84501dee3b516893/external/com_github_ali5h_rules_pip/src/piptool.py", line 246, in <module>
    main()
  File "/home/jcb/.cache/bazel/_bazel_jcb/e69664926aacd6cf84501dee3b516893/external/com_github_ali5h_rules_pip/src/piptool.py", line 171, in main
    reqs = sorted(get_requirements(args.input), key=as_tuple)
  File "/home/jcb/.cache/bazel/_bazel_jcb/e69664926aacd6cf84501dee3b516893/external/com_github_ali5h_rules_pip/src/piptool.py", line 51, in as_tuple
    req = Requirement(preq.requirement)
  File "/home/jcb/.cache/bazel/_bazel_jcb/e69664926aacd6cf84501dee3b516893/external/com_github_ali5h_rules_pip/third_party/py/pip/_vendor/packaging/requirements.py", line 106, in __init__
    raise InvalidRequirement(
pip._vendor.packaging.requirements.InvalidRequirement: Parse error at "'://githu'": Expected stringEnd
)

The above error occurs for both v3.0.0 and current HEAD.

Let me know if you think I should create a new issue.

[ali5h]

fixed in #55