ali5h / rules_pip

pip package rules for bazel that are fast (incremental fetch), support different python versions and work with all kinds of packages (i.e. packages with namespaces)
MIT License
60 stars 23 forks source link

Generate BUILD file with aliases for wheels packages #36

Closed oxidase closed 4 years ago

oxidase commented 4 years ago

Add convenience aliases to a BUILD file that is a sibling file to args.output one. Aliases allow to avoid loading @external_pip//:requirements.bzl file and use @external_pip//:pip targets directly. The alias target names are taken from the args.input as-is without lower-casing and have public visibility.


This change is Reviewable

ali5h commented 4 years ago

would u provide a sample for the new build files?

oxidase commented 4 years ago

for pre-compiled requirements

google_cloud_language
numpy
PyQt5
scipy

the generated BUILD file will be

# Generated BUILD file
[alias(name=name, actual=pkg,  visibility=["//visibility:public"]) for name, pkg in {
  "cachetools": "@pypi__36__cachetools_4_1_1//:pkg",
  "certifi": "@pypi__36__certifi_2020_6_20//:pkg",
  "chardet": "@pypi__36__chardet_3_0_4//:pkg",
  "google-api-core": "@pypi__36__google_api_core_1_21_0//:pkg",
  "google-api-core[grpc]": "@pypi__36__google_api_core_1_21_0//:grpc",
  "google-auth": "@pypi__36__google_auth_1_18_0//:pkg",
  "google-cloud-language": "@pypi__36__google_cloud_language_1_3_0//:pkg",
  "googleapis-common-protos": "@pypi__36__googleapis_common_protos_1_52_0//:pkg",
  "grpcio": "@pypi__36__grpcio_1_30_0//:pkg",
  "idna": "@pypi__36__idna_2_10//:pkg",
  "numpy": "@pypi__36__numpy_1_19_0//:pkg",
  "protobuf": "@pypi__36__protobuf_3_12_2//:pkg",
  "pyasn1": "@pypi__36__pyasn1_0_4_8//:pkg",
  "pyasn1-modules": "@pypi__36__pyasn1_modules_0_2_8//:pkg",
  "pyqt5": "@pypi__36__pyqt5_5_15_0//:pkg",
  "pyqt5-sip": "@pypi__36__pyqt5_sip_12_8_0//:pkg",
  "pytz": "@pypi__36__pytz_2020_1//:pkg",
  "requests": "@pypi__36__requests_2_24_0//:pkg",
  "rsa": "@pypi__36__rsa_4_6//:pkg",
  "scipy": "@pypi__36__scipy_1_5_0//:pkg",
  "setuptools": "@pypi__36__setuptools_47_3_1//:pkg",
  "six": "@pypi__36__six_1_15_0//:pkg",
  "urllib3": "@pypi__36__urllib3_1_25_9//:pkg"
}.items()]

Aliases are somewhat limited wrt the requirement macro because there is no way for the target parameter use case.

ali5h commented 4 years ago

this won't work when there are multiple python versions in the same repo, which is one of the main use cases

ali5h commented 4 years ago

how about adding the aliases under the pip repo (name of the pip_import rule)

oxidase commented 4 years ago

I was not thinking about multiple python runtimes, but this is how it works for google_cloud_language requirement

pip_import(name = "pip35", requirements = "//:requirements.txt", python_interpreter = "python3.5")
load("@pip35//:requirements.bzl", pip35_install = "pip_install")
pip35_install()
pip_import(name = "pip36", requirements = "//:requirements.txt", python_interpreter = "python3.6")
load("@pip36//:requirements.bzl", pip36_install = "pip_install")
pip36_install()

BUILD files will be external/pip35/BUILD

# Generated BUILD file
[alias(name=name, actual=pkg,  visibility=["//visibility:public"]) for name, pkg in {
  "certifi": "@pypi__35__certifi_2020_6_20//:pkg",
  "googleapis-common-protos": "@pypi__35__googleapis_common_protos_1_52_0//:pkg",
  "protobuf": "@pypi__35__protobuf_3_12_2//:pkg",
  "pyasn1": "@pypi__35__pyasn1_0_4_8//:pkg",
  "setuptools": "@pypi__35__setuptools_47_3_1//:pkg",
  "chardet": "@pypi__35__chardet_3_0_4//:pkg",
  "google-cloud-language": "@pypi__35__google_cloud_language_1_3_0//:pkg",
  "grpcio": "@pypi__35__grpcio_1_30_0//:pkg",
  "pyasn1-modules": "@pypi__35__pyasn1_modules_0_2_8//:pkg",
  "urllib3": "@pypi__35__urllib3_1_25_9//:pkg",
  "rsa": "@pypi__35__rsa_4_6//:pkg",
  "cachetools": "@pypi__35__cachetools_4_1_1//:pkg",
  "google-api-core": "@pypi__35__google_api_core_1_21_0//:pkg",
  "six": "@pypi__35__six_1_15_0//:pkg",
  "requests": "@pypi__35__requests_2_24_0//:pkg",
  "pytz": "@pypi__35__pytz_2020_1//:pkg",
  "google-auth": "@pypi__35__google_auth_1_18_0//:pkg",
  "google-api-core[grpc]": "@pypi__35__google_api_core_1_21_0//:grpc",
  "idna": "@pypi__35__idna_2_10//:pkg"
}.items()]

and in external/pip36/BUILD

# Generated BUILD file
[alias(name=name, actual=pkg,  visibility=["//visibility:public"]) for name, pkg in {
  "cachetools": "@pypi__36__cachetools_4_1_1//:pkg",
  "certifi": "@pypi__36__certifi_2020_6_20//:pkg",
  "chardet": "@pypi__36__chardet_3_0_4//:pkg",
  "google-api-core": "@pypi__36__google_api_core_1_21_0//:pkg",
  "google-api-core[grpc]": "@pypi__36__google_api_core_1_21_0//:grpc",
  "google-auth": "@pypi__36__google_auth_1_18_0//:pkg",
  "google-cloud-language": "@pypi__36__google_cloud_language_1_3_0//:pkg",
  "googleapis-common-protos": "@pypi__36__googleapis_common_protos_1_52_0//:pkg",
  "grpcio": "@pypi__36__grpcio_1_30_0//:pkg",
  "idna": "@pypi__36__idna_2_10//:pkg",
  "protobuf": "@pypi__36__protobuf_3_12_2//:pkg",
  "pyasn1": "@pypi__36__pyasn1_0_4_8//:pkg",
  "pyasn1-modules": "@pypi__36__pyasn1_modules_0_2_8//:pkg",
  "pytz": "@pypi__36__pytz_2020_1//:pkg",
  "requests": "@pypi__36__requests_2_24_0//:pkg",
  "rsa": "@pypi__36__rsa_4_6//:pkg",
  "setuptools": "@pypi__36__setuptools_47_3_1//:pkg",
  "six": "@pypi__36__six_1_15_0//:pkg",
  "urllib3": "@pypi__36__urllib3_1_25_9//:pkg"
}.items()]

and aliases are "@pip35//:google-cloud-language" or "@pip36//:google-cloud-language"

ali5h commented 4 years ago

oh, yeah, this build is in that external repo, my bad

ali5h commented 4 years ago

please add this to readme too, very useful. thank you

oxidase commented 4 years ago

added two usage examples

ali5h commented 4 years ago

one more thing, would you change one of the packages in the examples to be used like this? this will act as a test too.

oxidase commented 4 years ago

added "@piptool_deps_tests_3//:numpy"