search

periareon / rules_venv

Bazel rules for Python.
1 stars 1 forks source link
bazel bazel-rules python python3 starlark

readme

rules_venv

Overview

This repository implements Bazel rules for Python and is designed to be a drop-in replacement for the existing rules_python where uses of @rules_python//python:defs.bzl can be replaced with @rules_venv//python:defs.bzl.

Improvements over rules_python

While rules_python has fantastic toolchain infrastructure which this repo relies on, rules_python ultimately suffers from a few issues which this repo aims to solve:

  1. Use of PYTHONPATH to construct the python environment leads to operating system limitations.

    Some details on MAX_ARG_STRLEN and ARG_MAX can be found here: https://unix.stackexchange.com/a/120842

  2. Slow startup on windows systems that do not support symlinks.

    rules_python creates zipapps on systems that do not support runfiles. For large projects, this can lead to large (~500MB+) zip files being constantly compressed and uncompressed to run simple actions which is a lot more expensive than systems which support runfiles.

Setup

WORKSPACE.bazel

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

# See releases for urls and checksums
http_archive(
    name = "rules_venv",
    sha256 = "{sha256}",
    urls = ["https://github.com/periareon/rules_venv/releases/download/{version}/rules_venv-v{version}.tar.gz"],
)

load("@rules_venv//venv:repositories.bzl", "venv_register_toolchains", "rules_venv_dependencies")

rules_venv_dependencies()

venv_register_toolchains()

load("@rules_venv//python/venv:repositories_transitive.bzl", "rules_venv_transitive_deps")

rules_venv_transitive_deps()

bzlmod

bazel_dep(name = "rules_venv", version = "{version}")

py_venv_common

An experimental API for creating python executables within user author rules.

Rules

py_venv_binary

py_venv_binary(name, deps, srcs, data, env, imports, main)

A py_venv_binary is an executable Python program consisting of a collection of .py source files (possibly belonging to other py_library rules), a *.runfiles directory tree containing all the code and data needed by the program at run-time, and a stub script that starts up the program with the correct initial environment and data.

load("@rules_venv//python/venv:defs.bzl", "py_venv_binary")

py_venv_binary(
    name = "foo",
    srcs = ["foo.py"],
    data = [":transform"],  # a cc_binary which we invoke at run time
    deps = [
        ":bar",  # a py_library
    ],
)

ATTRIBUTES

Name Description Type Mandatory Default
name A unique name for this target. Name required
deps Other python targets to link to the current target. List of labels optional []
srcs The list of source (.py) files that are processed to create the target. List of labels optional []
data Files needed by this rule at runtime. May list file or rule targets. Generally allows any target. List of labels optional []
env Dictionary of strings; values are subject to $(location) and "Make variable" substitution. Dictionary: String -> String optional {}
imports List of import directories to be added to the PYTHONPATH. List of strings optional []
main The name of the source file that is the main entry point of the application. This file must also be listed in srcs. If left unspecified, name is used instead. If name does not match any filename in srcs, main must be specified. Label optional None

py_venv_library

py_venv_library(name, deps, srcs, data, imports)

A library of Python code that can be depended upon.

ATTRIBUTES

Name Description Type Mandatory Default
name A unique name for this target. Name required
deps Other python targets to link to the current target. List of labels optional []
srcs The list of source (.py) files that are processed to create the target. List of labels optional []
data Files needed by this rule at runtime. May list file or rule targets. Generally allows any target. List of labels optional []
imports List of import directories to be added to the PYTHONPATH. List of strings optional []

py_venv_test

py_venv_test(name, deps, srcs, data, env, env_inherit, imports, main)

A py_venv_test rule compiles a test. A test is a binary wrapper around some test code.

ATTRIBUTES

Name Description Type Mandatory Default
name A unique name for this target. Name required
deps Other python targets to link to the current target. List of labels optional []
srcs The list of source (.py) files that are processed to create the target. List of labels optional []
data Files needed by this rule at runtime. May list file or rule targets. Generally allows any target. List of labels optional []
env Dictionary of strings; values are subject to $(location) and "Make variable" substitution. Dictionary: String -> String optional {}
env_inherit Specifies additional environment variables to inherit from the external environment when the test is executed by bazel test. List of strings optional []
imports List of import directories to be added to the PYTHONPATH. List of strings optional []
main The name of the source file that is the main entry point of the application. This file must also be listed in srcs. If left unspecified, name is used instead. If name does not match any filename in srcs, main must be specified. Label optional None

py_venv_toolchain

py_venv_toolchain(name, zipapp_shebang)

Declare a toolchain for rules_venv rules.

ATTRIBUTES

Name Description Type Mandatory Default
name A unique name for this target. Name required
zipapp_shebang The shebang to use when creating zipapps (OutputGroupInfo.python_zip_file). String optional ""

py_venv_common.create_dep_info

py_venv_common.create_dep_info(ctx, deps)

Construct dependency info required for building PyInfo

PARAMETERS

Name Description Default Value
ctx The rule's context object. none
deps A list of python dependency targets none

RETURNS

struct: Dependency info.

py_venv_common.create_py_info

py_venv_common.create_py_info(ctx, imports, srcs, dep_info)

Construct a PyInfo provider

PARAMETERS

Name Description Default Value
ctx The rule's context object. none
imports The raw imports attribute. none
srcs A list of python (.py) source files. none
dep_info Dependency info from the current target. None

RETURNS

PyInfo: A PyInfo provider.

py_venv_common.create_python_zip_file

py_venv_common.create_python_zip_file(ctx, venv_toolchain, py_info, main, runfiles, py_toolchain,
                                      name)

Create a zipapp.

PARAMETERS

Name Description Default Value
ctx The rule's context object. none
venv_toolchain A py_venv_toolchain toolchain. none
py_info The PyInfo provider for the current target. none
main The main python entrypoint. none
runfiles Runfiles associated with the executable. none
py_toolchain A py_toolchain toolchain. If one is not provided one will be acquired via py_venv_toolchain. None
name An alternate name to use in the output instead of ctx.label.name. None

RETURNS

File: The generated zip file.

py_venv_common.create_runfiles_collection

py_venv_common.create_runfiles_collection(ctx, venv_toolchain, py_toolchain, runfiles,
                                          exclude_files, name, use_zip)

Generate a runfiles directory

This functionality exists due to the lack of native support for generating runfiles in an action. For details see: https://github.com/bazelbuild/bazel/issues/15486

PARAMETERS

Name Description Default Value
ctx The rule's context object. none
venv_toolchain A py_venv_toolchain toolchain. none
py_toolchain A py_toolchain toolchain. none
runfiles The runfiles to render into a directory none
exclude_files A collection of files to exclude from the collection despite them appearing in runfiles. depset([])
name An alternate name to use in the output instead of ctx.label.name. None
use_zip If True, a zip file will be generated instead of a json manifest. False

RETURNS

Tuple[File, Runfiles]: The generated runfiles collection and associated runfiles.

py_venv_common.create_venv_attrs

py_venv_common.create_venv_attrs()

py_venv_common.create_venv_entrypoint

py_venv_common.create_venv_entrypoint(ctx, venv_toolchain, py_info, main, runfiles, py_toolchain,
                                      name, use_runfiles_in_entrypoint, force_runfiles)

Create an executable which constructs a python venv and subprocesses a given entrypoint.

PARAMETERS

Name Description Default Value
ctx The rule's context object. none
venv_toolchain A py_venv_toolchain toolchain. none
py_info The PyInfo provider for the current target. none
main The main python entrypoint. none
runfiles Runfiles associated with the executable. none
py_toolchain A py_toolchain toolchain. If one is not provided one will be acquired via py_venv_toolchain. None
name An alternate name to use in the output instead of ctx.label.name. None
use_runfiles_in_entrypoint If true, an entrypoint will be created that relies on runfiles. True
force_runfiles If True, a rendered runfiles directory will be used over builtin runfiles where RUNFILES_DIR would be provided. False

RETURNS

Tuple[File, Runfiles]: The generated entrypoint and associated runfiles.

py_venv_common.get_toolchain

py_venv_common.get_toolchain(ctx, cfg)

PARAMETERS

Name Description Default Value
ctx

-

none
cfg

-

"target"

py_global_venv_aspect

py_global_venv_aspect(name)

An aspect for generating metadata required to include Python targets in a global venv.

ASPECT ATTRIBUTES

ATTRIBUTES

Name Description Type Mandatory Default
name A unique name for this target. Name required