Pypi libraries have a dependencies field that takes a list of dependencies. These are only needed at runtime ("build"ing a pypi dependency is just downloading the wheel--don't need deps to download the wheel), but they are treated as build-time dependencies by builder. This is by builder's design--it doesn't allow targets to distinguish between build- and run-time dependencies; it assumes all dependencies are both. The consequence is that updates to a deep dependency will trigger rebuilds (which is mostly "redownloads" for Pypi libraries). Since pip seems to do a good job of caching, this doesn't seem like a big problem in practice, but if there is some downstream target that does have an expensive build step which depends on these (at runtime, but not at build time), it will have to be rebuilt--for now, this is purely hypothetical.
The runtime dependencies aren't necessary until we hit some target that bundles the runtime dependencies into a single payload (e.g., virtualenv or pex). This target will need to be able to distinguish between runtime dependencies and build dependencies so that it doesn't package a bunch of build dependencies into the final package.
Incomplete list/analysis of solutions:
builder distinguishes between buildtime and runtime dependencies
pypi packages don't represent their own runtime dependencies, but rather downstream targets represent them
Con: Every downstream target must know about each pypi package's runtime dependencies
This could be mitigated if the pypi package provides a string list of dependency names (instead of a list of targets) and then every downstream package can vet its list of dependency targets against that list. Much more tedious than putting the dependencies on the upstream pypi target but much less tedious than keeping all of the downstream targets in sync without an authoritative list.
Maybe a variation of this is that these dependencies aren't even targets at all, but rather a simple tree that exists solely in starlark, but it is pinned onto a runtime target type (e.g., pytest) which analyzes and downloads dependencies. For example, each dependency is either a Python library target of some kind or a Dict with a "pypi_package_name" key and a "dependencies" key (wherein the dependencies field holds a list of dependencies which are either Python libraries or this dict structure). The downloading of the wheels only happens when the actual target (e.g., pytest, py_binary, py_virtualenv, etc) is built. This could lead to redundant package downloading, but the pip installation that we're calling into should handle caching for us, so maybe not so bad?
Pypi libraries have a
dependenciesfield that takes a list of dependencies. These are only needed at runtime ("build"ing a pypi dependency is just downloading the wheel--don't need deps to download the wheel), but they are treated as build-time dependencies bybuilder. This is bybuilder's design--it doesn't allow targets to distinguish between build- and run-time dependencies; it assumes all dependencies are both. The consequence is that updates to a deep dependency will trigger rebuilds (which is mostly "redownloads" for Pypi libraries). Sincepipseems to do a good job of caching, this doesn't seem like a big problem in practice, but if there is some downstream target that does have an expensive build step which depends on these (at runtime, but not at build time), it will have to be rebuilt--for now, this is purely hypothetical.The runtime dependencies aren't necessary until we hit some target that bundles the runtime dependencies into a single payload (e.g., virtualenv or pex). This target will need to be able to distinguish between runtime dependencies and build dependencies so that it doesn't package a bunch of build dependencies into the final package.
Incomplete list/analysis of solutions:
builderdistinguishes between buildtime and runtime dependencies