How to get boto3 modules in dbx_py_pytest_test

[pavloneiman]

Please explain how to add additional module https://pypi.org/project/boto3/ to python integration tests run in dbx_py_pytest_test I've found dbx_py_pypi_piplib but do not understand how to apply it to get working example.

[armooo]

We keep a top level //pip directly with a bazel package based on the pypi distribution name. So in this case we would create //pip/boto3/BUILD starting with something like this:

load("//build_tools/py:py.bzl", "dbx_py_pypi_piplib")

dbx_py_pypi_piplib(
    name = "boto3",
    pip_version = "1.16.25",
)

The next step is to manually find the dependencies from setup.py and add them to the deps attr.

dbx_py_pypi_piplib(
    name = "boto3",
    pip_version = "1.16.25",
    deps = [
        "//pip/botocore",
        "//pip/jmespath",
        "//pip/s3transfer",
    ]
)

Then repeat this process for each new dependency you discover. As you are doing this you can run the auto-generated import tests with bazel test //pip/... as quick way to see if import <foo> would fail.

Once this is complete you can add //pip/boto3 to the deps of a dbx_py_pytest_test, dbx_py_binary, dbx_py_test, or dbx_py_library where it is imported.

[pavloneiman]

One of boto3 dependency is jmespath which does not have dependencies in setup.py so //pip/jmespath/BUILD is:

dbx_py_pypi_piplib(
    name = "jmespath",
    pip_version = "0.10.0",
    visibility = ["//visibility:public"],
)

bazel test //pip/jmespath runs without any problems but running dbx_py_pytest_test target fails with an error:

Traceback (most recent call last):
    File ".../BUILD.bazel", line 164
        dbx_py_test(name = 'demo-tests')
    File ".../external/dbx_build_tools/build_tools/py/py.bzl", line 631, in _dbx_py_binary_impl
        dbx_py_binary_base_impl(ctx, <1 more arguments>)
    File ".../external/dbx_build_tools/build_tools/py/py.bzl", line 680, in dbx_py_binary_base_impl
        emit_py_binary(ctx, main = main, <10 more arguments>)
    File ".../external/dbx_build_tools/build_tools/py/common.bzl", line 445, in emit_py_binary
        fail(<1 more arguments>)
We must know the contents of 
struct(
    archive = <generated file pip/jmespath/jmespath-cpython-38/jmespath-0.0.0-py2.py3-none-any.whl>,
    extracted_dir = "bazel-out/k8-fastbuild/bin/pip/jmespath/jmespath-cpython-38",
    extracted_files = depset([]),
    label = Label("//pip/jmespath:jmespath"),
    namespace_pkgs = []
)

How to troubleshoot this problem and what can be a problem?

[armooo]

Are you using an older version of the rules? The extracted_files logic was removed in https://github.com/dropbox/dbx_build_tools/commit/e95b030dfbd2f9882450f1d73c31f7c2631c1c65 when we started using tree artifacts. Before this change you were required to list the contents of the wheel in a contents attr.

Also the auto-generated tests have names like //pip/jmespath:jmespath_cpython-38_import_test so bazel test //pip/jmespath/... would be better to select them.