search

bazelbuild / bazel

a fast, scalable, multi-language and extensible build system
https://bazel.build
Apache License 2.0
23.32k stars 4.09k forks source link

`py_binary` should support a PEX output #11931

Open shs96c opened 4 years ago

shs96c commented 4 years ago

Description of the problem / feature request:

A java_binary has an implicit *_deploy.jar that creates a single file that can be executed. There is no equivalent implicit output for a py_binary. Any such equivalent should support packaging native libraries (for multiple OSs) for obvious reasons.

PEX provides a mechanism to do this.

Feature requests: what underlying problem are you trying to solve with this feature?

Create a single deployable artifact from a py_binary target without specifying additional command line options.

What's the output of bazel info release?

3.4.1

Have you found anything relevant by searching the web?

There's a par_binary rule that already exists, but it handles neither native files, nor runfiles. In addition, it's a different rule than py_binary rather than implicit output.

FB have also released XAR, but this relies on squashfs and so is pretty hostile to Windows users.

xhuoroku commented 3 years ago

Found this bazel rule repo: https://github.com/benley/bazel_rules_pex however it's outdated and not actively maintained, is there any update on this topic?

UebelAndre commented 3 years ago

Why not solve for this in rules_python with a rule that ingests a py_binary/py_library and outputs a .pex (pex)? I think a rule like that would be ideal since from what I've gathered .pex handles multiple platforms better than .par (subpar)

Also, seems related to https://github.com/bazelbuild/rules_python/issues/436

alexeagle commented 2 years ago

I learned from @groodt that https://github.com/bazelbuild/bazel/pull/9453 indicates that this is already possible with the existing python rules, by requesting the python_zip_file from the OutputGroupInfo of a py_binary.

Here's a demo:

alexeagle@system76-pc:~/Projects/rules_python/examples/pip_parse$ bazel build :main --output_groups=python_zip_file
Starting local Bazel server and connecting to it...
INFO: Analyzed target //:main (23 packages loaded, 232 targets configured).
INFO: Found 1 target...
Target //:main up-to-date:
  bazel-bin/main.zip
INFO: Elapsed time: 3.459s, Critical Path: 0.07s
INFO: 4 processes: 3 internal, 1 linux-sandbox.
INFO: Build completed successfully, 4 total actions

alexeagle@system76-pc:~/Projects/rules_python/examples/pip_parse$ unzip -l bazel-bin/main.zip
Archive:  bazel-bin/main.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
    15054  2010-01-01 00:00   __main__.py
        0  2010-01-01 00:00   __init__.py
        0  2010-01-01 00:00   runfiles/example_repo/__init__.py
        0  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__requests/__init__.py
        0  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__idna/__init__.py
        0  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__certifi/__init__.py
        0  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/__init__.py
        0  2010-01-01 00:00   runfiles/__init__.py
        0  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/__init__.py
       65  2010-01-01 00:00   runfiles/example_repo/main.py
     4141  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__requests/requests/__init__.py
      441  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__requests/requests/__version__.py
     1096  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__requests/requests/_internal_utils.py
    21344  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__requests/requests/adapters.py
     6496  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__requests/requests/api.py
    10207  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__requests/requests/auth.py
      453  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__requests/requests/certs.py
     1782  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__requests/requests/compat.py
    18430  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__requests/requests/cookies.py
     3161  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__requests/requests/exceptions.py
     3515  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__requests/requests/help.py
      757  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__requests/requests/hooks.py
    34308  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__requests/requests/models.py
      542  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__requests/requests/packages.py
    30137  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__requests/requests/sessions.py
     4188  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__requests/requests/status_codes.py
     3005  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__requests/requests/structures.py
    30529  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__requests/requests/utils.py
    10142  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__requests/requests-2.25.1.dist-info/LICENSE
     4168  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__requests/requests-2.25.1.dist-info/METADATA
     1776  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__requests/requests-2.25.1.dist-info/RECORD
      110  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__requests/requests-2.25.1.dist-info/WHEEL
        9  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__requests/requests-2.25.1.dist-info/top_level.txt
       62  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__certifi/certifi/__init__.py
      243  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__certifi/certifi/__main__.py
     2303  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__certifi/certifi/core.py
     1048  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__certifi/certifi-2020.12.5.dist-info/LICENSE
     2994  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__certifi/certifi-2020.12.5.dist-info/METADATA
      704  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__certifi/certifi-2020.12.5.dist-info/RECORD
      110  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__certifi/certifi-2020.12.5.dist-info/WHEEL
        8  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__certifi/certifi-2020.12.5.dist-info/top_level.txt
   263774  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__certifi/certifi/cacert.pem
     1559  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/__init__.py
    31254  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/big5freq.py
     1757  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/big5prober.py
     9411  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/chardistribution.py
     3787  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/charsetgroupprober.py
     5110  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/charsetprober.py
        1  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/cli/__init__.py
     2738  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/cli/chardetect.py
     3590  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/codingstatemachine.py
     1134  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/compat.py
     1855  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/cp949prober.py
     1661  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/enums.py
     3950  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/escprober.py
    10510  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/escsm.py
     3749  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/eucjpprober.py
    13546  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/euckrfreq.py
     1748  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/euckrprober.py
    31621  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/euctwfreq.py
     1747  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/euctwprober.py
    20715  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/gb2312freq.py
     1754  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/gb2312prober.py
    13838  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/hebrewprober.py
    25777  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/jisfreq.py
    19643  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/jpcntx.py
    12839  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/langbulgarianmodel.py
    17948  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/langcyrillicmodel.py
    12688  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/langgreekmodel.py
    11345  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/langhebrewmodel.py
    12592  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/langhungarianmodel.py
    11290  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/langthaimodel.py
    11102  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/langturkishmodel.py
     5370  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/latin1prober.py
     3413  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/mbcharsetprober.py
     2012  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/mbcsgroupprober.py
    25481  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/mbcssm.py
     5657  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/sbcharsetprober.py
     3546  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/sbcsgroupprober.py
     3774  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/sjisprober.py
    12485  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/universaldetector.py
     2766  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/utf8prober.py
      242  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet/version.py
     2174  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet-3.0.4.dist-info/DESCRIPTION.rst
     3239  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet-3.0.4.dist-info/METADATA
     3916  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet-3.0.4.dist-info/RECORD
      110  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet-3.0.4.dist-info/WHEEL
       60  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet-3.0.4.dist-info/entry_points.txt
     1375  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet-3.0.4.dist-info/metadata.json
        8  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__chardet/chardet-3.0.4.dist-info/top_level.txt
       58  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__idna/idna/__init__.py
     3299  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__idna/idna/codec.py
      232  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__idna/idna/compat.py
    11951  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__idna/idna/core.py
    42350  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__idna/idna/idnadata.py
     1749  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__idna/idna/intranges.py
       22  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__idna/idna/package_data.py
   202084  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__idna/idna/uts46data.py
     1565  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__idna/idna-2.10.dist-info/LICENSE.rst
     9104  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__idna/idna-2.10.dist-info/METADATA
      950  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__idna/idna-2.10.dist-info/RECORD
      110  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__idna/idna-2.10.dist-info/WHEEL
        5  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__idna/idna-2.10.dist-info/top_level.txt
     2763  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/__init__.py
    10811  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/_collections.py
       63  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/_version.py
    18750  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/connection.py
    37131  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/connectionpool.py
        0  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/contrib/__init__.py
      957  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/contrib/_appengine_environ.py
        0  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/contrib/_securetransport/__init__.py
    17637  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/contrib/_securetransport/bindings.py
    13908  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/contrib/_securetransport/low_level.py
    11010  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/contrib/appengine.py
     4160  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/contrib/ntlmpool.py
    16874  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/contrib/pyopenssl.py
    34417  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/contrib/securetransport.py
     7097  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/contrib/socks.py
     8217  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/exceptions.py
     8579  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/fields.py
     2440  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/filepost.py
      108  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/packages/__init__.py
        0  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/packages/backports/__init__.py
     1417  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/packages/backports/makefile.py
    34665  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/packages/six.py
      927  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/packages/ssl_match_hostname/__init__.py
     5679  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/packages/ssl_match_hostname/_implementation.py
    19763  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/poolmanager.py
     5985  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/request.py
    28203  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/response.py
     1155  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/util/__init__.py
     4908  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/util/connection.py
     1604  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/util/proxy.py
      498  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/util/queue.py
     4123  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/util/request.py
     3510  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/util/response.py
    21391  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/util/retry.py
    17110  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/util/ssl_.py
     6907  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/util/ssltransport.py
    10003  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/util/timeout.py
    14030  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/util/url.py
     5404  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3/util/wait.py
     1115  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3-1.26.5.dist-info/LICENSE.txt
    43687  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3-1.26.5.dist-info/METADATA
     3678  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3-1.26.5.dist-info/RECORD
      110  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3-1.26.5.dist-info/WHEEL
        8  2010-01-01 00:00   runfiles/pip_parsed_deps_pypi__urllib3/urllib3-1.26.5.dist-info/top_level.txt
     3561  2010-01-01 00:00   runfiles/bazel_tools/tools/python/py3wrapper.sh
---------                     -------
  1562831                     148 files

It can be run with python bazel-bin/main.zip, following https://www.python.org/dev/peps/pep-0441/

So the remaining question for this issue is, do we actually need any other implementation like PEX? Maybe we are just missing some documentation for this output group.

Cartman75 commented 2 years ago

I feel like it being a command-line option is the wrong way to go. It should just be part of setting up py_binary. The all-or-nothing nature of this is kind of lacking.

It's really why I still use .par (subpar).

Also, PEX supports actually having the interpreter in the file as well, plus .so support. Its not just python only.

alexeagle commented 2 years ago

It's included in py_binary, you can use a filegroup to select the zip output group without a command line flag. It's a bit awkward compared with java_binary's _deploy.jar implicit output, but not unusable.

Seems like embedding interpreter, cross-platform, and .so are all reasons that the current zip implementation isn't sufficient, so this FR should stay open.

groodt commented 2 years ago

I can clarify a few things here as well.

It's included in py_binary, you can use a filegroup to select the zip output group without a command line flag. It's a bit awkward compared with java_binary's _deploy.jar implicit output, but not unusable.

I wonder if we can add a macro or something for it, but yes the output filegroup is available within Bazel to add the zip to a Docker container or pkg_tar etc. It's not only something that needs to be accessed on the command line.

Seems like embedding interpreter,

Both subpar and the Bazel zip output group do not support this, correct. The approach I see used is to either layer the zip into a Docker container with rules_docker, or use rules_pkg to prepare a .deb or .rpm if you truly need to ship the interpreter.

cross-platform

subpar works on linux and macos, but not Windows. The Bazel zip output group works on linux and macos and Windows (to the extent that Bazel works on Windows, but I have no first-hand experience of this). So it is cross-platform in that way. I do know for certain that the Windows Python Launcher does support the POSIX shebang because I've used this in the past.

So, by this, are you really asking for multi-platform? Some archive format that will support all OS in a single archive with all the platform-specific interpreters and binaries? Wouldn't that get quite bloated if it had to ship 3 interpreters and the binaries for all 3 platforms? Not even sure that is possible in Bazel at the moment. How would it transition to produce the different binaries and then finally switch back to bundle them all together?

and .so

.so (and sometimes .dylib on macos) are included in the zip produced by the Bazel zip output group.

Here is the output from a zip produced by a py_binary I have that includes scikit-learn. You can see the .so from the third_party wheels are included in the runfiles.

    testing: runfiles/pypi/pypi__scipy/scipy/stats/_boost/beta_ufunc.cpython-39-darwin.so   OK
    testing: runfiles/pypi/pypi__scipy/scipy/stats/_boost/binom_ufunc.cpython-39-darwin.so   OK
    testing: runfiles/pypi/pypi__scipy/scipy/stats/_boost/nbinom_ufunc.cpython-39-darwin.so   OK
    testing: runfiles/pypi/pypi__scipy/scipy/stats/_qmc_cy.cpython-39-darwin.so   OK
    testing: runfiles/pypi/pypi__scipy/scipy/stats/_qmc_cy.pyi   OK
    testing: runfiles/pypi/pypi__scipy/scipy/stats/_sobol.cpython-39-darwin.so   OK
    testing: runfiles/pypi/pypi__scipy/scipy/stats/_sobol.pyi   OK
    testing: runfiles/pypi/pypi__scipy/scipy/stats/_sobol_direction_numbers.npz   OK
    testing: runfiles/pypi/pypi__scipy/scipy/stats/_stats.cpython-39-darwin.so   OK
    testing: runfiles/pypi/pypi__scipy/scipy/stats/biasedurn.cpython-39-darwin.so   OK
    testing: runfiles/pypi/pypi__scipy/scipy/stats/biasedurn.pxd   OK
    testing: runfiles/pypi/pypi__scipy/scipy/stats/mvn.cpython-39-darwin.so   OK
    testing: runfiles/pypi/pypi__scipy/scipy/stats/statlib.cpython-39-darwin.so   OK
Cartman75 commented 2 years ago

Here is my par version:

par_binary(
    name = "change_instruments",
    srcs = ["change_instruments.py"],
    deps = [
        "//database/common/pylib:connect_db",
        "//pypi/absl_py",
    ],
    python_version = "PY3",
)

This works, other than par/py, I did not have to do anything special.

py_binary(
    name = "change_instruments",
    srcs = ["change_instruments.py"],
    deps = [
        "//database/common/pylib:connect_db",
        "//pypi/absl_py",
    ],
)

filegroup(
    name = "change_instruments_zip",
    srcs = [":change_instruments"],
    output_group = "python_zip_file",
)

Does NOT just work: 

bazel-bin/database/mysql/bin/change_instruments.zip --help
bazel-bin/database/mysql/bin/change_instruments.zip: line 1: $'PK\003\004': command not found
bazel-bin/database/mysql/bin/change_instruments.zip: line 4: syntax error near unexpected token `)'

I am probably doing something wrong here but people keep seeing zip is easy, just add this or do this, but it does not seem to work out of the box.

groodt commented 2 years ago

Please try to execute it as follows: python bazel-bin/database/mysql/bin/change_instruments.zip

The functionality being leveraged is similar to: https://www.python.org/dev/peps/pep-0441/ https://docs.python.org/3/library/zipapp.html

So to execute a py_binary "in-tree", you can bazel build/run the py_binary target.

To package your artifact for execution outside the source tree, you can build and ship the "python_zip_file" output group.

Cartman75 commented 2 years ago

Thanks @groodt, I did not realize it had to be run differently. Seems like some documentation around this feature would be good. A google search does not seem to yield anything but maybe its already out there.

I will keep plugging and this and see if we can get some non-python in here working, if so, then I would be less likely to want pex. PAR does not support anything but python currently which has worked for us, so far.

borancar commented 2 years ago

Seems like embedding interpreter,

Both subpar and the Bazel zip output group do not support this, correct. The approach I see used is to either layer the zip into a Docker container with rules_docker, or use rules_pkg to prepare a .deb or .rpm if you truly need to ship the interpreter.

I actually got the interpreter in by defining my own python toolchain as type @rules_python//python:toolchain_type. Here's the py_runtime definition:

#Configuring the Python 2 runtime
py_runtime(
    name = "python2_runtime",
    files = [
        "@python_2//:py_files",
    ],
    interpreter = "@python_2//:python/bin/python",
    python_version = "PY2",
    visibility = ["//visibility:public"],
)

This not only made sure that these files made it in as runfiles, but also the __main__.py in the zip root (the entry-point, if you will), had the following line:

PYTHON_BINARY = 'python_2/python/bin/python'

Allows one to go into the past, based on what you put :)

sys.version: 2.7.5 (default, Jun 11 2019, 14:33:56) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]
sys.path: ['/tmp/Bazel.runfiles__B635J/runfiles/__main__', '/tmp/Bazel.runfiles__B635J/runfiles', '/tmp/Bazel.runfiles__B635J/runfiles/__main__', '/tmp/Bazel.runfiles__B635J/runfiles/python_2', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib64/python2.7/site-packages/gtk-2.0', '/usr/lib/python2.7/site-packages']

The remaining issue are the .so libraries the Python interpreter itself would depend on - it still tries to hit /usr/lib64 rather than the zipfile provided files. I guess an LD_LIBRARY_PATH addition within __main__.py?

borancar commented 2 years ago

OT: Maybe as a bit of side-feedback, this (as in zip file via output group and the entrypoint generation) should be rolled into rules_py rather than rely on Bazel itself? In the absence of missing documentation, people are likely to search source files, and I can see that happening for rules, but usually not for Bazel source itself. Also makes it easier to iterate on.

alexeagle commented 2 years ago

If you use the whl_requirement from the generated pip_deps repo, you can get hold of the original wheel files Bazel downloaded.

Sadly py_library and PyInfo are stuck in the Bazel sources, so users have to put the whl_requirement in their pex_binary rule. But you can just validate in the rule implementation that the whl files match the transitive third-party py_library graph, and print a buildozer command when they don't match, so in practice it's pretty easy to use.

You can then write a pex_binary rule that's less Bazel-idiomatic and more like standard PEX, where you construct the pex file using their loader and hand the wheel files along with the py_library of first-party sources.

I think we can post an example under rules_python soon, I have a client making this work.

borancar commented 2 years ago

Happy to offer any help with PEX rule if people are interested. I already have a minimal functioning example - https://github.com/borancar/mishmash/tree/928cd7bb75c93472c3931f3f8f56941786ec6116, and here's the changes needed to make rules pex work for Bazel 4 https://github.com/borancar/mishmash/commit/928cd7bb75c93472c3931f3f8f56941786ec6116#diff-6b9d2d187ff982cd6100e3a5e0813e450351d5164262d0c1d9acc43522d994db

hrfuller commented 2 years ago

At twitter we have a rule that does exactly this, though it takes a py_library and outputs a pex. I can speak with the team about the potential to open source it.

libratiger commented 2 years ago

At twitter we have a rule that does exactly this, though it takes a py_library and outputs a pex. I can speak with the team about the potential to open source it.

this is very useful, is there any progress now ?

hrfuller commented 2 years ago

Hey @libratiger I haven't got around to bringing this up sorry. Its still a possibility. A couple things need to happen. Permission from close source repo and agreement on direction with the rest of rules_python maintainers.

korjek commented 1 year ago

@hrfuller is there any progress on open-sourcing rule that make pex?

hrfuller commented 1 year ago

No longer at Twitter. However, I'm open to working on a clean slate design based on use of the PEX CLI, which probably would a better fit for the OSS world.

arrdem commented 1 year ago

we have a rule that does exactly this, though it takes a py_library and outputs a pex.

Ftr this is basically how https://github.com/arrdem/rules_zapp functions

binaryaaron commented 1 year ago

No longer at Twitter. However, I'm open to working on a clean slate design based on use of the PEX CLI, which probably would a better fit for the OSS world.

hey @hrfuller / @arrdem ! either of you interested in working on this still? would be happy to help.

alexeagle commented 1 week ago

https://github.com/aspect-build/rules_py/blob/main/docs/pex.md solves this, if you're willing to adopt a ruleset on top of rules-python.

I think this issue should be closed, or at least transferred to rules_python.