search

tweag / clodl

Turn dynamically linked ELF binaries and libraries into self-contained closures.
BSD 3-Clause "New" or "Revised" License
170 stars 6 forks source link

Need Help Fixing Bazel Build Issue with Clodl #46

Open evanlu14 opened 9 months ago

evanlu14 commented 9 months ago

Hi,

I'm new to clodl and using it to package a binary in a Bazel project for cross-platform compatibility. However, I'm facing some problems and hope to get some help.

I'm not sure where to ask for help, so I'm posting it here. If this isn't the right place, please let me know.

In my WORKSPACE file, I added the following:

http_archive(
    name = "com_github_bazelbuild_buildtools",
    sha256 = "ae34c344514e08c23e90da0e2d6cb700fcd28e80c02e23e4d5715dddcb42f7b3",
    strip_prefix = "buildtools-4.2.2",
    urls = [
        "https://github.com/bazelbuild/buildtools/archive/refs/tags/4.2.2.tar.gz",
    ],
)

http_archive(
    name = "rules_sh",
    sha256 = "d668bb32f112ead69c58bde2cae62f6b8acefe759a8c95a2d80ff6a85af5ac5e",
    strip_prefix = "rules_sh-0.3.0",
    urls = ["https://github.com/tweag/rules_sh/archive/v0.3.0.tar.gz"],
)

http_archive(
    name = "rules_haskell",
    sha256 = "298f6f0db23391274b4eca215daa01797d05699469048ef94540d5829b466377",
    strip_prefix = "rules_haskell-0.17",
    url = "https://github.com/tweag/rules_haskell/archive/refs/tags/v0.17.tar.gz",
)

http_archive(
    name = "io_tweag_clodl",
    urls = ["https://github.com/tweag/clodl/archive/b75ce067f9c95546b553faac76c07e99346251e5.tar.gz"],
    strip_prefix = "clodl-b75ce067f9c95546b553faac76c07e99346251e5",
)

(I'm not sure why there are so many dependencies. If there is anyway to avoid it, please let me know).

And in the BUILD file:

load(
    "@io_tweag_clodl//clodl:clodl.bzl",
    "binary_closure",
)
binary_closure(
    name = "my_binary_closure",
    src = "my_binary",
)

When I run:

bazel build //path:my_binary_closure --verbose_failures

I get this error:

ERROR: /home/user/workspace/path/BUILD.bazel:195:15: output 'path/patchelf' was not created
ERROR: /home/user/workspace/path/BUILD.bazel:195:15: output 'path/scanelf' was not created
ERROR: /home/user/workspace/path/BUILD.bazel:195:15: Action path/bash failed: not all outputs were created or valid
Target //path:my_binary_closure failed to build
ERROR: /home/user/workspace/path/BUILD.bazel:195:15 Action path/my_binary_closure.zip failed: not all outputs were created or valid

Any help is appreciated. Thanks!

facundominguez commented 9 months ago

Hello @evanlu14, thanks for the report.

I think rules_haskell is only needed for testing clodl and buildtools is only needed for formatting clodl sources. Perhaps you could drop those.

The error that you are getting is not revealing the cause to me. If you want to debug it further, I would try cloning the clodl repo and build it first with the configuration in the repo.

If that works, then upgrading bazel to whatever version you are using could be tried. I don't know what OS you need, but keep in mind that clodl has only been tested in some linuxes and MacOSs.

Let me know your OS and your bazel version. If you can, you could also share the example source code and build configuration. I'm soon out for vacations, but I could try it myself next year.

evanlu14 commented 9 months ago

Hi @facundominguez, thanks for your response. I'm currently using Ubuntu 18.04 with Bazel version 6.3.2

When attempting to build clodl, I encountered the following error:

➜  clodl git:(master) bazel build //...
WARNING: Option 'host_javabase' is deprecated
WARNING: Option 'host_javabase' is deprecated
ERROR: /home/zecao/workspace/clodl/BUILD:40:26: While resolving toolchains for target //:base: No matching toolchains found for types @rules_haskell//haskell:toolchain.
To debug, rerun with --toolchain_resolution_debug='@rules_haskell//haskell:toolchain'
If platforms or toolchains are a new concept for you, we'd encourage reading https://bazel.build/concepts/platforms-intro.
ERROR: Analysis of target '//:base' failed; build aborted: 
INFO: Elapsed time: 0.940s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)

I have already installed Nix and scanelf:

➜  clodl git:(master) nix --version  
nix (Nix) 2.19.2
➜  clodl git:(master) scanelf --version
pax-utils-v1.2.2: <unknown>
scanelf written for Gentoo by <solar and vapier @ gentoo.org>

Could you please help me identify and resolve the issue with the build? Thanks!

avdv commented 9 months ago

Hi @evanlu14,

It seems we only register a haskell toolchain from nixpkgs and do not use a GHC bindist in this project. That means the local platform needs to have the support_nix constraint, ie. you have to set the host platform like this: --host_platform=@io_tweag_rules_nixpkgs//nixpkgs/platforms:host.

Also, if you want to build clodl, it would be best to enter the nix shell at first (this provides an environment with all required development tools at the right version):

$ cd clodl
$ nix-shell
these 265 paths will be fetched (1349.36 MiB download, 2512.89 MiB unpacked):
  /nix/store/22vvkika81ahw77875ik1q89s25p8lpr-ORBit2-2.14.19
  /nix/store/pgc40k0gc57v36n6m0p27daypnyp0zr6-SDL2-2.0.14
  /nix/store/xdr0w3b6zvb7xvish4f736029rx2xhcj-acl-2.3.0
  /nix/store/fzyzjj5sx74makb3v4hpb5481561m257-alsa-lib-1.2.4
  /nix/store/m20nz4vr0m0qba35raxfzxy3fwl6nzqg-alsa-topology-conf-1.2.4
  /nix/store/1n702bpj9c6lxvzm8vcy9janwvwimiy0-alsa-ucm-conf-1.2.4
  /nix/store/3xma4y1qsb1xfqy9qxcj4na9xr91pwpy-at-spi2-atk-2.38.0
  /nix/store/7zvlghwrm0yc7f7qi6qibvlln9hs44ic-at-spi2-core-2.40.1
...
copying path '/nix/store/0ix6nx7w5vvxhliz5pp88kk07kaksw2h-bazel-rc' from 'https://cache.nixos.org'...
copying path '/nix/store/cvb2w69nkbir6737b7vk8qzmywz96vll-bazel-3.7.2' from 'https://cache.nixos.org'...

$ bazel build //... --host_platform=@io_tweag_rules_nixpkgs//nixpkgs/platforms:host
DEBUG: /home/claudio/.cache/bazel/_bazel_claudio/7ec1d07ccb757a36801b07c1fa19d949/external/rules_haskell/haskell/private/versions.bzl:60:10: WARNING: bazel version is too recent. Supported versions range from 2.1.0 to 3.3.1, but found: 3.7.2- (@non-git)
INFO: Build options --host_platform, --platforms, and --toolchain_resolution_debug have changed, discarding analysis cache.
warning: unknown setting 'experimental-features'
warning: unknown setting 'extra-trusted-public-keys'
warning: unknown setting 'experimental-features'
warning: unknown setting 'extra-trusted-public-keys'
waiting for lock on '/home/claudio/.cache/nix/tarballs/074yi2s0mzvdpckjc5lnr2s248c3rvf5cx5niglkqaqiw0smdfhl-file'...
warning: unknown setting 'experimental-features'
warning: unknown setting 'extra-trusted-public-keys'
waiting for lock on '/home/claudio/.cache/nix/tarballs/074yi2s0mzvdpckjc5lnr2s248c3rvf5cx5niglkqaqiw0smdfhl-file'...
these derivations will be built:
  /nix/store/hvch8nq9df6w6wiv08my9p35di4655bc-posix-toolchain.drv
  /nix/store/qgihfdkyf4lvnppq91achb25ah27ii26-bazel-nixpkgs-posix-toolchain.drv
these paths will be fetched (0.15 MiB download, 0.15 MiB unpacked):
  /nix/store/0nab63qw5a87mpl94gci4aywmml41px0-bash-4.4-p23-man
  /nix/store/3lv5swbfq9fspzwac0n99cq65rdqf46p-gnumake-4.3-man
  /nix/store/fjrvp7ngp4fzshdbz44p7yyr382rzzyw-gawk-5.1.0-man
  /nix/store/x6cgr428209324hwx52v2x9bfdgalb7z-gzip-1.10-man
these paths will be fetched (417.97 MiB download, 681.76 MiB unpacked):
  /nix/store/p21v8k2jzmkvr2zlqpv1y11jrdsfm43w-openjdk-16+36
/nix/store/d7m34xx275d6pwjsh7l43rqkzhpn6pz1-bazel-nixpkgs-posix-toolchain
these paths will be fetched (140.38 MiB download, 1848.82 MiB unpacked):
  /nix/store/1pfdb37qm067ib7dnj5arbhl5q2kam3l-libffi-3.3-dev
  /nix/store/afplpil77lg20xn1wnnbbbgk4z1s31n7-ghc-9.0.1
  /nix/store/f2psw0phlmp7h7gk14rfsqdmjz4d1arb-gmp-6.2.1-dev
  /nix/store/fcs3ial6x1jqnan2yfgcr64g0xiin8bj-ghc-9.0.1-doc
/nix/store/afplpil77lg20xn1wnnbbbgk4z1s31n7-ghc-9.0.1
warning: unknown setting 'experimental-features'
warning: unknown setting 'extra-trusted-public-keys'
warning: unknown setting 'experimental-features'
warning: unknown setting 'extra-trusted-public-keys'
waiting for lock on '/home/claudio/.cache/nix/tarballs/074yi2s0mzvdpckjc5lnr2s248c3rvf5cx5niglkqaqiw0smdfhl-file'...
these derivations will be built:
  /nix/store/9z6njgfxi76fsc8a3bwbipz29rfl0m4m-bazel-nixpkgs-python-toolchain.drv
/nix/store/prc9nr0r27bqhlixpcri8i7h0hqargvp-bazel-nixpkgs-python-toolchain
warning: unknown setting 'experimental-features'
warning: unknown setting 'extra-trusted-public-keys'
/nix/store/prc9nr0r27bqhlixpcri8i7h0hqargvp-bazel-nixpkgs-python-toolchain
these paths will be fetched (10.97 MiB download, 214.79 MiB unpacked):
  /nix/store/in621vh2kj0ayqa6qc9pqnjvx6hzj5h5-glibc-locales-2.32-46
/nix/store/in621vh2kj0ayqa6qc9pqnjvx6hzj5h5-glibc-locales-2.32-46
/nix/store/p21v8k2jzmkvr2zlqpv1y11jrdsfm43w-openjdk-16+36
INFO: Analyzed 21 targets (19 packages loaded, 1581 targets configured).
INFO: Found 21 targets...
INFO: Elapsed time: 269.372s, Critical Path: 6.04s
INFO: 180 processes: 142 internal, 34 linux-sandbox, 4 worker.
INFO: Build completed successfully, 180 total actions

Note, that this uses Bazel 3.7.2 currently. Using Bazel 6 might not be possible because of some incompatible changes between those versions (PRs welcome!).

\edit: I have created PR https://github.com/tweag/clodl/pull/47 to make this work out of the box.

facundominguez commented 8 months ago

@evanlu14, does the answer from @avdv help?