pkg_tar rule break the symlink and cause link error at runtime on MacOS

[bfsoyc]

The attribute include_runfiles = True is useful for me since I would like to keep things organized as they are in workspace dir tree.

However, I found the pkg_tar rule keep two copys of dynamic lib:

• One in the dir tree as we declare them by BUILD file.
• Another is located in _solib_darwin_arm64 for MacOS (_solib_k8 for Linux).

In the project workspace, the one in bazel-bin/_solib_darwin_arm64 is actually a symlink to the one in dir tree. However, in package tar both are duplicated files.

It introduces linking error when I run the binary. (The binary links the .so lib with relative path by using @loader_path)

This behavior makes no sense to me.

Could anyone tell if it's desired behavior or not? many thanks.

An simple example:

- bin/executable
- lib
  |- module_a
    |- a.so
  |- module_b
    |- deeper_dir
      |- b.so 

We have executable deps on b and b deps on a.

After pack with pkg_tar, we have the following in tar ball:

- _solib_darwin_arm64
  |- encoded_path_to_a
    |- a.so. # no longer a symlink to lib/module_a/a.so
  |- encoded_path_to_b
    |- b.so. # no longer a symlink
- bin/executable
- lib
  |- module_a
    |- a.so
  |- module_b
    |- deeper_dir
      |- b.so 

When running the executable binary, it try to link _solib_darwin_arm64/encoded_path_to_b/b.so, which is no longer a symlink but the same content with the origin one. Therefore every relative path broken when b.so trying to load a.so by relative path.

[dieortin]

Are those shared libraries prebuilt? Or are you building them with Bazel?

In my case, I’m packaging binaries with lots of shared libraries and haven’t encountered this problem. But I don’t have any prebuilt .so in my sources tree: most libraries are built by Bazel, and those that are not are included from external repositories.

[bfsoyc]

Are those shared libraries prebuilt? Or are you building them with Bazel?

In my case, I’m packaging binaries with lots of shared libraries and haven’t encountered this problem. But I don’t have any prebuilt .so in my sources tree: most libraries are built by Bazel, and those that are not are included from external repositories.

Thanks for your attention at first.

We are working on a huge project, where there's some prebuilt .so library.

However, I think that not the point. As long as we have a binary require dynamic linking to lib B, and lib B require dynamic linking to A. The linking phase will fail as I mention above. (I use otool -L to query the dynamic link of a binary or .so.)

# the way we generate a lib B, needed to link lib A at runtime.
cc_binary(
    name = "libb.so",
    srcs = ["b_static"],
    linkshared = 1,
    linkstatic = 1,
    visibility = ["//visibility:public"],
    deps = [
        "//lib/module_a:a",  # this explicit deps forces B linking to A, such that we can reproduce the problem.
    ],
)

So my point is, if we can preserve the object in _solib_darwin_arm64 symlink to the object in real dir tree. The behavior can remain the same as they were in Bazel workspace.

[aiuto]

I don't know enough about how mac dynamic linking as you used it (with both linkstatic and linkshard) is supposed to work to comment yet. To me, the cc_binary example above seems like module_a should be statically linked. Can you point to a small reproduction? Is module_a a cc_shared_library?

[bfsoyc]

@aiuto , thanks for your attention. FYI: https://github.com/bfsoyc/bar_repo/tree/main

bazel build //bin:deploy

tar -zxf bazel-bin/bin/deploy.tar --directory {test_dir}

cd {test_dir}
./bin/main  # this should crash due to library loading failure.