Build directories are included in the RPATH of lib_lightgbm_swig.so

[nviets]

Description

Build directories are included in the RPATH of lib_lightgbm_swig.so. Check the rpath of lib_lightgbm_swig.so to find a reference to /build/source.

[jameslamb]

Thanks for using LightGBM.

Can you please provide some more information?

• what version of LightGBM are you using?
• what commands did you use to install it?
• what operating system and compiler are you using?
• what commands are you running to check the RPATH entries, what specific content do you see there, and why is it problematic?

We'd be happy to help you, but you need to provide enough information for us to investigate this.

[nviets]

Hi @jameslamb - thanks for your quick response. I noticed this when updating the lightgbm build to 4.1.0 in nixpkgs. See the referenced PR above and the build log below. You could reproduce the build on my branch with something like:

nix build --impure --expr "with import ./. { }; let lgb = lightgbm.override{javaWrapper = true;}; in lgb"

@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/41yf0amc91n7wbrgklpanzz4gwr1fjxh-source
source root is source
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
@nix { "action": "setPhase", "phase": "updateAutotoolsGnuConfigScriptsPhase" }
updateAutotoolsGnuConfigScriptsPhase
@nix { "action": "setPhase", "phase": "configurePhase" }
configuring
fixing cmake files...
cmake flags: -DCMAKE_FIND_USE_SYSTEM_PACKAGE_REGISTRY=OFF -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF -DCMAKE_INSTALL_LOCALEDIR=/nix/store
/678ps1pgix78qx0mb5lnimgz152xn869-lightgbm-4.1.0/share/locale -DCMAKE_INSTALL_LIBEXECDIR=/nix/store/678ps1pgix78qx0mb5lnimgz152xn869-lightgbm-4.1.0/libexec -DCMAKE_INSTALL_LIBDIR=/nix/store/678ps1pgix78qx0mb5lnimgz152
xn869-lightgbm-4.1.0/lib -DCMAKE_INSTALL_DOCDIR=/nix/store/678ps1pgix78qx0mb5lnimgz152xn869-lightgbm-4.1.0/share/doc/lightgbm -DCMAKE_INSTALL_INFODIR=/nix/store/678ps1pgix78qx0mb5lnimgz152xn869-lightgbm-4.1.0/share/in
fo -DCMAKE_INSTALL_MANDIR=/nix/store/678ps1pgix78qx0mb5lnimgz152xn869-lightgbm-4.1.0/share/man -DCMAKE_INSTALL_OLDINCLUDEDIR=/nix/store/678ps1pgix78qx0mb5lnimgz152xn869-lightgbm-4.1.0/include -DCMAKE_INSTALL_INCLUDEDI
R=/nix/store/678ps1pgix78qx0mb5lnimgz152xn869-lightgbm-4.1.0/include -DCMAKE_INSTALL_SBINDIR=/nix/store/678ps1pgix78qx0mb5lnimgz152xn869-lightgbm-4.1.0/sbin -DCMAKE_INSTALL_BINDIR=/nix/store/678ps1pgix78qx0mb5lnimgz15
2xn869-lightgbm-4.1.0/bin -DCMAKE_INSTALL_NAME_DIR=/nix/store/678ps1pgix78qx0mb5lnimgz152xn869-lightgbm-4.1.0/lib -DCMAKE_POLICY_DEFAULT_CMP0025=NEW -DCMAKE_OSX_SYSROOT= -DCMAKE_FIND_FRAMEWORK=LAST -DCMAKE_STRIP=/nix/
store/ygi2qf01dyx5015lqlqshrdh9gqijkxa-gcc-wrapper-12.3.0/bin/strip -DCMAKE_RANLIB=/nix/store/ygi2qf01dyx5015lqlqshrdh9gqijkxa-gcc-wrapper-12.3.0/bin/ranlib -DCMAKE_AR=/nix/store/ygi2qf01dyx5015lqlqshrdh9gqijkxa-gcc-w
rapper-12.3.0/bin/ar -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DCMAKE_INSTALL_PREFIX=/nix/store/678ps1pgix78qx0mb5lnimgz152xn869-lightgbm-4.1.0 -DUSE_SWIG=ON
-- The C compiler identification is GNU 12.3.0
-- The CXX compiler identification is GNU 12.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /nix/store/ygi2qf01dyx5015lqlqshrdh9gqijkxa-gcc-wrapper-12.3.0/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /nix/store/ygi2qf01dyx5015lqlqshrdh9gqijkxa-gcc-wrapper-12.3.0/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found SWIG: /nix/store/awl9xyqql8m9p30rivlk3ri097bibinl-swig-3.0.12/bin/swig (found version "3.0.12")
-- Found Java: /nix/store/f1nj8bk8p063ajjjvkc3jrbcfca81mqa-openjdk-19.0.2+7/bin/java (found version "19.0.2")
-- Found JNI: /nix/store/f1nj8bk8p063ajjjvkc3jrbcfca81mqa-openjdk-19.0.2+7/include  found components: AWT JVM
-- Found OpenMP_C: -fopenmp (found version "4.5")
-- Found OpenMP_CXX: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")
-- Performing Test MM_PREFETCH
-- Performing Test MM_PREFETCH - Success
-- Using _mm_prefetch
-- Performing Test MM_MALLOC
-- Performing Test MM_MALLOC - Success
-- Using _mm_malloc
CMake Warning (dev) at /nix/store/p6nyqsw5qbg114gprcj4ci8j4nx9nhnb-cmake-3.26.4/share/cmake-3.26/Modules/UseSWIG.cmake:809 (message):
  Policy CMP0078 is not set: UseSWIG generates standard target names.  Run
  "cmake --help-policy CMP0078" for policy details.  Use the cmake_policy
  command to set the policy and suppress this warning.

Call Stack (most recent call first):
  CMakeLists.txt:466 (swig_add_library)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at /nix/store/p6nyqsw5qbg114gprcj4ci8j4nx9nhnb-cmake-3.26.4/share/cmake-3.26/Modules/UseSWIG.cmake:651 (message):
  Policy CMP0086 is not set: UseSWIG honors SWIG_MODULE_NAME via -module
  flag.  Run "cmake --help-policy CMP0086" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

Call Stack (most recent call first):
  /nix/store/p6nyqsw5qbg114gprcj4ci8j4nx9nhnb-cmake-3.26.4/share/cmake-3.26/Modules/UseSWIG.cmake:922 (SWIG_ADD_SOURCE_TO_MODULE)
  CMakeLists.txt:466 (swig_add_library)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Configuring done (1.9s)
-- Generating done (0.0s)
CMake Warning:
  Manually-specified variables were not used by the project:

    BUILD_TESTING
    CMAKE_EXPORT_NO_PACKAGE_REGISTRY
    CMAKE_INSTALL_BINDIR
    CMAKE_INSTALL_DOCDIR
    CMAKE_INSTALL_INCLUDEDIR
    CMAKE_INSTALL_INFODIR
    CMAKE_INSTALL_LIBDIR
    CMAKE_INSTALL_LIBEXECDIR
    CMAKE_INSTALL_LOCALEDIR
    CMAKE_INSTALL_MANDIR
    CMAKE_INSTALL_OLDINCLUDEDIR
    CMAKE_INSTALL_SBINDIR
    CMAKE_POLICY_DEFAULT_CMP0025

-- Build files have been written to: /build/source/build
cmake: enabled parallel building
cmake: enabled parallel installing
@nix { "action": "setPhase", "phase": "buildPhase" }
building
build flags: -j192 SHELL=/nix/store/ir0j7zqlw9dc49grmwplppc7gh0s40yf-bash-5.2-p15/bin/bash
[  1%] Building CXX object CMakeFiles/lightgbm_capi_objs.dir/src/c_api.cpp.o
[  3%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/boosting/boosting.cpp.o
[  5%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/boosting/cuda/cuda_score_updater.cpp.o
[  6%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/boosting/gbdt_model_text.cpp.o
[  8%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/boosting/gbdt_prediction.cpp.o
[ 12%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/boosting/prediction_early_stop.cpp.o
[ 12%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/boosting/gbdt.cpp.o
[ 15%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/boosting/sample_strategy.cpp.o
[ 15%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/cuda/cuda_utils.cpp.o
[ 17%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/bin.cpp.o
[ 18%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/config.cpp.o
[ 20%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/config_auto.cpp.o
[ 22%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/cuda/cuda_column_data.cpp.o
[ 24%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/cuda/cuda_metadata.cpp.o
[ 25%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/cuda/cuda_row_data.cpp.o
[ 27%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/cuda/cuda_tree.cpp.o
[ 29%] Swig compile swig/lightgbmlib.i for java
[ 31%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/dataset.cpp.o
[ 32%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/dataset_loader.cpp.o
[ 34%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/file_io.cpp.o
[ 36%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/json11.cpp.o
[ 37%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/metadata.cpp.o
[ 39%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/parser.cpp.o
[ 41%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/train_share_states.cpp.o
[ 43%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/tree.cpp.o
[ 44%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/metric/cuda/cuda_binary_metric.cpp.o
[ 46%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/metric/cuda/cuda_pointwise_metric.cpp.o
[ 48%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/metric/cuda/cuda_regression_metric.cpp.o
[ 50%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/metric/dcg_calculator.cpp.o
[ 51%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/metric/metric.cpp.o
[ 53%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/network/linkers_mpi.cpp.o
[ 55%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/network/linker_topo.cpp.o
[ 56%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/network/linkers_socket.cpp.o
[ 58%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/network/network.cpp.o
[ 60%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/objective/cuda/cuda_binary_objective.cpp.o
[ 62%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/objective/cuda/cuda_multiclass_objective.cpp.o
[ 63%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/objective/cuda/cuda_rank_objective.cpp.o
[ 65%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/objective/cuda/cuda_regression_objective.cpp.o
[ 67%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/objective/objective_function.cpp.o
[ 68%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_best_split_finder.cpp.o
[ 70%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_data_partition.cpp.o
[ 72%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_histogram_constructor.cpp.o
[ 74%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_leaf_splits.cpp.o
[ 75%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_single_gpu_tree_learner.cpp.o
[ 77%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/data_parallel_tree_learner.cpp.o
[ 79%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/feature_parallel_tree_learner.cpp.o
[ 81%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/gpu_tree_learner.cpp.o
[ 82%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/gradient_discretizer.cpp.o
[ 84%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/linear_tree_learner.cpp.o
[ 86%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/serial_tree_learner.cpp.o
[ 87%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/tree_learner.cpp.o
[ 89%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/voting_parallel_tree_learner.cpp.o
[ 89%] Built target _lightgbm_swig_swig_compilation
[ 89%] Built target lightgbm_capi_objs
[ 89%] Built target lightgbm_objs
[ 91%] Building CXX object CMakeFiles/lightgbm.dir/src/main.cpp.o
[ 93%] Linking CXX shared library /build/source/lib_lightgbm.so
[ 94%] Building CXX object CMakeFiles/lightgbm.dir/src/application/application.cpp.o
[ 94%] Built target _lightgbm
[ 96%] Building CXX object CMakeFiles/_lightgbm_swig.dir/java/lightgbmlibJAVA_wrap.cxx.o
In file included from /build/source/build/java/lightgbmlibJAVA_wrap.cxx:237:
/build/source/include/../include/LightGBM/c_api.h: In function 'void Java_com_microsoft_ml_lightgbm_lightgbmlibJNI_LGBM_1SetLastError(JNIEnv*, jclass, jstring)':
/build/source/include/../include/LightGBM/c_api.h:1569:42: warning: '%s' directive argument is null [-Wformat-truncation=]
 1569 |   snprintf(LastErrorMsg(), err_buf_len, "%s", msg);
      |                                          ^~
[ 98%] Linking CXX shared module /build/source/lib_lightgbm_swig.so
java/CSRDirect.java:24: warning: [removal] finalize() in Object has been deprecated and marked for removal
  protected void finalize() {
                 ^
java/doubleChunkedArray.java:24: warning: [removal] finalize() in Object has been deprecated and marked for removal
  protected void finalize() {
                 ^
java/floatChunkedArray.java:24: warning: [removal] finalize() in Object has been deprecated and marked for removal
  protected void finalize() {
                 ^
java/int32ChunkedArray.java:24: warning: [removal] finalize() in Object has been deprecated and marked for removal
  protected void finalize() {
                 ^
4 warnings
[ 98%] Built target _lightgbm_swig
[100%] Linking CXX executable /build/source/lightgbm
[100%] Built target lightgbm
buildPhase completed in 1 minutes 26 seconds
@nix { "action": "setPhase", "phase": "installPhase" }
installing
@nix { "action": "setPhase", "phase": "fixupPhase" }
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/678ps1pgix78qx0mb5lnimgz152xn869-lightgbm-4.1.0
shrinking /nix/store/678ps1pgix78qx0mb5lnimgz152xn869-lightgbm-4.1.0/lib/lib_lightgbm.so
shrinking /nix/store/678ps1pgix78qx0mb5lnimgz152xn869-lightgbm-4.1.0/bin/lightgbm
shrinking /nix/store/678ps1pgix78qx0mb5lnimgz152xn869-lightgbm-4.1.0/com/microsoft/ml/lightgbm/linux/x86_64/lib_lightgbm.so
shrinking /nix/store/678ps1pgix78qx0mb5lnimgz152xn869-lightgbm-4.1.0/com/microsoft/ml/lightgbm/linux/x86_64/lib_lightgbm_swig.so
checking for references to /build/ in /nix/store/678ps1pgix78qx0mb5lnimgz152xn869-lightgbm-4.1.0...
RPATH of binary /nix/store/678ps1pgix78qx0mb5lnimgz152xn869-lightgbm-4.1.0/com/microsoft/ml/lightgbm/linux/x86_64/lib_lightgbm_swig.so contains a forbidden reference to /build/
/nix/store/rk3g4mxrs84bv1vanqnzpdkhm73ksr0z-stdenv-linux/setup: line 87: pop_var_context: head of shell_variables not a function context
/nix/store/rk3g4mxrs84bv1vanqnzpdkhm73ksr0z-stdenv-linux/setup: line 1423: pop_var_context: head of shell_variables not a function context
/nix/store/rk3g4mxrs84bv1vanqnzpdkhm73ksr0z-stdenv-linux/setup: line 1596: pop_var_context: head of shell_variables not a function context

[jameslamb]

Thanks for that. So the relevant issue, specifically, is this?

RPATH of binary /nix/store/678ps1pgix78qx0mb5lnimgz152xn869-lightgbm-4.1.0/com/microsoft/ml/lightgbm/linux/x86_64/lib_lightgbm_swig.so contains a forbidden reference to /build/

Are you interested in contributing a fix here?

[nviets]

Yes, this is what nix is complaining about. My rudimentary understanding is that the inclusion of temp directories like /build/ creates a security vulnerability. You can find the source of nix's audit here.

Happy to have a look, but I'm afraid I don't know much about swig's mechanics. Has anything changed in the swig build that might have led to the inclusion of /build/?

In the nixpkgs build of lightgbm, I think I could add something like "-DCMAKE_SKIP_BUILD_RPATH=ON" to avoid the issue, but I had read it's better to remove the temp directory upstream. See this. If the issue I opened here is a red herring, the extra argument will probably suffice.

[jameslamb]

Has anything changed in the swig build that might have led to the inclusion of /build/?

I'm not aware of anything specifically related to this. You can try looking through the commit history if you'd like: https://github.com/microsoft/LightGBM/commits/master/CMakeLists.txt

Otherwise...yes if you want this changed but aren't sure how to detect it outside of nix tooling, and aren't sure why it's important that it be changed, it probably isn't something we'll prioritize very high. So the best way to see this changed in LightGBM is to do the research yourself and submit pull requests changing the build.

You can @ me here any time with questions as you do that and I'll try to help.

[nviets]

Thanks again, @jameslamb. The CMake flag worked in my case, which cleanly removes the RPATH issue. Closing this issue.

[jameslamb]

Alright sure, no problem. Thabks for re-packaging LightGBM for nix, come back and contribute any time 👋🏻

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.