ExecutionEngine produces expected result but also error Failed to create MemoryBuffer for: libmlir_c_runner_utils.so

[bixia1]

Bugzilla Link	49236
Version	unspecified
OS	All
CC	@jpienaar,@River707

Extended Description

I first set up LD_LIBRARY_PATH so that dlopen can find libmlir_c_runner_utils.so. Then I run "mlir-cpu-runner -shared-libs=libmlir_c_runner_utils.so ". The command runs fine and produces the expected result. However, I also got this bogus error message:

Failed to create MemoryBuffer for: libmlir_c_runner_utils.so Error: No such file or directory

It is a bogus error message for my case, but could be a real problem for other situations.

The error message is from this code block, which doesn't consider LD_LIBRARY_PATH when trying to open the shared library: https://github.com/llvm/llvm-project/blob/48d2e04152366890e0b219a5f7c6f5b4905af480/mlir/lib/ExecutionEngine/ExecutionEngine.cpp#L260-L265

On the other hand, the shared library was correctly loaded from this code block, as it takes LD_LIBRARY_PATH into consideration -- which is the reason that my command works: https://github.com/llvm/llvm-project/blob/2628e9146120507b2cf025f5c4ccc857cc3724c4/mlir/lib/ExecutionEngine/JitRunner.cpp#L175

You can use mlir/integration_test in the current LLVM tree to demonstrate this:

first set up PATH for mlir-opt and mlir-cpu-runner, LD_LIBRARY_PATH for libmlir_c_runner_utils.so

mlir-opt mlir/integration_test/Sparse/CPU/sparse_sum.mlir \ --test-sparsification="lower" --convert-linalg-to-loops --func-bufferize --tensor-constant-bufferize --tensor-bufferize --finalizing-bufferize --convert-scf-to-std --convert-vector-to-llvm --convert-std-to-llvm | \ TENSOR0="mlir/integration_test/data/test.mtx" \ -cpu-runner -e entry -entry-point-result=void -shared-libs=libmlir_c_runner_utils.so

Output from the command: Failed to create MemoryBuffer for: libmlir_c_runner_utils.so Error: No such file or directory 28.2

If you unset LD_LIBRARY_PATH and run the same command again, you should get a different error and no output from the mlir program being jit: Failed to create MemoryBuffer for: libmlir_c_runner_utils.so Error: No such file or directory JIT session error: Symbols not found: [ delSparseTensor, sparseValsF64, getTensorFilename, sparsePtrsI64, newSparseTensor, printF64, printNewline, sparseIndxsI64 ] Error: Failed to materialize symbols: { (main, { _mlir_entry, entry, kernel_sum_reduce, _mlir_kernel_sum_reduce }) }

[bixia1]

mentioned in issue llvm/llvm-bugzilla-archive#49237

[bixia1]

Why not supporting LD_LIBRARY_PATH is better than supporting it?

As a user, I would prefer supporting LD_LIBRARY_PATH.

From the implementation point of view, is there an easy way to make dlopen not look at LD_LIBRARY_PATH? I am not aware of such a way, but you can always save, change and restore its value.

[bixia1]

Bug llvm/llvm-bugzilla-archive#49237 has been marked as a duplicate of this bug.

[joker-eph]

Should we just require a valid path to the library and reject LD_LIBRARY_PATH?

[bixia1]

assigned to @joker-eph