ExecutionEngine produces expected result but also error Failed to create MemoryBuffer for: libmlir_c_runner_utils.so

[Quuxplusone]

Bugzilla Link	PR49236
Status	NEW
Importance	P enhancement
Reported by	Bixia Zheng (bixia@google.com)
Reported on	2021-02-17 17:32:55 -0800
Last modified on	2021-02-18 08:04:24 -0800
Version	unspecified
Hardware	Other All
CC	jpienaar@google.com, riddleriver@gmail.com
Fixed by commit(s)
Attachments
Blocks
Blocked by
See also

I first set up LD_LIBRARY_PATH so that dlopen can find
libmlir_c_runner_utils.so. Then I run "mlir-cpu-runner -shared-
libs=libmlir_c_runner_utils.so <other options>". The command runs fine and
produces the expected result. However, I also got this bogus error message:

Failed to create MemoryBuffer for: libmlir_c_runner_utils.so
Error: No such file or directory

It is a bogus error message for my case, but could be a real problem for other
situations.

The error message is from this code block, which doesn't consider
LD_LIBRARY_PATH when trying to open the shared library:
https://github.com/llvm/llvm-project/blob/48d2e04152366890e0b219a5f7c6f5b4905af480/mlir/lib/ExecutionEngine/ExecutionEngine.cpp#L260-L265

On the other hand, the shared library was correctly loaded from this code
block, as it takes LD_LIBRARY_PATH into consideration -- which is the reason
that my command works:
https://github.com/llvm/llvm-project/blob/2628e9146120507b2cf025f5c4ccc857cc3724c4/mlir/lib/ExecutionEngine/JitRunner.cpp#L175

You can use mlir/integration_test in the current LLVM tree to demonstrate this:
# first set up PATH for mlir-opt and mlir-cpu-runner, LD_LIBRARY_PATH for
libmlir_c_runner_utils.so
mlir-opt mlir/integration_test/Sparse/CPU/sparse_sum.mlir \
--test-sparsification="lower" --convert-linalg-to-loops --func-bufferize --
tensor-constant-bufferize --tensor-bufferize --finalizing-bufferize --convert-
scf-to-std --convert-vector-to-llvm --convert-std-to-llvm | \
TENSOR0="mlir/integration_test/data/test.mtx" \
-cpu-runner -e entry -entry-point-result=void -shared-
libs=libmlir_c_runner_utils.so

Output from the command:
Failed to create MemoryBuffer for: libmlir_c_runner_utils.so
Error: No such file or directory
28.2

If you unset LD_LIBRARY_PATH and run the same command again, you should get a
different error and no output from the mlir program being jit:
Failed to create MemoryBuffer for: libmlir_c_runner_utils.so
Error: No such file or directory
JIT session error: Symbols not found: [ delSparseTensor, sparseValsF64,
getTensorFilename, sparsePtrsI64, newSparseTensor, printF64, printNewline,
sparseIndxsI64 ]
Error: Failed to materialize symbols: { (main, { _mlir_entry, entry,
kernel_sum_reduce, _mlir_kernel_sum_reduce }) }

[Quuxplusone]

Should we just require a valid path to the library and reject LD_LIBRARY_PATH?

[Quuxplusone]

_Bug 49237 has been marked as a duplicate of this bug._

[Quuxplusone]

Why not supporting LD_LIBRARY_PATH is better than supporting it?

As a user, I would prefer supporting LD_LIBRARY_PATH.

From the implementation point of view, is there an easy way to make dlopen not look at LD_LIBRARY_PATH? I am not aware of such a way, but you can always save, change and restore its value.