Closed sogartar closed 1 year ago
This is likely a codegen issue. You can try debugging it by enabling ASAN; our docs seem to be rather poor on that but here's the gist:
--iree-llvm-sanitize=address
and --iree-llvm-link-embedded=false
to iree-compile-DIREE_ENABLE_ASAN=ON
and delete cache/reconfigure(docs are https://github.com/iree-org/iree/blob/67c04b9de75f40b0ad83b949fd0f5c92a6b74dd7/docs/developers/developing_iree/sanitizers.md but the ASAN section doesn't mention the iree-compile flags and the TSAN section while similar is very focused on internal tests and such and not what you do when compiling your own programs)
I tried compiling the module with
iree-compile \
mnist_train.mlir \
--iree-input-type=mhlo \
--iree-hal-target-backends=llvm-cpu \
--iree-llvm-sanitize=address \
--iree-llvm-link-embedded=false \
-o mnist_train.vmfb
I got this error
ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol __asan_option_detect_stack_use_after_return; recompile with -fPIC
>>> defined in /tmp/mnist_module_linked_llvm_cpu-e64538.o
>>> referenced by mnist_module_linked_llvm_cpu
>>> /tmp/mnist_module_linked_llvm_cpu-e64538.o:(update_dispatch_18_matmul_1024x784x128)
Linking failed; escaped command line returned exit code 256:
/usr/local/bin/ld.lld -o /tmp/mnist_module_linked_llvm_cpu-e64538.so -nostdlib -static -shared /tmp/mnist_module_linked_llvm_cpu-e64538.o
I think I have to pass -fPIC
to the module object file compilation flags.
It seems that PIC is already specified https://github.com/iree-org/iree/blob/e5d71f5a0b2635cfaf5153364cc699434ac92eb7/compiler/src/iree/compiler/Dialect/HAL/Target/LLVM/LLVMIRPasses.cpp#L39
That seems to be a separate issue. Can you try dropping iree-llvm-link-embedded=false
.
Without it I get
lld: error: undefined symbol: __asan_stack_free_8
>>> referenced by mnist_module_linked_llvm_cpu
>>> /tmp/mnist_module_linked_llvm_cpu-ab1148.o:(update_dispatch_18_matmul_1024x784x128)
Linking failed; escaped command line returned exit code 256:
LLD_VERSION=IREE /home/petkantchin/ws/iree/build/ninja/Debug/third_party/llvm-project/llvm/bin/lld -flavor gnu -o /tmp/mnist_module_linked_llvm_cpu-ab1148.so --build-id=none -nostdlib -static -shared --no-undefined --no-allow-shlib-undefined --allow-multiple-definition --gc-sections -z now -z relro --discard-all --icf=all --ignore-data-address-equality --ignore-function-address-equality --hash-style=sysv /tmp/mnist_module_linked_llvm_cpu-ab1148.o
I think it needs to link to libasan
.
I am closing this. It seems it got fixed somehow. I don't see this problem on 8bedd4b249caa06ed346557214c016fe9b685dbf.
This was likely related to the stack issues that we've recently been squashing. Glad it's working now - please let us know if you hit more issues of this type and we can dig in!
What happened?
I ran into a segmentation fault when trying to run mnist_train.mlir.tar.gz.
The stack trace before getting the error is
Steps to reproduce your issue
iree-run-module \ --module_file=mnist_train.vmfb \ --device=local-sync \ --entry_function=initialize \ "--function_input=2xui32=[232, 843]"