LLVM ERROR: Program used external function '__multi3' which could not be resolved!

[nalimilan]

I used to be able to run tests without failure on ARM when building the Fedora RPM for Julia 1.0.0, after skipping some tests (numbers, ccall, vecelement, stress, errorshow and threads, but let's keep that for another issue). But with 1.0.1, there's a new failure in the Distributed test:

      From worker 32:   LLVM ERROR: Program used external function '__multi3' which could not be resolved!
      From worker 31:   LLVM ERROR: Program used external function '__multi3' which could not be resolved!
      From worker 30:   LLVM ERROR: Program used external function '__multi3' which could not be resolved!
      From worker 29:   LLVM ERROR: Program used external function '__multi3' which could not be resolved!
      From worker 5:    Worker 32 terminated.
      From worker 5:    Worker 29 terminated.ERROR: 
      From worker 5:    LoadError: ProcessExitedException()Worker 30 terminated.
      From worker 5:    Stacktrace:
      From worker 5:    Worker 31 terminated.
      From worker 5:    
      From worker 5:     [1] try_yieldto(::typeof(Base.ensure_rescheduled), ::Base.RefValue{Task}) at ./event.jl:196
      From worker 5:     [2] wait() at ./event.jl:255
      From worker 5:     [3] wait(::Condition) at ./event.jl:46
      From worker 5:     [4] wait(::Task) at ./task.jl:188
      From worker 5:     [5] fetch at ./task.jl:202 [inlined]
      From worker 5:     [6] iterate at ./generator.jl:47 [inlined]
      From worker 5:     [7] collect(::Base.Generator{Array{Task,1},typeof(fetch)}) at ./array.jl:619
      From worker 5:     [8] preduce(::Function, ::Function, ::Base.OneTo{UInt64}) at /builddir/build/BUILD/julia/build/usr/share/julia/stdlib/v1.0/Distributed/src/macros.jl:263
      From worker 5:     [9] top-level scope at /builddir/build/BUILD/julia/build/usr/share/julia/stdlib/v1.0/Distributed/test/distributed_exec.jl:1532
      From worker 5:     [10] include at ./boot.jl:317 [inlined]
      From worker 5:     [11] include_relative(::Module, ::String) at ./loading.jl:1041
      From worker 5:     [12] include(::Module, ::String) at ./sysimg.jl:29
      From worker 5:     [13] exec_options(::Base.JLOptions) at ./client.jl:229
      From worker 5:     [14] _start() at ./client.jl:421
      From worker 5:    in expression starting at /builddir/build/BUILD/julia/build/usr/share/julia/stdlib/v1.0/Distributed/test/distributed_exec.jl:1531

This is with USE_SYSTEM_LLVM=0, but with custom CFLAGS (which are the same as those used to build 1.0.0). Full log at https://kojipkgs.fedoraproject.org//work/tasks/5654/29975654/build.log

[vchuravy]

Hm that is peculiar __multi3 is part of glibc, that is a MUL_I128 has the libc configuration changed?

[yuyichao]

No this is a part of gcc/compiler_rt. GCC doesn't seem to have this for ARM though.

Also note that the signature of this function is long long __multi3 (long long a, long long b), which is 64bit on arm. I don't see why LLVM should ask for this.

[nalimilan]

FWIW, I've found this: https://github.com/numba/numba/issues/969 and https://github.com/numba/numba/pull/1068.

It is necessary for working around an issue in LLVM (see issue #969). The symbol does not exist in 32-bit platform, and should not be used by LLVM. However, optimization passes will create i65 multiplication that is then lowered to __multi3.

The LLVM bug is still open: https://bugs.llvm.org/show_bug.cgi?id=20871

Any idea which commits I could try to revert?

[vchuravy]

Looks like the road of least resistant it to do https://reviews.llvm.org/D38668 for ARM 32bit as well...

[vtjnash]

All the code for this exists in https://github.com/llvm-project/llvm-project/tree/master/compiler-rt, if we build it with clang and set -DCRT_HAS_128BIT. How do we feel about building and linking that? Unfortunately, the TargetLowering class (ala https://reviews.llvm.org/D38668) is not accessible to us to adjust to these issues directly.

[vchuravy]

Previous attempt at that was https://github.com/JuliaLang/julia/pull/18734