search

JuliaLang / PackageCompiler.jl

Compile your Julia Package
https://julialang.github.io/PackageCompiler.jl/dev/
MIT License
1.42k stars 190 forks source link

Undefined symbol error with VectorizationBase 0.14.12 on Julia 1.5.2 #487

Closed mmiller-max closed 3 years ago

mmiller-max commented 3 years ago

This may be similar to #445 and #456 but this is occurring for me in Julia 1.5.2 if a project or project dependency uses VectorizationBase v0.14.12. The same projects are fine with v0.13.12

MWE

Building this package:

module TestPkg

using VectorizationBase

greet() = print("Hello World!")

end # module

With this Project.toml:

name = "TestPkg"
uuid = "5e43b693-7ccd-488e-962a-6f83670885ee"
version = "0.1.0"

[deps]
PackageCompiler = "9b87118b-4619-50d2-8e1e-99f35a4d4d9d"
VectorizationBase = "3d5dd08c-fd9d-11e8-17fa-ed2836048c2f"

[compat]
VectorizationBase = "0.13.12"

Builds successfully with

create_sysimage([:TestPkg]; sysimage_path="TestPkg.so")

Changing Project.toml to:

name = "TestPkg"
uuid = "5e43b693-7ccd-488e-962a-6f83670885ee"
version = "0.1.0"

[deps]
PackageCompiler = "9b87118b-4619-50d2-8e1e-99f35a4d4d9d"
VectorizationBase = "3d5dd08c-fd9d-11e8-17fa-ed2836048c2f"

[compat]
VectorizationBase = "0.14.12"

Results in this error:

julia> create_sysimage([:TestPkg2]; sysimage_path="TestPkg2.so")
[ Info: PackageCompiler: creating system image object file, this might take a while...
Undefined symbols for architecture x86_64:
  "_jfptr_llvmcall_expr_72687", referenced from:
      _tojlinvoke76774 in jl_qVNoRE.o(text.o)
  "_jfptr_llvmname_72469", referenced from:
      _tojlinvoke76801 in jl_qVNoRE.o(text.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
ERROR: failed process: Process(`gcc -m64 -march=x86-64 -shared -L/Applications/Julia-1.5.app/Contents/Resources/julia/lib -o TestPkg2.so -Wl,-all_load /var/folders/29/f6ksj93n5dz5zwdbz89rt9kh95rtb6/T/jl_qVNoRE.o -ljulia`, ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error at ./process.jl:525 [inlined]
 [2] run(::Cmd; wait::Bool) at ./process.jl:440
 [3] run at ./process.jl:438 [inlined]
 [4] run_with_env at /Users/malmiller/.julia/packages/PackageCompiler/KeZOk/src/PackageCompiler.jl:101 [inlined]
 [5] create_sysimg_from_object_file(::String, ::String) at /Users/malmiller/.julia/packages/PackageCompiler/KeZOk/src/PackageCompiler.jl:497
 [6] create_sysimage(::Array{Symbol,1}; sysimage_path::String, project::String, precompile_execution_file::Array{String,1}, precompile_statements_file::Array{String,1}, incremental::Bool, filter_stdlibs::Bool, replace_default::Bool, cpu_target::String, script::Nothing, base_sysimage::Nothing, isapp::Bool) at /Users/malmiller/.julia/packages/PackageCompiler/KeZOk/src/PackageCompiler.jl:451
 [7] top-level scope at REPL[11]:1

Context that may help

I found this issues when working with a larger package that uses DiffEqBase. In that case, the sys image build successfully (possibly because it was in a Docker Julia v1.5 container) but when loading I got a similar error to #445. Comparing the manifests between a previous working version of this package I found the VectorizationBase version change was the culprit.

Versioninfo

julia> versioninfo()
Julia Version 1.5.2
Commit 539f3ce943 (2020-09-23 23:17 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin18.7.0)
  CPU: Intel(R) Core(TM) i5-8279U CPU @ 2.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, skylake)
Environment:
  JULIA_NUM_THREADS = 8
DilumAluthge commented 3 years ago

cc: @chriselrod

chriselrod commented 3 years ago

I guess git bisect is the best way to find out what's going wrong?

Is there any info on things to be suspicious of?

chriselrod commented 3 years ago

Oddly, I can reproduce when I've added a version of VectorizationBase from the registry, but not when it is deved. create_sysimage always works when deved, meaning git bisect is useless =/.

DilumAluthge commented 3 years ago

You can write a script manually to iterate over commit shas. And then do Pkg.add(Pkg.PackageSpec(name = "VectorizationBase", rev = COMMIT_SHA)).

chriselrod commented 3 years ago

Is there a convenient way to get a vector of commit shas?

EDIT: defined an emacs macro that turned a copy and paste of git log into a vector of commit strings.

chriselrod commented 3 years ago

Maybe I'm doing something wrong, but it keeps working, snapshot of script output:

  Resolving package versions...
Updating `~/Documents/progwork/julia/env/pkgcompiler_vecbase/TestPkg/Project.toml`
  [3d5dd08c] ~ VectorizationBase v0.14.13 `https://github.com/chriselrod/VectorizationBase.jl.git#95c1994` ⇒ v0.14.12 `https://github.com/chriselrod/VectorizationBase.jl.git#b4a3479`
Updating `~/Documents/progwork/julia/env/pkgcompiler_vecbase/TestPkg/Manifest.toml`
  [3d5dd08c] ~ VectorizationBase v0.14.13 `https://github.com/chriselrod/VectorizationBase.jl.git#95c1994` ⇒ v0.14.12 `https://github.com/chriselrod/VectorizationBase.jl.git#b4a3479`
sha = "b4a3479fde575824836d06b4a7b1ee5cff9925d1"
Project TestPkg v0.1.0
Status `~/Documents/progwork/julia/env/pkgcompiler_vecbase/TestPkg/Project.toml`
  [9b87118b] PackageCompiler v1.2.4
  [3d5dd08c] VectorizationBase v0.14.12 `https://github.com/chriselrod/VectorizationBase.jl.git#b4a3479`
[ Info: PackageCompiler: creating system image object file, this might take a while...
succeeded = true

That is the commit of the 0.14.12 release: https://github.com/chriselrod/VectorizationBase.jl/commit/b4a3479fde575824836d06b4a7b1ee5cff9925d1

I have this Julia script to call package compiler:

using PackageCompiler
using Pkg
Pkg.status()
try
    create_sysimage([:TestPkg]; sysimage_path="TestPkg.so")
    exit(0)
catch
    exit(1)
end

and then:


shas = ["37cc93559e0689416062c00f38ce250c36e3977b",
"49a2c5ad37eb425ba89c5e68ac8d2c8cb1b72ba1",
"743212e2a1ce7fe50a5c2d182522e9ea09e1859a",
"55d6767719da52b5be9370c7c66d1434e543cb37",
"fa74949d6d62462386cac249ed2760ca95b291b1",
"52aeb5f07958be203c274c1fd673612de973297c",
"95c1994a295087ac5ec88b095aa6a2acb4eece2b",
"b4a3479fde575824836d06b4a7b1ee5cff9925d1",
"98523a8f01cc3d182e8a089d55e3bf8e5e659799",
"6c94aae6be2e84019e628ea408a645547811d3e4",
"04455624b7cf29c74b70b0e0abda441b1aede38e",
"6c15e62288e41754db90328a82e38338405dd3c6",
"bda85512ec5b8d0abb7fa03a5a3764871542c6a4",
"0fc85e9c92b59e003d078ebf3d759184c31841fb",
"7b1ed5ada2315cf1ba25c4480b2f5b75843fc750",
"0206e8f044dbaab44f039480b7b85742ce7b88c4",
"2f22c9936fa6b513f8ecf65d01b16f14b4c6d141",
"77caf46f05db34fda1a4b55da3bb08ae38a2df29",
"67105aecf114785efad7ec00a0ed1cf2c49e0edc",
"739e61467aac3eb5cb30fe37724dc5bfca957c91",
"6e90140813bb7e066b18eea5a606cf8bd4998ab1",
"cc9e6fd41da9128f7fb7f3ac93fba97d79ee9273",
"3ecbf2733f64ec170296dc42e731d06067391253",
"67eb26b3ec265cb1ee217a54d2dfac947c0df85a",
"8fc9e091d0073e9397d56bd37ba1f762977b0ac5",
"e1065829ed59cc020daa3fa0de1f2e9f31fc873a",
"87989d2006bf8b411ceb234463f73f72649ee512",
"5fc71d7b43fcfff138309fb413eb0040add951c2",
"af60d116efcbfb7b6579c77ac4ae7bf985cbf681",
"b0deb36285975463228fbfa087ccce97dd3d2098",
"642f14656519bfaf7194658b0ed0ab17122f45db",
"870f87f73986c1f30f1e2d740cd6821d2e7ef82e",
"2d793a72f82188fec340a3c33332f7239f5d8575",
"2abed879a356bd1a2e9129a66ddcdfabe68b9708",
"436adbd8511a01e182d46f6a352a250b1a954c42",
"d8ad96d1a49a1e457e6ae6e716750d564388ec72",
"51d535b1a7bf8ebcbe7b65545999467ea97ba81a",
"04a12fb4b5509558150e713fe04005aed789ecf1",
"0e70775e8481420fae601371addad9033f0d66ae",
"5e128ff2ff44615a4d85e2099f1ff2b4531e6c06",
"10c5a19eaca03b23023c84df9eaad6e0714766a1",
"2c3f22c4d04e2de89cc34050ebe5f5b6189c52b1",
"f88b149df807951194645f824c91bf0978b6ddf0",
"fa5cd3a2f103bb585845b69b41a60c9d8599ded1",
"c89ac76d962b9f7ec18d9cefd9579bd0099e8565",
"781606fa68bd900e4b0293f40a70b85f6b9fd09f",
"f83a2dc291e38126af82a922994ea0efd94ede30",
"3e9ffde47f6ff4b0f4a76ed6c236a0cf40194320",
"ac538d2101f0774a6b923d8afc7179bcae89b0db",
"7ee02be90722c22d6e70d83f75b15af7f477a1a3",
"c24f966d3a50f04b1555aaed36b1e5118c36b234",
"12fa5577fc023a482b084a7c74c49dc0f5386747",
"f9c8585718f8bf5d1a9e697cbe028fa6cc36e6f8",
"f682f7bb2dabb086ebff12552be08a06ab105542",
"cf15a1385ac62d3cc546fea32db104c3e2fc9faa",
"cfb98e0b5aa08d55b6f44b7bc04f2bb47b632b08",
"2e457043b8463a49cbd5d0133428e6fd78858709",
"bfde9bfef9d126966d3fd04783030ed41a55ead6",
"6578c0a619f2ae63c8105e8e772fb44045c4bd7a",
"9d02c9e4dcbc22561743a78d54acf2a0c3cade3e",
"f42f38e06bb398d3947304c299c058f3a01114ec",
"9e0b8efae185d94858fe5ff949ed74b1a8b277f9",
"dd7bfcd45e2b4d488e2db79cda92ef564b1c75d8",
"95935bbeaa152dd1ca66f8454a3e25404de37112",
"ac538d2101f0774a6b923d8afc7179bcae89b0db",
"7ee02be90722c22d6e70d83f75b15af7f477a1a3",
"3e9ffde47f6ff4b0f4a76ed6c236a0cf40194320",
"ac538d2101f0774a6b923d8afc7179bcae89b0db",
"7ee02be90722c22d6e70d83f75b15af7f477a1a3",
"c24f966d3a50f04b1555aaed36b1e5118c36b234",
"12fa5577fc023a482b084a7c74c49dc0f5386747",
"f83a2dc291e38126af82a922994ea0efd94ede30",
"3e9ffde47f6ff4b0f4a76ed6c236a0cf40194320",
"ac538d2101f0774a6b923d8afc7179bcae89b0db",
"7ee02be90722c22d6e70d83f75b15af7f477a1a3",
"c24f966d3a50f04b1555aaed36b1e5118c36b234",
"12fa5577fc023a482b084a7c74c49dc0f5386747",
"f9c8585718f8bf5d1a9e697cbe028fa6cc36e6f8",
"f682f7bb2dabb086ebff12552be08a06ab105542",
"cf15a1385ac62d3cc546fea32db104c3e2fc9faa",
"cfb98e0b5aa08d55b6f44b7bc04f2bb47b632b08",
"2e457043b8463a49cbd5d0133428e6fd78858709",
"bfde9bfef9d126966d3fd04783030ed41a55ead6",
"6578c0a619f2ae63c8105e8e772fb44045c4bd7a",
"9d02c9e4dcbc22561743a78d54acf2a0c3cade3e",
"f42f38e06bb398d3947304c299c058f3a01114ec",
"9e0b8efae185d94858fe5ff949ed74b1a8b277f9",
"dd7bfcd45e2b4d488e2db79cda92ef564b1c75d8"]

using Pkg, PackageCompiler
successes = fill(false, length(shas));
for (i,sha) ∈ enumerate(shas)
    Pkg.add(Pkg.PackageSpec(name = "VectorizationBase", rev = sha))
    @show sha
    succeeded = success(run(`/home/chriselrod/Documents/languages/julia-polly/usr/bin/julia --project=/home/chriselrod/Documents/progwork/julia/env/pkgcompiler_vecbase/TestPkg --startup=no /home/chriselrod/Documents/progwork/julia/env/pkgcompiler_vecbase/testpkgcompiler.jl`))
    @show succeeded
    successes[i] = succeeded
end

to switch package versions and call the previous script.

Just tested and realized I actually need try/catch around the run command, because it'd throw an error if building the sys image actually failed.

chriselrod commented 3 years ago

Maybe without having compat restrict to a specific VectorizationBase version, PkgCompiler ignores the package's environment?

chriselrod commented 3 years ago

Anyway, the stack trace you the OP shared:

julia> create_sysimage([:TestPkg2]; sysimage_path="TestPkg2.so")
[ Info: PackageCompiler: creating system image object file, this might take a while...
Undefined symbols for architecture x86_64:
  "_jfptr_llvmcall_expr_72687", referenced from:
      _tojlinvoke76774 in jl_qVNoRE.o(text.o)
  "_jfptr_llvmname_72469", referenced from:
      _tojlinvoke76801 in jl_qVNoRE.o(text.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
ERROR: failed process: Process(`gcc -m64 -march=x86-64 -shared -L/Applications/Julia-1.5.app/Contents/Resources/julia/lib -o TestPkg2.so -Wl,-all_load /var/folders/29/f6ksj93n5dz5zwdbz89rt9kh95rtb6/T/jl_qVNoRE.o -ljulia`, ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error at ./process.jl:525 [inlined]
 [2] run(::Cmd; wait::Bool) at ./process.jl:440
 [3] run at ./process.jl:438 [inlined]
 [4] run_with_env at /Users/malmiller/.julia/packages/PackageCompiler/KeZOk/src/PackageCompiler.jl:101 [inlined]
 [5] create_sysimg_from_object_file(::String, ::String) at /Users/malmiller/.julia/packages/PackageCompiler/KeZOk/src/PackageCompiler.jl:497
 [6] create_sysimage(::Array{Symbol,1}; sysimage_path::String, project::String, precompile_execution_file::Array{String,1}, precompile_statements_file::Array{String,1}, incremental::Bool, filter_stdlibs::Bool, replace_default::Bool, cpu_target::String, script::Nothing, base_sysimage::Nothing, isapp::Bool) at /Users/malmiller/.julia/packages/PackageCompiler/KeZOk/src/PackageCompiler.jl:451
 [7] top-level scope at REPL[11]:1

Mentions llvmname, which is defined as:

function llvmname(op, WR, WA, T, TA)
    lret = LLVM_TYPES[T]
    ln = "llvm.$op.$(suffix(WR,T))"
    (isone(abs(WR)) || T !== first(TA)) ? ln * '.' * suffix(maximum(WA),first(TA)) : ln
end

LLVM_TYPES is:

struct Bit; data::Bool; end # Dummy for Ptr
const NativeTypesExceptBit = Union{Bool,HWReal}
const NativeTypes = Union{NativeTypesExceptBit, Bit}
const LLVM_TYPES = IdDict{Type{<:NativeTypes},String}(
    Float32 => "float",
    Float64 => "double",
    Int8 => "i8",
    Int16 => "i16",
    Int32 => "i32",
    Int64 => "i64",
    UInt8 => "i8",
    UInt16 => "i16",
    UInt32 => "i32",
    UInt64 => "i64",
    Bool => "i8",
    Bit => "i1"
)

Could the IdDict be causing problems? This didn't actually change between VectorizationBase 0.13 and 0.14, but I'm grasping at straws.

DilumAluthge commented 3 years ago

@mmiller-max Can you reproduce the error on the latest Julia nightly or the latest Julia master?

chriselrod commented 3 years ago

FWIW, I was using Julia 1.5.3 above.

mmiller-max commented 3 years ago

@DilumAluthge will give it a go

mmiller-max commented 3 years ago

Whilst that's running, @chriselrod does your script remove the Manifest.toml? When I was trying to reproduce I had to manually delete it, setting compat to 0.14 didn't update the manifest if it already at 0.13 in it.

chriselrod commented 3 years ago

I didn't remove the Manifest.toml, and I set the [compat] to

[compat]
VectorizationBase = "0.13, 0.14"

And my Manifest.toml does seem to be updating, e.g. it currently says:

[[VectorizationBase]]
deps = ["ArrayInterface", "Hwloc", "IfElse", "Libdl", "LinearAlgebra"]
git-tree-sha1 = "51adc974cea83f935a34526b15c552edf39cfae8"
repo-rev = "2abed879a356bd1a2e9129a66ddcdfabe68b9708"
repo-url = "https://github.com/chriselrod/VectorizationBase.jl.git"
uuid = "3d5dd08c-fd9d-11e8-17fa-ed2836048c2f"
version = "0.14.6"

while my script says

  Resolving package versions...
Updating `~/Documents/progwork/julia/env/pkgcompiler_vecbase/TestPkg/Project.toml`
  [3d5dd08c] ~ VectorizationBase v0.14.6 `https://github.com/chriselrod/VectorizationBase.jl.git#2d793a7` ⇒ v0.14.6 `https://github.com/chriselrod/VectorizationBase.jl.git#2abed87`
Updating `~/Documents/progwork/julia/env/pkgcompiler_vecbase/TestPkg/Manifest.toml`
  [3d5dd08c] ~ VectorizationBase v0.14.6 `https://github.com/chriselrod/VectorizationBase.jl.git#2d793a7` ⇒ v0.14.6 `https://github.com/chriselrod/VectorizationBase.jl.git#2abed87`
sha = "2abed879a356bd1a2e9129a66ddcdfabe68b9708"
Project TestPkg v0.1.0
Status `~/Documents/progwork/julia/env/pkgcompiler_vecbase/TestPkg/Project.toml`
  [9b87118b] PackageCompiler v1.2.4
  [3d5dd08c] VectorizationBase v0.14.6 `https://github.com/chriselrod/VectorizationBase.jl.git#2abed87`
[ Info: PackageCompiler: creating system image object file, this might take a while...

The 2abed87 matches.

mmiller-max commented 3 years ago

Hmm, I get a similar error on latest master:

[ Info: PackageCompiler: creating system image object file, this might take a while...
Undefined symbols for architecture x86_64:
  "_jsysw_llvmname_170", referenced from:
      _tojlinvoke54137 in jl_bYDbWe.o(text.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
ERROR: failed process: Process(`gcc -m64 -march=x86-64 -shared -L/Users/malmiller/repos/julia/usr/lib -L/Users/malmiller/repos/julia/usr/lib -o TestPkg.so -Wl,-all_load /var/folders/29/f6ksj93n5dz5zwdbz89rt9kh95rtb6/T/jl_bYDbWe.o -ljulia-internal -ljulia`, ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error
   @ ./process.jl:525 [inlined]
 [2] run(::Cmd; wait::Bool)
   @ Base ./process.jl:440
 [3] run
   @ ./process.jl:438 [inlined]
 [4] run_with_env
   @ ~/.julia/packages/PackageCompiler/KeZOk/src/PackageCompiler.jl:101 [inlined]
 [5] create_sysimg_from_object_file(input_object::String, sysimage_path::String)
   @ PackageCompiler ~/.julia/packages/PackageCompiler/KeZOk/src/PackageCompiler.jl:497
 [6] create_sysimage(packages::Vector{Symbol}; sysimage_path::String, project::String, precompile_execution_file::Vector{String}, precompile_statements_file::Vector{String}, incremental::Bool, filter_stdlibs::Bool, replace_default::Bool, cpu_target::String, script::Nothing, base_sysimage::Nothing, isapp::Bool)
   @ PackageCompiler ~/.julia/packages/PackageCompiler/KeZOk/src/PackageCompiler.jl:451
 [7] top-level scope
   @ REPL[9]:1
julia> versioninfo()
Julia Version 1.7.0-DEV.236
Commit 83bee67631 (2021-01-08 02:28 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin18.7.0)
  CPU: Intel(R) Core(TM) i5-8279U CPU @ 2.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.0 (ORCJIT, skylake)
Environment:
  OR_JULIA_REGISTRY_SSH_DIR = /Users/malmiller/.ssh/registry_id_rsa
  JULIA_NUM_THREADS = 8
chriselrod commented 3 years ago

Odd. And llvmname came up again: "_jsysw_llvmname_170"

mmiller-max commented 3 years ago

0.14.6 worked for me too (on 1.5.3), I'll try stepping through some more

mmiller-max commented 3 years ago

With some more testing, everything up to 0.14.11 is fine too

DilumAluthge commented 3 years ago

Presumably then the problem is somewhere in here:

https://github.com/chriselrod/VectorizationBase.jl/compare/v0.14.11...v0.14.12

chriselrod commented 3 years ago

My first guess would be that it's the diff in precompile (and that llvmname actually doesn't have anything to do with it). I suspect that because of: gcc -m64 -march=x86-64 Maybe it's trying to generate code invalid for generic x86-64 in the precompile statement.

Could you try commenting it out, and seeing if that works?

If so, is there any way I can detect if the code is being run by PackageCompiler?

KristofferC commented 3 years ago

If so, is there any way I can detect if the code is being run by PackageCompiler?

Not really because package compiler just uses the precompiled files from a normal julia session.

chriselrod commented 3 years ago

@mmiller-max, could you check if that precompile statement is to blame (i.e., dev the library and comment it out)? If it is, because I can't detect PackageCompiler is running the code, I'll have to avoid such statements (which is unfortunate, because inference time was moderately high for it).

If it's something else, then I still don't know what the problem is.

chriselrod commented 3 years ago

I'm not sure what happened earlier when I did say I got an error, but I cannot reproduce this issue at all (even with 0.14.12 set in compat at =) or in any other way, so I'm entirely dependent on @mmiller-max for debugging this.

(TestPkg) pkg> st
Project TestPkg v0.1.0
Status `~/Documents/progwork/julia/env/pkgcompiler_vecbase/TestPkg/Project.toml`
  [9b87118b] PackageCompiler v1.2.4
  [3d5dd08c] VectorizationBase v0.14.12

julia> using PackageCompiler

julia> create_sysimage([:TestPkg]; sysimage_path="TestPkg.so")
[ Info: PackageCompiler: creating system image object file, this might take a while...

shell> cat Project.toml
name = "TestPkg"
uuid = "171afbbf-58d5-4f59-932d-e7f3f3aafb15"
authors = ["chriselrod <elrodc@gmail.com>"]
version = "0.1.0"

[deps]
PackageCompiler = "9b87118b-4619-50d2-8e1e-99f35a4d4d9d"
VectorizationBase = "3d5dd08c-fd9d-11e8-17fa-ed2836048c2f"

[compat]
VectorizationBase = "=0.14.12"
mmiller-max commented 3 years ago

@chriselrod yep sure I'll give commenting that out a test, will have to be tomorrow though. Thanks for looking into this so quickly today

chriselrod commented 3 years ago

Great, thanks for helping track the problem down!

mmiller-max commented 3 years ago

As you suspected @chriselrod it's this line that causes the problem:

https://github.com/chriselrod/VectorizationBase.jl/blob/37cc93559e0689416062c00f38ce250c36e3977b/src/precompile.jl#L13

When commented out the sysimage builds successfully.

Another way I was able to reproduce it by was just by building a sysimage with just VectorizationBase:

# From /path/to/VectorizationBase.jl, commit 6c15e62288e41754db90328a82e38338405dd3c6
(VectorizationBase) pkg>add PackageCompiler
julia> create_sysimage([:VectorizationBase]; sysimage_path="VectorizationBase.so")
Undefined symbols for architecture x86_64:
  "_jfptr_llvmcall_expr_67028", referenced from:
      _tojlinvoke71686 in jl_EjMi4u.o(text.o)
  "_jfptr_llvmname_66763", referenced from:
      _tojlinvoke71713 in jl_EjMi4u.o(text.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
ERROR: failed process: Process(`gcc -m64 -march=x86-64 -shared -L/Applications/Julia-1.5.app/Contents/Resources/julia/lib -o VectorizationBase.so -Wl,-all_load /var/folders/29/f6ksj93n5dz5zwdbz89rt9kh95rtb6/T/jl_EjMi4u.o -ljulia`, ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error at ./process.jl:525 [inlined]
 [2] run(::Cmd; wait::Bool) at ./process.jl:440
 [3] run at ./process.jl:438 [inlined]
 [4] run_with_env at /Users/malmiller/.julia/packages/PackageCompiler/KeZOk/src/PackageCompiler.jl:101 [inlined]
 [5] create_sysimg_from_object_file(::String, ::String) at /Users/malmiller/.julia/packages/PackageCompiler/KeZOk/src/PackageCompiler.jl:497
 [6] create_sysimage(::Array{Symbol,1}; sysimage_path::String, project::String, precompile_execution_file::Array{String,1}, precompile_statements_file::Array{String,1}, incremental::Bool, filter_stdlibs::Bool, replace_default::Bool, cpu_target::String, script::Nothing, base_sysimage::Nothing, isapp::Bool) at /Users/malmiller/.julia/packages/PackageCompiler/KeZOk/src/PackageCompiler.jl:451
 [7] top-level scope at REPL[5]:1

Hopefully this method works for you.

EDIT: I updated to Julia 1.5.3 just to make sure it wasn't anything with 1.5.2 and it didn't make a difference.

chriselrod commented 3 years ago

What happens if you

create_sysimage([:VectorizationBase]; sysimage_path="VectorizationBase.so", cpu_target = "native")

?

mmiller-max commented 3 years ago

create_sysimage([:VectorizationBase]; sysimage_path="VectorizationBase.so", cpu_target = "native")

I get the same error with this.

Interestingly, if I use the following dockerfile:

FROM julia:1.5.3

RUN apt-get update && apt-get install -y gcc

WORKDIR /app

COPY . .

RUN julia --project=. -e 'using Pkg; Pkg.add("PackageCompiler"); using PackageCompiler; create_sysimage([:VectorizationBase]; sysimage_path="VectorizationBase.so")'

ENTRYPOINT ["julia", "-JVectorizationBase.so", "-e", "println(\"Started\")"]

The sysimage builds in the docker contianer fine, but the sysimage fails to load:

$ docker build -t vectorisation-base -t
...
$ docker run vectorization-base
ERROR: could not load library "/app/VectorizationBase.so"
/app/VectorizationBase.so: undefined symbol: jfptr_llvmname_27369
ranocha commented 3 years ago

I get the same error using Julia v1.5.3 on Ubuntu with an Intel i7-8700K.

using Pkg, Libdl
Pkg.activate(temp=true)
Pkg.add("PackageCompiler")
packages = [:VectorizationBase]
map(Pkg.add ∘ string, packages)
Pkg.precompile()
using PackageCompiler
PackageCompiler.create_sysimage(
  packages,
  sysimage_path="custom_sysimage." * Libdl.dlext,
  cpu_target="native"
)

works and creates a sysimage for VectorizationBase v0.15.5, but 

julia --sysimage=custom_sysimage.so
ERROR: could not load library ".../custom_sysimage.so"
.../custom_sysimage.so: undefined symbol: jfptr_llvmname_27801
Segmentation fault (core dumped)

I get the same kind of error on an Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz (Ubuntu) and an Intel(R) Core(TM) i5-4590S CPU @ 3.00GHz (Debian).

chriselrod commented 3 years ago 
> julia -q                          (base)
julia> using Pkg, Libdl

julia> Pkg.activate(temp=true)
  Activating new environment at `/tmp/jl_yw3fft/Project.toml`

julia> packages = [:VectorizationBase];

julia> map(Pkg.add ∘ string, packages);
    Updating registry at `~/.julia/registries/General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Resolving package versions...
    Updating `/tmp/jl_yw3fft/Project.toml`
  [3d5dd08c] + VectorizationBase v0.15.5
    Updating `/tmp/jl_yw3fft/Manifest.toml`
  [4fba245c] + ArrayInterface v2.14.14
  [0e44f5e4] + Hwloc v1.2.0
  [615f187c] + IfElse v0.1.0
  [692b3bcd] + JLLWrappers v1.2.0
  [ae029012] + Requires v1.1.2
  [3d5dd08c] + VectorizationBase v0.15.5
  [e33a78d0] + Hwloc_jll v2.4.0+0
  [0dad84c5] + ArgTools
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [f43a241f] + Downloads
  [b77e0a4c] + InteractiveUtils
  [b27032c2] + LibCURL
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [ca575930] + NetworkOptions
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [fa267f1f] + TOML
  [a4e569a6] + Tar
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [deac9b47] + LibCURL_jll
  [29816b5a] + LibSSH2_jll
  [c8ffd9c3] + MbedTLS_jll
  [14a3606d] + MozillaCACerts_jll
  [83775a58] + Zlib_jll
  [8e850ede] + nghttp2_jll

julia> Pkg.precompile()

julia> using PackageCompiler

julia> PackageCompiler.create_sysimage(
         packages,
         sysimage_path="custom_sysimage." * Libdl.dlext,
         cpu_target="native"
       )
[ Info: PackageCompiler: creating system image object file, this might take a while...

julia> exit()
> julia -q --sysimage=custom_sysimage.so
julia> 1+2
3

julia> @time using VectorizationBase
  0.000091 seconds (1.03 k allocations: 76.703 KiB, 2450.97% compilation time)

Still can't reproduce. =/ FWIW, the fully verbose versioninfo

julia> versioninfo(verbose=true)
Julia Version 1.7.0-DEV.346
Commit 3df941f415* (2021-01-20 14:07 UTC)
Platform Info:
  OS: Linux (x86_64-generic-linux)
  uname: Linux 5.10.7-1014.native #1 SMP Tue Jan 12 15:03:30 PST 2021 x86_64 unknown
  CPU: Intel(R) Core(TM) i9-7900X CPU @ 3.30GHz:
                 speed         user         nice          sys         idle          irq
       #1-20  3572 MHz    1081601 s        125 s     202952 s  108173727 s      36471 s

  Memory: 31.043872833251953 GB (9033.39453125 MB free)
  Uptime: 547554.0 sec
  Load Avg:  0.84  0.85  0.7
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake-avx512)
Environment:
  JULIA_NUM_THREADS = auto
  CFLAGS = -O3 -march=native -mprefer-vector-width=512 -feliminate-unused-debug-types  -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=32 -Wformat -Wformat-security -m64  -fasynchronous-unwind-tables -Wp,-D_REENTRANT -ftree-loop-distribute-patterns -Wl,-z -Wl,now -Wl,-z -Wl,relro -fno-semantic-interposition -ffat-lto-objects  -fno-signed-zeros -fno-trapping-math  -fassociative-math -Wl,-sort-common -Wl,--enable-new-dtags
  CLASSPATH = /opt/intel/compilers_and_libraries_2019.4.243/linux/mpi/intel64/lib/mpi.jar:/opt/intel/compilers_and_libraries_2019.4.243/linux/daal/lib/daal.jar
  CPATH = /opt/intel/compilers_and_libraries_2019.4.243/linux/ipp/include:/opt/intel/compilers_and_libraries_2019.4.243/linux/mkl/include:/opt/intel/compilers_and_libraries_2019.4.243/linux/pstl/include:/opt/intel/compilers_and_libraries_2019.4.243/linux/tbb/include:/opt/intel/compilers_and_libraries_2019.4.243/linux/tbb/include:/opt/intel/compilers_and_libraries_2019.4.243/linux/daal/include
  CXXFLAGS = -O3 -march=native -mprefer-vector-width=512 -feliminate-unused-debug-types  -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=32 -Wformat -Wformat-security -m64  -fasynchronous-unwind-tables -Wp,-D_REENTRANT -ftree-loop-distribute-patterns -Wl,-z -Wl,now -Wl,-z -Wl,relro -fno-semantic-interposition -ffat-lto-objects  -fno-signed-zeros -fno-trapping-math  -fassociative-math -Wl,-sort-common -Wl,--enable-new-dtags -fvisibility-inlines-hidden -Wl,--enable-new-dtags
  FCFLAGS = -Ofast -march=native -mprefer-vector-width=512 -feliminate-unused-debug-types  -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=32 -Wformat -Wformat-security -m64  -fasynchronous-unwind-tables -Wp,-D_REENTRANT -ftree-loop-distribute-patterns -Wl,-z -Wl,now -Wl,-z -Wl,relro -fno-semantic-interposition -ffat-lto-objects  -fno-signed-zeros -fno-trapping-math  -fassociative-math -Wl,-sort-common -Wl,--enable-new-dtags
  FFLAGS = -Ofast -march=native -mprefer-vector-width=512 -feliminate-unused-debug-types  -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=32 -Wformat -Wformat-security -m64  -fasynchronous-unwind-tables -Wp,-D_REENTRANT -ftree-loop-distribute-patterns -Wl,-z -Wl,now -Wl,-z -Wl,relro -fno-semantic-interposition -ffat-lto-objects  -fno-signed-zeros -fno-trapping-math  -fassociative-math -Wl,-sort-common -Wl,--enable-new-dtags
  FI_PROVIDER_PATH = /opt/intel/compilers_and_libraries_2019.4.243/linux/mpi/intel64/libfabric/lib/prov
  HOME = /home/chriselrod
  LA_PATH = /usr/lib64/
  LD_LIBRARY_PATH = /opt/intel/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2019.4.243/linux/mpi/intel64/libfabric/lib:/opt/intel/compilers_and_libraries_2019.4.243/linux/mpi/intel64/lib/release:/opt/intel/compilers_and_libraries_2019.4.243/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2019.4.243/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2019.4.243/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2019.4.243/linux/tbb/lib/intel64/gcc4.1:/opt/intel/compilers_and_libraries_2019.4.243/linux/tbb/lib/intel64/gcc4.1:/opt/intel/compilers_and_libraries_2019.4.243/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2019.4.243/linux/daal/../tbb/lib/intel64_lin/gcc4.4
  LIBRARY_PATH = /opt/intel/compilers_and_libraries_2019.4.243/linux/mpi/intel64/libfabric/lib:/opt/intel/compilers_and_libraries_2019.4.243/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2019.4.243/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2019.4.243/linux/tbb/lib/intel64/gcc4.1:/opt/intel/compilers_and_libraries_2019.4.243/linux/tbb/lib/intel64/gcc4.1:/opt/intel/compilers_and_libraries_2019.4.243/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2019.4.243/linux/daal/../tbb/lib/intel64_lin/gcc4.4
  MANPATH = /opt/intel/man/common:/opt/intel/compilers_and_libraries_2019.4.243/linux/mpi/man:/usr/local/share/man:/usr/share/man:/usr/man
  MPI_PATH = /usr/lib64/
  NLSPATH = /opt/intel/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/compilers_and_libraries_2019.4.243/linux/mkl/lib/intel64_lin/locale/%l_%t/%N
  PATH = /home/chriselrod/miniconda3/bin:/home/chriselrod/miniconda3/condabin:/opt/intel/compilers_and_libraries_2019.4.243/linux/bin/intel64:/opt/intel/compilers_and_libraries_2019.4.243/linux/mpi/intel64/libfabric/bin:/opt/intel/compilers_and_libraries_2019.4.243/linux/mpi/intel64/bin:/usr/bin/haswell/avx512_1:/usr/bin/haswell:/usr/local/bin:/usr/local/sbin:/usr/bin:/opt/3rd-party/bin
  PKG_CONFIG_PATH = /opt/intel/compilers_and_libraries_2019.4.243/linux/mkl/bin/pkgconfig
  TERM = screen
  THEANO_FLAGS = floatX=float32,openmp=true,gcc.cxxflags="-ftree-vectorize -mavx"
  WINDOWPATH = 2
  FONTCONFIG_PATH = /usr/share/defaults/fonts
  CMDSTAN_HOME = /home/chriselrod/Documents/languages/cmdstan
  R_HOME = /usr/lib64/R

OS is Clear Linux.

ranocha commented 3 years ago

Thanks for the feedback! I can confirm that it works for me if I use the latest Julia nightly build (Version 1.7.0-DEV.358 (2021-01-21), Commit a40335766e*). However, I get the same errors on the latest release branch of v1.6 (Version 1.6.0-beta1.1 (2021-01-11), release-1.6/8240225853).

chriselrod commented 3 years ago

Okay, I'll try 1.5 or 1.6. If the problem is the precompute statements, good news is that it sounds like I should be able to turn them back on in Julia >= 1.7.

chriselrod commented 3 years ago

I could reproduce with Julia 1.5. Should be fixed in the latest release (VectorizationBase 0.15.7).

ranocha commented 3 years ago

I can confirm that everything works for me using VectorizationBase v0.15.7. Thanks for the quick fix, @chriselrod :+1:

mmiller-max commented 3 years ago

Cheers @chriselrod!