search

SciML / LinearSolve.jl

LinearSolve.jl: High-Performance Unified Interface for Linear Solvers in Julia. Easily switch between factorization and Krylov methods, add preconditioners, and all in one interface.
https://docs.sciml.ai/LinearSolve/stable/
Other
248 stars 53 forks source link

LinearSolve fails to precompile on julia 1.8.5 #266

Closed BPMinaker closed 1 year ago

BPMinaker commented 1 year ago

[ Info: Precompiling LinearSolve [7ed4a6bd-45f5-4d41-b270-4a48e9bafcae] ERROR: LoadError: MethodError: convert(::Type{Union{}}, ::SparseArrays.SparseMatrixCSC{Float64, Int64}) is ambiguous. ... ... [1] Sparspak.SpkSparseSolver.SparseSolver{Int64, Float64}(p::SparseArrays.SparseMatrixCSC{Float64, Int64}, slvr::Sparspak.SpkSparseBase._SparseBase{Int64, Float64}, n::Int64, ma::Int64, na::Int64, mc::Int64, nc::Int64, _inmatrixdone::Bool, _orderdone::Bool, _symbolicdone::Bool, _factordone::Bool, _trisolvedone::Bool, _refinedone::Bool, _condestdone::Bool) (repeats 2 times) @ Sparspak.SpkSparseSolver ~/.julia/packages/Sparspak/ZSfSg/src/SparseMethod/SpkSparseSolver.jl:18 [2] Sparspak.SpkSparseSolver.SparseSolver(m::SparseArrays.SparseMatrixCSC{Float64, Int64}) @ Sparspak.SparseCSCInterface ~/.julia/packages/Sparspak/ZSfSg/src/SparseCSCInterface/SparseCSCInterface.jl:189 [3] sparspaklu(m::SparseArrays.SparseMatrixCSC{Float64, Int64}; factorize::Bool) @ Sparspak.SparseCSCInterface ~/.julia/packages/Sparspak/ZSfSg/src/SparseCSCInterface/SparseCSCInterface.jl:219 [4] init_cacheval(#unused#::LinearSolve.SparspakFactorization, A::SparseArrays.SparseMatrixCSC{Float64, Int64}, b::Vector{Float64}, u::Vector{Float64}, Pl::IterativeSolvers.Identity, Pr::IterativeSolvers.Identity, maxiters::Int64, abstol::Float64, reltol::Float64, verbose::Bool, assumptions::LinearSolve.OperatorAssumptions{Nothing}) @ LinearSolve ~/.julia/packages/LinearSolve/dxfUd/src/factorization.jl:514 ... ...

j-fu commented 1 year ago

Strange. I can't reproduce this with 1.8.5 in a clean environment. Do you have more information about the packages in your environment, and also the full error log ?

BPMinaker commented 1 year ago

Running on Arch Linux, fully updated. Clean environment. My experiments (after seeing your post) show this only occurs with the Arch repository binaries for Julia. If I pull binaries from the Julia website instead, no problem. I think that this might have something to do with the suitesparse library on Arch, which was just upgrade to v7. That upgrade caused a bug on Julia on launch that was related to the linear algebra library. Arch updated Julia to 1.8.5-2, which solved the first problem, but I suspect that this is somehow related. Where is best to report this?

ChrisRackauckas commented 1 year ago

oh that's expected. Arch builds the binaries incorrectly and changes to the wrong version of LLVM. Julia throws a warning during the startup process when one does this, but they specifically patch Julia to not throw the warning that it's broken. It's wild, but it's not LinearSolve's problem. Report this to the package manager, they somehow think that broken is okay.

antonio-rojas commented 1 year ago

No, this has nothing to do with different LLVM versions or with anybody thinking that "broken is OK". This can be easily reproduced by compiling Julia with USE_BLAS64=0 and no external dependencies.

ChrisRackauckas commented 1 year ago

Okay interesting. Thanks for a non-broken reproducer. For reference (https://github.com/SciML/Sundials.jl/issues/358), the Arch Linux build states (https://wiki.archlinux.org/title/Julia#Installation):

Warning: Package julia is compiled against system libraries and is therefore > unsupported by the upstream. While it may be suitable for basic usage it is incompatible with external packages (e.g. MKL, > LLVM, LLVMExtra, or packages that depend on these, like CUDA or Flux) which rely on Julia's bundled versions of its > dependencies. Bugs reported against this binary will be https://github.com/JuliaLinearAlgebra/MKL.jl/issues/105#issuecomment-1025985667.

and has patches to it which remove the printing of warnings that Julia naturally throws for version incompatibilities, which is why in general we will not dive into issues from the Arch Linux build. It doesn't pass Base Julia's tests and any bug reports are closed, so it's generally just not expected to work.

But since there's a reproducer that doesn't require the Arch Linux build, reopening.

j-fu commented 1 year ago

Is there any place where one can find an actual pre-built julia with USE_BLAS64=0 ?

And does this give the same error ?

antonio-rojas commented 1 year ago

in general we will not dive into issues from the Arch Linux build. It doesn't pass Base Julia's tests and any bug reports are closed, so it's generally just not expected to work.

And that is perfectly fine. What is not fine is saying we think that "broken is OK". I personally have spent countless hours diving in julia's git repo picking patches to make sure it works with our system libraries, And no package is released to the repos until all tests are checked to pass (except those that test the version number of used libraries).

Most reported issues on Arch concern external packages, which of course we can't realistically test.

Anyway: I have just released a 64-bit integers openblas package and rebuilt the julia package with it, which should solve this issue on Arch and in general improve compatibility with upstream binaries.

j-fu commented 1 year ago

See also https://discourse.julialang.org/t/precompile-error-in-differentialequations-jl/94086/1

Related to https://github.com/JuliaLang/julia/pull/42735 ?

j-fu commented 1 year ago

It has been tagged as 0.3.8, thank you @PetrKryslUCSD.

j-fu commented 1 year ago

Fingers crossed - I am not sure if this was the reason, as this bug so far was visible only in the wild...

j-fu commented 1 year ago

So the problem is not gone:

https://discourse.julialang.org/t/precompile-error-in-differentialequations-jl/94086/5

BPMinaker commented 1 year ago

I can share that I first discovered the problem with LinearSolve when attempting to install OrdinaryDiffEq because LinearSolve is a dependency. But the updated packages from antonio-rojas fix the problem with LinearSolve for me and allow me to build OrdinaryDiffEq.

PetrKryslUCSD commented 1 year ago

Should we convert the sparse matrix to a problem? And store the problem instead of a union?

j-fu commented 1 year ago

I understand that this might be a workaround. But in fact this bug appears to be a canary for some problems with some "non-official" Julia builds on aarch, and IMHO similar bugs could occur elsewhere. As it appears in https://github.com/SciML/LinearSolve.jl/issues/271 the problem seems to go away when switching from a "non-official" homebrew version to the official one. So, unless someone with an "official" Julia build comes up with the problem I would keep the code as it is now - this even may help with hunting down the bug.

BTW on this occasion I think we could have the Sparspak CI also running on nightly - will make a PR... Unfortunately, there seems to be no way in the moment to run CI tests on M1 Macs with aarch.