Closed olaayeko closed 4 years ago
Still getting the same error, is there a hard reset I need to complete first ?
Still getting the same error, is there a hard reset I need to complete first ?
Do you mean when testing DifferentialEquations, or trying to run vfmadd231(::Vec{8,Float64}...)
?
If the latter, the point is: your CPU can do it, but LLVM doesn't think it can. So the change to VectorizationBase is meant to make some of DifferentialEquations.jl's dependencies use vfmadd231(::Vec{4,Float64}...)
instead (basically, replace all the 8s with 4s, and it'll work).
When trying to run vfmadd231(::Vec{4,Float64}...), after changing the VectorizationBase settings
Are you sure that this does not work (replacing all the 8s with 4s)?
const Vec{W,T} = NTuple{W,Core.VecElement{T}}
@inline function vfmadd231(a::Vec{4,Float64}, b::Vec{4,Float64}, c::Vec{4,Float64})
Base.llvmcall("""%res = call <4 x double> asm "vfmadd231pd \$3, \$2, \$1", "=v,0,v,v"(<4 x double> %2, <4 x double> %1, <4 x double> %0)
ret <4 x double> %res""",
Vec{4,Float64},
Tuple{Vec{4,Float64},Vec{4,Float64},Vec{4,Float64}}, a, b, c)
end
x = ntuple(Val(4)) do _ Core.VecElement(rand()) end;
y = ntuple(Val(4)) do _ Core.VecElement(rand()) end;
z = ntuple(Val(4)) do _ Core.VecElement(rand()) end;
vfmadd231(x,y,z)
Sorry, yes that does work
So, the problem is that the host's CPU features AVX512 support, but that LLVM thinks it is Haswell or something. Unfortunately,
unsafe_string(LLVM.API.LLVMGetHostCPUFeatures())
returns -avx512f and all the other cpu feature flags.
So LLVM gets it wrong? https://github.com/llvm/llvm-project/commit/82921bf2baed96b700f90b090d5dc2530223d9c0 looks relevant then, and not included in our current LLVM (avx512f is set later based on HasLeaf7 && ((EBX >> 16) & 1) && HasAVX512Save
).
Is there any action I can take to rectify this and any theory of why this bug appeared suddenly when it was working well before ?
So LLVM gets it wrong? llvm/llvm-project@82921bf looks relevant then, and not included in our current LLVM (avx512f is set later based on HasLeaf7 && ((EBX >> 16) & 1) && HasAVX512Save).
Yes, great.
So it looks like the simplest approach to testing would be building Julia master with LLVM_VER=10.0.0
in the Make.user
on that architecture.
If that solves the problem, we could patch Julia's LLVM 9 for the 1.5 release?
EDIT: Or is there a way olaayeko could check if
bool HasAVX512Save = HasAVX && ((EAX & 0xe0) == 0xe0);
is currently set to false?
For the first one, how do I build that, sorry I am new to Julia. On the second one I am getting syntax/UndefVarError when I try to check that.
@maleadt
Ha ha, I didn't realize that the feature string listed all the features, indicating with a +
vs -
whether or not they're present.
julia> using LLVM
julia> unsafe_string(LLVM.API.LLVMGetHostCPUFeatures())
"+sse2,+cx16,+sahf,-tbm,-avx512ifma,-sha,-gfni,-fma4,-vpclmulqdq,+prfchw,+bmi2,-cldemote,+fsgsbase,-ptwrite,+xsavec,+popcnt,+aes,-avx512bitalg,-movdiri,+xsaves,-avx512er,-avx512vnni,-avx512vpopcntdq,-pconfig,+clwb,+avx512f,-clzero,-pku,+mmx,-lwp,-rdpid,-xop,+rdseed,-waitpkg,-movdir64b,-sse4a,+avx512bw,+clflushopt,+xsave,-avx512vbmi2,+64bit,+avx512vl,+invpcid,+avx512cd,+avx,-vaes,+rtm,+fma,+bmi,+rdrnd,-mwaitx,+sse4.1,+sse4.2,+avx2,-wbnoinvd,+sse,+lzcnt,+pclmul,-prefetchwt1,+f16c,+ssse3,-sgx,-shstk,+cmov,-avx512vbmi,+movbe,+xsaveopt,+avx512dq,+adx,-avx512pf,+sse3"
julia> versioninfo()
Julia Version 1.4.1
Commit 381693d3df* (2020-04-14 17:20 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
CPU: Intel(R) Core(TM) i9-7900X CPU @ 3.30GHz
So the 7900X does correctly have +avx512f
, but, for example, not -avx512er
as it doesn't have exponentiation or accurate reciprocals.
While olaayeko's CPU shows -avx512*
, matching LLVM's behavior (although not the CPU's capabilities). So using this would let me generate code that agrees with LLVM. But it'd still be nice to stop underutilizing the host CPU on Apple.
Thanks @YingboMa for pointing out my misunderstanding.
So I think parsing this feature string is the right thing to do.
I could drop the CpuId.jl
dependency entirely if LLVM can somehow provide the number of physical cores as well as cache sizes? Is this possible at all?
I don't mind an LLVM.jl
dependency too much, as I may consider using it in some cases instead of relying on Base.llvmcall
. Any thoughts/concerns/recommendations? E.g., how do they compare on compile time? (llvmcall improved dramatically in Julia 1.4 versus early versions.)
How robust is it versus changing underlying LLVM version?
I've had a few issues with needing to checks based on Base.libllvm_version
, but this is something I can easily handle myself. How stable/brittle is LLVM.jl with respect to this?
I can load the library and the feature string works with LLVM 10, but I get 1 test failure.
I don't mind an
LLVM.jl
dependency too much, as I may consider using it in some cases instead of relying onBase.llvmcall
. Any thoughts/concerns/recommendations?
LLVM.jl is a lightweight dependency, so OK for querying features. For replacing llvmcall
, although it's a better (albeit sometimes verbose) way of emitting IR, the generated code is currently not precompilable because of how llvmcall
with llvm::Function*
arguments works (the package itself is perfectly precompilable though). Robustness is much better, since it uses the stable C ap vs. using the explicitly non-portable textual IR format. So it depends on your needs.
I'll wait until it is precompilable to switching to it more broadly. For now, I can use it for feature detection: https://github.com/chriselrod/VectorizationBase.jl/commit/1f1a22b3e1fbdcc67532320ad787a551545c7fcf
This also means I could add invariant.start
and invariant.end
methods, which I couldn't get to work with llvmcall
(errors because it doesn't like {}*
); these may be useful when working with arrays to which we have pointers but that are also static:
using LLVM, LLVM.Interop
struct Invariant{L,T}
ivp::Ptr{Cvoid}
ptr::Ptr{T}
end
@generated function invariant_start!(ptr::Ptr{T}, ::Val{L}) where {L,T}
T_int = LLVM.IntType(sizeof(Int)*8, JuliaContext())
paramtyps = [ T_int ]
ret_typ = T_int # returning a Ptr{Cvoid}
llvmf, _ = create_function(ret_typ, paramtyps)
i8 = convert(LLVMType, Int8)
i8_ptr = LLVM.PointerType(i8)
emptystruct = LLVM.StructType(typeof(i8)[])
emptystruct_ptr = LLVM.PointerType(emptystruct)
mod = LLVM.parent(llvmf)
intrinsic_typ = LLVM.FunctionType(emptystruct_ptr, [convert(LLVMType, Int), i8_ptr])
intrinsic = LLVM.Function(mod, "llvm.invariant.start.p0i8", intrinsic_typ)
Builder(JuliaContext()) do builder
entry = BasicBlock(llvmf, "entry", JuliaContext())
position!(builder, entry)
ptr = inttoptr!(builder, parameters(llvmf)[1], i8_ptr)
ivt_ptr = call!(builder, intrinsic, [ConstantInt(T_int, sizeof(T)*L), ptr])
ivt_int = ptrtoint!(builder, ivt_ptr, T_int)
ret!(builder, ivt_int)
end
call_function(llvmf, Ptr{Cvoid}, Tuple{Ptr{T}}, :(ptr,))
end
@generated function invariant_end!(ivt_ptr::Ptr{Cvoid}, ptr::Ptr{T}, ::Val{L}) where {L,T}
T_int = LLVM.IntType(sizeof(Int)*8, JuliaContext())
voidtype = LLVM.VoidType(JuliaContext())
paramtyps = [ T_int, T_int ]
ret_typ = voidtype # returning a Ptr{Cvoid}
llvmf, _ = create_function(ret_typ, paramtyps)
i8 = convert(LLVMType, Int8)
i8_ptr = LLVM.PointerType(i8)
emptystruct = LLVM.StructType(typeof(i8)[])
emptystruct_ptr = LLVM.PointerType(emptystruct)
mod = LLVM.parent(llvmf)
intrinsic_typ = LLVM.FunctionType(voidtype, [emptystruct_ptr, convert(LLVMType, Int), i8_ptr])
intrinsic = LLVM.Function(mod, "llvm.invariant.end.p0i8", intrinsic_typ)
Builder(JuliaContext()) do builder
entry = BasicBlock(llvmf, "entry", JuliaContext())
position!(builder, entry)
invt_ptr = inttoptr!(builder, parameters(llvmf)[1], emptystruct_ptr)
ptr = inttoptr!(builder, parameters(llvmf)[2], i8_ptr)
ret = call!(builder, intrinsic, [invt_ptr, ConstantInt(T_int, sizeof(T)*L), ptr])
ret!(builder)
end
call_function(llvmf, Cvoid, Tuple{Ptr{Cvoid},Ptr{T}}, :(ivt_ptr, ptr))
end
@inline function freeze!(A_ptr::Ptr{T}, ::Val{L}) where {T,L}
Invariant{L,T}(invariant_start!(A_ptr, Val(L)), A_ptr)
end
@inline function melt!(ivt::Invariant{L,T}) where {L,T}
invariant_end!(ivt.ivp, ivt.ptr, Val(L))
end
The new release of VectorizationBase should be registered soon.
It can only use LLVMGetHostCPUFeatures
with LLVM is >= 8, so this wont be fixed on Julia versions <= 1.3. I hope not many people are still using these old versions.
I made the changes suggested here https://github.com/JuliaLang/julia/issues/36329, DifferentialEquations is passing Pkg.test(). In Jupyter notebook the DifferentialEquations module is still crashing the kernel. Any suggestions on next steps for debugging ? Thank you for the help so far as well.
When is DifferentialEquations crashing the kernel? Also, could you try running RecursiveFactorization's tests?
The kernel crashes when I attempt to run "solve(prob, Rodas4())". The RecursiveFactorization tests passed
and fails when I let it choose the solving method by emitting Rodas4()
What I find especially bizarre is that it was working perfectly fine earlier this month. Of note was I downloaded a packaged "FourierTransform", coincidentally(or not) it stopped working after this
And solve(prob, Rodas4())
works outside of Jupyter?
Is versioninfo()
the same inside and outside of Jupyter?
No its 1.5 in Julia and 1.4.2 in Jupyter, and it is working fine in terminal strangely
Could you try either of
using Pkg; Pkg.update()
in Jupyter, to make sure you're on the latest versions.using IJulia; installkernel()
in the terminal, and then using the newly installed kernel in Jupyter so that you're using the same Julia version in bothand then let me know if you're still seeing the problem?
I am using the 1.5 kernel in Jupyter, its sorted now thank you for the help!
Great! Sounds like we can close this?
Thanks, everyone!
I am failing to use DifferentialEquations.jl (v6.14.0) in Julia 1.4.1 on Mac: