olaayeko
commented
4 years ago Still getting the same error, is there a hard reset I need to complete first ?
Closed olaayeko closed 4 years ago
olaayeko
commented
4 years ago Still getting the same error, is there a hard reset I need to complete first ?
chriselrod
commented
4 years ago Still getting the same error, is there a hard reset I need to complete first ?
Do you mean when testing DifferentialEquations, or trying to run vfmadd231(::Vec{8,Float64}...)?
If the latter, the point is: your CPU can do it, but LLVM doesn't think it can. So the change to VectorizationBase is meant to make some of DifferentialEquations.jl's dependencies use vfmadd231(::Vec{4,Float64}...) instead (basically, replace all the 8s with 4s, and it'll work).
olaayeko
commented
4 years ago When trying to run vfmadd231(::Vec{4,Float64}...), after changing the VectorizationBase settings
chriselrod
commented
4 years ago Are you sure that this does not work (replacing all the 8s with 4s)?
const Vec{W,T} = NTuple{W,Core.VecElement{T}}
@inline function vfmadd231(a::Vec{4,Float64}, b::Vec{4,Float64}, c::Vec{4,Float64})
Base.llvmcall("""%res = call <4 x double> asm "vfmadd231pd \$3, \$2, \$1", "=v,0,v,v"(<4 x double> %2, <4 x double> %1, <4 x double> %0)
ret <4 x double> %res""",
Vec{4,Float64},
Tuple{Vec{4,Float64},Vec{4,Float64},Vec{4,Float64}}, a, b, c)
end
x = ntuple(Val(4)) do _ Core.VecElement(rand()) end;
y = ntuple(Val(4)) do _ Core.VecElement(rand()) end;
z = ntuple(Val(4)) do _ Core.VecElement(rand()) end;
vfmadd231(x,y,z)Sorry, yes that does work
maleadt
commented
4 years ago So, the problem is that the host's CPU features AVX512 support, but that LLVM thinks it is Haswell or something. Unfortunately,
unsafe_string(LLVM.API.LLVMGetHostCPUFeatures())returns -avx512f and all the other cpu feature flags.
So LLVM gets it wrong? https://github.com/llvm/llvm-project/commit/82921bf2baed96b700f90b090d5dc2530223d9c0 looks relevant then, and not included in our current LLVM (avx512f is set later based on HasLeaf7 && ((EBX >> 16) & 1) && HasAVX512Save).
olaayeko
commented
4 years ago Is there any action I can take to rectify this and any theory of why this bug appeared suddenly when it was working well before ?
chriselrod
commented
4 years ago So LLVM gets it wrong? llvm/llvm-project@82921bf looks relevant then, and not included in our current LLVM (avx512f is set later based on HasLeaf7 && ((EBX >> 16) & 1) && HasAVX512Save).
Yes, great.
So it looks like the simplest approach to testing would be building Julia master with LLVM_VER=10.0.0 in the Make.user on that architecture.
If that solves the problem, we could patch Julia's LLVM 9 for the 1.5 release?
EDIT: Or is there a way olaayeko could check if
bool HasAVX512Save = HasAVX && ((EAX & 0xe0) == 0xe0);is currently set to false?
olaayeko
commented
4 years ago For the first one, how do I build that, sorry I am new to Julia. On the second one I am getting syntax/UndefVarError when I try to check that.
chriselrod
commented
4 years ago @maleadt
Ha ha, I didn't realize that the feature string listed all the features, indicating with a + vs - whether or not they're present.
julia> using LLVM
julia> unsafe_string(LLVM.API.LLVMGetHostCPUFeatures())
"+sse2,+cx16,+sahf,-tbm,-avx512ifma,-sha,-gfni,-fma4,-vpclmulqdq,+prfchw,+bmi2,-cldemote,+fsgsbase,-ptwrite,+xsavec,+popcnt,+aes,-avx512bitalg,-movdiri,+xsaves,-avx512er,-avx512vnni,-avx512vpopcntdq,-pconfig,+clwb,+avx512f,-clzero,-pku,+mmx,-lwp,-rdpid,-xop,+rdseed,-waitpkg,-movdir64b,-sse4a,+avx512bw,+clflushopt,+xsave,-avx512vbmi2,+64bit,+avx512vl,+invpcid,+avx512cd,+avx,-vaes,+rtm,+fma,+bmi,+rdrnd,-mwaitx,+sse4.1,+sse4.2,+avx2,-wbnoinvd,+sse,+lzcnt,+pclmul,-prefetchwt1,+f16c,+ssse3,-sgx,-shstk,+cmov,-avx512vbmi,+movbe,+xsaveopt,+avx512dq,+adx,-avx512pf,+sse3"
julia> versioninfo()
Julia Version 1.4.1
Commit 381693d3df* (2020-04-14 17:20 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
CPU: Intel(R) Core(TM) i9-7900X CPU @ 3.30GHzSo the 7900X does correctly have +avx512f, but, for example, not -avx512er as it doesn't have exponentiation or accurate reciprocals.
While olaayeko's CPU shows -avx512*, matching LLVM's behavior (although not the CPU's capabilities). So using this would let me generate code that agrees with LLVM. But it'd still be nice to stop underutilizing the host CPU on Apple.
Thanks @YingboMa for pointing out my misunderstanding.
So I think parsing this feature string is the right thing to do.
I could drop the CpuId.jl dependency entirely if LLVM can somehow provide the number of physical cores as well as cache sizes? Is this possible at all?
I don't mind an LLVM.jl dependency too much, as I may consider using it in some cases instead of relying on Base.llvmcall. Any thoughts/concerns/recommendations? E.g., how do they compare on compile time? (llvmcall improved dramatically in Julia 1.4 versus early versions.)
How robust is it versus changing underlying LLVM version?
I've had a few issues with needing to checks based on Base.libllvm_version, but this is something I can easily handle myself. How stable/brittle is LLVM.jl with respect to this?
I can load the library and the feature string works with LLVM 10, but I get 1 test failure.
maleadt
commented
4 years ago I don't mind an
LLVM.jldependency too much, as I may consider using it in some cases instead of relying onBase.llvmcall. Any thoughts/concerns/recommendations?
LLVM.jl is a lightweight dependency, so OK for querying features. For replacing llvmcall, although it's a better (albeit sometimes verbose) way of emitting IR, the generated code is currently not precompilable because of how llvmcall with llvm::Function* arguments works (the package itself is perfectly precompilable though). Robustness is much better, since it uses the stable C ap vs. using the explicitly non-portable textual IR format. So it depends on your needs.
chriselrod
commented
4 years ago I'll wait until it is precompilable to switching to it more broadly. For now, I can use it for feature detection: https://github.com/chriselrod/VectorizationBase.jl/commit/1f1a22b3e1fbdcc67532320ad787a551545c7fcf
This also means I could add invariant.start and invariant.end methods, which I couldn't get to work with llvmcall (errors because it doesn't like {}*); these may be useful when working with arrays to which we have pointers but that are also static:
using LLVM, LLVM.Interop
struct Invariant{L,T}
ivp::Ptr{Cvoid}
ptr::Ptr{T}
end
@generated function invariant_start!(ptr::Ptr{T}, ::Val{L}) where {L,T}
T_int = LLVM.IntType(sizeof(Int)*8, JuliaContext())
paramtyps = [ T_int ]
ret_typ = T_int # returning a Ptr{Cvoid}
llvmf, _ = create_function(ret_typ, paramtyps)
i8 = convert(LLVMType, Int8)
i8_ptr = LLVM.PointerType(i8)
emptystruct = LLVM.StructType(typeof(i8)[])
emptystruct_ptr = LLVM.PointerType(emptystruct)
mod = LLVM.parent(llvmf)
intrinsic_typ = LLVM.FunctionType(emptystruct_ptr, [convert(LLVMType, Int), i8_ptr])
intrinsic = LLVM.Function(mod, "llvm.invariant.start.p0i8", intrinsic_typ)
Builder(JuliaContext()) do builder
entry = BasicBlock(llvmf, "entry", JuliaContext())
position!(builder, entry)
ptr = inttoptr!(builder, parameters(llvmf)[1], i8_ptr)
ivt_ptr = call!(builder, intrinsic, [ConstantInt(T_int, sizeof(T)*L), ptr])
ivt_int = ptrtoint!(builder, ivt_ptr, T_int)
ret!(builder, ivt_int)
end
call_function(llvmf, Ptr{Cvoid}, Tuple{Ptr{T}}, :(ptr,))
end
@generated function invariant_end!(ivt_ptr::Ptr{Cvoid}, ptr::Ptr{T}, ::Val{L}) where {L,T}
T_int = LLVM.IntType(sizeof(Int)*8, JuliaContext())
voidtype = LLVM.VoidType(JuliaContext())
paramtyps = [ T_int, T_int ]
ret_typ = voidtype # returning a Ptr{Cvoid}
llvmf, _ = create_function(ret_typ, paramtyps)
i8 = convert(LLVMType, Int8)
i8_ptr = LLVM.PointerType(i8)
emptystruct = LLVM.StructType(typeof(i8)[])
emptystruct_ptr = LLVM.PointerType(emptystruct)
mod = LLVM.parent(llvmf)
intrinsic_typ = LLVM.FunctionType(voidtype, [emptystruct_ptr, convert(LLVMType, Int), i8_ptr])
intrinsic = LLVM.Function(mod, "llvm.invariant.end.p0i8", intrinsic_typ)
Builder(JuliaContext()) do builder
entry = BasicBlock(llvmf, "entry", JuliaContext())
position!(builder, entry)
invt_ptr = inttoptr!(builder, parameters(llvmf)[1], emptystruct_ptr)
ptr = inttoptr!(builder, parameters(llvmf)[2], i8_ptr)
ret = call!(builder, intrinsic, [invt_ptr, ConstantInt(T_int, sizeof(T)*L), ptr])
ret!(builder)
end
call_function(llvmf, Cvoid, Tuple{Ptr{Cvoid},Ptr{T}}, :(ivt_ptr, ptr))
end
@inline function freeze!(A_ptr::Ptr{T}, ::Val{L}) where {T,L}
Invariant{L,T}(invariant_start!(A_ptr, Val(L)), A_ptr)
end
@inline function melt!(ivt::Invariant{L,T}) where {L,T}
invariant_end!(ivt.ivp, ivt.ptr, Val(L))
endThe new release of VectorizationBase should be registered soon.
It can only use LLVMGetHostCPUFeatures with LLVM is >= 8, so this wont be fixed on Julia versions <= 1.3. I hope not many people are still using these old versions.
olaayeko
commented
4 years ago I made the changes suggested here https://github.com/JuliaLang/julia/issues/36329, DifferentialEquations is passing Pkg.test(). In Jupyter notebook the DifferentialEquations module is still crashing the kernel. Any suggestions on next steps for debugging ? Thank you for the help so far as well.
chriselrod
commented
4 years ago When is DifferentialEquations crashing the kernel? Also, could you try running RecursiveFactorization's tests?
olaayeko
commented
4 years ago The kernel crashes when I attempt to run "solve(prob, Rodas4())". The RecursiveFactorization tests passed
olaayeko
commented
4 years ago and fails when I let it choose the solving method by emitting Rodas4()
olaayeko
commented
4 years ago What I find especially bizarre is that it was working perfectly fine earlier this month. Of note was I downloaded a packaged "FourierTransform", coincidentally(or not) it stopped working after this
chriselrod
commented
4 years ago And solve(prob, Rodas4()) works outside of Jupyter?
Is versioninfo() the same inside and outside of Jupyter?
olaayeko
commented
4 years ago No its 1.5 in Julia and 1.4.2 in Jupyter, and it is working fine in terminal strangely
chriselrod
commented
4 years ago Could you try either of
using Pkg; Pkg.update() in Jupyter, to make sure you're on the latest versions.using IJulia; installkernel() in the terminal, and then using the newly installed kernel in Jupyter so that you're using the same Julia version in bothand then let me know if you're still seeing the problem?
olaayeko
commented
4 years ago I am using the 1.5 kernel in Jupyter, its sorted now thank you for the help!
chriselrod
commented
4 years ago Great! Sounds like we can close this?
YingboMa
commented
4 years ago Thanks, everyone!
I am failing to use DifferentialEquations.jl (v6.14.0) in Julia 1.4.1 on Mac: