OpenMendel / SnpArrays.jl

Compressed storage for SNP data
https://openmendel.github.io/SnpArrays.jl/latest
Other
44 stars 9 forks source link

Linear Algebra is not actually parallel #118

Closed biona001 closed 2 years ago

biona001 commented 2 years ago

Starting Julia with 8 threads, the following code never reaches over 100% CPU usage when monitoring with htop

Threads.nthreads() # returns 8

using Revise, SnpArrays, LinearAlgebra
x = SnpArray(SnpArrays.datadir("EUR_subset.bed")) # 379 by 54051
A = SnpLinAlg{Float64}(x, center=true, scale=true, impute=true)
B = rand(size(x, 1), 1800)
C = zeros(54051, 1800)
mul!(C, Transpose(A), B)

# check answer
truth = Transpose(convert(Matrix{Float64}, x, center=true, scale=true, impute=true))*B
all(isapprox.(truth, C, atol=1e-10)) # true

However, if I replace our custom-defined macros with Threads.@spawn and @sync, I get 800% CPU usage. Is there a reason we don't use the standard macros?

biona001 commented 2 years ago

Actually, the current code is multithreaded. Sorry for the confusion.

biona001 commented 2 years ago

I am unfortunately still having this problem with SnpArrays v0.3.14. SnpLinAlg sometimes does not run in parallel, but not sure why. Re-opening this issue.

kose-y commented 2 years ago

resolved by #119.