JuliaSparse / SparseArrays.jl

SparseArrays.jl is a Julia stdlib
https://sparsearrays.juliasparse.org/
Other
90 stars 52 forks source link

getindex is very slow for adjoints of sparse arrays #36

Open Pbellive opened 5 years ago

Pbellive commented 5 years ago

Calling getindex on an adjoint of a sparse array falls back (tested on 1.2 and on master built from source today, both on Linux) to a general AbstractArray method, making it unusable for non-scalar indexing with large sparse arrays. Consider the following example:

julia> versioninfo()
Julia Version 1.2.0
Commit c6da87ff4b (2019-08-20 00:03 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i9-9820X CPU @ 3.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)
julia> using SparseArrays
julia> n = 100000;m = 10000; nf = 25000;

julia> a = sprandn(n,m, 0.0005);

julia> bt = sprandn(m,n, 0.0005)';

julia> @time c = a[1:nf, :];
  0.091680 seconds (208.92 k allocations: 12.387 MiB)

julia> @time c = a[1:nf, :];
  0.007136 seconds (12 allocations: 1.976 MiB)

julia> @time c = bt[1:nf, :];
  9.522862 seconds (444.40 k allocations: 37.878 MiB, 0.55% gc time)

julia> @time c = bt[1:nf, :];
  9.280306 seconds (17 allocations: 15.308 MiB)

This was discussed last year on Discourse. That thread contains another example of the issue. A PR with a fix for sparse vectors (https://github.com/JuliaLang/julia/pull/28654) was opened not long after that discussion but seems to have been abandoned. I don't have time to resurrect the PR at the moment but I thought it would be good to have an issue open to track this. Is fixing the issue on anyone's radar at the moment?

stevengj commented 4 months ago

See also https://discourse.julialang.org/t/operation-on-transpose-sparse-array-is-slow/114330