search

GPUPeople / spECK

Efficient SpGEMM on GPU using CUDA and CSR
MIT License
50 stars 16 forks source link

What is the largest scale spECK can work on? #5

Closed BarclayII closed 2 years ago

BarclayII commented 2 years ago

Thanks for building this library! I was looking for alternatives of cuSPARSE's CSRGEMM for quite a while and I was wondering what is the largest scale (number of nnzs, rows, and columns) you have conducted in your experiments.

The context is that I was working on GPU SpSpMM functions in DGL using cuSPARSE and people reported that cuSPARSE has some trouble handling matrices with a lot of NNZs (e.g. ~10M), even if the number of rows and columns are small (e.g. ~9000). cuSPARSE's SpSpMM also has some other problems which I wasn't able to debug. So I'm thinking of migrating to spECK.

I was also wondering whether it's easy to adapt spECK for int64 sparse matrices.

Thanks!

dabeschte commented 2 years ago

Hi!

We definitely also used larger matrices in our experiments, so that should not be a problem. But I have to say that cuSPARSE was very reliable and able to solve all matrices correctly as long as it did not run out of memory. But that was already some years ago, they could have updated their algorithms and use more memory now. With around 9k x 9k matrices, in the worst case the result is a dense matrix with 81M elements. That should not be a problem, even on entry level GPUs with less memory.

@int64: this unfortunately requires a little more work since spECK relies on atomic operations and cuda does not support signed int64 atomic operations. However, you can use int32 or uint64 by adding a line for the required datatype here. Duplicate the line for double datatype and replace all "double" with one of the supported datatypes. You may also need to do the same for all the definitions here. I added uint64 support in the latest commit now.

BarclayII commented 2 years ago

Thanks for the quick addition! Really appreciate it.

Sorry that I didn't make myself clear, but I intended to ask what I should change to support int64 indptr and indices (or making row_offsets and col_ids of your dCSR int64 arrays).

dabeschte commented 2 years ago

Sorry for taking so long.

I quickly tried changing the typed of row_offsets and col_ids, but it seems to require more work than I hoped...even though I used the "IndexType" as template in the entire project. Does DGL only support int64? seems like a waste of resources anyways in most cases, as matrices with > 4B elements or >4B columns might require huge memory. Are you working on such large matrices? Or could you convert the indices to uint32 before using spECK?

BarclayII commented 2 years ago

It's OK since DGL does support int32. I was just curious.