search

BIDData / BIDMat

A CPU and GPU-accelerated matrix library for data mining
BSD 3-Clause "New" or "Revised" License
265 stars 73 forks source link

Creating a Matrix with more than 2,147,483,647 elements #71

Open amedeedaboville opened 8 years ago

amedeedaboville commented 8 years ago

Hi, The matrix dimensions in BidMat use the Int type, which seems to be limiting the number of elements I can fit in a matrix. After 2147483647 elements there are calculations in the Matrix class about the total number of elements that overflow and return a java.lang.NegativeArraySizeException.

As a background, my dataset is at most 15 billion numbers, as I am doing a PCA of ~ 1 million rows x 15,000 columns.. My use case is comparing the speed of this implementation to randomized SVD in BidMach/Mat and randomized SVD in numpy. With 1.5 billion elements BidMach performed excellently (10s for the BidMach SVD with dim=50 on a beefy machine) but I cannot go further with it because I can't fit any more values in my matrices.

Cheers, Amedee

jcanny commented 8 years ago

Hi, the 2B limit for basic matrices is pretty fundamental. Its not really a BIDMat decision - since we build on Java, we inherit Java's limitation for array size to 2B elements.

But, you can use TMats. TMat is a new matrix type in BIDMat that contains tiles. Each tile will be limited to 2B elements, but your overall matrix can be larger. TMat's support many of the standard array ops, so you may not to do anything to use them in your code. If something you need is not implemented, let us know.

several27 commented 7 years ago

@jcanny So what would be the best way to convert the matrix (or load it from a file) using TMat? Is there any automated process of doing that, a function that would divide the huge matrix automatically into smaller ones and put them into TMat? I looked into it, but there seem to be no documentation for this matrix type and it looks like one has to specify all of the parameters (number of matrices, sizes etc) manually to create TMat.