search

jblas-project / jblas

Linear Algebra for Java
http://jblas.org
BSD 3-Clause "New" or "Revised" License
590 stars 149 forks source link

Int overflow when doing QR decomposition on 1 million rows #100

Closed jimlad999 closed 6 years ago

jimlad999 commented 6 years ago

This may be out of scope for what jblas was intended for but we are trying to decompose a large matrix (> 1 million rows). When we call Decompose.qr(m) we get the following error

java.lang.NegativeArraySizeException
        at org.jblas.DoubleMatrix.<init>(DoubleMatrix.java:342)
        at org.jblas.DoubleMatrix.eye(DoubleMatrix.java:525)
        at org.jblas.Decompose.qr(Decompose.java:211)

Digging a little bit, this is because it will try can create the eye matrix of size m.rows^2 which will overflow the 32-bit version of Int and try to create a negative sized array.

Is there a 64-bit version we could use that wouldn't face this or is there some better way to get the Q matrix for a matrix of this size? Any help with this would be much appreciated.

If this is out of scope then please feel free to close this ticket.

Thank you in advanced.

mikiobraun commented 6 years ago

Hey, jimlad999!

I think switching to 64bit won't help as well, as the m.row^2 matrix also won't fit into memory (1e6^2 is 1e12 * 8 bytes per double results in about 7500 GB, not speaking of the time it will take to converge. I assume the matrix itself is rectangular? I think you'd probably have to dig deeper into the right algorithm to make this kind of decomposition work.

Hope this helps!

mikiobraun commented 6 years ago

And congrats in being issue #100! ;)

jimlad999 commented 6 years ago

@mikiobraun, thank you for your response.

Yes. It is a rectangular matrix (tall and skinny) with only 3 columns. I kind of figured that the matrix was going to be too big. I will look into other algorithms as you suggested.

Thank you for all you help :)