search

jblas-project / jblas

Linear Algebra for Java
http://jblas.org
BSD 3-Clause "New" or "Revised" License
590 stars 149 forks source link

QR decomp OutOfMemory #108

Closed spinicrus closed 4 years ago

spinicrus commented 6 years ago

Hi, thanks for jblas!

got stuck doing qr decomp on a 40k:500 DoubleMatrix. -Xmx9g, but it doesn't reach it, somewhere at 2.5g+ it happens. Any advice would be appreciated, thanks a lot. (maven installed on os x 10.11.6 (el capitan) i5)

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at org.jblas.DoubleMatrix.<init>(DoubleMatrix.java:342)
    at org.jblas.DoubleMatrix.eye(DoubleMatrix.java:525)
    at org.jblas.Decompose.qr(Decompose.java:211)
mikiobraun commented 6 years ago

Hi spinicrus,

the problem I think the problem is that your (40k, 500) matrix would be decomposed into Q (40k, 40k) and R (40k, 500) and Q would consume about 12GB of memory (40k40k8), and you'd also have a copy to call the native side of things. What do you want to achieve? Maybe if you'd do a QR decomposition on the transpose, would that help?

-M

spinicrus commented 6 years ago

yes you are right M, thank you very much! i need it in a method that does merging of two simple lsa models (for some text analysis), and for that i would need q&r. but maybe i can dance a little :) although as a quick temp resolve i used mtj just for this particular task.