dievsky
commented
2 years ago The problem here is that the JVM's array size is inherently limited by the maximum value of the (signed) int32. The best that we can do for now is to produce a better error message (and also account for the case where the int32 truncation leads to a positive number and causes problems further down the line). We can in theory go around this limitation by allocating nested arrays and providing some sort of access to them, but that's a much more difficult undertaking.
When trying to load a very large matrix (8312274035 elements) stored in npy format, I get a "NegativeArraySizeException". It seems, that the operation to calculate the size being int32 limits the maximal number of elements (line 184 in NpyFile.class, int32(8312274035) = -277660557).
Maybe, there is a possibility to make this fit for large arrays?