Open squirvel opened 3 years ago
Actually, I think I might have found what I was looking for... Should have a patch to push sometime in the next day or two
Update, this is in fact an issue with Math.Net.
Specifically, the constructor:
internal DenseColumnMajorMatrixStorage(int rows, int columns)
: base(rows, columns)
{
Data = new T[rows*columns];
}
in the DenseColumnMajorMatrixStorage.cs fails when multiplying rows*columns.
This cannot be worked around by compiling math.net and Ariadne as 64bit, as the C# int definition (even on 64 bit platforms) is specifically noted as being 32bits.
If the ints in math.net were 64bit and largeObjects were enabled (which LargeObjects is when forcing it to compile as 64bit) than this error would not exist. (Better yet, using BigInteger and Lists would ensure no size limits at all, without the 64bit requirement).
That said, I did write up a patch which swaps out the arrays for F# lists. (With the exception of the hyperparameter arrays.) I couldn't get the tests to run on my machine, but it seems to be running properly. Assuming Math.Net had BigIntegers and Lists over arrays, that patch might be useful. Regardless, if anyone would like me to post the patch, just let me know.
For now, I guess I will have to Guassian process a a set of Gaussian process outputs, or just lazily average them.
Re-opened. No good reason to close just because this library isn't the root cause of the problem.
I am working with a rather large sub-set of data (922854 location-position pairs). When attempting to do GuassianProcess.Predict() an integer overflow occurs, which results in an index out of bounds error. (Specifically, the index is being set to -2146233066).
I have narrowed in this issue starting somewhere between 45,000 and 50,000 pairs, with the library running and producing results without error when in the valid range.
A trace of the error is listed below:
I am calling the library from some C# code, so I am unsure if that would affect anything. I do not think that that is the case, but for the sake of completion, here is what my code looks like:
As it can be seen in the trace, this is not specifically occurring in Ariadne, but rather the underlying MathNet.Numerics library. Specifically in LinearAlgebra.MatrixBuilder
My current speculation is that: The list contained in xEnumerable is getting casted to an array. Because the MathNet library would be pre-compiled, likely without gcAllowVeryLargeObjects being enabled, we are literally running out of space, and overflowing. This could be avoided if we can work directly with lists rather than Int32 arrays.
However I am not sure if that is the case, and if were, I have not been able to find code that hints at such a casting occurring.