search

haifengl / smile

Statistical Machine Intelligence & Learning Engine
https://haifengl.github.io
Other
5.99k stars 1.12k forks source link

GP Regression HPO IndexOutOfBoundsException #693

Closed DirkToewe closed 2 years ago

DirkToewe commented 2 years ago

The following code fails with an IndexOutOfBoundsException:

double[][] X = {{3.0}, {4.0}, {4.0}, {4.0}, {9.0}, {10.0}, {10.0}, {10.0}, {10.0}, {10.0}};
double[]   y = {9.0, 12.0, 11.0, 9.0, 4.0, 0.0, 0.0, 0.0, 0.0, 0.0};

var gps = GaussianProcessRegression.fit(
  X,y, new GaussianKernel(3),
  /*noise=*/1e-5, /*normalize=*/false, /*tol=*/1e-5, /*masIter=*/1024
);

Stack trace:

Exception in thread "main" java.lang.IndexOutOfBoundsException: 16
    at java.base/java.nio.HeapDoubleBuffer.put(HeapDoubleBuffer.java:218)
    at smile.math.matrix.Matrix.set(Matrix.java:684)
    at smile.math.BFGS.minimize(BFGS.java:719)
    at smile.regression.GaussianProcessRegression.fit(GaussianProcessRegression.java:368)
    at com.github.jaaa.GPRExample.main(GPRExample.java:11)

If I am not mistaken, the resizing of W in L-BFGS-B optimizer seems to be tied to the wrong condition (BFGS.java L702).

if (iter <= m)

should be

if( W.ncols() < 2*h )

right?

Additional context

haifengl commented 2 years ago

It is L-BFGS. The first m steps need special handling. Can you try a real multi-dimensional data?

DirkToewe commented 2 years ago

Can you try a real multi-dimensional data?

In my case, the real data is in fact univariate (noisy performance curves). But I can try to find a multivariate example as well.

The first m steps need special handling.

It should be the first m updates not the first m steps, right? I do not understand the code in its entirety, but it seems like sometime updates are rejected if they are not guaranteed to be positive definite or not positive definite "enough", see BFGS.java#L687. That means there may be more steps than updates, if I am not mistaken.

Edit: Here is a 3D example using the Cheddar Cheese Dataset which produced the same IndexOutOfBoundsException:

double[][] X = {
  {4.543,  3.135, 0.86},
  {5.159,  5.043, 1.53},
  {5.366,  5.438, 1.57},
  {5.759,  7.496, 1.81},
  {4.663,  3.807, 0.99},
  {5.697,  7.601, 1.09},
  {5.892,  8.726, 1.29},
  {6.078,  7.966, 1.78},
  {4.898,  3.85,  1.29},
  {5.242,  4.174, 1.58},
  {5.74 ,  6.142, 1.68},
  {6.446,  7.908, 1.9 },
  {4.477,  2.996, 1.06},
  {5.236,  4.942, 1.3 },
  {6.151,  6.752, 1.52},
  {6.365,  9.588, 1.74},
  {4.787,  3.912, 1.16},
  {5.412,  4.7  , 1.49},
  {5.247,  6.174, 1.63},
  {5.438,  9.064, 1.99},
  {4.564,  4.949, 1.15},
  {5.298,  5.22,  1.33},
  {5.455,  9.242, 1.44},
  {5.855, 10.199, 2   },
  {5.366,  3.664, 1.31},
  {6.043,  3.219, 1.46},
  {6.458,  6.962, 1.72},
  {5.328,  3.912, 1.25},
  {5.802,  6.685, 1.08},
  {6.176,  4.787, 1.25}
};
double[] y = {
  12.3, 20.9, 39, 47.9, 5.6, 25.9, 37.3, 21.9, 18.1, 21, 34.9, 57.2, 0.7, 25.9, 54.9,
  40.9, 15.9, 6.4, 18, 38.9, 14, 15.2, 32, 56.71, 16.8, 11.6, 26.5, 0.7, 13.4, 5.5
};
var gps = GaussianProcessRegression.fit(
  X,y, new GaussianKernel(3),
  /*noise=*/1e-5, /*normalize=*/false, /*tol=*/1e-5, /*masIter=*/1024
);
haifengl commented 2 years ago

I fixed it. Please try master branch. thanks!

DirkToewe commented 2 years ago

No more OOB issues. Fitting works really well now. Thank You!

ZeliangSu commented 2 years ago

I'm still getting OOB issues, using the above two examples