cross_validate_nmf giving NAN tolerance and overfit

Hello singlet team!

First off, thank you so much for creating this package. This and RcppML have really sped up my workflow and made my life much easier!

I have been using cross_validate_nmf to decide the number of ranks for NMF on my datasets. However, I am occasionally running into this error:

> singlet::cross_validate_nmf(A, ranks = ranks,
                                    n_replicates = 3,
                                    verbose=3)
running with dense optimization
k = 50, rep = 1 (1/9):

iter |      tol |  overfit 
---------------------------
   1 | 8.67e-01 | 0.00e+00
   2 | 5.52e-02 |        -
   3 | 2.07e-02 |        -
   4 | 1.12e-02 |        -
   5 | 7.07e-03 |        -
   6 | 4.92e-03 | 0.00e+00
   7 | 3.63e-03 |        -
   8 | 2.79e-03 |        -
   9 | 2.23e-03 |        -
  10 | 1.84e-03 |        -
  11 | 1.57e-03 | 1.12e-03
test set error: 6.3858e-02

k = 100, rep = 1 (2/9):

iter |      tol |  overfit 
---------------------------
   1 | 8.56e-01 | 0.00e+00
   2 | 6.45e-02 |        -
   3 | 2.49e-02 |        -
   4 | 1.44e-02 |        -
   5 | 9.87e-03 |        -
   6 | 7.48e-03 | 0.00e+00
   7 | 6.10e-03 |        -
   8 | 5.27e-03 |        -
   9 | 4.96e-03 |        -
  10 | 5.14e-03 |        -
  11 | 5.65e-03 | 1.96e-02
test set error: 6.7135e-02

overfitting detected, lower rank recommended
k = 200, rep = 1 (3/9):

iter |      tol |  overfit 
---------------------------
   1 | 8.40e-01 | 0.00e+00
   2 | 8.00e-02 |        -
   3 | 2.89e-02 |        -
   4 | 1.73e-02 |        -
   5 | 1.27e-02 |        -
   6 | 1.08e-02 | 1.29e-02
test set error: 6.6980e-02

overfitting detected, lower rank recommended
k = 50, rep = 2 (4/9):

iter |      tol |  overfit 
---------------------------
   1 | 8.65e-01 | 0.00e+00
   2 | 5.76e-02 |        -
   3 | 2.10e-02 |        -
   4 | 1.09e-02 |        -
   5 | 6.78e-03 |        -
   6 | 4.75e-03 | 0.00e+00
   7 | 3.60e-03 |        -
   8 | 2.82e-03 |        -
   9 | 2.29e-03 |        -
  10 | 1.92e-03 |        -
  11 | 1.63e-03 | 1.24e-03
test set error: 6.4045e-02

k = 100, rep = 2 (5/9):

iter |      tol |  overfit 
---------------------------
   1 | 8.54e-01 | 0.00e+00
   2 | 6.63e-02 |        -
   3 | 2.49e-02 |        -
   4 | 1.41e-02 |        -
   5 | 9.59e-03 |        -
   6 | 7.17e-03 | 0.00e+00
   7 | 5.71e-03 |        -
   8 | 4.86e-03 |        -
   9 | 4.52e-03 |        -
  10 | 4.29e-03 |        -
  11 | 4.19e-03 | 1.02e-02
test set error: 6.6021e-02

overfitting detected, lower rank recommended
k = 200, rep = 2 (6/9):

iter |      tol |  overfit 
---------------------------
   1 |      nan |      nan
test set error: NaN

Error in if (model$test_mse[[length(model$test_mse)]]/model$test_mse[[1]] >  : 
  missing value where TRUE/FALSE needed

Is there any explanation for why the tolerance and overfit are nan here? I am on singlet_0.99.38, RcppML_0.5.6, and RcppEigen_0.3.3.9.4. The replicate on which the error occurs also changes between runs.

Thank you so much!

PS: another issue I have run into is having to manually set options(RcppML.threads = 0) when I am using RcppML and singlet on Mac M2. On a linux system, this seems to be set automatically when I call library(RcppML).

In case it helps, here is a reproducible example:

> set.seed(1234)
> X <- matrix(runif(200), 10, 20)
> dim(X)
[1] 10 20
> cv <- singlet::cross_validate_nmf(X, ranks=c(5,10,15), n_replicates=3, verbose=3)
running with dense optimization
k = 5, rep = 1 (1/9):

iter |      tol |  overfit 
---------------------------
   1 | 1.93e-01 | 0.00e+00
   2 | 3.58e-02 |        -
   3 | 5.95e-03 |        -
   4 | 2.99e-03 |        -
   5 | 2.35e-03 |        -
   6 | 2.07e-03 | 0.00e+00
   7 | 1.95e-03 |        -
   8 | 1.50e-03 |        -
   9 | 1.15e-03 |        -
  10 | 8.44e-04 |        -
  11 | 4.72e-04 | 0.00e+00
  12 | 3.95e-04 |        -
  13 | 3.43e-04 |        -
  14 | 2.98e-04 |        -
  15 | 2.13e-04 |        -
  16 | 1.41e-04 | 0.00e+00
  17 | 1.01e-04 |        -
  18 | 7.69e-05 |        -
test set error: 4.2967e-02

k = 10, rep = 1 (2/9):

iter |      tol |  overfit 
---------------------------
   1 | 1.63e-01 | 0.00e+00
   2 | 4.49e-02 |        -
   3 | 1.95e-02 |        -
   4 | 7.14e-03 |        -
   5 | 2.60e-03 |        -
   6 | 1.70e-03 | 0.00e+00
   7 | 1.19e-03 |        -
   8 | 9.31e-04 |        -
   9 | 7.93e-04 |        -
  10 | 5.74e-04 |        -
  11 | 4.39e-04 | 2.18e-01
test set error: 1.6453e-01

k = 15, rep = 1 (3/9):

iter |      tol |  overfit 
---------------------------
   1 | 1.68e-01 | 0.00e+00
   2 | 5.81e-02 |        -
   3 | 1.62e-02 |        -
   4 | 3.30e-03 |        -
   5 | 1.16e-03 |        -
   6 | 7.85e-04 | 0.00e+00
   7 | 5.46e-04 |        -
   8 | 4.72e-04 |        -
   9 | 3.62e-04 |        -
  10 | 2.77e-04 |        -
  11 | 1.86e-04 | 0.00e+00
  12 | 1.27e-04 |        -
  13 | 9.41e-05 |        -
test set error: 8.0109e-02

k = 5, rep = 2 (4/9):

iter |      tol |  overfit 
---------------------------
   1 | 1.89e-01 | 0.00e+00
   2 | 8.65e-02 |        -
   3 | 1.84e-02 |        -
   4 | 7.67e-03 |        -
   5 | 2.85e-03 |        -
   6 | 1.77e-03 | 3.50e-01
test set error: 1.9986e-01

overfitting detected, lower rank recommended
k = 10, rep = 2 (5/9):

iter |      tol |  overfit 
---------------------------
   1 | 1.59e-01 | 0.00e+00
   2 | 4.66e-02 |        -
   3 | 1.26e-02 |        -
   4 | 8.73e-03 |        -
   5 | 5.77e-03 |        -
   6 | 2.25e-03 | 6.91e-03
test set error: 2.1501e-01

overfitting detected, lower rank recommended
k = 15, rep = 2 (6/9):

iter |      tol |  overfit 
---------------------------
   1 |      nan |      nan
test set error: NaN

Error in if (model$test_mse[[length(model$test_mse)]]/model$test_mse[[1]] >  : 
  missing value where TRUE/FALSE needed

zdebruine / singlet

cross_validate_nmf giving NAN tolerance and overfit #49