fminscg gets stuck in a while loop

[hyunjik11]

I'm fitting a VAR GP on some Mauna Loa CO2 time series data, using a rather complicated kernel. Here is a snippet of the code I'm using:

m=160; %number of inducing points
signal_var=0.1;l1=0.10;sf1=0.1;cs2=1.00;l2=1.00;sf2=0.1;lper=1.00;sfper=0.1;l3=1.00;sf3=0.1;lrq=1.00;sfrq=0.1; per=1/x_std; alpha=2; %initial hyperparameter values
lik = lik_gaussian('sigma2', signal_var);
gpcf_se1 = gpcf_sexp('lengthScale', l1, 'magnSigma2',sf1); 
gpcf_lin=gpcf_linear('coeffSigma2',cs2);
gpcf1=gpcf_prod('cf',{gpcf_se1,gpcf_lin});
gpcf_se2 = gpcf_sexp('lengthScale', l2, 'magnSigma2', sf2);
gpcf_per = gpcf_periodic('lengthScale',lper,'period',per,'magnSigma2',sfper);
gpcf2=gpcf_prod('cf',{gpcf_se2,gpcf_per});
gpcf_se3 = gpcf_sexp('lengthScale', l3, 'magnSigma2', sf3); 
gpcf_rat = gpcf_rq('lengthScale',lrq,'magnSigma2',sfrq,'alpha',2); 
gpcf3=gpcf_prod('cf',{gpcf_se3,gpcf_rat});
X_u=datasample(x,m,1,'Replace',false);
gp_var = gp_set('type', 'VAR', 'lik', lik, 'cf',{gpcf1,gpcf2,gpcf3},'X_u', X_u); 
gp_var = gp_set(gp_var, 'infer_params', 'covariance+likelihood+inducing');

opt=optimset('TolFun',1e-3,'TolX',1e-4,'Display','iter','MaxIter',1000);
gp_var=gp_optim(gp_var,x,y,'opt',opt);

For some values of X_u, the optimiser gets stuck at the following section of 'fminscg.m' (lines184-189):

while ((any(isinf(gplus)) || any(isnan(gplus)))) && ~isnan(fold)
      sigma=2*sigma;
      kappa=0.5.^2*kappa;
      xplus=x+sigma*d;
      [tmp,gplus] = fun(xplus);
end

Here fun is @(ww)gp_eg(ww,gp,x,y,'z',z), so tmp and gplus are the energy and its gradients with respect to the hyperparameters and the inducing points. Both are constantly computed to be NaN, which gets the while loop stuck. Have you ever experienced this problem before? Do you have any suggestions?

[avehtari]

Could you test if fminlbfgs gets stuck? gp_var=gp_optim(gp_var,x,y,'opt',opt,'optimf',@fminlbfgs);

[hyunjik11]

Thanks for the quick reply. I get a similar problem with fminlbfgs. This time for m=320, it gets stuck at the following section of 'fminlbfgs.m' (lines 745-751):

    while isnan(f_alpha) || isinf(f_alpha) || any(isnan(grad)) || any(isinf(grad))
      % NaN or Inf encountered, switch to safe mode
      here_be_dragons=true;
      alphaMax=alpha;
      alpha = alphaPrev+0.25*(alpha-alphaPrev);
      [data,f_alpha, grad]=gradient_function(data.xInitial(:)+alpha*data.dir(:),funfcn, data, optim);
    end

For both cg and bfgs, it seems as though the optimiser is more likely to get stuck with more inducing points.

[avehtari]

Better priors on hyperparameters would probably help. You could check whether some of the hyperparameters have ridiculous values when it gets stuck and then think about weakly informative priors to keep the optimization in a more feasible region.

[hyunjik11]

Ah yes indeed the length scale for gpcf3 was becoming a bit big (not ridiculous though, something like exp(1.14)=3.1; the inputs have been whitened to have unit variance), which was causing the NaN. Using a Gaussian prior with variance 0.5, instead of the default student-t distribution with nu=4, does the job. This stronger prior does however give me a looser variational lower bound when the optimiser doesn't get stuck with the default prior, but I guess a compromise is needed. I think it would be good to have a flag for when the optimiser spends too long on a while loop. Thanks for the help!