Only run generated quantities when needed for optimizing, sampling, and ADVI

alashworth commented 5 years ago

Issue by bob-carpenter Thursday Jan 18, 2018 at 17:40 GMT Originally opened as https://github.com/stan-dev/stan/issues/2459

Summary:

Right now,

optimization and ADVI create generated quantities each iteration
sampling creates generated quantities during warmup which aren't needed

Provide options so they're not saved when not needed.

Current Version:

v2.17.1

alashworth commented 5 years ago

Comment by bgoodri Thursday Jan 18, 2018 at 18:35 GMT

For MCMC sampling, there is already an option to save the warmup (of everything) or not. For optimization, ADVI, and many future algorithms, it only makes sense to generate quantities after the optimum has been found.

On Thu, Jan 18, 2018 at 12:40 PM, Bob Carpenter notifications@github.com wrote:

Summary:

Right now,

-

optimization and ADVI create generated quantities each iteration

sampling creates generated quantities during warmup which aren't needed

Provide options so they're not saved when not needed. Current Version:

v2.17.1

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/stan-dev/stan/issues/2459, or mute the thread https://github.com/notifications/unsubscribe-auth/ADOrqtY3E9ledBXfm-t5Jz7BhmkvstRlks5tL4ICgaJpZM4RjTXD .

alashworth commented 5 years ago

Comment by bob-carpenter Thursday Jan 18, 2018 at 22:20 GMT

Even if they're not saved, aren't the generated quantities executed during warmup?

If you save warmup, then you probably need to run generated quantities to make sure the number of "parameters" matches.

alashworth commented 5 years ago

Comment by VMatthijs Thursday Feb 01, 2018 at 15:03 GMT

Are we sure that generated quantities are computed at every step for ADVI? I've just been looking at the source code and get the impression that they are only computed once after the optimisation has terminated. For LBFGS, it does seem like they are computed at every step, but only if save_iterations is set to true.

My experiments confirm the observations above. Isn't this already the behaviour that we want? What should change about it?

Am I right in thinking that generated quantities are only computed by the write_array method? This even has a flag which you can set to not compute the generated quantities, so it should be really easy to fix this (based on my first impression).

What to do with HMC/NUTS during the warm-up, I am less sure about, because of the reasons outlined by Bob.

alashworth commented 5 years ago

Comment by bob-carpenter Thursday Feb 01, 2018 at 22:16 GMT

I don't know what ADVI does. It doesn't use the L-BFGS in algorithms for its optimization.

Sampling computes generated quantities during warmup whether the warmup iterations are saved or not.

I thought optimization computed generated quantities every iteration, but it should also only need them at the end.

How much are any of these saved values used for debugging? Probably not much because generated quatnities don't affect sampling.

You'll need to be careful that nothing requires rectangular input for "parameters" (everything dumped out including transformed parameters and generated quantities) in RStan or PyStan.

alashworth commented 5 years ago

Comment by betanalpha Thursday Feb 01, 2018 at 22:36 GMT

Huh? Generated quantities is called only when the parameters are requested to be written out and that occurs only if the “save_warmup” argument is set to true,

https://github.com/stan-dev/stan/blob/develop/src/stan/services/util/run_adaptive_sampler.hpp https://github.com/stan-dev/stan/blob/develop/src/stan/services/util/run_adaptive_sampler.hpp

If interfaces other than CmdStan modify this to always save warmup iterations then this is an interface issue, not a core Stan issue.

On Feb 1, 2018, at 5:16 PM, Bob Carpenter notifications@github.com wrote:

I don't know what ADVI does. It doesn't use the L-BFGS in algorithms for its optimization.

Sampling computes generated quantities during warmup whether the warmup iterations are saved or not.

I thought optimization computed generated quantities every iteration, but it should also only need them at the end.

How much are any of these saved values used for debugging? Probably not much because generated quatnities don't affect sampling.

You'll need to be careful that nothing requires rectangular input for "parameters" (everything dumped out including transformed parameters and generated quantities) in RStan or PyStan.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/stan-dev/stan/issues/2459#issuecomment-362420626, or mute the thread https://github.com/notifications/unsubscribe-auth/ABdNloleFN8DHTMquS-1AF41kUHQ4nJHks5tQjfDgaJpZM4RjTXD.

alashworth commented 5 years ago

Comment by bob-carpenter Friday Feb 02, 2018 at 18:08 GMT

On Feb 1, 2018, at 5:36 PM, Michael Betancourt notifications@github.com wrote:

Huh? Generated quantities is called only when the parameters are requested to be written out and that occurs only if the “save_warmup” argument is set to true,

Great. That's what I think it should do. I think we were running into issues where running generated quantities during warmup (I believe RStan saves warmup by default) led to crashing behavior due to numerical issues. I think the right behavior is to write out NaN values if generated quantities throws an exception, but I don't know if that ever got merged.

If we can turn off saving warmup, I don't see there's much else we can do at the interface level.

alashworth commented 5 years ago

Comment by bgoodri Friday Feb 02, 2018 at 19:52 GMT

This issue is supposed to be about optimization and ADVI, which do not have a save warmup flag. Run this

library(rstan)
m <- stan_model(model_code = 'parameters {real y;} model {y ~ normal(0,1);} generated quantities {reject("this gets executed");}')
optimizing(m) # fails on initial evaluation
optimizing(m, save_warmup = TRUE) # fails due to lack of a save_warmup option
vb(m) # fails after adaptation but before optimization
vb(m, save_warmup = TRUE) # fails due to lack of a save_warmup option

alashworth commented 5 years ago

Comment by VMatthijs Friday Feb 02, 2018 at 19:54 GMT

Could this be an RStan issue? I just don't think I saw this behaviour in CmdStan, either in my experiments or in the source code.

alashworth commented 5 years ago

Comment by bgoodri Friday Feb 02, 2018 at 20:02 GMT

Fine. Save this to a file called gc.stan

parameters {
  real y;
} 
model {
  y ~ normal(0,1);
} 
generated quantities {
  reject("this gets executed");
}

build it in CmdStan and try to run it with optimizing or vb.

./gc optimize # fails on initial evaluation
./gc optimize save_iterations=0 # fails on initial evaluaion

alashworth commented 5 years ago

Comment by VMatthijs Friday Feb 02, 2018 at 20:12 GMT

Done. This is the output:

method = optimize optimize algorithm = lbfgs (Default) lbfgs init_alpha = 0.001 (Default) tol_obj = 9.9999999999999998e-13 (Default) tol_rel_obj = 10000 (Default) tol_grad = 1e-08 (Default) tol_rel_grad = 10000000 (Default) tol_param = 1e-08 (Default) history_size = 5 (Default) iter = 2000 (Default) save_iterations = 0 (Default) id = 0 (Default) data file = (Default) init = 2 (Default) random seed = 1598959552 output file = output.csv (Default) diagnostic_file = (Default) refresh = 100 (Default)

Initial log joint probability = -0.835716 Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes 2 -2.46519e-32 1.16356 2.22045e-16 1 1 5 Exception: this gets executed (in 'stan/src/test/test-models/good/gqfail.stan' at line 8)

method = variational variational algorithm = meanfield (Default) meanfield iter = 10000 (Default) grad_samples = 1 (Default) elbo_samples = 100 (Default) eta = 1 (Default) adapt engaged = 1 (Default) iter = 50 (Default) tol_rel_obj = 0.01 (Default) eval_elbo = 100 (Default) output_samples = 1000 (Default) id = 0 (Default) data file = (Default) init = 2 (Default) random seed = 1598968164 output file = output.csv (Default) diagnostic_file = (Default) refresh = 100 (Default)

EXPERIMENTAL ALGORITHM: This procedure has not been thoroughly tested and may be unstable or buggy. The interface is subject to change.

Gradient evaluation took 0 seconds 1000 transitions using 10 leapfrog steps per transition would take 0 seconds. Adjust your expectations accordingly!

Begin eta adaptation. Iteration: 1 / 250 [ 0%] (Adaptation) Iteration: 50 / 250 [ 20%] (Adaptation) Iteration: 100 / 250 [ 40%] (Adaptation) Iteration: 150 / 250 [ 60%] (Adaptation) Iteration: 200 / 250 [ 80%] (Adaptation) Success! Found best value [eta = 1] earlier than expected.

Begin stochastic gradient ascent. iter 100 -0.019 200 -0.053 300 -0.072 400 0.138 500 -0.072 600 -0.002 700 -0.044 800 0.110 900 -0.000 1000 0.111 1100 -0.036 1200 0.004 1300 -0.058 1400 -0.007 1500 -0.005 1600 0.038 1700 -0.080 1800 -0.021 1900 0.023 2000 0.050 2100 -0.020 2200 -0.161 2300 -0.061 2400 -0.164 2500 0.002 2600 0.042 2700 0.069 2800 0.006 2900 0.100 3000 0.081 3100 0.050 3200 -0.042 3300 0.175 3400 -0.012 3500 -0.024 3600 0.087 3700 0.002 3800 -0.144 3900 -0.028 4000 -0.109 4100 -0.137 4200 0.098 4300 0.133 4400 0.050 4500 0.048 4600 0.108 4700 -0.030 4800 0.004 4900 0.065 5000 -0.105 5100 -0.002 5200 0.058 5300 0.011 5400 -0.073 5500 0.105 5600 -0.073 5700 -0.126 5800 -0.064 5900 -0.123 6000 0.011 6100 0.075 6200 -0.005 6300 0.052 6400 0.052 6500 0.024 6600 0.047 6700 0.034 6800 -0.046 6900 -0.150 7000 0.035 7100 0.044 7200 -0.123 7300 0.065 7400 -0.000 7500 0.059 7600 -0.086 7700 0.035 7800 -0.007 7900 -0.033 8000 -0.018 8100 -0.028 8200 -0.034 8300 -0.235 8400 -0.016 8500 0.086 8600 0.037 8700 -0.080 8800 0.082 8900 0.047 9000 -0.042 9100 0.010 9200 -0.080 9300 -0.068 9400 -0.024 9500 -0.198 9600 -0.014 9700 -0.146 9800 0.054 9900 0.066 10000 -0.019 Informational This variational Exception: this gets executed ELBO delta_ELBO_mean delta_ELBO_med notes 1.000 1.000 0.822 1.000 0.633 0.643 0.855 1.000 1.269 1.000 6.053 1.521 5.323 1.000 4.833 1.399 40.428 1.399 36.486 1.399 36.790 1.521 MAY BE DIVERGING... INSPECT ELBO 37.650 2.923 MAY BE DIVERGING... INSPECT ELBO 37.732 2.923 MAY BE DIVERGING... INSPECT ELBO 38.265 4.037 MAY BE DIVERGING... INSPECT ELBO 38.034 4.037 MAY BE DIVERGING... INSPECT ELBO 35.149 1.399 MAY BE DIVERGING... INSPECT ELBO 35.201 1.471 MAY BE DIVERGING... INSPECT ELBO 35.350 2.888 MAY BE DIVERGING... INSPECT ELBO 3.019 1.885 MAY BE DIVERGING... INSPECT ELBO 2.972 1.885 MAY BE DIVERGING... INSPECT ELBO 2.916 1.885 MAY BE DIVERGING... INSPECT ELBO 2.079 1.471 MAY BE DIVERGING... INSPECT ELBO 2.138 1.667 MAY BE DIVERGING... INSPECT ELBO 1.516 1.471 MAY BE DIVERGING... INSPECT ELBO 12.296 1.667 MAY BE DIVERGING... INSPECT ELBO 12.280 1.667 MAY BE DIVERGING... INSPECT ELBO 12.172 1.667 MAY BE DIVERGING... INSPECT ELBO 12.913 1.667 MAY BE DIVERGING... INSPECT ELBO 12.818 0.964 MAY BE DIVERGING... INSPECT ELBO 12.789 0.964 MAY BE DIVERGING... INSPECT ELBO 12.504 0.939 MAY BE DIVERGING... INSPECT ELBO 12.636 0.964 MAY BE DIVERGING... INSPECT ELBO 12.593 0.964 MAY BE DIVERGING... INSPECT ELBO 14.151 1.238 MAY BE DIVERGING... INSPECT ELBO 3.362 0.964 MAY BE DIVERGING... INSPECT ELBO 3.393 1.238 MAY BE DIVERGING... INSPECT ELBO 6.993 1.276 MAY BE DIVERGING... INSPECT ELBO 6.064 1.238 MAY BE DIVERGING... INSPECT ELBO 6.391 1.276 MAY BE DIVERGING... INSPECT ELBO 6.442 1.276 MAY BE DIVERGING... INSPECT ELBO 6.399 1.276 MAY BE DIVERGING... INSPECT ELBO 6.419 1.276 MAY BE DIVERGING... INSPECT ELBO 6.321 1.276 MAY BE DIVERGING... INSPECT ELBO 4.867 1.276 MAY BE DIVERGING... INSPECT ELBO 4.818 1.276 MAY BE DIVERGING... INSPECT ELBO 4.746 1.016 MAY BE DIVERGING... INSPECT ELBO 1.567 1.016 MAY BE DIVERGING... INSPECT ELBO 2.247 1.667 MAY BE DIVERGING... INSPECT ELBO 1.920 0.932 MAY BE DIVERGING... INSPECT ELBO 2.007 1.615 MAY BE DIVERGING... INSPECT ELBO 6.202 1.667 MAY BE DIVERGING... INSPECT ELBO 6.067 1.615 MAY BE DIVERGING... INSPECT ELBO 6.467 1.667 MAY BE DIVERGING... INSPECT ELBO 6.416 1.615 MAY BE DIVERGING... INSPECT ELBO 6.582 1.690 MAY BE DIVERGING... INSPECT ELBO 6.770 2.433 MAY BE DIVERGING... INSPECT ELBO 6.351 1.690 MAY BE DIVERGING... INSPECT ELBO 5.665 1.615 MAY BE DIVERGING... INSPECT ELBO 5.619 1.615 MAY BE DIVERGING... INSPECT ELBO 6.644 1.690 MAY BE DIVERGING... INSPECT ELBO 2.513 1.153 MAY BE DIVERGING... INSPECT ELBO 4.042 1.690 MAY BE DIVERGING... INSPECT ELBO 3.725 1.153 MAY BE DIVERGING... INSPECT ELBO 3.611 1.095 MAY BE DIVERGING... INSPECT ELBO 3.561 1.095 MAY BE DIVERGING... INSPECT ELBO 3.367 0.952 MAY BE DIVERGING... INSPECT ELBO 3.362 0.952 MAY BE DIVERGING... INSPECT ELBO 3.442 1.095 MAY BE DIVERGING... INSPECT ELBO 3.464 1.095 MAY BE DIVERGING... INSPECT ELBO 2.812 1.095 MAY BE DIVERGING... INSPECT ELBO 2.749 1.095 MAY BE DIVERGING... INSPECT ELBO 1.252 1.095 MAY BE DIVERGING... INSPECT ELBO 1.432 1.188 MAY BE DIVERGING... INSPECT ELBO 461.421 1.360 MAY BE DIVERGING... INSPECT ELBO 461.402 1.360 MAY BE DIVERGING... INSPECT ELBO 461.521 1.682 MAY BE DIVERGING... INSPECT ELBO 461.829 1.752 MAY BE DIVERGING... INSPECT ELBO 462.243 2.893 MAY BE DIVERGING... INSPECT ELBO 462.251 2.893 MAY BE DIVERGING... INSPECT ELBO 461.798 1.682 MAY BE DIVERGING... INSPECT ELBO 461.812 1.682 MAY BE DIVERGING... INSPECT ELBO 461.693 1.682 MAY BE DIVERGING... INSPECT ELBO 461.489 1.000 MAY BE DIVERGING... INSPECT ELBO 2.857 1.000 MAY BE DIVERGING... INSPECT ELBO 2.875 1.187 MAY BE DIVERGING... INSPECT ELBO 2.837 1.187 MAY BE DIVERGING... INSPECT ELBO 2.639 1.187 MAY BE DIVERGING... INSPECT ELBO 2.249 1.187 MAY BE DIVERGING... INSPECT ELBO 2.243 1.187 MAY BE DIVERGING... INSPECT ELBO 2.375 1.301 MAY BE DIVERGING... INSPECT ELBO 2.867 1.467 MAY BE DIVERGING... INSPECT ELBO 2.963 1.467 MAY BE DIVERGING... INSPECT ELBO 2.895 1.467 MAY BE DIVERGING... INSPECT ELBO 1.722 1.467 MAY BE DIVERGING... INSPECT ELBO 1.691 1.467 MAY BE DIVERGING... INSPECT ELBO 2.844 1.838 MAY BE DIVERGING... INSPECT ELBO 2.787 1.838 MAY BE DIVERGING... INSPECT ELBO 2.958 1.838 MAY BE DIVERGING... INSPECT ELBO 2.902 1.838 MAY BE DIVERGING... INSPECT ELBO 3.132 1.838 MAY BE DIVERGING... INSPECT ELBO Message: The maximum number of iterations is reached! The algorithm may not have converged. approximation is not guaranteed to be meaningful. (in 'stan/src/test/test-models/good/gqfail.stan' at line 8)

alashworth commented 5 years ago

Comment by bgoodri Friday Feb 02, 2018 at 20:16 GMT

The vb seems different than what I got in rstan, but that may have just been a random failure. The optimization is what Andrew has been complaining about for months if not years. He wants it to be faster than literally every other optimization algorithm, which it will never be, but one reason is that it executes generating quantities at all the suboptimal points, which is why you get that exception after 1 function evaluation.

On Fri, Feb 2, 2018 at 3:12 PM, Matthijs Vákár notifications@github.com wrote:

Done. This is the output: `method = optimize optimize algorithm = lbfgs (Default) lbfgs init_alpha = 0.001 (Default) tol_obj = 9.9999999999999998e-13 (Default) tol_rel_obj = 10000 (Default) tol_grad = 1e-08 (Default) tol_rel_grad = 10000000 (Default) tol_param = 1e-08 (Default) history_size = 5 (Default) iter = 2000 (Default) save_iterations = 0 (Default) id = 0 (Default) data file = (Default) init = 2 (Default) random seed = 1598959552 output file = output.csv (Default) diagnostic_file = (Default) refresh = 100 (Default)

Initial log joint probability = -0.835716 Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes 2 -2.46519e-32 1.16356 2.22045e-16 1 1 5 Exception: this gets executed (in 'stan/src/test/test-models/good/gqfail.stan' at line 8)`

`method = variational variational algorithm = meanfield (Default) meanfield iter = 10000 (Default) grad_samples = 1 (Default) elbo_samples = 100 (Default) eta = 1 (Default) adapt engaged = 1 (Default) iter = 50 (Default) tol_rel_obj = 0.01 (Default) eval_elbo = 100 (Default) output_samples = 1000 (Default) id = 0 (Default) data file = (Default) init = 2 (Default) random seed = 1598968164 output file = output.csv (Default) diagnostic_file = (Default) refresh = 100 (Default)

EXPERIMENTAL ALGORITHM: This procedure has not been thoroughly tested and may be unstable or buggy. The interface is subject to change.

Gradient evaluation took 0 seconds 1000 transitions using 10 leapfrog steps per transition would take 0 seconds. Adjust your expectations accordingly!

Begin eta adaptation. Iteration: 1 / 250 [ 0%] (Adaptation) Iteration: 50 / 250 [ 20%] (Adaptation) Iteration: 100 / 250 [ 40%] (Adaptation) Iteration: 150 / 250 [ 60%] (Adaptation) Iteration: 200 / 250 [ 80%] (Adaptation) Success! Found best value [eta = 1] earlier than expected.

Begin stochastic gradient ascent. iter ELBO delta_ELBO_mean delta_ELBO_med notes 100 -0.019 1.000 1.000 200 -0.053 0.822 1.000 300 -0.072 0.633 0.643 400 0.138 0.855 1.000 500 -0.072 1.269 1.000 600 -0.002 6.053 1.521 700 -0.044 5.323 1.000 800 0.110 4.833 1.399 900 -0.000 40.428 1.399 1000 0.111 36.486 1.399 1100 -0.036 36.790 1.521 MAY BE DIVERGING... INSPECT ELBO 1200 0.004 37.650 2.923 MAY BE DIVERGING... INSPECT ELBO 1300 -0.058 37.732 2.923 MAY BE DIVERGING... INSPECT ELBO 1400 -0.007 38.265 4.037 MAY BE DIVERGING... INSPECT ELBO 1500 -0.005 38.034 4.037 MAY BE DIVERGING... INSPECT ELBO 1600 0.038 35.149 1.399 MAY BE DIVERGING... INSPECT ELBO 1700 -0.080 35.201 1.471 MAY BE DIVERGING... INSPECT ELBO 1800 -0.021 35.350 2.888 MAY BE DIVERGING... INSPECT ELBO 1900 0.023 3.019 1.885 MAY BE DIVERGING... INSPECT ELBO 2000 0.050 2.972 1.885 MAY BE DIVERGING... INSPECT ELBO 2100 -0.020 2.916 1.885 MAY BE DIVERGING... INSPECT ELBO 2200 -0.161 2.079 1.471 MAY BE DIVERGING... INSPECT ELBO 2300 -0.061 2.138 1.667 MAY BE DIVERGING... INSPECT ELBO 2400 -0.164 1.516 1.471 MAY BE DIVERGING... INSPECT ELBO 2500 0.002 12.296 1.667 MAY BE DIVERGING... INSPECT ELBO 2600 0.042 12.280 1.667 MAY BE DIVERGING... INSPECT ELBO 2700 0.069 12.172 1.667 MAY BE DIVERGING... INSPECT ELBO 2800 0.006 12.913 1.667 MAY BE DIVERGING... INSPECT ELBO 2900 0.100 12.818 0.964 MAY BE DIVERGING... INSPECT ELBO 3000 0.081 12.789 0.964 MAY BE DIVERGING... INSPECT ELBO 3100 0.050 12.504 0.939 MAY BE DIVERGING... INSPECT ELBO 3200 -0.042 12.636 0.964 MAY BE DIVERGING... INSPECT ELBO 3300 0.175 12.593 0.964 MAY BE DIVERGING... INSPECT ELBO 3400 -0.012 14.151 1.238 MAY BE DIVERGING... INSPECT ELBO 3500 -0.024 3.362 0.964 MAY BE DIVERGING... INSPECT ELBO 3600 0.087 3.393 1.238 MAY BE DIVERGING... INSPECT ELBO 3700 0.002 6.993 1.276 MAY BE DIVERGING... INSPECT ELBO 3800 -0.144 6.064 1.238 MAY BE DIVERGING... INSPECT ELBO 3900 -0.028 6.391 1.276 MAY BE DIVERGING... INSPECT ELBO 4000 -0.109 6.442 1.276 MAY BE DIVERGING... INSPECT ELBO 4100 -0.137 6.399 1.276 MAY BE DIVERGING... INSPECT ELBO 4200 0.098 6.419 1.276 MAY BE DIVERGING... INSPECT ELBO 4300 0.133 6.321 1.276 MAY BE DIVERGING... INSPECT ELBO 4400 0.050 4.867 1.276 MAY BE DIVERGING... INSPECT ELBO 4500 0.048 4.818 1.276 MAY BE DIVERGING... INSPECT ELBO 4600 0.108 4.746 1.016 MAY BE DIVERGING... INSPECT ELBO 4700 -0.030 1.567 1.016 MAY BE DIVERGING... INSPECT ELBO 4800 0.004 2.247 1.667 MAY BE DIVERGING... INSPECT ELBO 4900 0.065 1.920 0.932 MAY BE DIVERGING... INSPECT ELBO 5000 -0.105 2.007 1.615 MAY BE DIVERGING... INSPECT ELBO 5100 -0.002 6.202 1.667 MAY BE DIVERGING... INSPECT ELBO 5200 0.058 6.067 1.615 MAY BE DIVERGING... INSPECT ELBO 5300 0.011 6.467 1.667 MAY BE DIVERGING... INSPECT ELBO 5400 -0.073 6.416 1.615 MAY BE DIVERGING... INSPECT ELBO 5500 0.105 6.582 1.690 MAY BE DIVERGING... INSPECT ELBO 5600 -0.073 6.770 2.433 MAY BE DIVERGING... INSPECT ELBO 5700 -0.126 6.351 1.690 MAY BE DIVERGING... INSPECT ELBO 5800 -0.064 5.665 1.615 MAY BE DIVERGING... INSPECT ELBO 5900 -0.123 5.619 1.615 MAY BE DIVERGING... INSPECT ELBO 6000 0.011 6.644 1.690 MAY BE DIVERGING... INSPECT ELBO 6100 0.075 2.513 1.153 MAY BE DIVERGING... INSPECT ELBO 6200 -0.005 4.042 1.690 MAY BE DIVERGING... INSPECT ELBO 6300 0.052 3.725 1.153 MAY BE DIVERGING... INSPECT ELBO 6400 0.052 3.611 1.095 MAY BE DIVERGING... INSPECT ELBO 6500 0.024 3.561 1.095 MAY BE DIVERGING... INSPECT ELBO 6600 0.047 3.367 0.952 MAY BE DIVERGING... INSPECT ELBO 6700 0.034 3.362 0.952 MAY BE DIVERGING... INSPECT ELBO 6800 -0.046 3.442 1.095 MAY BE DIVERGING... INSPECT ELBO 6900 -0.150 3.464 1.095 MAY BE DIVERGING... INSPECT ELBO 7000 0.035 2.812 1.095 MAY BE DIVERGING... INSPECT ELBO 7100 0.044 2.749 1.095 MAY BE DIVERGING... INSPECT ELBO 7200 -0.123 1.252 1.095 MAY BE DIVERGING... INSPECT ELBO 7300 0.065 1.432 1.188 MAY BE DIVERGING... INSPECT ELBO 7400 -0.000 461.421 1.360 MAY BE DIVERGING... INSPECT ELBO 7500 0.059 461.402 1.360 MAY BE DIVERGING... INSPECT ELBO 7600 -0.086 461.521 1.682 MAY BE DIVERGING... INSPECT ELBO 7700 0.035 461.829 1.752 MAY BE DIVERGING... INSPECT ELBO 7800 -0.007 462.243 2.893 MAY BE DIVERGING... INSPECT ELBO 7900 -0.033 462.251 2.893 MAY BE DIVERGING... INSPECT ELBO 8000 -0.018 461.798 1.682 MAY BE DIVERGING... INSPECT ELBO 8100 -0.028 461.812 1.682 MAY BE DIVERGING... INSPECT ELBO 8200 -0.034 461.693 1.682 MAY BE DIVERGING... INSPECT ELBO 8300 -0.235 461.489 1.000 MAY BE DIVERGING... INSPECT ELBO 8400 -0.016 2.857 1.000 MAY BE DIVERGING... INSPECT ELBO 8500 0.086 2.875 1.187 MAY BE DIVERGING... INSPECT ELBO 8600 0.037 2.837 1.187 MAY BE DIVERGING... INSPECT ELBO 8700 -0.080 2.639 1.187 MAY BE DIVERGING... INSPECT ELBO 8800 0.082 2.249 1.187 MAY BE DIVERGING... INSPECT ELBO 8900 0.047 2.243 1.187 MAY BE DIVERGING... INSPECT ELBO 9000 -0.042 2.375 1.301 MAY BE DIVERGING... INSPECT ELBO 9100 0.010 2.867 1.467 MAY BE DIVERGING... INSPECT ELBO 9200 -0.080 2.963 1.467 MAY BE DIVERGING... INSPECT ELBO 9300 -0.068 2.895 1.467 MAY BE DIVERGING... INSPECT ELBO 9400 -0.024 1.722 1.467 MAY BE DIVERGING... INSPECT ELBO 9500 -0.198 1.691 1.467 MAY BE DIVERGING... INSPECT ELBO 9600 -0.014 2.844 1.838 MAY BE DIVERGING... INSPECT ELBO 9700 -0.146 2.787 1.838 MAY BE DIVERGING... INSPECT ELBO 9800 0.054 2.958 1.838 MAY BE DIVERGING... INSPECT ELBO 9900 0.066 2.902 1.838 MAY BE DIVERGING... INSPECT ELBO 10000 -0.019 3.132 1.838 MAY BE DIVERGING... INSPECT ELBO Informational Message: The maximum number of iterations is reached! The algorithm may not have converged. This variational approximation is not guaranteed to be meaningful. Exception: this gets executed (in 'stan/src/test/test-models/good/gqfail.stan' at line 8)`

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/stan-dev/stan/issues/2459#issuecomment-362693906, or mute the thread https://github.com/notifications/unsubscribe-auth/ADOrqiItbP-1LFSr50EywCJ08ns9VHbAks5tQ2wvgaJpZM4RjTXD .

alashworth commented 5 years ago

Comment by VMatthijs Friday Feb 02, 2018 at 20:24 GMT

OK so is the behaviour of ADVI as intended?

For LBFGS, what's wrong about the current behaviour? It takes 5 steps before it's done optimising and then evaluates the GQs, right? Based on the source code, it should only evaluate GQs once, after it's done optimising.

alashworth commented 5 years ago

Comment by bgoodri Friday Feb 02, 2018 at 20:36 GMT

I think ADVI is as intended. For LBFGS, if you change the generated quantities to a print message rather than a reject, in CmdStan it is only printing once at the end. But in rstan, it is printing multiple times before it reaches the optimum despite the C++ just calling

        stan::services::optimize::lbfgs(model, *init_context_ptr,
                                          random_seed, id, init_radius,
                                          history_size,
                                          init_alpha,
                                          tol_obj,
                                          tol_rel_obj,
                                          tol_grad,
                                          tol_rel_grad,
                                          tol_param,
                                          num_iterations,
                                          save_iterations,
                                          refresh,
                                          interrupt, logger,
                                          init_writer, sample_writer);

Which of those arguments is supposed to turn off generated quantities at suboptimal points during warmup?

alashworth commented 5 years ago

Comment by martinmodrak Monday Jun 04, 2018 at 08:49 GMT

Related to #2355

alashworth / test-issue-import

Only run generated quantities when needed for optimizing, sampling, and ADVI #180

Summary:

Current Version:

optimization and ADVI create generated quantities each iteration

EXPERIMENTAL ALGORITHM: This procedure has not been thoroughly tested and may be unstable or buggy. The interface is subject to change.