compute generated quantities only when necessary

[alashworth]

Issue by bob-carpenter Thursday Jul 13, 2017 at 16:58 GMT Originally opened as https://github.com/stan-dev/stan/issues/2355

---

Summary:

The generated quantities block gets executed every iteration when we only need it at the end.

We could leave it out of optimization until the last iteration

We can leave it out of sampling until after warmup.

Warning: Care will have to be taken if warmup is being saved.

Current Version:

v2.16.0

[alashworth]

Comment by bgoodri Thursday Jul 13, 2017 at 18:11 GMT

---

I think generated quantities is already skipped when sampling if save_warmup = 0. So, we are really just talking about optimization and ADVI.

On Thu, Jul 13, 2017 at 12:58 PM, Bob Carpenter notifications@github.com wrote:

Summary:

The generated quantities block gets executed every iteration when we only need it at the end.

We could leave it out of optimization until the last iteration

We can leave it out of sampling until after warmup.

Warning: Care will have to be taken if warmup is being saved. Current Version:

v2.16.0

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/stan-dev/stan/issues/2355, or mute the thread https://github.com/notifications/unsubscribe-auth/ADOrqpuWOVcLJRj5CCiEG9M0LW4UOXCLks5sNkybgaJpZM4OXReg .

[alashworth]

Comment by betanalpha Thursday Jul 13, 2017 at 18:41 GMT

---

Generated quantities are computed when the constrained parameters are requested as a method in the model instance. That means at no point are they controlled by the algorithms. It also means that you cannot calculate constrained parameters or transformed parameters without also computing the generated quantities.

This means that generated quantities are not computed during warmup and they are computed during optimizing only because optimizing tries to write the entire state down.

In order to avoid computing the generated quantities there we’d need model methods that perform the constraining transformations and compute transformed parameters separately of the generated quantities. And then you’d be dealing with the issue that the intermediate output would have a different schema than the final output.

On Jul 13, 2017, at 2:11 PM, bgoodri notifications@github.com wrote:

I think generated quantities is already skipped when sampling if save_warmup = 0. So, we are really just talking about optimization and ADVI.

On Thu, Jul 13, 2017 at 12:58 PM, Bob Carpenter notifications@github.com wrote:

Summary:

The generated quantities block gets executed every iteration when we only need it at the end.

We could leave it out of optimization until the last iteration

We can leave it out of sampling until after warmup.

Warning: Care will have to be taken if warmup is being saved. Current Version:

v2.16.0

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/stan-dev/stan/issues/2355, or mute the thread https://github.com/notifications/unsubscribe-auth/ADOrqpuWOVcLJRj5CCiEG9M0LW4UOXCLks5sNkybgaJpZM4OXReg .

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/stan-dev/stan/issues/2355#issuecomment-315158580, or mute the thread https://github.com/notifications/unsubscribe-auth/ABdNloIXVb3AynEQvYTRMKNsj56PYtE2ks5sNl3dgaJpZM4OXReg.

[alashworth]

Comment by rgerkin Monday Jul 17, 2017 at 03:54 GMT

---

Andrew asked a question about this a while back. Isn't implementing this still the path towards efficient all-in-Stan computation of out-of-sample model fit (e.g. by cross-validation)? Or is there another way of doing this?

[alashworth]

Comment by bob-carpenter Monday Jul 17, 2017 at 19:01 GMT

---

There's a separate issue to be able to run generated quantities standalone. And it's on our short-term to-do list. This one may not get done any time soon because most of the problems will be solved by just running generated quantities later.

[alashworth]

Comment by rgerkin Monday Jul 17, 2017 at 19:54 GMT

---

@bob-carpenter Are you talking about #1641 or another one?

[alashworth]

Comment by sakrejda Monday Jul 17, 2017 at 20:02 GMT

---

Just putting a note in here that the current design for output seems to allow for this by splitting the generated quantities out into a separate stream. We would probably need to track iteration/chain so things could be matched up later.

[alashworth]

Comment by bob-carpenter Monday Jul 17, 2017 at 20:59 GMT

---

@rgerkin: Yes, #1641

@sakrejda I keep thinking we want a more structured output for the samplers. The problem is how to keep it generic enough---the fear was requiring too many implementations for the interfaces.

[alashworth]

Comment by sakrejda Monday Jul 17, 2017 at 21:11 GMT

---

@bob-carpenter If you have specific implementation worries could you add them to the discourse discussion or the wiki? I want this to move forward so I can figure out more details as needed.

[alashworth]

Comment by bob-carpenter Monday Jul 17, 2017 at 21:14 GMT

---

No worries for the standalone generated quantities. This issue is for turning off generated quantities while optimizing or warming up.

You mean this discussion:

https://github.com/stan-dev/stan/wiki/Standalone-Generated-Quantities:-Functional-Specification

[alashworth]

Comment by sakrejda Monday Jul 17, 2017 at 21:15 GMT

---

Nope, I mean this one (linked above): https://github.com/stan-dev/stan/wiki/Design:-Consolidated-Output-for-Sample,-Optimize,-ADVI,-and-Diagnose