search

PalEON-Project / stepps-calibration

STEPPS pollen-veg calibration model code and paper.
0 stars 1 forks source link

effective sample size estimator #16

Open andydawson opened 9 years ago

andydawson commented 9 years ago

Now that we have multiple chains, use the multi-chain estimator that Stan implements.

andydawson commented 9 years ago

A bit of a pain. Right now, it is not possible to estimate multi-chain ESS for chains with different length using Stan methods. I started to think about how to do this, but then decided to truncate chains based on the shortest chain length. Ideally, I would run all chains for the same number of iterations, but this is not always possible. Some runs take more than the max job run time, and get cut short.

Also, Stan uses a conservative ESS estimator. In the calibration paper, I reported only the ESSs for the log posterior. For the variable PL model, the ESS is ~500. Not as much as I had hoped.

Next time I will run more short chains.

I could estimate ESS for each chain independently, and then add these numbers (but that seems a bit generous). @paciorek how do people usually do this?

paciorek commented 9 years ago

I don't see any reason not to estimate ESS for each independently and add.

Also, I forget if we ever found a reference that justifies giving ESS for log posterior as opposed to parameters. Perry has questioned my use of that and I'm not sure where the idea originally came from...

On Thu, Aug 20, 2015 at 1:22 PM, Andria Dawson notifications@github.com wrote:

A bit of a pain. Right now, it is not possible to estimate multi-chain ESS for chains with different length using Stan methods. I started to think about how to do this, but then decided to truncate chains based on the shortest chain length. Ideally, I would run all chains for the same number of iterations, but this is not always possible. Some runs take more than the max job run time, and get cut short.

Also, Stan uses a conservative ESS estimator. In the calibration paper, I reported only the ESSs for the log posterior. For the variable PL model, the ESS is ~500. Not as much as I had hoped.

Next time I will run more short chains.

I could estimate ESS for each chain independently, and then add these numbers (but that seems a bit generous). @paciorek https://github.com/paciorek how do people usually do this?

— Reply to this email directly or view it on GitHub https://github.com/PalEON-Project/stepps-calibration/issues/16#issuecomment-133159864 .

andydawson commented 9 years ago

Well I think Gelman argues that it is better to use a multi-chain variance estimator when possible. But I don't think we have to do this. I can just do single-chain ESS and add.

No, I never found a ref that justifies looking at log posterior ESS. I assumed it would in general underestimate the parameter ESS, so if we are happy with the log posterior ESS than all is well. But I don't know if this is really the case...

(Also, we have many parameters and I didn't want to make another table. The ESS range is quite large among parameters.)

paciorek commented 9 years ago

yeah, I'm not sure if we can justify log posterior ESS. I would just try to give a sense for the range of values in the text.

On Thu, Aug 20, 2015 at 2:05 PM, Andria Dawson notifications@github.com wrote:

Well I think Gelman argues that it is better to use a multi-chain variance estimator when possible. But I don't think we have to do this. I can just do single-chain ESS and add.

No, I never found a ref that justifies looking at log posterior ESS. I assumed it would in general underestimate the parameter ESS, so if we are happy with the log posterior ESS than all is well. But I don't know if this is really the case...

(Also, we have many parameters and I didn't want to make another table. The ESS range is quite large among parameters.)

— Reply to this email directly or view it on GitHub https://github.com/PalEON-Project/stepps-calibration/issues/16#issuecomment-133172646 .

andydawson commented 9 years ago

OK. Will update accordingly. I will post to the Stan list to ask though, because I am also interested to know if this is really justified. Will let you know.