stan-dev / posteriordb

Database with posteriors of interest for Bayesian inference
176 stars 36 forks source link

Add test that checks compatibility of data and model for each posterior #142

Closed eerolinna closed 2 years ago

eerolinna commented 4 years ago

In #140 we have cases where the model and data for a posterior are not compatible, for example the data might be missing a variable that is declared in the model data block.

I guess the easiest way to test this for Stan models is to run inference for one iteration and make sure there's no errors.

This might be too computationally heavy to include in Travis (or it might not, we should try it out). Based on a quick search the time limit seems to be 50 minutes https://github.com/travis-ci/travis-ci/issues/6232

In case the CI builds start timing out we have a few options

If the tests don't time out but just take too long for a comfortable development workflow we might want to run them only on pushes to master

MansMeg commented 4 years ago

Yes. I agree. I think we, in the long run, could create a test branch where we push before we push to master with more extensive tests.

But in general, we need to check these full tests. I guess compilation would take roughly 10s so at leaste a couple of hundred models should be possible to check. We could also have checks that we run locally once in a while.

eerolinna commented 4 years ago

I think it should also be possible to cache the compilation between builds (for unchanged models), which would shorten the test time by a lot, especially for small posteriors

MansMeg commented 4 years ago

Indeed, although I do not know how to do that using Travis.

MansMeg commented 2 years ago

This has now been fixed in the last version.