Add new test model to catch run time errors

[Cole-Monnahan-NOAA]

I think it would be wise for developers to have a test model which includes:

• All possible variable types: scalars, vectors, matrices, arrays, dev_vectors etc., including bounded and unbounded versions. I would think you could do obj_fun+=dnorm(, 0,1) to get standard normal -loglikelihoods for each one.
• Have each parameter be it's own phase (N parameter types = N phases total), and at least one or two which have hardcoded negative phases (test for inactive parameters)
• Modify it slightly to be a RE model as well
• Add one correlation and initialize away from mode so the optimizer has to work a little
• Run this with a myriad of command line options. Combinations of things like -noest, -nohess, -hess_step 10, -phase N, -maxph, -maxfn, -output {0,1,2}, -sdonly, -mcmc -nuts/rwm 10, -hbf, . The more the better, especially combinations.

The model will run trivially fast, and this is not a statistical test. It would be designed to catch runtime errors. The main idea is to find edge cases where new features may fail but are not in the current test suite.

@wStockhausen any other ideas to add?

@johnoel if you can build the variable type part and how to integrate it in as a test I can help with the other parts

[johnoel]

So, you want something like make verify, but all put into a single model?

[Cole-Monnahan-NOAA]

No I meant to add a new model to the make verify suite. The point is that the current set of test models don't necessarily cover a wide usage and combination of features. It appears to me that they're setup from real statistical examples and so have a lot of similarities.

So e.g., if none of the models use a 4d array type, and our new output stuff breaks on those, we won't know until it's released. This would be most useful for developers to have a faith that a new feature or bug fix doesn't break some unusual model setup.