Unit tests?

[arrayofletters]

Are there unit tests available?

[s-mishra]

No we don't have it. It is a research related code not a production code.

[arrayofletters]

Cripes... All code is production if you think the results are important.

[dooglio]

Thanks guys. Nice work. A model whose output has affected all our lives and didn't bother to put in unit tests?!

[dsjoerg]

The best/quickest way to get confidence in models like this is not to unit test the code (although of course that is nice too) but to compare the outputs of multiple models built by different groups and reconcile/compare them.

[turzes]

Exactly! Its a domain specific model validation exercise, predictive or causal- preferably both, through rigorous statistical method that justifies the model- the code is by necessity validated through the research and current process given we have all pulled it apart and put it back together.

[dooglio]

The problem with that is, we programmers don't have access to those other models so we're kinda taking your word that the data matches up. If you include unit tests, however, it give us a chance to verify that the code is right in the archive itself.

It also saves you time in the long run because you can catch errors immediately rather than having to wait until you have the output from the other models then go through and compare everything.

Forgive me, but it feels like you're making excuses instead of agreeing that best practices are, well, best followed.

[dsjoerg]

You have access to those other models as well. In the USA there are about 20 modeling groups I'm tracking. A bunch of them publish to https://reichlab.io/covid19-forecast-hub/. Their forecasts are all displayed there and you can compare them nationally or per state. The good ones also share a whitepaper and/or code.

Again, I'm not saying that unit tests are bad — they're clearly good — but when evaluating a model, checking its unit tests should not be high on your priority list of things to do.

Let me be specific. If you're concerned about the accuracy of this model, since this model uses deaths as its principal input, compare it to https://covid-19.tacc.utexas.edu/mortality-projection-faq/. Or go one step broader and compare to http://www.healthdata.org/covid

[dooglio]

Well, if you are writing models for internal use and just playing with data, I don't really care. But if your models are going to be used for public policy, then shoddy development practices are in my mind unacceptable. Your code better be stellar.

[dsjoerg]

Your code better be stellar.

People aren't using the code. They're using the outputs of the model. It's the outputs that had better be stellar. And speed is a feature. Great outputs in two years do nothing to save lives. So the design challenge is the speed-vs-quality tradeoff.

If this model were the only model in the world, and public policy teams were slavishly relying on it regardless of what it said, it would be a different story. In a world without other models, it's super important to get top quality.

But it's not the only model. Any sane public policy team looks at multiple models, and decides how much to believe each of them, and is skeptical about outliers.

So if there's a bug in this model that materially affects its outputs, it will look like an outlier, and get extra scrutiny / less weight from public policy teams. And each team's errors are likely to be in different directions.

So your insistence on unit tests is effectively saying that in the speed-vs-quality tradeoff, whatever impact unit tests have on that tradeoff are utility-improving. But you give no support for that assertion. Especially in the multiple-models context it's not at all obvious to me that the lower-speed-but-higher-quality you get from unit tests is a net win.

And I know that for many dev teams, unit tests ultimately mean you can go faster. But this isn't any dev team, it's the ICL dev team. And if they're not familiar with test-first development, and they don't have volunteers swooping in to teach/help them, then it's not so relevant. You go to war with the army you have.

[dooglio]

Lack of unit tests is just a symptom of not caring about best practices in your code. Since you seem so determined to avoid following best practices, and shooting the messenger instead, I'm not going to think highly of your project.

[dsjoerg]

I have nothing to do with this project. I'm just an interested bystander like yourself.

[dsjoerg]

You also aren't interacting with anything I've actually said.

[dooglio]

I've said my piece, I consider your objections as nothing more than red herrings to my argument. Thanks. I'm unsubscribing now from this thread.

[dsjoerg]

They care more about saving lives than about best practices and your approval. Nice having you here.