Validation and calibration options

[dsivdnbe]

Dear all,

I'm very new to agent-based modeling but have already familiarized myself with the basics of this Mesa package. For my specific application of this ABM, I will also need to calibrate certain parameters in order to validate it for real-world data.

Does anyone have experience doing this or can anyone refer me to an example/other code/... that could set me in the right direction.

Thanks a lot!

[jfunction]

I scratched around my Mendeley and found this paper which I quite enjoyed.

I'm not yet sufficiently familiar with Mesa to say whether there is explicit support for calibration. There are batch runners which can give you some sense of how parameter variation causes output variation which is a start at least. You should probably first decide on a calibration technique before considering what code to write though. Here are some common techniques to research if you have not yet done so:

• Manual tuning of the parameters with inspection of output done by a domain expert (often a good first step)
• Approximate Bayesian Computation
• Particle Swarm Optimization

There are more techniques in the linked paper. Maybe someone like @jackiekazil, @dmasad or @Corvince (going by most/recent commit contribution) can comment on something more specific to Mesa or if something like that is on a wish list / road map? I am also interested to learn more and may even contribute to the project given that I am currently doing work in this area.

Good luck

[Corvince]

Thanks for the answer @jfunction! You are correct regarding the state of mesa: There is no explicit calibration module, but the batch runners are the essential link between model runs and calibration. You are also correct that there are multiple calibration approaches and one would need to decide which one to use.

That being said I think a calibration module sounds like a great addition to mesa to facilitate that process. But I don't think building such a module is a straight-forward process, so we should probably continue this discussion on the mailing list. @jfunction you are more then welcomed to initiate that discussion there on how such module may look like.

Also, thanks for the paper, from skimming through it looks indeed very interesting!