joecastiglione
commented
1 year ago Revised Description:
The purpose of this work element is to develop either a stand-alone software tool or an integrated ActivitySim core capability, that provides the ability to automate the calibration of selected model coefficients and alternative specific constants. This capability must be able to run user selected model components and compare estimated results with the observed (or expected / asserted) values and mathematically adjust one or more parameters and/or alternative specific constants. This capability not intended to replace the value and central importance of more detailed examinations and comparisons between observed and estimated values. Rather, it is to provide a more efficient method for reaching the point where these more detailed comparisons can be made and a wide range of solutions considered where the model under development fails to achieve an acceptable level of calibration.
This feature should have the ability to address the entire scope of ActivitySim model component types and component type elements. For example, the feature should be able to address basic calibration use cases where model system component calibration typically involves adjusting one or more alternative specific constants, such as for the auto ownership model, as well as more complex calibration use cases such as where calibration involves adjusting distance polynomial terms in a destination choice model. The features should provide clear, user-friendly summary statistics and displays, including comparative summaries of observed and estimated values (shares or absolute values) for each alternative specific constant in both tabular and graphic forms, by iteration, allowing the user to determine if the value of the constant displays a monotonic pattern or oscillates across the iterations
Plan to design and implement for two different types of models: 1) an alternative specific constants model such as auto ownership and 2) a distances terms model such as work location choice.
See MTC's specification for more detail.
Develop software to automatically run the model, compare select sub-model results to expected values, and intelligently automatically adjust alternative specific constants based on the differences. For example, run trip mode choice, summarize trips by mode, compare trips by mode to the surveyed trips by mode, and adjust the mode specific constants based on the log of the difference ratio * a dampening factor. Repeat the procedure until difference is minimized and respect min and max adjustment constraints. This has been done before for simpler trip-based models and could be done for ABMs. This feature needs to be used with caution and would require design related to how to specify tracked and expected outputs for each sub-model.
May also want to consider Automated Calibration Pilot. Calibrating an activity-based model is challenging, as the number of parameters is large and the observed data is often scarce. In Version 1.6, the Consortium will begin experimenting with automated calibration procedures via one or more pilot demonstrations. The Consortium expects to continue improving the automated calibration procedures in subsequent ActivitySim releases.