bettinardi
commented
1 year ago I agree with moving forward with the new proposal. Features that we might need to consider in this phase of the work and throughout ActivitySims further development:
- The input checker runs first and needs to run in a non fatal way where all checks are completed and reported as a summary log.
- This leads to the need to have a input checker log different from the full ActivitySim log series, as well as a log summary table (easy to read in a couple of lines) that summarizes how many fatals, warnings, and or logical errors were found in the input check.
- Based on the summary table (log) the code needs to come to a soft landing with a clear exit message to refer back to the input checker log(s) if any fatals were found. Anything better then "fatal" should allow the code to continue without issue, but it would be nice if there are any tripped checks (warnings, logic problems) for the code to do something different than if all the checks are clean. For example, I would suggest that if any checks are tripped but no fatals, so the code keeps running, that the input checker dumps a messy - "Warnings.log" in the root directory of the scenario - so that it's super easy to see.
- There should not be a separate "inputChecker" working folder in the scenario. Input checker should work within the existing folder structure of ActivitySim (noting that the data model seems to add a folder, which I am fine with).
- The data model / input checker, needs to allow for a user to specify "side checks" that are not digested into the ActivitySim environment. The common case for this will be network attribute exports that ActivitySim doesn't need, but that need to be checked up front to ensure a healthy and correct run. Note - we have talked in our meetings a lot about the desire to check skims. But I have always found that a healthy network produces a healthy skim and that it's much easier to track down skim oddities by tracking down the network bad spots that cause the skim problems. Meaning - reviewing skims is difficult and time consuming - it's much easier to build in all the issues that have caused skim problems in the past - into a series of checks - and ensure that none of those exist in your network
Side note - I just finished "immune", which is amazing and should be required reading for the entire population. Writing about the input checker feels like building the immunity memory in the adaptive system so we will never feel sick from those errors again:
https://www.amazon.com/Immune-Kurzgesagt-gorgeously-illustrated-immune-ebook/dp/B08YR8FNCP
RSG is developing an input checker as part of the phase 8 work.
Input checker features The input checker is a series of checks to be run on ActivitySim inputs (synthetic population, landuse, skims) to try to catch problems in the input data that might otherwise crash ActivitySim downstream or lead to bad model results. The input checker will be run as the first "model" in an ActivitySim run and should run quickly before starting any subsequent ActivitySim sub-models.
Original Design The original design of the input checker aligns with the current paradigm of ActivitySim configuration files: there is a csv file that contains a list of python expressions that evaluate to True or False to pass the check. (For more details, see the presentations on Feb 16 and April 20).
A new proposal The ActivitySim consortium has been discussing the possibility of adding data model to the ActivitySim ecosystem. This data model (see #617) would leverage the pydantic and pandera packages to enumerate allowed values and provide documentation on what each data field represents. Instead of creating an input checker in line with the original design, a different approach would be to leverage the data model. The input checker code would then just validate the input data against what is available in the data model.
If moving towards the data model approach, RSG would implement the input checker and focus on the input side of the data model integration. This would include writing validator functions and checks in the data model as opposed to the csv "spec" file in the original approach. Both approaches would be fundamentally similar in function -- the data model would still be validating the input data via a series of checks as defined by the user.
Discussion of pros & cons between the original design and the new approach took place at the May 4 and May 11 meetings. Please view the meeting notes and slides on those meeting pages for more details and in-depth discussion.
Deadline for decision As decided in today's meeting, we are requesting further discussion and questions to be in before a decision is made at the May 18 meeting.