This PR creates an option to use the simulation to log & print events and their effect on individuals across any number of modules. By collating this information in post-processing, we can create chains of events and outcomes for each simulated individual across different diseases, demographic changes, etc.
This will be useful to generate training data for module emulators/time series analysis, but also for general debugging. In order to assist in the former, the PR additionally gives the user the option to “ignore” the epidemiology of diseases in order to prioritise infecting as many individuals as possible during the runs. This (for now) is controlled by the “generate_event_chains” parameter in the Simulation class. Post-processing is done via postprocess_event_chains.py.
NOTE: This is working as intended, but not really feasible (RAM+CPU-time wise) if intending to use it for a large number of individuals. Next step will be to think about how to make it so.
How the PR works:
Events are collected when fired. This allows us to collect info centrally without modifying any of the existing modules and/or their event functions. This occurs at the Simulation-class level and inside the HealthSystemScheduler function. (In addition, birth “events” are recorded inside the do_birth function as a one-off).
In the case of Population-wide events, the event is only logged for those individuals for which the event resulted in a meaningful (i.e. parameter) change. E.g. TbActiveEvent, which is a population-wide event, is only stored for individuals who become actively infected by Tb as a result of the event having been fired. This is not optimal from a run time perspective (see below).
The user can specify for which modules events should be logged via “generate_event_chains_modules_of_interest” (e.g. could chose to log events belonging to Tb+Hiv modules only), and also specify which events to ignore completely via “generate_event_chains_ignore_events“, if the user has established a priori which event would be particularly cumbersome+uninteresting (e.g. TbActiveCasePollGenerateData, which only schedules the date on which infection will actually occur).
Properties of individuals are currently printed in full, and both before and after the event has occurred (in order to identify, in post-processing, which events resulted in meaningful changes in the individual’s parameters). This is far from ideal, given that there could be hundreds of events per individual, and x2 for before+after, and hence not sustainable from a memory standpoint (see discussion below).
In post processing, events are grouped by individual (while chronological order is already observed by construction), and all those “fired” after death are eliminated from the chain. For each individual, the chain is scanned to identify which events resulted in meaningful changes for the individual, therefore reconstructing the individual trajectory to outcome.
OPTIMISATION
To reduce CPU requirements:
At the moment dataframe is extended row by row every time; this is of course not ideal, should be extended by N rows at a time like currently done in population dataframe.
Detecting changes in the entire population dataframe is extremely CPU costly. On the other hand, printing this info for the entire population without any change-based pre-selection would be extremely RAM intensive…
To reduce RAM requirements:
Print to hdm5 during run time?
Reduce the number of individual properties to be logged. Not ideal at this stage if we want to print and collect data in a way that is agnostic as to what parameters are relevant. However, there are certainly some that we know we won’t need (district_num_of_residence, district_of_residence, region_of_residence, age_exact_years, age_years, age_range, age_days). However, there could be a potential draw-back in CPU time from having to eliminate columns from the dataframe.
To avoid printing properties before/after (Note: maximum benefit to this is x ½ reduction, so not huge):
--Assume that the individual has not been modified since the previous event, such that the last “After” state is the “Before” state of the subsequent event. This is a bit risky given that the user can decide not to print all events the individual has experienced. In addition to this, one could also only log the changes . Again, potential draw-back in CPU time from having to compare rows.
-- Decide to ignore all intermediate events, and clearly define a “start” (i.e. onset) and end (“death”/”resolution”) of the episode. This is what we would need for emulator, however loads of very interesting information in between that ultimately we would want to capture to explore the possibility of training time series / to capture “infectiousness” of individual (e.g. viral load status for HIV).
This PR creates an option to use the simulation to log & print events and their effect on individuals across any number of modules. By collating this information in post-processing, we can create chains of events and outcomes for each simulated individual across different diseases, demographic changes, etc. This will be useful to generate training data for module emulators/time series analysis, but also for general debugging. In order to assist in the former, the PR additionally gives the user the option to “ignore” the epidemiology of diseases in order to prioritise infecting as many individuals as possible during the runs. This (for now) is controlled by the “generate_event_chains” parameter in the Simulation class. Post-processing is done via postprocess_event_chains.py.
NOTE: This is working as intended, but not really feasible (RAM+CPU-time wise) if intending to use it for a large number of individuals. Next step will be to think about how to make it so.
How the PR works:
OPTIMISATION
To reduce CPU requirements:
To reduce RAM requirements: