JoeJimFlood
commented
1 year ago Update: I tried compiling Sharrow where the only model step was initialize_landuse so that the only thing that would really be done was the skims would be converted to zarr. I then copied that (as well as the flows) into an empty output folder and ran ActivitySim to write in said output folder. Doing so resulted in the skims only taking a couple minutes to load.
dhensle
When running our ABM3 setup, it takes about 20 minutes to load our skims (which we do have a lot of) without Sharrow. However, when Sharrow is turned on (but not zarr), this increases to 35 minutes. Last month I tried a test of ABM3 with zarr turned on, where I put the skims into zarr format while compiling the flows. When running Sharrow in production mode, the skims loaded in just a couple of minutes. However, we would like to avoid compiling Sharrow on each global iteration as that would add a lot of runtime. Because we are going to need to convert the updated skims from OMX to zarr after running assignment and skimming, I tried running ActivitySim with Sharrow in production mode but the skim cache empty to see if the skims conversion would happen. It took 55 minutes to convert the skims, and then another 55 minutes to load the skims into memory. I then tried another run with the skims already in the cache. According to the timing_log the setup_skims step took 2.5 minutes, but it seemed longer when I had the window up. The reported times in the timing log added up to 123.4 minutes, but the log shows that it took 197.4 minutes to run all of the models, so I'm wondering if the skim loading was involved in the runtime that is unaccounted for.