Current 00_daily_update sources scripts to the global environment by default, and clears it with rm after each source call. Inside of individual scripts, dependency objects are obtained by decompressing the saved RData files from previously sourced scripts, which is an expensive operation. We can omit the need for this by explicitly declaring environments for scripts to be sourced into and passing the necessary environment to the subsequently sourced scripts that depend on it, thus eliminating the need to load the RData file. The code to load RData files is retained for working on scripts individually, though now it is run only when key objects from that script do not exist in the environment that the script is using.
These changes result in an approx. 44% reduction in the total run time for the COHHIO_HMIS data crunching.
feather_save
Rationale
RAM constraints have caused the apps to be unavailable to some users. Server settings were tweaked to reduce the number of users per instance.
This code modification makes it such that all large data.frame objects are no longer loaded into the global workspace on app bootup. All data.frames are saved to feather files in the app's data/db folder and the objects that were data.frames in the RData file are replaced with identically named "accessor" functions that read the df from the feather file.
Requires Rminor_elevated#51 & Rminor#50
env_optimization
Rationale
Current 00_daily_update
sources scripts to the global environment by default, and clears it withrmafter eachsourcecall. Inside of individual scripts, dependency objects are obtained by decompressing the saved RData files from previously sourced scripts, which is an expensive operation. We can omit the need for this by explicitly declaring environments for scripts to be sourced into and passing the necessary environment to the subsequentlysourced scripts that depend on it, thus eliminating the need to load the RData file. The code to load RData files is retained for working on scripts individually, though now it is run only when key objects from that script do not exist in the environment that the script is using.These changes result in an approx. 44% reduction in the total run time for the COHHIO_HMIS data crunching.
feather_save
Rationale
RAM constraints have caused the apps to be unavailable to some users. Server settings were tweaked to reduce the number of users per instance. This code modification makes it such that all large
data.frameobjects are no longer loaded into the global workspace on app bootup. Alldata.frames are saved to feather files in the app's data/db folder and the objects that weredata.frames in the RData file are replaced with identically named "accessor" functions that read the df from the feather file.Requires