Note: I'm pulling this discussion (previously in #2) out into a new issue, since it's now a defined task, but this is currently TABLED until we progress further on other higher priority tasks for the GLM modeling work
During the last full run of all 64,494 models, I had vmstat tracking going in order to track memory usage, CPU usage, and I/O demands.
Jesse dug into those results, and is pretty certain that the performance bottleneck is I/O. The system is never coming anywhere near running out of RAM, and the CPU is only ever used ~30% at most:
David suggested looking into writing GLM output to a scratch directory, so that "each model was writing to an independent file system and not competing with each other on /cxfs". Jesse explored options for Denali and identified the "DataWarp" burst buffer as the best supported option. He's thinking the process would involve:
Estimating the total highs-speed scratch space storage needs,
Getting comfortable with requesting the necessary DataWarp resources through slurm and accessing them in the script, and
Doing the actual changes to the pipeline to make it use this resource. It looks to me like we'd be using what they describe as "scratch" here, so the necessary changes would be (a) Setting GLM's input to use a DataWarp scratch directory, which would be named by the environment variable Sys.getenv("DW_JOB_STRIPED"), and (b) setting the outputs to go first to a scratch directory in Sys.getenv("DW_JOB_STRIPED"), then be moved to the regular targets folder hierarchy and the scratch directory deleted.
I did some investigation into (1), and Jesse is thinking that we could ask for ~10 GB of the 46 TB of this space on Denali.
For reference, Jordan also noted
GLM outputs scale predictably based on number of timesteps in the output (we do one per day) and the "max_layers" param in the .nml, since GLM populates those regardless of whether they are wet or dry. So for every year and every 100 layers, your file grows by ~2.9MB.
Note: I'm pulling this discussion (previously in #2) out into a new issue, since it's now a defined task, but this is currently TABLED until we progress further on other higher priority tasks for the GLM modeling work
During the last full run of all 64,494 models, I had vmstat tracking going in order to track memory usage, CPU usage, and I/O demands.
Jesse dug into those results, and is pretty certain that the performance bottleneck is I/O. The system is never coming anywhere near running out of RAM, and the CPU is only ever used ~30% at most:
David suggested looking into writing GLM output to a scratch directory, so that "each model was writing to an independent file system and not competing with each other on /cxfs". Jesse explored options for Denali and identified the "DataWarp" burst buffer as the best supported option. He's thinking the process would involve:
Sys.getenv("DW_JOB_STRIPED"), and (b) setting the outputs to go first to a scratch directory inSys.getenv("DW_JOB_STRIPED"), then be moved to the regular targets folder hierarchy and the scratch directory deleted.I did some investigation into (1), and Jesse is thinking that we could ask for ~10 GB of the 46 TB of this space on Denali.
For reference, Jordan also noted
I have not yet started work on items (2) or (3)