Monitoring performance

[rortizmerino]

Two main strategies are necessary, and there are different ways to implement them. Main points go as follow:

• [ ] 1. End point: For example, using seff on DefltBlue

• [ ] 2. Snapshot: Like memory_profiler, cProfile and psutil

• [ ] 3. Interactive: Logging into the snellius node, visualising the dask cluster(s) peformance

Specific issues will be raised for each point. Their main goal is to effectively calculate resources for each environment.

[mgeraeds]

For reading/writing, there are a few tests that we can do: on the one hand, to see how many cores/processes should be used for reading/writing for optimal performance, and on the other hand whether writing multiple time steps per file would increase or decrease writing time.

Testing for timestep writing optimization:

• [x] 96 cores, 96 timesteps, 1 timestep per output file, 00:22:14
• [ ] 96 cores, 96 timesteps, 12 timesteps per output file
• [ ] 96 cores, 96 timesteps, 24 timesteps per output file
• [ ] 96 cores, 96 timesteps, 48 timesteps per output file

Testing for cores/processes optimization:

• [ ] 24 cores, 96 timesteps, 1 timestep per output file
• [x] 48 cores, 96 timesteps, 1 timestep per output file 00:18:28, 28.28% of 336.0GB
• [ ] 192 cores, 96 timesteps, 1 timestep per output file

More tests might be added if needed.