search

CliMA / CalibrateEDMF.jl

A package to calibrate atmospheric turbulence and convection parameterizations using gradient-free ensemble Kalman methods
https://clima.github.io/CalibrateEDMF.jl/dev/
Apache License 2.0
22 stars 1 forks source link

Temp TC.jl output not deleted in pmap pipeline until calibration is finished #89

Open ilopezgp opened 3 years ago

ilopezgp commented 3 years ago

The removal of TC.jl output is now done implicitly through its storage in temporary files that are deleted once the Julia process ends. This approach works for the HPC pipeline, where control is returned to the system after every step of the process, but not for the pmap pipeline, which is a single Julia process.

The result is the buildup of TC.jl output data that can lead to memory overflow in the system used. We should handle the output better so that this does not happen and the TC.jl outputs are deleted after every EKP step.

haakon-e commented 2 years ago

Tangential to this, if the hpc pipeline fails. temporary files are not deleted. My motivation for using the temporary directories was that files are deleted even if Julia crashes. However, if we are good at properly handling Julia errors, I suppose we can find a compromise.