🐛[BUG]: graphcast race condition when caching graph data structure

NVIDIA / earth2mip

Earth-2 Model Intercomparison Project (MIP) is a python framework that enables climate researchers and scientists to inter-compare AI models for weather and climate.

https://nvidia.github.io/earth2mip/

Apache License 2.0

183 stars 40 forks source link

🐛[BUG]: graphcast race condition when caching graph data structure #151

Open nbren12 opened 8 months ago

nbren12 commented 8 months ago

Version

main

On which installation method(s) does this occur?

No response

Describe the issue

When running in parallel all processes try to write to this .cache.pkl file: https://github.com/NVIDIA/earth2mip/blob/2b6d3f816e68db3f53d6fb3d795620086e641041/earth2mip/networks/graphcast.py#L107

Environment details

No response