search

ai2cm / fv3net

explore the FV3 data for parameterization
MIT License
16 stars 3 forks source link

Offline reports can clobber each other #1202

Open nbren12 opened 3 years ago

nbren12 commented 3 years ago

When running the train-diags-prog workflow, the offline reports saved can easily clobber each other. Two ML models with the same "tag" will save to the same location even if they are performed with different buckets, dates, etc.

cc @brianhenn

brianhenn commented 2 years ago

Just like the step output locations, it seems like we should have some rules about where offline reports (and maybe online ones) are written instead of requiring the user to provide a basically freeform path. I've also "lost" reports temporarily because the current interface requires manually recording the output location somewhere, particularly an issue for the prognostic report and the random string in its address (though that does prevent clobbering).

First suggestion that comes to mind would be to enforce a {project}/{date}/{tag} hierarchy in the relevant root directory (e.g., inside gs://vcm-ml-public/offline_ml_diags. This wouldn't solve the clobbering on its own but would be structured and could be combined with a warning or failure if clobbering is happening.