Training machine learning (ML) models require access to the grid information of the domain, wind rotation matrix parameters and land_sea_mask data. The data are stored in .zarr format with the locations to the datasets being specified in the catalog.yaml file within vcm directory. The paths are given in the form of "gs"-strings - the style of the path strings used on the cloud platform. This pull request introduces the code modifications needed for performing ML training on supercomputing systems.
Significant internal changes:
Added catalog_path attribute to the BatchesFromMapperConfig dataclass (_batch.py script).
Refactored load_batches method of BatchesFromMapperConfig dataclass (_batch.py script).
Refactored batches_from_mapper function (_batch.py script).
Added catalog_path parameter to add_grid_info andadd_rotation_infofunctions (_utils.py` script)
Refactored _load_grid and _load_wind_rotation_matrix functions (_utils.py script).
Training machine learning (ML) models require access to the grid information of the domain, wind rotation matrix parameters and land_sea_mask data. The data are stored in
.zarrformat with the locations to the datasets being specified in thecatalog.yamlfile withinvcmdirectory. The paths are given in the form of "gs"-strings - the style of the path strings used on the cloud platform. This pull request introduces the code modifications needed for performing ML training on supercomputing systems.Significant internal changes:
catalog_pathattribute to theBatchesFromMapperConfigdataclass (_batch.pyscript).load_batchesmethod ofBatchesFromMapperConfigdataclass (_batch.pyscript).batches_from_mapperfunction (_batch.pyscript).catalog_pathparameter toadd_grid_info andadd_rotation_infofunctions (_utils.py` script)_load_gridand_load_wind_rotation_matrixfunctions (_utils.pyscript).Resolves #2204 (partially)