Closed sadamov closed 1 month ago
From a quick glance this looks simply amazing @sadamov! Thanks for doing this work. I will give a thorough review later today/tomorrow. Just tagging @SimonKamuk to have a read and give your thoughts too. I've added @ThomasRieutord to the organisation too. I'll also send Thomas an email so that he definitely sees the PR.
I implemented most requested changes in the latest commit and requested one more review. From my side we are clear to merge. The latest changes were again tested for model training and evaluation.
Remember to update the changelog before you merge @sadamov! I think I would call this a new (very useful!) feature. A few things also change here (where constants are stored and thus how they are accessed in the code)
Hurraay! :partying_face:
Wonderful job with this @sadamov!
Summary This PR replaces the
constants.py
file with adata_config.yaml
file. Dataset related settings can be defined by the user in the new yaml file. Training specific settings were added as additional flags to thetrain_model.py
routine. All respective calls to the old files were replaced.Rationale
/data
folder.constants.py
actually combined both constants and variables, many "constants" should rather be flags totrain_models.py
utils.py
allows for very specific queries of the yaml and calculations based thereon. This branch shows future possibilities of such a class https://github.com/joeloskarsson/neural-lam/tree/feature_dataset_yamlTesting Both training and evaluation of the model were succesfully tested with the
meps_example
dataset.Note @leifdenby Could you invite Thomas R. to this repo, in case he wanted to give his input on the yaml file? This PR should mostly serve as a basis for discussion. Maybe we should add more information to the yaml file as you outline in https://github.com/mllam/mllam-data-prep. I think we should always keep in mind how the repository will look like with realistic boundary conditions and zarr-archives as data-input.
This PR solves parts of https://github.com/joeloskarsson/neural-lam/issues/23