tommylees112
commented
5 years ago Thank you for your comments!
Please make sure all your notebooks contain fully reproducible examples. Non-reproducible procedures will not be acceptable.
Yes that makes sense. We will tailor and test the notebooks to ensure that they are fully reproducible! Which bits specifically did you try and didn’t work? We can focus on those first! Unfortunately the notebooks are not necessarily the best way of interacting with the pipeline because processes take a long time to run. This means that the best way of interacting is probably through scripts/. The notebooks are useful because they show the diagrams next to function calls but actually running them is less than ideal.
It would be great to add small sample datasets to allow readers to test notebooks and functions.
In order to demonstrate the full functionality of the pipeline we felt that it made sense to download the data using the Exporters. This data is minimal (it is a small subset of all total data that we could use) in that sense and shouldn’t take too long.
If you think an example dataset and skipping the Exporters is necessary that is definitely something that we can look at!
For every model you test, please provide a summary of used hyperparameters (e.g. activation function, loss function, learning rate, neurons in each layer, any hidden layers, number of epochs, etc.)
This is ongoing but we do currently have a spreadsheet with all of the parameters and the results (RMSE & R^2) for each experimental setup. We are still finalizing the best way to present these results.
Please comment on the machine/system used to run the notebooks. Would you have achieved your results faster using a machine with better specs? If possible, suggest ideal specs.
We are currently using 2 google cloud instances. Both have a 500Gb hard drive, which we have found to be sufficient so far.
To download data, we have been using an n1-standard-1, which has 1 vCPU with 3.75Gb memory. This is adequate for data downloads and preprocessing, but is not enough to train the models. To train the models, we are using an n1-highmemn-2, which has 2vCPUs and 13Gb memory (as described in the scripts readme).
In addition to these two instances, we have access to an ECMWF machine, but we don’t know it’s specs.
Please make sure all your notebooks contain fully reproducible examples. Non-reproducible procedures will not be acceptable.
It would be great to add small sample datasets to allow readers to test notebooks and functions.
For every model you test, please provide a summary of used hyperparameters (e.g. activation function, loss function, learning rate, neurons in each layer, any hidden layers, number of epochs, etc.)
Please comment on the machine/system used to run the notebooks. Would you have achieved your results faster using a machine with better specs? If possible, suggest ideal specs.
The notebooks need some tidying up and, if possible, please run spelling/linter checks.