Tutorial for code

[odunbar]

Purpose

I am planning to basically follow the README instructions given like an idiot, and will highlight steps where i hit issues, and we can resolve them. I hope this will lead to a nice tutorial for future users, and myself for getting a grip on the codebase.

Current Status

• [ ] Finding the data: Right now, examples are not runnable data is provided only as a series of generic URLs and paths to unknown files rather than calling in a specific data configuration. All of these must be filled in.

To be added to the README

The following will be edited as I go through the code

The Training Data

• [ ] To initialize training of the LSTM, one must download data from the ERA5 (environment series), Hydrosheds (basins), GRDC (gauge data? runoff?) ... . The data sets used for the LSTM are ... insert summary information. Note that this is ~2TB of data.
• [ ] Extract the data and place them into files: ... insert files and in the formats insert formats
• [ ] The data must then be processed and made more consistent with each other. We do so by using the script example/engineering/engineer.jl. This script contains links to the files above to extract the data
• [ ] Having processed the data, we see that there are 3 types, timeseries, static_attributes and basin_list found in directories of the same name.
• [ ] the training data for the different configurations used in this document can be extracted from the processed data by using the files USA_data_config, Global_data_config

  Training the LSTM

[maumlima]

Sounds good!

• Regarding "finding the code": yes in theory the way to start (from scratch) is to run the code in examples/engineering/engineer.jl. I must say this involves downloading a very big volume of data (~2TB) and some days of computer processing on a HPC.
• Regarding "finding the data": I don't currently know the best solution for providing all the source data... but I agree that in order for someone to jump directly into the machine learning ("neuralhydrology") part it would be good to have only the data used by the LSTM model (i.e. timeseries/, attributes/ and the list of basins). I can make this available after the data consolidation for the paper.

[odunbar]

I understand that in general it is not a good idea for me to run this myself. But for reproducibility of code and results we should write a config for someone who wishes to run the code in this repository from scratch. Therefore, if possible we should include the steps to download the code, and what scripts to call for the processing etc.