FLARE-forecast / FLARE-containers

Docker containers to run FLARE forecast
0 stars 2 forks source link

Uploading FCRE Forecasts from Bruno to S3 Storage #42

Closed vahid-dan closed 2 years ago

vahid-dan commented 3 years ago

Current forecasts on Bruno are located at /opt/flare/shared/flare-generate-forecast-2.0-dev/forecast/. For instance, for 2021-08-12, the forecast files look as follows:

wrr_runs_H_2021_08_11_2021_08_12_F_16_20210813T133554-eml.xml
wrr_runs_H_2021_08_11_2021_08_12_F_16_20210813T133554.nc
wrr_runs_H_2021_08_11_2021_08_12_F_16_20210813T133554.pdf
wrr_runs_H_2021_08_11_2021_08_12_F_16_20210813T133554_turnover.png

These files are generated for each day when the forecasts are completed for that date. If there is no forecast running for a day, we don't have these forecast files for that specific date. If the forecasts have been running multiple times for a day, we have multiple versions of the files for that date. But, in most cases, we have these 4 files for each date.

We would like to have a script to read the names of the forecast files, extract the date from the names, create a directory for each date, and move the forecast files to their respective directories. For instance, the above files will be moved to the following structure:

2021-08-12/00/GLM/wrr_runs_H_2021_08_11_2021_08_12_F_16_20210813T133554-eml.xml
2021-08-12/00/GLM/wrr_runs_H_2021_08_11_2021_08_12_F_16_20210813T133554.nc
2021-08-12/00/GLM/wrr_runs_H_2021_08_11_2021_08_12_F_16_20210813T133554.pdf
2021-08-12/00/GLM/wrr_runs_H_2021_08_11_2021_08_12_F_16_20210813T133554_turnover.png

Then we need to move them to a permanent S3 storage.