Open raspstephan opened 3 years ago
@HirtM We need to decide which variables to download from TIGGE. You can see the available variables here: https://apps.ecmwf.int/datasets/data/tigge/levtype=sfc/type=cf/ (Also note the different levels on the left). Unfortunately, TIGGE doesn't archive the stratiform and convective precipitation separately. This could have been important information but oh well.
In addition to Total Precipitation, what do you think could be useful? Note the potential storage limitations: https://github.com/raspstephan/nwp-downscale/issues/4#issuecomment-744003517
Orography (this should be time-independent) Land-sea mask (also time-independant)
In addition, the following variables may be worthwhile:
Others we might consider, but I don't fully understand how they would contribute helpful information:
Also, do we want to distinguish between snow and rain? This would mean that our output is not just one 2d field, but two. I would assume it's okay to ignore snow for now.
Thanks a lot @HirtM! That's a lot of fields. We probably should start with the most important ones to keep data volumes down and prevent making the training too slow.
Here is what I would start with based on your considerations:
I think it's also really important to include orography. But ideally we would feed in the high-resolution orography. So it would need to come in at a different point in the network. Same for land-sea mask even though we probably will start with mostly land points?
Do you agree with my ranking? We can always download more later.
Do you agree with my ranking? We can always download more later.
Yes, I agree! Sounds like a good start.
@anna-184702, here are some pointers for downloading the MRMS data.
Here are the links for the data archive: https://mesonet.agron.iastate.edu/archive/
There is a link to the Google Drive files and there also is a local cache on a server but it doesn't contain all the data. I have so far only used the cached data. So unfortunately, we will have to use Google Drive but you mentioned that you have some experience with that.
The data comes in hourly zip files. Each zip file contains all the variables, which is why they are quite big (4G) and take a while to download. Here are all the variables.
We only need a small subset of all these variables. I would suggest the following but am open to suggestions from you and @HirtM.
'MultiSensor_QPE_06H_Pass1', 'MultiSensor_QPE_06H_Pass2', 'RadarOnly_QPE_06H', 'MultiSensor_QPE_03H_Pass1', 'MultiSensor_QPE_03H_Pass2', 'RadarOnly_QPE_03H', 'RadarQualityIndex'
In fact, we only need the 6h accumulations every 6 hours (0/6/12/18) to match the TIGGE data. However, I would also download the 3h accumulations every 3 hours, just in case we want to use the YOPP data later.
I already wrote some code to download, extract (first .zip, then each variable is in a .gz) and save the relevant files here. Note that there will be some missing files, i.e. some variables are missing at some time steps. The code should already catch that.
One thing I did not do in my code is convert from grib to nc since xarray can already read grib. Do you think we should do this right away so we never have to deal with grib? We can probably worry about this later when we are thinking about how to get the data into the NN.
I hacked together a tigge download script. Pretty ugly but it works for deterministic. For ensemble there is still some bug and the data are quite large so I can't download monthly batches for all 50 ensemble members... I will fix it somehow. Next, I will write a regridding script.
The raw tigge data for the control run should be downloaded in /datadrive/tigge/raw
. @HirtM Feel free to do some analysis :)
@raspstephan thanks for this! I plan to try downloading the MRMS data tomorrow - I have some colab scripts for downloading from google drive which will hopefully work 😁
@anna-184702 If you haven't started yet, maybe wait with the MRMS downloading. I might have found a private data source that has MRMS data for several years.
I also wrote a quick download script for the MRMS data that is cached on the server. It's currently running so that we should have a few months of data soon. Soon means a couple of days because downloading the data takes forever.
@raspstephan I've finally got the MRMS data from September 2019 - May 2020 off that google drive 🎉. Do we want the 3hr accumulations or only 6hr at this stage? It looks like /datadrive/mrms/raw/MultiSensor_QPE_03H_Pass1 and /datadrive/mrms/raw/MultiSensor_QPE_03H_Pass2 are empty
Wow, amazing! Thanks so much. For now we are only using 6h but keeping 3h probably doesn't hurt. Yes, it seems the multi-sensor data in only available for the most recent few months, so we have to use RadarOnly. Let me know if you need help getting the data over to Azure.
HRRR data now seems to be on AWS! https://registry.opendata.aws/noaa-hrrr-pds/
HREF is here: https://data.nssl.noaa.gov/thredds/catalog/FRDD/HREFv2.html
Wow, amazing! Thanks so much. For now we are only using 6h but keeping 3h probably doesn't hurt. Yes, it seems the multi-sensor data in only available for the most recent few months, so we have to use RadarOnly. Let me know if you need help getting the data over to Azure.
It seems that the multi-sensor is available from September 2019 on in the google drive - I'll download as much as I can find. Still looking for archived data further back too.
@raspstephan the radar only 1hr accumulations are available back to 2014. So if we need more data one option would be to generate 6hrly accumulations from that. The multi-sensor seems to be better quality though
The new data is available at /datadrive/mrms/raw_drive
HRRR data is now downloading and regridding. Should be done for a year of data in a day or two.
@HirtM One thing to be aware of is that the HRRR domain is a little smaller than the radar coverage. You can see it at the bottom of the figure below, where the precipitation is cut off. Currently, these values are simply zero. We just need to make sure to consider this for evaluation.
Now that we have more MRMS data (yay!), let's use 2018/19 for training and 2020 for validation. I am currently downloading all the required data. This will take a week at least though. In the meantime, we can just keep using the few months of data we already have.
Now that we have more MRMS data (yay!), let's use 2018/19 for training and 2020 for validation. I am currently downloading all the required data. This will take a week at least though. In the meantime, we can just keep using the few months of data we already have.
Thanks @raspstephan!
@annavaughan @HirtM I just saw that we've run out of disk space on the CPU machine. I will add more storage tomorrow.
@annavaughan @HirtM I just saw that we've run out of disk space on the CPU machine. I will add more storage tomorrow.
We now have 3TB on the CPU machine which should be enough for a little bit.
After switch to Azure
Data period is controlled by available MRMS data. Region is CONUS. We could use Snakemake as for WeatherBench for the data downloads to catch errors.