mitchellmanware
commented
1 year ago Example: NCEP North American Regional Reanalysis
Notes
- Absolute paths included in script for exploratory purposes only. Relative paths to be included in final script
- Does not included metadata
- This script creates a
.txtfile containing all thewgetcommands for downloading data from NCEP-NARR Thredds Catalog
## Script to write 'wget' commands for batch downloading of covariate data from
## NCEP North America Regional Reanalysis
## Metadata and links available at:
## https://psl.noaa.gov/data/gridded/data.narr.html
## Mitchell Manware
## September 1, 2023
## example url for format structure
## https://downloads.psl.noaa.gov//Datasets/NARR/Dailies/monolevel/weasd.1979.nc
## define base variable for building URLs
base <- "https://psl.noaa.gov//Datasets/NARR/Dailies/"
## define list of years
years <- c("2018", "2019", "2020", "2021", "2022")
## define list of months
months <- stringr::str_pad(as.character(1:12), 2, pad="0")
## define list of variables obtained from NCEP-NARR Reanalysis
variables <- c("monolevel/weasd.", # accumulated snow
"monolevel/evap.", # accumulated total evaporation
"monolevel/apcp.", # accumulated total precipitation
"monolevel/air.sfc.", # surface air temperature
"monolevel/albedo.", # surface albedo
"monolevel/tcdc.", # total cloud coverage
"monolevel/dswrf.", # downward shortwave radiation flux
"monolevel/hcdc.", # high cloud area fraction
"monolevel/lhtfl.", # latent heat flux
"monolevel/lcdc.", # low cloud area fraction
"monolevel/mcdc.", # medium cloud area fraction
"pressure/omega.", # omega
"monolevel/hpbl.", # planetary boundary layer height
"monolevel/pr_wtr.", # precipitable water for entire atmosphere
"monolevel/prate.", # precipitation rate
"monolevel/pres.sfc.", # pressure at surface
"monolevel/shtfl.", # sensible heat flux
"monolevel/snowc.", # snow cover
"monolevel/soilm.", # soil moisture content
"pressure/shum.", # specific humidity
"monolevel/uwnd.10m.", # u-wind
"monolevel/ulwrf.sfc.", # upward longwave radiation flux
"monolevel/vwnd.10m.", # v-wind
"monolevel/vis.") # visibility)
## initiate NCEP-NARR-Reanalysis_wget_commands.txt file
sink("/ddn/gs1/home/manwareme/NRT-AP-Model/code/Data_Download/NCEP-NARR-Reanalysis_wget_commands.txt")
## for loop to download data for each variable
for(v in 1:length(variables)){
if(stringr::str_starts(variables[v], "m") == TRUE){
## define current variable
variable <- variables[v]
## define folder to save data based on current variable
folder <- sub(".*/", "", variable)
for(y in 1:length(years)){
## define url
url <- paste0(base, variable, years[y], ".nc")
## define command to be saved to text file
command <- paste0("wget --no-check-certificate -P /ddn/gs1/home/manwareme/NRT-AP-Model/input/NCEP-NARR-Reanalysis/",
folder, " ", url, "\n")
## test by printing command
print(command)
## add commands to text file
cat(command, file = "NCEP-NARR-Reanalysis_wget_commands.txt", sep = '"\n"',
append = TRUE)
}
} else {
## month indicator required for data from "Dailies/pressure/"
## example: August 2023 is 202308
## define current variable
variable <- variables[v]
## define folder to save data based on current variable
folder <- sub(".*/", "", variable)
for(y in 1:length(years)){
for(m in 1:length(months)){
## define url
url <- paste0(base, variable, years[y], months[m], ".nc")
## define command to be saved to text file
command <- paste0("wget --no-check-certificate -P /ddn/gs1/home/manwareme/NRT-AP-Model/input/NCEP-NARR-Reanalysis/",
folder, " ", url, "\n")
## test by printing command
print(command)
## add commands to text file
cat(command, file = "NCEP-NARR-Reanalysis_wget_commands.txt", sep = '"\n"',
append = TRUE)
}
}
}
}
## finish NCEP-NARR-Reanalysis_wget_commands.txt file
sink()- The following bash script runs the above
Rscript and executes each command listed in the newly generated.txtfile
#!/bin/bash
R CMD BATCH NCEP-NARR-Reanalysis_download.R
. NCEP-NARR-Reanalysis_wget_commands.txt
sigmafelix
Rscripts, bash scripts or combined for data download?wget, could use R script to generate.txtfile with allwgetcommandsRscript and.txtfile in one bash script could make reanalysis easier because the only changes would be to the dates of interest in theRscripts