add mass download function

[mdwhitby]

suggested function attached to download NEXRAD 2 .gz files from AWS

NEXRAD_L2_massdownload.txt

##SET STATION ID - four letter character string
STATION <- "KOAX"

##vector of dates as a string in the YYMMDD format (
#DATES <- c("20150708","20150715", "20150722", "20150730", "20150905", "20150912","20150919", "20150927")
DATES <- c("20150722", "20150905")
##Start and end hours of interest for each day (must be same for all days) as integer 0-23
startHOUR <- 0
endHOUR <- 14

##folder to save radard.gz files to
saveLoc <- "./Bioscatter_data"

AWSbucket = "https://noaa-nexrad-level2.s3.amazonaws.com/"

system.time(
AWS_objects <- get_NEXRAD2(STATION,DATES,saveLoc,startHOUR,endHOUR)
)

# Function to download NEXRAD2 files from AWS -----------------------------
##version: 0.1.1
##Author: Michael Whitby
##EMAIL: michael.whitby@gmail.com
##July 28, 2017 

get_NEXRAD2 <- function(STATION,##SET STATION ID - four letter character string
                        DATES,##vector of dates as a string in the YYMMDD format
                        saveLoc=getwd(),
                        startHOUR = 00,##Start and end hours of interest for each day (must be same for all days) as integer 0-23
                        endHOUR = 23,
                        AWSbucket = "https://noaa-nexrad-level2.s3.amazonaws.com/"
                        ) {

dates.year <- substring(DATES, 1, 4)
dates.month <- substring(DATES, 5, 6)
dates.day <- substring(DATES, 7,9)

times <- seq(startHOUR, endHOUR)

times <- stringr::str_pad(times, 2, pad="0")

#files <- paste0(STATION,DATES,"_", times)
files <- c(t(outer(DATES, times, paste)))
files <- sub(" ", "_", files)
files <- paste0(STATION, files)
location <- paste0(dates.year, "/", dates.month, "/", dates.day, "/", STATION, "/")
obj <- c(t(outer(location, files, paste0)))

library(aws.s3)
all <- list()
  all <- lapply( obj, function(o){
    get_bucket_df(bucket = "noaa-nexrad-level2", o)
  }
  )

toDWNLD <- plyr::ldply(all)$Key

DWNLDURLS <- c(t(outer(AWSbucket, toDWNLD, paste0)))

saveLoc=paste0(saveLoc, "/", STATION, "/")
saveFile=substr(toDWNLD, 17, 42)
dub <- (saveFile  %in% list.files(saveLoc))
saveFile = saveFile[!dub]

saveFile=paste0(saveLoc, saveFile)
toDWNLD =toDWNLD[!dub]

lapply(1:length(DWNLDURLS), function (i){ 
  download.file(DWNLDURLS[i], destfile = saveFile[i])
})

}

[adokter]

write a download_pvolfiles function very similar to download_vpfiles written by @stijnvanhoey

[Rafnuss]

Would be interested in a function like that. Has anyone use it or something similar? Is it worth if I try to write a function for that? If so, a main question is about authentification with the AWS API. If I understand it correctly, you need credential to use the API. Is that correct?

[adokter]

Hi @Rafnuss that would be nice - the aws command line interface (CLI) has a flag --no-sign-request which allows you to do requests without an AWS account, not sure if something similar exists for the AWS API

[adokter]

closed by #487