search

AustralianAntarcticDataCentre / raadsync

Other
2 stars 0 forks source link

download speeds #39

Open mdsumner opened 7 years ago

mdsumner commented 7 years ago

This is just a placeholder/observation, today I'm seeing unuseable performance downloading files in the Tas node, it maybe related to the ongoing issues with ocean colour sync?

Manual download script, works fine in Melbourne node, hopeless in Hobart today 2017-03-06)


## extract all L3 bin chlorophyll-a for September 2015 February 2016
getter <- function(x, out) {
  u <- file.path("https://oceandata.sci.gsfc.nasa.gov/cgi/getfile", x)
  outf <- file.path(out, x)
  if (!file.exists(outf)) {
    download.file(u, outf, mode = "wb")

  }
   outf
}

templates <- c("A%s.L3b_DAY_CHL.nc", "V%s.L3b_DAY_SNPP_CHL.nc")
dates <- seq(as.Date("2015-09-01"), as.Date("2016-03-01"), by = "1 day")

outfolder <- tempdir()
#print(outfolder)

files <- unlist(lapply(templates, function(x) sprintf(x, format(dates, "%Y%j"))))
for (i in seq_along(files)) getter(files[i], outfolder)