runGdal not downloading TIF files

[pabuveery]

Hello -- I'm using the eBird best practices guidelines for downloading MODIS tiles, and while the runGdal function runs for a couple of hours, in my output folder I am only finding .hdf files (each only 7 bytes), and no tif files. In addition, the following error is thrown:

Error in setPath(path.expand(paste0(opts$localArcPath, paste0(unlist(string),  : 
  dir.create(path, recursive = TRUE, showWarnings = showWarnings) is not TRUE

I have double checked that my EarthDataUSA credentials are entered properly using the command file.edit("~/.netrc")

Any help would be greatly appreciated. I've pasted my code below, and will paste the session info in followup comments. Thank you!

tiles <- getTile(bcr12)
tifs <- runGdal(product = "MCD12Q1", collection = "061", SDSstring = "01", 
                tileH = tiles@tileH, tileV = tiles@tileV,
                begin = "2010.01.01", end = "2018.12.31", 
                job = "warbler_bcr_12", MODISserverOrder = "LPDAAC") %>% 
  pluck("MCD12Q1.061") %>% 
  unlist()

When I run class(bcr12), I get the following:

class(bcr12)
[1] "sf"         "tbl_df"     "tbl"        "data.frame"

[pabuveery]

Here is my session info:

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] gdalUtils_2.0.3.2 curl_4.3.2 rnaturalearth_0.1.0 forcats_0.5.1 stringr_1.4.0
[6] dplyr_1.0.9 purrr_0.3.4 readr_2.1.2 tidyr_1.2.0 tibble_3.1.8
[11] ggplot2_3.3.6 tidyverse_1.3.1 viridis_0.6.2 viridisLite_0.4.0 MODIStsp_2.0.8
[16] MODIS_1.2.3 mapdata_2.3.0 maps_3.4.0 devtools_2.4.4 usethis_2.1.6
[21] raster_3.5-29 sp_1.5-0 remotes_2.4.2 sf_1.0-8

loaded via a namespace (and not attached): [1] colorspace_2.0-3 ellipsis_0.3.2 class_7.3-20 rgdal_1.5-32 fs_1.5.2
[6] rstudioapi_0.13 proxy_0.4-27 bit64_4.0.5 fansi_1.0.3 lubridate_1.8.0
[11] xml2_1.3.3 codetools_0.2-18 R.methodsS3_1.8.2 auk_0.5.1 cachem_1.0.6
[16] pkgload_1.3.0 jsonlite_1.8.0 ptw_1.9-16 broom_0.8.0 dbplyr_2.2.0
[21] R.oo_1.25.0 rgeos_0.5-9 shiny_1.7.2 RcppDE_0.1.6 compiler_4.2.0
[26] httr_1.4.3 backports_1.4.1 assertthat_0.2.1 fastmap_1.1.0 cli_3.3.0
[31] later_1.3.0 s2_1.1.0 htmltools_0.5.3 prettyunits_1.1.1 tools_4.2.0
[36] gtable_0.3.0 glue_1.6.2 wk_0.6.0 gdalUtilities_1.2.1 Rcpp_1.0.9
[41] cellranger_1.1.0 vctrs_0.4.1 iterators_1.0.14 ps_1.7.1 rvest_1.0.2
[46] mime_0.12 miniUI_0.1.1.1 lifecycle_1.0.1 terra_1.6-7 zoo_1.8-10
[51] scales_1.2.0 vroom_1.5.7 hms_1.1.1 promises_1.2.0.1 parallel_4.2.0
[56] mapedit_0.6.0 memoise_2.0.1 gridExtra_2.3 stringi_1.7.8 maptools_1.1-4
[61] foreach_1.5.2 e1071_1.7-11 pkgbuild_1.3.1 rlang_1.0.4 pkgconfig_2.0.3
[66] bitops_1.0-7 evaluate_0.16 lattice_0.20-45 htmlwidgets_1.5.4 bit_4.0.4
[71] processx_3.7.0 tidyselect_1.1.2 magrittr_2.0.3 R6_2.5.1 generics_0.1.3
[76] profvis_0.3.7 DBI_1.1.3 pillar_1.8.0 haven_2.5.0 foreign_0.8-82
[81] withr_2.5.0 units_0.8-0 xts_0.12.1 modelr_0.1.8 crayon_1.5.1
[86] KernSmooth_2.23-20 utf8_1.2.2 tzdb_0.3.0 urlchecker_1.0.1 grid_4.2.0
[91] readxl_1.4.0 data.table_1.14.2 callr_3.7.1 reprex_2.0.1 digest_0.6.29
[96] classInt_0.4-7 xtable_1.8-4 httpuv_1.6.5 R.utils_2.12.0 munsell_0.5.0
[101] sessioninfo_1.2.2

[pabuveery]

I've pasted MODIS options below. Also, if it's relevant, I'm downloading the code onto a remote storage system that I'm accessing through a personal laptop.

MODISoptions()

STORAGE:

---

localArcPath : /Volumes/lsa-wingerb1/hackm/phylo_comm_ecol/data/ebird/data outDirPath : /Volumes/lsa-wingerb1/hackm/phylo_comm_ecol/data/ebird/data/PROCESSED

DOWNLOAD:

---

MODISserverOrder : LPDAAC, LAADS dlmethod : curl stubbornness : high wait : 0.5 quiet : TRUE

PROCESSING:

---

GDAL : 3.4.2 MRT : Enabled pixelSize : asIn outProj : asIn resamplingType : NN dataFormat : GTiff cellchunk : 1

[fdetsch]

Can you try again with the latest version from main? Installation instructions are commented in the code below. I changed a couple of things on a feature branch over the last couple of months (years?), which have now finally been merged.

# remotes::install_github("fdetsch/MODIS", ref = "main")
library(MODIS)

## modis options
lap = "~/Documents/data/MODIS_ARC"

MODISoptions(
  localArcPath = lap
  , outDirPath = file.path(
    lap
    , "PROCESSED"
  )
  , MODISserverOrder = "LPDAAC"
)

## sample area of interest
dnk = getData(
  country = "DNK"
  , level = 0L
  , path = tempdir()
)

product = "MCD12Q1"

## get tile
tiles = getTile(
  dnk
)

## get newest collection
collection = getCollection(
  product
  , forceCheck = TRUE
)

## get data
tifs = runGdal(
  product = product
  , collection = collection
  , SDSstring = "01"
  , tileH = tiles@tileH
  , tileV = tiles@tileV
  , begin = "2018.01.01"
  , end = "2018.12.31"
  , job = "mcd12q1_dnk"
)

tifs
# $MCD12Q1.061
# $MCD12Q1.061$`2018-01-01`
# [1] "~/Documents/data/MODIS_ARC/PROCESSED/mcd12q1_dnk/MCD12Q1.A2018001.LC_Type2.tif"

[fdetsch]

@pabuveery Any success?

[pabuveery]

Hi, thank you for this response! I installed the latest version and reran my code, and I'm not receiving any errors but the job also hasn't completed after several hours (and no new hdf or TIF files have appeared or existing files updated, though I see a .hdf.curltmp file appearing and disappearing). What sort of ETA might I expect for this sort of job? Thanks!

[fdetsch]

Can you try again with

MODISoptions(
  MODISserverOrder = "LAADS"
)

as suggested in https://github.com/fdetsch/MODIS/issues/106#issuecomment-1230027766? LP DAAC seems to be having authorization issues due to heavy traffic right now (see forum links in comment).

[pabuveery]

I ran your above comment, along with your code for product and collection variables, and then tried the following code chunk:

tifs <- runGdal(product = product, collection = collection, SDSstring = "01", 
                tileH = tiles@tileH, tileV = tiles@tileV,
                begin = "2012.01.01", end = "2020.12.31", 
                job = "warbler_bcr_12", MODISserverOrder = "LAADS") %>% 
  pluck("MCD12Q1.061") %>% 
  unlist()

and received the following error:

Error in if (FtpDayDirs[1] == FALSE) { : 
  missing value where TRUE/FALSE needed

And the same error occurred when I tried removing the pluck and unlist commands. I see that this same issue occurred here -- is the LAADS server perhaps also not reachable? My traceback pasted below if helpful. Thanks for your continued help on this.

3: getStruc(product = strsplit(todo[u], "\\.")[[1]][1], collection = strsplit(todo[u], 
       "\\.")[[1]][2], begin = tLimits$begin, end = tLimits$end, 
       server = server)
2: as.Date(getStruc(product = strsplit(todo[u], "\\.")[[1]][1], 
       collection = strsplit(todo[u], "\\.")[[1]][2], begin = tLimits$begin, 
       end = tLimits$end, server = server)$dates)
1: runGdal(product = product, collection = collection, SDSstring = "01", 
       tileH = tiles@tileH, tileV = tiles@tileV, begin = "2012.01.01", 
       end = "2020.12.31", job = "warbler_bcr_12", MODISserverOrder = "LAADS")

[fdetsch]

When accessing the LAADS server at https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/61/MCD12Q1/, it says "No files found", which would explain this last error. Through LP DAAC, I would guess ETA should be in the order of seconds depending on your internet connection. I'll need to dig a little deeper..

[fdetsch]

For what it's worth, this should work when querying the older collection "006", which is available on LAADS:

tifs <- runGdal(product = product, collection = "006", SDSstring = "01", 
                tileH = tiles@tileH, tileV = tiles@tileV,
                begin = "2012.01.01", end = "2020.12.31", 
                job = "warbler_bcr_12", MODISserverOrder = "LAADS")

[pabuveery]

Thanks for taking an additional look, I appreciate your help! FYI, I tried the above code, and it generated an empty .tif file (0 bytes); after about eight hours, nothing new has been created beyond that empty tif.

[fdetsch]

I made major adjustments to the download routines (see issue #105 and pull request https://github.com/fdetsch/MODIS/pull/126). Can you try running the script from above that pulls data from LP DAAC once again using the latest package version on main?

[fdetsch]

@pabuveery Any new insights?

[fdetsch]

Closing due to inactivity.