getHDF cannot open connection

[fischhoff]

Hello! Apologies if this issue has already been raised elsewhere -- I didn't see it in the list.

I'm hitting an error with getHDF: b1 <- getHdf(product = "M.D11A1", begin = "2016.12.01", tileH = 18:19, tileV = 4)

cannot create dir '/var/folders/0d/qm_pqljx11s_ddc42g1_yscr0000gn/T/RtmpQjGo6F/MODIS_ARC/PROCESSED/.auxiliaries', reason 'No such file or directory'cannot create file '/var/folders/0d/qm_pqljx11s_ddc42g1_yscr0000gn/T/RtmpQjGo6F/MODIS_ARC/PROCESSED/.auxiliaries//collections.RData', reason 'No such file or directory'cannot open compressed file '/var/folders/0d/qm_pqljx11s_ddc42g1_yscr0000gn/T/RtmpQjGo6F/MODIS_ARC/PROCESSED/.auxiliaries//collections.RData', probable reason 'No such file or directory'Error in readChar(con, 5L, useBytes = TRUE) : cannot open the connection

The long string above refers to my TMPDIR folder (on Mac). After hitting this error, I've tried creating the folder structure within my TMPDIR of MODIS_ARC/PROCESSED/.auxilliaries, and also created collections.Rdata, to no avail. Thanks a lot for any advice!

[fdetsch]

No problems over here using Windows tempdir() (make sure to use the 'develop' version until the next CRAN release).

# devtools::install_github("MatMatt/MODIS", ref = "develop")
library(MODIS)

lap = file.path(tempdir(), "MODIS_ARC")
odp = file.path(lap, "PROCESSED")
MODISoptions(lap, odp)

b1 <- getHdf(product = "M.D11A1", begin = "2016.12.01", end = "2016.12.02"
             , tileH = 18:19, tileV = 4)
b1
# $`MOD11A1.006`
# [1] "C:/Users/florianD/AppData/Local/Temp/RtmpU7Rre8/MODIS_ARC/MODIS/MOD11A1.006/2016.12.01/MOD11A1.A2016336.h18v04.006.2016340141659.hdf"
# [2] "C:/Users/florianD/AppData/Local/Temp/RtmpU7Rre8/MODIS_ARC/MODIS/MOD11A1.006/2016.12.01/MOD11A1.A2016336.h19v04.006.2016340140014.hdf"
# [3] "C:/Users/florianD/AppData/Local/Temp/RtmpU7Rre8/MODIS_ARC/MODIS/MOD11A1.006/2016.12.02/MOD11A1.A2016337.h18v04.006.2016340142327.hdf"
# [4] "C:/Users/florianD/AppData/Local/Temp/RtmpU7Rre8/MODIS_ARC/MODIS/MOD11A1.006/2016.12.02/MOD11A1.A2016337.h19v04.006.2016340140534.hdf"
# 
# $MYD11A1.006
# [1] "C:/Users/florianD/AppData/Local/Temp/RtmpU7Rre8/MODIS_ARC/MODIS/MYD11A1.006/2016.12.01/MYD11A1.A2016336.h18v04.006.2016338081708.hdf"
# [2] "C:/Users/florianD/AppData/Local/Temp/RtmpU7Rre8/MODIS_ARC/MODIS/MYD11A1.006/2016.12.01/MYD11A1.A2016336.h19v04.006.2016338081708.hdf"
# [3] "C:/Users/florianD/AppData/Local/Temp/RtmpU7Rre8/MODIS_ARC/MODIS/MYD11A1.006/2016.12.02/MYD11A1.A2016337.h18v04.006.2016338180949.hdf"
# [4] "C:/Users/florianD/AppData/Local/Temp/RtmpU7Rre8/MODIS_ARC/MODIS/MYD11A1.006/2016.12.02/MYD11A1.A2016337.h19v04.006.2016338180947.hdf"

I have no experience with MAC, so this will be hard for me to address. MODIS's 'localArcPath' used to be created in the user's home directory by default; however, we recently had to change this to tempdir() due to a violation of CRAN policies. I assume R does have write access to tempdir()? What happens if you change 'localArcPath' and 'outDirPath' to some other folder, say your home directory?

[fdetsch]

@fischhoff, any updates on that matter?

[fischhoff]

Hi @fdetsch thanks for looking into this and sorry for the slow response! I am working on getting the MODIS reprojection tool on my Mac so I can give this a proper test.

[fdetsch]

In fact, you don't need the MRT for testing. A valid GDAL installation should be sufficient.

[gktech]

Hello, I am having the same or a similar issue on Windows:

# devtools::install_github("MatMatt/MODIS", ref = "develop")
library(MODIS)

> lap = file.path('C:\\temp\\', "MODIS_ARC")
> odp = file.path(lap, "PROCESSED")
> MODISoptions(lap, odp)
  'MRT_HOME' not set/found! MRT is NOT enabled! See: 'https://lpdaac.usgs.gov/tools/modis_reprojection_tool'
File '~/.MODIS_Opts.R' does not exist. Create it now to make settings permanent? [y/n]: n

STORAGE:
_______________
localArcPath : C:/temp/MODIS_ARC/ 
outDirPath   : C:/temp/MODIS_ARC/PROCESSED/ 

DOWNLOAD:
_______________
MODISserverOrder : LAADS, LPDAAC 
dlmethod         : auto 
stubbornness     : high 
wait             : 0.5 
quiet            : TRUE 

PROCESSING:
_______________
GDAL           : GDAL 1.7.0b2, FWTools 2.4.7, released 2010/01/19 
MRT            : Not available. Use 'MODIS:::checkTools('MRT')' for more information! 
pixelSize      : asIn 
outProj        : asIn 
resamplingType : NN 
dataFormat     : GTiff 

a <- getHdf('MOD09GQ', begin='2003219', end='2003219', extent='Austria')
Updating collection from LPDAAC for product: MOD09GQ 
Error in readLines(con) : 
  Timeout was reached: Operation timed out after 10000 milliseconds with 0 out of 0 bytes received
Error in readLines(con) : 
  Timeout was reached: Operation timed out after 10000 milliseconds with 0 out of 0 bytes received
Error in readLines(con) : 
  Timeout was reached: Operation timed out after 10000 milliseconds with 0 out of 0 bytes received
Error in readLines(con) : 
  Timeout was reached: Operation timed out after 10015 milliseconds with 0 out of 0 bytes received
Error in readLines(con) : 
  Timeout was reached: Operation timed out after 10000 milliseconds with 0 out of 0 bytes received
Error in file(file, "rt") : cannot open the connection
Error in file(file, "rt") : cannot open the connection

> traceback()
9: readLines(con)
8: FUN(X[[i]], ...)
7: lapply(X = X, FUN = FUN, ...)
6: sapply(file.path(ftp, dirs, "/"), function(ftpdir) {
       h <- curl::new_handle()
       if (grepl("nsidc", ftpdir)) {
           curl::handle_setopt(handle = h, httpauth = 1, userpwd = paste0(credentials()$login, 
               ":", credentials()$password))
       }
       con = curl::curl(ftpdir, handle = h)
       on.exit(try(close(con), silent = TRUE))
       cnt = readLines(con)
       close(con)
       dts = regmatches(cnt, regexpr("[[:digit:]]{4}\\.[[:digit:]]{2}\\.[[:digit:]]{2}", 
           cnt))
       return(length(dts) > 0)
   })
5: getCollection(product = product, quiet = TRUE, forceCheck = TRUE)
4: unlist(getCollection(product = product, quiet = TRUE, forceCheck = TRUE))
3: .local(product, ...)
2: getHdf("MOD09GQ", begin = "2003219", end = "2003219", extent = "Austria")
1: getHdf("MOD09GQ", begin = "2003219", end = "2003219", extent = "Austria")

Tried running RStudio as administrator to circumvent possible permission problems.

Regards, Georg

[fischhoff]

@fdetsch Apologies for the long delay!

library(devtools)
install_github("MatMatt/MODIS")
library(MODIS)

lap = file.path(tempdir(), "MODIS_ARC")
odp = file.path(lap, "PROCESSED")
MODISoptions(lap, odp)

'outDirPath' does not exist and will be created in '/var/folders/0d/qm_pqljx11s_ddc42g1_yscr0000gn/T/RtmpW0ftDL/MODIS_ARC/PROCESSED/'. HDF4 driver seems to be lacking. Please install GDAL with HDF4 support. 'MRT_HOME' not set/found! MRT is NOT enabled! See: 'https://lpdaac.usgs.gov/tools/modis_reprojection_tool'

I'd appreciate any advice on getting HDF4 to be recognized. When I check gdal-config --formats in terminal, hdf4 is on the list. There must be something wrong with paths. Thanks!

[melody1006]

Hello, I am having the same or a similar issue on Windows: lap = "D:/R_MODIS/MODIS_ARC"

odp = file.path(lap, "PROCESSED") MODISoptions(lap,odp,MODISserverOrder = "LPDAAC")

STORAGE:

---

localArcPath : D:/R_MODIS/MODIS_ARC outDirPath : D:/R_MODIS/MODIS_ARC/PROCESSED

DOWNLOAD:

---

MODISserverOrder : LPDAAC dlmethod : auto stubbornness : high wait : 0.5 quiet : TRUE

PROCESSING:

---

GDAL : 3.2.1 MRT : Version 4.1 (March 2011) pixelSize : asIn outProj : asIn resamplingType : NN dataFormat : GTiff cellchunk : 1

EarthdataLogin(usr = '*', pwd = '**') getHdf(product = "MOD09A1", collection = "006",begin = "2021129", end = "2021153",extent = list(xmin = -180, xmax = 180, ymin = 60, ymax = 90)) Error in readLines(con) : Timeout was reached: [e4ftl01.cr.usgs.gov] Connection timed out after 10001 milliseconds