runGdal "Error 4"

[paulauvirage]

Hello

I am running MODIS, for the first time, in R 4.0.1 via R Studio on Ubuntu 20

runGdal returns "ERROR 4 /tmp .....htf' not recognised as a supported file format. ... gdalinfo failed - unable to open /tmp ...htf

When I run

MODIS:::checkTools()

I get OK, GDAL 3.0.4... found! and an error message for MRT (not enabled), but i think that is deprecated anyway.

If I run gdalinfo on the command line, outside of R, it works fine, as does gdal_translate to convert .htfs to tifs for .htfs I have downloaded directly from USGS Earth Data.

whereis gdal returns usr/include/gdal (a lot of .h files) and usr/share/gdal (files of mixxed types).

Many thanks for any help

[fdetsch]

I believe it should be .hdf. If this doesn't solve it, please provide some code to reproduce the issue.

[paulauvirage]

Sorry, I did use .hdf in the code i ran, just wrote .htf by mistake in the code I posted above! I was using the eBird tutorial in addition to the MODIS package help.

After MODIS::Earthdatalogin() and MODIS:::checkTools("GDAL") I ran this code:

sntifs <- runGdal(product = "MCD12Q1",
                collection = "006",
                begin = "2018-01-01",
                end = "2019-01-01",
                tileH = "16",
                tileV = "07",
                SDSstring = "01")

and go this error message repeatedly

Downloading structure on 'LPDAAC' for: MCD12Q1.006 ERROR 4: `/tmp/Rtmpb44T3g/MODIS_ARC/MODIS/MCD12Q1.006/2018.01.01/MCD12Q1.A2018001.h16v07.006.2019199202821.hdf' not recognized as a supported file format. gdalinfo failed - unable to open '/tmp/Rtmpb44T3g/MODIS_ARC/MODIS/MCD12Q1.006/2018.01.01/MCD12Q1.A2018001.h16v07.006.2019199202821.hdf'.

[fdetsch]

Can you please attach your sessionInfo()?

[paulauvirage]

Here it is:

sessionInfo() R version 4.0.1 (2020-06-06) Platform: x86_64-pc-linux-gnu (64-bit) Running under: Ubuntu 20.04 LTS

Matrix products: default BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0 LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0

locale: [1] LC_CTYPE=en_GB.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_GB.UTF-8 LC_COLLATE=en_GB.UTF-8
[5] LC_MONETARY=en_GB.UTF-8 LC_MESSAGES=en_GB.UTF-8
[7] LC_PAPER=en_GB.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=C

attached base packages: [1] stats graphics grDevices utils datasets [6] methods base

other attached packages: [1] MODIS_1.2.0 raster_3.1-5 sp_1.4-2
[4] mapdata_2.3.0 maps_3.3.0 gridExtra_2.3
[7] sf_0.9-4 lubridate_1.7.9 auk_0.4.1
[10] mapview_2.7.8 data.table_1.12.8 forcats_0.5.0
[13] stringr_1.4.0 dplyr_1.0.0 purrr_0.3.4
[16] readr_1.3.1 tidyr_1.1.0 tibble_3.0.1
[19] ggplot2_3.3.1 tidyverse_1.3.0

loaded via a namespace (and not attached): [1] leafem_0.1.1 colorspace_1.4-1
[3] ellipsis_0.3.1 class_7.3-17
[5] leaflet_2.0.3 rgdal_1.5-10
[7] rprojroot_1.3-2 satellite_1.0.2
[9] base64enc_0.1-3 fs_1.4.1
[11] rstudioapi_0.11 remotes_2.1.1
[13] fansi_0.4.1 xml2_1.3.2
[15] codetools_0.2-16 knitr_1.28
[17] pkgload_1.1.0 jsonlite_1.6.1
[19] nloptr_1.2.2.1 ptw_1.9-15
[21] packrat_0.5.0 broom_0.5.6
[23] dbplyr_1.4.4 png_0.1-7
[25] rgeos_0.5-3 shiny_1.4.0.2
[27] compiler_4.0.1 httr_1.4.1
[29] backports_1.1.7 assertthat_0.2.1
[31] fastmap_1.0.1 cli_2.0.2
[33] later_1.1.0.1 htmltools_0.4.0
[35] prettyunits_1.1.1 tools_4.0.1
[37] gtable_0.3.0 glue_1.4.1
[39] Rcpp_1.0.4.6 cellranger_1.1.0
[41] vctrs_0.3.1 nlme_3.1-147
[43] crosstalk_1.1.0.1 xfun_0.14
[45] ps_1.3.3 testthat_2.3.2
[47] rvest_0.3.5 mime_0.9
[49] lifecycle_0.2.0 devtools_2.3.0
[51] scales_1.1.1 hms_0.5.3
[53] promises_1.1.1 parallel_4.0.1
[55] mapedit_0.6.0 yaml_2.2.1
[57] curl_4.3 memoise_1.1.0
[59] stringi_1.4.6 maptools_1.0-1
[61] desc_1.2.0 e1071_1.7-3
[63] pkgbuild_1.0.8 rlang_0.4.6
[65] pkgconfig_2.0.3 bitops_1.0-6
[67] evaluate_0.14 lattice_0.20-41
[69] htmlwidgets_1.5.1 processx_3.4.2
[71] tidyselect_1.1.0 magrittr_1.5
[73] R6_2.4.1 generics_0.0.2
[75] DBI_1.1.0 pillar_1.4.4
[77] haven_2.3.1 foreign_0.8-79
[79] withr_2.2.0 units_0.6-7
[81] modelr_0.1.8 crayon_1.3.4
[83] KernSmooth_2.23-17 rmarkdown_2.2
[85] usethis_1.6.1 grid_4.0.1
[87] readxl_1.3.1 blob_1.2.1
[89] callr_3.4.3 reprex_0.3.0
[91] digest_0.6.25 classInt_0.4-3
[93] webshot_0.5.2 xtable_1.8-4
[95] httpuv_1.5.4 stats4_4.0.1
[97] munsell_0.5.0 viridisLite_0.3.0 [99] sessioninfo_1.1.1

[fdetsch]

This is weird, your code executes flawlessly on my Linux test machines. As of MODIS-1.2.0, all GDAL related operations are carried out via sf::gdal_utils(). Could you try the following to narrow the problem down a little bit?

snhdfs = getHdf(
  product = "MCD12Q1"
  , collection = "006"
  , begin = "2018-01-01"
  , end = "2019-01-01"
  , tileH = "16"
  , tileV = "07"
)

sf::gdal_utils(
  util = "info"
  , source = unlist(snhdfs)
)

This should print the usual gdalinfo output to your console. In addition, make sure to double check your Earthdata login credentials stored in "~/.netrc". If file.size(unlist(snhdfs)) is equal or close to zero, this usually indicates that something went wrong during authentication/file download.

[paulauvirage]

Thanks for your patience!

On running the first block of R code above (snhdfs = getHdf etc.. ) I get the same error as previously:

ERROR 4: `/tmp/RtmpKvE374/MODIS_ARC/MODIS/MCD12Q1.006/2018.01.01/MCD12Q1.A2018001.h16v07.006.2019199202821.hdf' not recognized as a supported file format. gdalinfo failed - unable to open '/tmp/RtmpKvE374/MODIS_ARC/MODIS/MCD12Q1.006/2018.01.01/MCD12Q1.A2018001.h16v07.006.2019199202821.hdf'.

So the second code block just tells me that object snhdfs not found and file.size(unlist(snhdfs) also tells me object snhdfs not found

I checked in "~/.netrc" and my Earthdata login is the same as that entered in MODIS::EarthdataLogin and I can log into the nasa Earthdata website with these login credentials. And I am logged in to the website whilst running the R MODIS code

[fdetsch]

How about file.size("/tmp/RtmpKvE374/MODIS_ARC/MODIS/MCD12Q1.006/2018.01.01/MCD12Q1.A2018001.h16v07.006.2019199202821.hdf") (path will change for a new session) then?

[paulauvirage]

If I run file.size() on the .hdf file generated in the new ERROR 4 message, after I have re-run the getHdf code in your previous message, the output is [1] 0.

As the ERROR 4 message repeats itself, each with a different long .hdf file name, I tried file.size() on the first four and the output for all was the same: [1] 0

[paulauvirage]

OK! Having uninstalled the development version of MODIS and replaced it with the CRAN version, the runGdal code posted six of my posts above, and which you found to be working fine on your Linux testing machines, now seems to be working fine on my laptop. I am sorry if I have given you unnecessary worry, as I realise I did not mention before that I was using the development version. I'll check everything is OK later this evening then close it Thanks for all your help.

[fdetsch]

No worries, glad to help.