search

fdetsch / MODIS

Download and processing framework for MODIS imagery. The package provides automated access to the global online data archives LP DAAC, LAADS and NSIDC as well as processing capabilities such as file conversion, mosaicking, subsetting and time series.
Other
57 stars 27 forks source link

MODIS problem under Mac OS 11 - Big Sur #104

Open vanderleidebastiani opened 3 years ago

vanderleidebastiani commented 3 years ago

Hi,

I updated my MacBook to Big Sur and now runGdal not work.

The package sf have some problems with installation under Big Sur. I followed the instructions on this issue (https://github.com/r-spatial/sf/issues/1536) to install the sf and worked for me. I installed MODIS via devtools::install_github("MatMatt/MODIS", ref = "develop").

I also checked if I had GDAL installed with HDF4 support (https://cornelllabofornithology.github.io/ebird-best-practices/covariates.html#covariates-dl). It looks ok.

In terminal:

gdal-config --formats "derived gtiff hfa mem vrt aaigrid adrg aigrid airsar arg blx bmp bsb cals ceos ceos2 coasp cosar ctg dimap dted e00grid elas envisat ers fit gff gsg gxf hf2 idrisi ignfheightasciigrid ilwis ingr iris iso8211 jaxapalsar jdem kmlsuperoverlay l1b leveller map mrf msgn ngsgeoid nitf northwood pds prf r raw rmf rs2 safe saga sdts sentinel2 sgi sigdem srtmhgt terragen til tsx usgsdem xpm xyz zmap rik ozi grib eeda plmosaic rda wcs wms wmts daas rasterlite mbtiles pdf webp epsilon dods openjpeg jpeg2000 netcdf hdf5 hdf4 gif jpeg png pcraster fits pcidsk postgisraster"

In R: library(sf) "Linking to GEOS 3.8.1, GDAL 3.2.0, PROJ 6.3.2"

library(MODIS) "ok"

MODIS:::checkTools("GDAL") "Checking availability of GDAL: OK, GDAL 3.2.0 found!"

However, then I try rum runGdal show the following:

MODIS:::runGdal(...) "GDAL not installed or configured, read in '?MODISoptions' for help"

MODIS:::checkHdf4Driver() "HDF4 driver seems to be lacking, please install GDAL with HDF4 support."

Is there any way to fix this?

Best, Vanderlei

jeroen commented 3 years ago

This problem is now fixed on CRAN. You can now install the sf and rgdal packages from CRAN in the regular way on MacOS big sur.

fdetsch commented 3 years ago

@vanderleidebastiani Can you give it another try? @jeroen Glad to hear this has now been fixed, thanks!

vanderleidebastiani commented 3 years ago

Hi @jeroen and @fdetsch

I removed and reinstall rgdal (current version 1.5.18), sf (0.9.6) and MODIS (1.2.3) via CRAN and apparently it worked appropriately. I needed to define a new timeout to download the packages, in my case options(timeout = 240). However, the function runGdal is taken a long time to download, and I am not sure if it is in fact working.

Best, Vanderlei

vanderleidebastiani commented 3 years ago

Hi @jeroen and @fdetsch

I am working on two Macs with MacOS 11. I can run the runGdal function just in one of them. When the function runGdal does not work, it takes a long time in the download process and never ends. I can not find the exact difference between them, but the Mac that it works with uses bash as the default shell. The Mac that does not work uses zsh (Z shell) as the default shell (see https://support.apple.com/en-us/HT208050). Can that make difference?

Best, Vanderlei

mickayla32 commented 3 years ago

Hi @jeroen and @fdetsch

I removed and reinstall rgdal (current version 1.5.18), sf (0.9.6) and MODIS (1.2.3) via CRAN and apparently it worked appropriately. I needed to define a new timeout to download the packages, in my case options(timeout = 240). However, the function runGdal is taken a long time to download, and I am not sure if it is in fact working.

Best, Vanderlei

Hi Vanderlei, I was having the same original problem as you (thank you for posting this!) Would you mind sharing how you correctly removed and reinstalled rdgal, sf, and MODIS? Thank you very much, Mickayla

vanderleidebastiani commented 3 years ago

Hi @mickayla32

I typed: remove.packages(c("sf", "MODIS", "rgdal"))

However, the runGdal function is still not working.

Best, Vanderlei

fdetsch commented 3 years ago

@vanderleidebastiani Does runGdal() create any new files in getOption("MODIS_localArcPath")? If not, this would point towards issues with getHdf(), which is running under the hood.

vanderleidebastiani commented 3 years ago

Hi @fdetsch

I tried with the example of the getHdf function. It creates some folders and a file .hdf, but the file have always zero bytes. Could you help me with that?

Some additional check:

MODISoptions() File '~/.MODIS_Opts.R' does not exist. Create it now to make settings permanent? [y/n]: n

STORAGE:

localArcPath : /private/var/folders/q_/jbn0s0r90ss2_0yxltdcz0zm0000gn/T/RtmpCtrNLZ/MODISARC outDirPath : /private/var/folders/q/jbn0s0r90ss2_0yxltdcz0zm0000gn/T/RtmpCtrNLZ/MODIS_ARC/PROCESSED

DOWNLOAD:

MODISserverOrder : LPDAAC, LAADS dlmethod : auto stubbornness : high wait : 0.5 quiet : TRUE

PROCESSING:

GDAL : 3.1.1 MRT : Enabled pixelSize : asIn outProj : asIn resamplingType : NN dataFormat : GTiff cellchunk : 1

MODIS:::checkTools("GDAL") OK, GDAL 3.1.1 found!

MODIS:::checkTools("WGET") Nothing was returned

Best, Vanderlei

fdetsch commented 3 years ago

Are you positive that login credentials in ~/.netrc are correct?

vanderleidebastiani commented 3 years ago

@fdetsch

I am working on two Macs with macOS 11. One of them is OK, I have done a mistake and change the credenctials. Thanks for the tip. The other it keeps not working, the downloading never finish. I think that it can be a PATH problem, but I am not able to fix this.

In Terminal: ~echo $0 -zsh ~R CMD -looks ok

In R/RStudio:

system("echo $0") sh system("R CMD") sh: R: command not found Warning message: In system("R CMD") : error in running command

Then I change the PATH with Sys.getenv to include "/Library/Frameworks/R.framework/Resources"

Sys.getenv("PATH") [1] "/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Library/TeX/texbin:/opt/X11/bin:/Library/Apple/usr/bin:/opt/local/bin:/Applications/RStudio.app/Contents/MacOS/postback" Sys.setenv(PATH="/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Library/TeX/texbin:/opt/X11/bin:/Library/Apple/usr/bin:/opt/local/bin:/Applications/RStudio.app/Contents/MacOS/postback:/Library/Frameworks/R.framework/Resources") system("R CMD") -looks ok

I tried to include the PATH permanently, so I edited ./bash_profile (additionally I edited it on ./zshrc)

export PATH="/Library/Frameworks/R.framework/Resources:$PATH"

However, it does not make any effect. The PATH does not recognize after I restart R, it goes back to initial settings. Anyway the function runGdal nor works on this computer. I have not idea if it is in fact the problem, but this is the difference that I can find at the moment.

Do I need to include PATH in another file or I made some mistake in the way to include it?

Best, Vanderlei

fdetsch commented 3 years ago

For me as a non-Mac person, this is extremely hard to debug. Since empty files are created, I would assume that there's something wrong with the file download, some proxy or firewall issue maybe. Can you

Like this, you should be able to find the exact point at which file download fails. Maybe this will provide some insights..

vanderleidebastiani commented 3 years ago

Hi @fdetsch

I tried to debug ModisFileDownloader in both macs that I use. I notice that the functions follow distinct paths between computers. The Mac that the download can be complete use the function curl::curl_download to try download, while the Mac that not complete the download try to download via utils::download.file. Using download.file I think that have some authentication problems (I rechecked my user and pass, this is not a problem).

To fix this issue I need to force the option dlmethod with MODISoptions.

The steps to fix:

Thanks for the help.

Best, Vanderlei

fdetsch commented 3 years ago

... which brings up two questions:

vanderleidebastiani commented 3 years ago

@fdetsch

I have not set a permanent download method. The mac that the download work without set a download method a warning message is showed ("sh: wget: command not found"). This message is not a problem, the download can be completed and it looks like a message from the bash shell. I think that this difference is because of the difference between the shell, I think that I have a PATH problem in the mac where the download can not finish without set the download method. I opened a question in stackoverflow (https://stackoverflow.com/questions/65013779/shell-path-to-r-under-macos-big-sur) to try to fix, but until now it has no answer.

Best, Vanderlei

fdetsch commented 3 years ago

Great, thanks. I'd appreciate if you could keep us updated!

fdetsch commented 3 years ago

Managed to narrow this down to some kind of change in wget LAADS download behavior.

On the command line, LP DAAC download works as expected and a ~20 MB .hdf file is downloaded:

wget --user=<your_user> --password=<your_pw> --no-check-certificate https://e4ftl01.cr.usgs.gov/MOLT/MOD13A3.006/2020.01.01/MOD13A3.A2020001.h21v09.006.2020034152427.hdf

By contrast, LAADS download downloads a ~12 KB file only:

wget --user=<your_user> --password=<your_pw> --no-check-certificate https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/6/MOD13A3/2020/001/MOD13A3.A2020001.h21v09.006.2020034152427.hdf

Any wget redirect/authentication etc. experts around here, @MatMatt maybe? I am fairly sure that once this is solved, 0 KB file download issues will be resolved.

MatMatt commented 3 years ago

When I try to access the two files on a browser the LP DAAC can be downloaded, but for the LAADS url I get an error, is that to be expected (due to login) or is there an issue with the server/service? Access stops here https://ladsweb.modaps.eosdis.nasa.gov/archive/

fdetsch commented 3 years ago

You can go to https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/6/MOD13A3/2020/001 and then download the above file manually. This should redirect you to urs.earthdata.nasa.gov where you need to sign in before you can download the file.