getGLDAS error "Access denied"

[rsmiller74]

getGLDAS returns access denied error.

utils::packageVersion("climateR") [1] ‘0.3.0’

checkNetrc() [1] TRUE

tmp <- getGLDAS(AOI=aoi_get(state = "conus"), varname = "snowdepth_tavg", model = "CLSM025_DA1_D.2.2", startDate="2021-01-01", endDate="2021-12-31", verbose = TRUE )

source: https://hydro1.gesdisc.eosdis.nasa.gov/dods/GLDAS_CLSM025_DA... varname(s): snowdepth_tavg [NA] (** snow depth [m] )

diminsions: 233, 101, 365 (names: lon,lat,time) resolution: 0.25, 0.25, 1 days extent: -125, -66.75, 24.25, 49.5 (xmin, xmax, ymin, ymax) crs: +proj=longlat +a=6378137 +f=0.00335281066474748 +p... time: 2021-01-01 to 2021-12-31

values: 8,589,545 (varsXY*T)syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR context: HTTP^ Basic: Access denied.

[mhweber]

I get the same error with:

sites_stack <- getGLDAS(AOI=AOI::aoi_get(state = "OR",
                        varname='rootmoist_inst',
                        model='NOAH025_M.2.1',
                        startDate='2006-10-01',
                        endDate='2007-09-30',
                        verbose = TRUE)

utils::packageVersion("climateR")
[1] ‘0.3.0’

checkNetrc()
[1] TRUE

But on one machine I get the same error of:

source:  https://hydro1.gesdisc.eosdis.nasa.gov/dods/GLDAS_NOAH025_M.... 
varname(s):
   > rootmoist_inst [NA] (** root zone soil moisture [kg m-2] )
==================================================
diminsions:  34, 19, 13 (names: lon,lat,time)
resolution:  0.25, 0.25, 1 months
extent:      -124.75, -116.25, 41.75, 46.5 (xmin, xmax, ymin, ymax)
crs:         +proj=longlat +a=6378137 +f=0.00335281066474748 +p...
time:        2006-10-01 to 2007-10-01
==================================================
values: 8,398 (vars*X*Y*T)syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: HTTP^ Basic: Access denied.

And on another machine (a work computer that may have some security issues) I get:

source:  https://hydro1.gesdisc.eosdis.nasa.gov/dods/GLDAS_NOAH025_M.... 
varname(s):
   > rootmoist_inst [NA] (** root zone soil moisture [kg m-2] )
==================================================
diminsions:  34, 19, 13 (names: lon,lat,time)
resolution:  0.25, 0.25, 1 months
extent:      -124.75, -116.25, 41.75, 46.5 (xmin, xmax, ymin, ymax)
crs:         +proj=longlat +a=6378137 +f=0.00335281066474748 +p...
time:        2006-10-01 to 2007-10-01
==================================================
values: 8,398 (vars*X*Y*T)Note:Caching=1
Error:curl error: SSL connect error
curl error details: 
Warning:oc_open: Could not read url

[mikejohnson51]

Hi @mhweber and @rsmiller74

Thanks for the note! I am not seeing this on my machine but am happy to try and work through it:

library(climateR)

sites_stack <- getGLDAS(AOI=AOI::aoi_get(state = "OR"),
                        varname='rootmoist_inst',
                        model='NOAH025_M.2.1',
                        startDate='2006-10-01',
                        endDate='2007-09-30',
                        verbose = TRUE)

#> source:   https://hydro1.gesdisc.eosdis.nasa.gov/dods/GLDAS_NOAH025_M.... 
#> varname(s):
#>    > rootmoist_inst [NA] (** root zone soil moisture [kg m-2] )
#> ==================================================
#> diminsions:  34, 19, 13 (names: lon,lat,time)
#> resolution:  0.25, 0.25, 1 months
#> extent:      -124.75, -116.25, 41.75, 46.5 (xmin, xmax, ymin, ymax)
#> crs:         +proj=longlat +a=6378137 +f=0.00335281066474748 +p...
#> time:        2006-10-01 to 2007-10-01
#> ==================================================
#> values: 8,398 (vars*X*Y*T)

terra::plot(sites_stack[[1]])

^{Created on 2023-06-27 by the reprex package (v2.0.1)}

Would you please check these and make sure you have these two files on your machine?

getDodsrcPath()
#> [1] "/Users/mjohnson/.dodsrc"
getNetrcPath()
#> [1] "/Users/mjohnson/.netrc"

Thanks!

Mike

[mhweber]

Hi @mikejohnson51 - here's what I'm currently getting:

library(climateR)
sites_stack <- getGLDAS(AOI=AOI::aoi_get(state = "OR"),
                        varname='rootmoist_inst',
                        model='NOAH025_M.2.1',
                        startDate='2006-10-01',
                        endDate='2007-09-30',
                        verbose = TRUE)
#> source:   https://hydro1.gesdisc.eosdis.nasa.gov/dods/GLDAS_NOAH025_M.... 
#> varname(s):
#>    > rootmoist_inst [NA] (** root zone soil moisture [kg m-2] )
#> ==================================================
#> diminsions:  34, 19, 13 (names: lon,lat,time)
#> resolution:  0.25, 0.25, 1 months
#> extent:      -124.75, -116.25, 41.75, 46.5 (xmin, xmax, ymin, ymax)
#> crs:         +proj=longlat +a=6378137 +f=0.00335281066474748 +p...
#> time:        2006-10-01 to 2007-10-01
#> ==================================================
#> values: 8,398 (vars*X*Y*T)syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
#>context: HTTP^ Basic: Access denied.
terra::plot(sites_stack[[1]])
#> Warning in xy.coords(x, y, xlabel, ylabel, log): NAs introduced by coercion
#> Warning in min(x): no non-missing arguments to min; returning Inf
#> Warning in max(x): no non-missing arguments to max; returning -Inf
#> Error in plot.window(...): need finite 'ylim' values

getDodsrcPath()
#> [1] "C:/Users/mwebe/OneDrive/Documents/_dodsrc"
getNetrcPath()
#> [1] "C:/Users/mwebe/OneDrive/Documents/_netrc"

sessionInfo()
#> R version 4.3.0 (2023-04-21 ucrt)
#> Platform: x86_64-w64-mingw32/x64 (64-bit)
#> Running under: Windows 10 x64 (build 19045)
#> 
#> Matrix products: default
#> 
#> 
#> locale:
#> [1] LC_COLLATE=English_United States.utf8 
#> [2] LC_CTYPE=English_United States.utf8   
#> [3] LC_MONETARY=English_United States.utf8
#> [4] LC_NUMERIC=C                          
#> [5] LC_TIME=English_United States.utf8    
#> 
#> time zone: America/Los_Angeles
#> tzcode source: internal
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> loaded via a namespace (and not attached):
#>  [1] digest_0.6.31   fastmap_1.1.1   xfun_0.39       glue_1.6.2     
#>  [5] knitr_1.42      htmltools_0.5.5 rmarkdown_2.21  lifecycle_1.0.3
#>  [9] cli_3.6.1       reprex_2.0.2    withr_2.5.0     compiler_4.3.0 
#> [13] rstudioapi_0.14 tools_4.3.0     evaluate_0.20   yaml_2.3.7     
#> [17] rlang_1.1.1     fs_1.6.2

^{Created on 2023-06-27 with reprex v2.0.2}

[mikejohnson51]

Hey Marc!

Thanks, I have a hunch this is a windows vs Unix issue. I need to double check that I am defining the rc files correctly for windows.

Those rc files are also ment to be in the home directory so I need to figure out where that is going wrong!

Thanks for bringing it to my attention.

[mikejohnson51]

Hey @mhweber, I am trying to get to the bottom of this but am hitting a wall.

Conceptually, these rc files need to live in the home directory:

I am assuming this

Sys.getenv("HOME")

gives you C:/Users/mwebe/OneDrive/Documents/ in R?

But I assume (in terminal)

cd ~
pwd

takes you elsewhere?

Would you mind confirming?

Thanks!

[mhweber]

Thanks Mike, sorry for the rabbit hole! I'm on a work computer where I'm also getting a slightly different error - here's what I have:

Sys.getenv("HOME")
#> [1] "C:/Users/mweber/OneDrive - Environmental Protection Agency (EPA)/Profile/Documents"

cd ~
pwd

$  C:/Users/mweber/OneDrive - Environmental Protection Agency (EPA)/Profile/Documents

And on this machine I get:

``` r
library(climateR)
sites_stack <- getGLDAS(AOI=AOI::aoi_get(state = "OR"),
                        varname='rootmoist_inst',
                        model='NOAH025_M.2.1',
                        startDate='2006-10-01',
                        endDate='2007-09-30',
                        verbose = TRUE)
#> Found Netrc file. Writing dodsrs file to: C:/Users/mweber/OneDrive - Environmental Protection Agency (EPA)/Profile/Documents/_dodsrc
#> source:   https://hydro1.gesdisc.eosdis.nasa.gov/dods/GLDAS_NOAH025_M.... 
#> varname(s):
#>    > rootmoist_inst [NA] (** root zone soil moisture [kg m-2] )
#> ==================================================
#> diminsions:  34, 19, 13 (names: lon,lat,time)
#> resolution:  0.25, 0.25, 1 months
#> extent:      -124.75, -116.25, 41.75, 46.5 (xmin, xmax, ymin, ymax)
#> crs:         +proj=longlat +a=6378137 +f=0.00335281066474748 +p...
#> time:        2006-10-01 to 2007-10-01
#> ==================================================
#> values: 8,398 (vars*X*Y*T)Error:curl error: SSL connect error
#> curl error details: 
Warning:oc_open: Could not read url
str(sites_stack)
#>  'glue' chr "https://hydro1.gesdisc.eosdis.nasa.gov/dods/GLDAS_NOAH025_M.2.1/?rootmoist_inst[81:1:93][407:1:425][221:1:254]"

^{Created on 2023-06-28 with reprex v2.0.2}

Fee free to reach out directly if helpful for me to test some things!

[mikejohnson51]

Hey Marc! @program-- was able to help me through this conceptually for windows OS... In climateR, Windows systems will now write the rc files to the UserProfile rather than HOME. On Windows HOME (in R) points to the documents directory instead on the root.

If you can do the following you should be good to go 🤞

1. Reinstall climateR, and restart session

remotes::install_github("mikejohnson51/climateR")

3. Rewrite netrc and dodsrc files

writeNetrc(login = ... , password = ...)
writeDodsrc()

4. Make sure they are truly in the root directory so curl can find them

getNetrcPath()
getDodsrcPath()

5. Run the test!

library(climateR)

sites_stack <- getGLDAS(AOI=AOI::aoi_get(state = "OR"),
                        varname='rootmoist_inst',
                        model='NOAH025_M.2.1',
                        startDate='2006-10-01',
                        endDate='2007-09-30',
                        verbose = TRUE)

[mhweber]

Thanks for pursuing Mike! Unfortunately I'm still getting error on my end -

Found Netrc file. Writing dodsrs file to: C:\Users\mwebe/_dodsrc
source:  https://hydro1.gesdisc.eosdis.nasa.gov/dods/GLDAS_NOAH025_M.... 
varname(s):
   > rootmoist_inst [NA] (** root zone soil moisture [kg m-2] )
==================================================
diminsions:  34, 19, 13 (names: lon,lat,time)
resolution:  0.25, 0.25, 1 months
extent:      -124.75, -116.25, 41.75, 46.5 (xmin, xmax, ymin, ymax)
crs:         +proj=longlat +a=6378137 +f=0.00335281066474748 +p...
time:        2006-10-01 to 2007-10-01
==================================================
values: 8,398 (vars*X*Y*T)syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: HTTP^ Basic: Access denied.

But perhaps something I still did not get updated correctly - followed steps above, here's my sessionInfo():

R version 4.3.1 (2023-06-16 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19045)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.utf8 
[2] LC_CTYPE=English_United States.utf8   
[3] LC_MONETARY=English_United States.utf8
[4] LC_NUMERIC=C                          
[5] LC_TIME=English_United States.utf8    

time zone: America/Los_Angeles
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets 
[6] methods   base     

other attached packages:
[1] reprex_2.0.2   climateR_0.3.1

loaded via a namespace (and not attached):
 [1] DBI_1.1.3                 remotes_2.4.2            
 [3] rlang_1.1.1               magrittr_2.0.3           
 [5] e1071_1.7-13              compiler_4.3.1           
 [7] callr_3.7.3               vctrs_0.6.3              
 [9] rvest_1.0.3               fastmap_1.1.1            
[11] pkgconfig_2.0.3           ncmeta_0.3.5             
[13] lwgeom_0.2-13             utf8_1.2.3               
[15] rmarkdown_2.22            ps_1.7.5                 
[17] bit_4.0.5                 xfun_0.39                
[19] jsonlite_1.8.5            collapse_1.9.6           
[21] terra_1.7-39              parallel_4.3.1           
[23] R6_2.5.1                  fipio_1.1.1              
[25] terrainr_0.7.4            parallelly_1.36.0        
[27] Rcpp_1.0.10               assertthat_0.2.1         
[29] knitr_1.43                future.apply_1.11.0      
[31] unifir_0.2.3              base64enc_0.1-3          
[33] dataRetrieval_2.7.12      tidyselect_1.2.0         
[35] yaml_2.3.7                RNetCDF_2.6-2            
[37] rnaturalearth_0.3.2       rstudioapi_0.14          
[39] codetools_0.2-19          processx_3.8.1           
[41] curl_5.0.1                listenv_0.9.0            
[43] lattice_0.21-8            tibble_3.2.1             
[45] withr_2.5.0               evaluate_0.20            
[47] future_1.33.0             sf_1.0-13                
[49] units_0.8-2               proxy_0.4-27             
[51] xml2_1.3.4                exactextractr_0.9.1      
[53] pillar_1.9.0              KernSmooth_2.23-21       
[55] generics_0.1.3            sp_1.6-1                 
[57] munsell_0.5.0             scales_1.2.1             
[59] globals_0.16.2            class_7.3-22             
[61] glue_1.6.2                clipr_0.8.0              
[63] tools_4.3.1               data.table_1.14.8        
[65] fs_1.6.2                  grid_4.3.1               
[67] colorspace_2.1-0          raster_3.6-20            
[69] cli_3.6.1                 fansi_1.0.4              
[71] dplyr_1.1.2               gtable_0.3.3             
[73] digest_0.6.32             classInt_0.4-9           
[75] AOI_0.2.1                 htmltools_0.5.5          
[77] lifecycle_1.0.3           httr_1.4.6               
[79] StreamCatTools_0.1.1.9002 bit64_4.0.5

[mikejohnson51]

Thanks @mhweber! This looks weird to me: C:\Users\mwebe/_dodsrc not sure how we got different backslashes. I think its because file.path() in R forces the / entry. Is \ the typical windows pattern?

[mhweber]

Hey @mikejohnson51 - yeah, windows is typically \ but I don't think that file path should be problematic (but could be wrong) - I get this:

file.exists("C:\\Users\\mwebe/_netrc")
[1] TRUE

I did notice when I opened it that _netrc had three login and password entries, perhaps because I had written with overwrite=TRUE? And one of the three had an error in the password. I fixed it, restarted R session, but got the same access denied error. I did check and my login and password are definitely good. Thanks for pursuing but don't burn too much time if not finding any solutions! Thanks!

[mikejohnson51]

Hey @mhweber ,

Could you please check that the _dodrc file only has one entry as well? I will check back with some windows folks to see if they have any ideas. This is something worth burning time one :) I dont want to alienate a whole group of users based on OS!

[mhweber]

Sure enough, it had multiple entries - but getting rid of all but one still didn't seem to solve the issue. I got rid of extra entries and edited to have all backslashes for windows in filepaths (just in case) so that it looked like:

IGNORE_EXPIRES=0
CACHE_ROOT=C:\Users\mwebe\.dods_cache/
DEFAULT_EXPIRES=86400
ALWAYS_VALIDATE=0
DEFLATE=0
VALIDATE_SSL=1
HTTP.COOKIEJAR=C:\Users\mwebe\.cookies
HTTP.NETRC=C:\Users\mwebe\_netrc

But every time I run test code:

library(climateR)
sites_stack <- getGLDAS(AOI=AOI::aoi_get(state = "OR"),
                        varname='rootmoist_inst',
                        model='NOAH025_M.2.1',
                        startDate='2006-10-01',
                        endDate='2007-09-30',
                        verbose = TRUE)

I get the same error and it writes another entry into the _dodrc file so that the file looks like:

IGNORE_EXPIRES=0
CACHE_ROOT=C:\Users\mwebe\.dods_cache/
DEFAULT_EXPIRES=86400
ALWAYS_VALIDATE=0
DEFLATE=0
VALIDATE_SSL=1
HTTP.COOKIEJAR=C:\Users\mwebe\.cookies
HTTP.NETRC=C:\Users\mwebe\_netrc

USE_CACHE=0
MAX_CACHE_SIZE=20
MAX_CACHED_OBJ=5
IGNORE_EXPIRES=0
CACHE_ROOT=C:/Users/mwebe/.dods_cache/
DEFAULT_EXPIRES=86400
ALWAYS_VALIDATE=0
DEFLATE=0
VALIDATE_SSL=1
HTTP.COOKIEJAR=C:/Users/mwebe/.cookies
HTTP.NETRC=C:\Users\mwebe/_netrc

Maybe some clues in what is going on there? Thanks-

[mikejohnson51]

@mhweber, @program-- and I worked through this today (learned a lot) and have a solution that produced results on his windows machine. I believe we have it!

As long as you have used writeNetrc() to create a Netrc file you should be able to run MODIS, GLDAS and NLDAS.

Thanks again for nudging us on this

[mhweber]

Hey @mikejohnson51 - I installed climateR again and getNetrcPath() returns file path with double backslashes - but when I run

sites_stack <- getGLDAS(AOI=AOI::aoi_get(state = "OR"),
                        varname='rootmoist_inst',
                        model='NOAH025_M.2.1',
                        startDate='2006-10-01',
                        endDate='2007-09-30',
                        verbose = TRUE)

I get a slightly different error now:

source:  https://hydro1.gesdisc.eosdis.nasa.gov/dods/GLDAS_NOAH025_M.... 
varname(s):
   > rootmoist_inst [NA] (** root zone soil moisture [kg m-2] )
==================================================
diminsions:  34, 19, 13 (names: lon,lat,time)
resolution:  0.25, 0.25, 1 months
extent:      -124.75, -116.25, 41.75, 46.5 (xmin, xmax, ymin, ymax)
crs:         +proj=longlat +a=6378137 +f=0.00335281066474748 +p...
time:        2006-10-01 to 2007-10-01
==================================================
values: 8,398 (vars*X*Y*T)syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: <!DOCTYPE^ HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"><html><head><title>400 Bad Request</title></head><body><h1>Bad Request</h1><p>Your browser sent a request that this server could not understand.<br /></p></body></html>

[mikejohnson51]

Sweet! I am sure it doesn't feel like it to you, but that is a good step forward... so that is pointing to something local with your machine...

When you can, could you run the following and share the results of each?: Thanks!

unlink(getNetrc())

writeNetrc(login    = '...',
           password = '...',
           overwrite = TRUE)

(writeDodsrc())
(getNetrcPath())

file.exists(".dodsrc")

x = climater_filter(AOI = AOI::aoi_get(state = "OR"),
                varname='rootmoist_inst',
                model='NOAH025_M.2.1',
                startDate='2006-10-01',
                endDate='2007-09-30')

x

x2 <- dap(catalog = x,
          AOI=AOI::aoi_get(state = "OR"),
          startDate='2006-10-01')

plot(x2$rootmoist_inst)

[mhweber]

Here's what I get:

unlink(climateR::getNetrcPath())

writeNetrc(login    = '...',  #filled in with my info
           password = '...',
           overwrite = TRUE)

[1] "C:\\Users\\mwebe\\.netrc"

file.exists(".dodsrc")

[1] TRUE

x = climater_filter(AOI = AOI::aoi_get(state = "OR"),
                    varname='rootmoist_inst',
                    model='NOAH025_M.2.1',
                    startDate='2006-10-01',
                    endDate='2007-09-30')

x

# A tibble: 1 × 30
  id    asset      URL   type  varname variable description units model ensemble
  <chr> <chr>      <glu> <chr> <chr>   <chr>    <chr>       <chr> <chr> <chr>   
1 GLDAS GLDAS_NOA… http… open… rootmo… rootmoi… "** root z… NA    NOAH… NA      
# ℹ 20 more variables: scenario <chr>, T_name <chr>, duration <chr>,
#   interval <chr>, nT <dbl>, X_name <chr>, Y_name <chr>, X1 <dbl>, Xn <dbl>,
#   Y1 <dbl>, Yn <dbl>, resX <dbl>, resY <dbl>, ncols <dbl>, nrows <dbl>,
#   crs <chr>, toptobottom <lgl>, tiled <chr>, s <chr>, e <chr>

x2 <- dap(catalog = x,
          AOI=AOI::aoi_get(state = "OR"),
          startDate='2006-10-01')

source:  https://hydro1.gesdisc.eosdis.nasa.gov/dods/GLDAS_NOAH025_M.... 
varname(s):
   > rootmoist_inst [NA] (** root zone soil moisture [kg m-2] )
==================================================
diminsions:  34, 19, 1 (names: lon,lat,time)
resolution:  0.25, 0.25, 1 months
extent:      -124.75, -116.25, 41.75, 46.5 (xmin, xmax, ymin, ymax)
crs:         +proj=longlat +a=6378137 +f=0.00335281066474748 +p...
time:        2006-10-01 to 2006-10-01
==================================================
values: 646 (vars*X*Y*T)syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: <!DOCTYPE^ HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"><html><head><title>400 Bad Request</title></head><body><h1>Bad Request</h1><p>Your browser sent a request that this server could not understand.<br /></p></body></html>

[mikejohnson51]

Hi @mhweber, One more try :) We have three success window uses at this point! Can you please try the following?

1. Login to EarthData
2. Go to Applications > Authorized Apps
3. If NASA GESDISC DATA ARCHIVE is not in the Approved Applications list, select APPROVE MORE APPLICATIONS
4. Find NASA GESDISC DATA ARCHIVE and click AUTHORIZE
5. Retry the R code in a restarted session

[mhweber]

@mikejohnson51 - Yikes! was it that simple all along? Success - looks like can close this one! Thanks for chasing this down

[mikejohnson51]

It was that, on top of all these other things :) they gave a wide range off possible places to mess up. Glad it's working for you!