search

r-spatial / rgee

Google Earth Engine for R
https://r-spatial.github.io/rgee/
Other
670 stars 146 forks source link

Unexpected behaviour during images download in a loop #181

Closed Leprechault closed 2 years ago

Leprechault commented 3 years ago

Matrix products: default BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1 LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1

locale: [1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C LC_TIME=C.UTF-8 LC_COLLATE=C.UTF-8 LC_MONETARY=C.UTF-8
[6] LC_MESSAGES=C.UTF-8 LC_PAPER=C.UTF-8 LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C

attached base packages: [1] parallel stats graphics grDevices utils datasets methods base

other attached packages: [1] kernlab_0.9-29 fpc_2.2-9 magrittr_2.0.1 ppclust_1.1.0 fclust_2.1.1
[6] sf_1.0-2 ggspatial_1.1.5 rgee_1.1.0 lubridate_1.7.10 pls_2.7-3
[11] ggpubr_0.4.0 data.table_1.14.0 rcompanion_2.4.1 multcompView_0.1-8 keras_2.6.0
[16] plyr_1.8.6 sen2r_1.5.0 multcomp_1.4-17 TH.data_1.0-10 MASS_7.3-54
[21] survival_3.2-11 mvtnorm_1.1-2 e1071_1.7-8 rgeos_0.5-5 vegan_2.5-7
[26] permute_0.9-5 doParallel_1.0.16 iterators_1.0.13 foreach_1.5.1 forcats_0.5.1
[31] stringr_1.4.0 purrr_0.3.4 readr_2.0.1 tidyr_1.1.3 tibble_3.1.4
[36] tidyverse_1.3.1 GGally_2.1.2 caret_6.0-88 lattice_0.20-44 mlbench_2.1-3
[41] factoextra_1.0.7 spatstat_2.2-0 spatstat.linnet_2.3-0 spatstat.core_2.3-0 rpart_4.1-15
[46] nlme_3.1-152 spatstat.geom_2.2-2 spatstat.data_2.1-0 fastDummies_1.6.3 mlr_2.19.0
[51] ParamHelpers_1.14 cluster_2.1.2 ggplot2_3.3.5 randomForest_4.6-14 dplyr_1.0.7
[56] raster_3.4-13 maptools_1.1-1 rgdal_1.5-23 sp_1.4-5

loaded via a namespace (and not attached): [1] utf8_1.2.2 reticulate_1.20 tidyselect_1.1.1 htmlwidgets_1.5.3 grid_3.6.3
[6] geojsonio_0.9.4 pROC_1.17.0.1 munsell_0.5.0 jqr_1.2.1 codetools_0.2-18
[11] units_0.7-2 future_1.22.1 withr_2.4.2 colorspace_2.0-2 inaparc_1.1.0
[16] rstudioapi_0.13 stats4_3.6.3 robustbase_0.93-8 tensorflow_2.6.0 DescTools_0.99.42
[21] ggsignif_0.6.2 tensor_1.5 listenv_0.8.0 kpeaks_1.1.0 polyclip_1.10-0
[26] parallelly_1.27.0 vctrs_0.3.8 generics_0.1.0 ipred_0.9-11 diptest_0.76-0
[31] R6_2.5.1 flexmix_2.3-17 lhs_1.1.1 spatstat.utils_2.2-0 reshape_0.8.8
[36] geojson_0.3.4 assertthat_0.2.1 scales_1.1.1 nnet_7.3-16 rootSolve_1.8.2.2
[41] gtable_0.3.0 globals_0.14.0 lwgeom_0.2-7 processx_3.5.2 goftest_1.2-2
[46] lmom_2.8 sandwich_3.0-1 timeDate_3043.102 rlang_0.4.11 zeallot_0.1.0
[51] BBmisc_1.11 splines_3.6.3 rstatix_0.7.0 lazyeval_0.2.2 gargle_1.2.0
[56] ModelMetrics_1.2.2.2 broom_0.7.9 checkmate_2.0.0 reshape2_1.4.4 abind_1.4-5
[61] modelr_0.1.8 crosstalk_1.1.1 backports_1.2.1 tools_3.6.3 lava_1.6.9
[66] ellipsis_0.3.2 RColorBrewer_1.1-2 proxy_0.4-26 Rcpp_1.0.7 base64enc_0.1-3
[71] parallelMap_1.5.1 classInt_0.4-3 ps_1.6.0 openssl_1.4.4 deldir_0.2-10
[76] RcppTOML_0.1.7 zoo_1.8-9 haven_2.4.3 ggrepel_0.9.1 fs_1.5.0
[81] crul_1.1.0 openxlsx_4.2.4 lmtest_0.9-38 reprex_2.0.1 googledrive_2.0.0
[86] whisker_0.4 matrixStats_0.60.1 hms_1.1.0 leaflet_2.0.4.1 XML_3.99-0.3
[91] rio_0.5.27 mclust_5.4.7 readxl_1.3.1 tfruns_1.5.0 compiler_3.6.3
[96] KernSmooth_2.23-20 V8_3.4.2 crayon_1.4.1 geojsonsf_2.0.1 htmltools_0.5.2
[101] mgcv_1.8-36 tzdb_0.1.2 libcoin_1.0-8 expm_0.999-6 Exact_2.1
[106] DBI_1.1.1 dbplyr_2.1.1 boot_1.3-28 Matrix_1.3-4 car_3.0-11
[111] cli_3.0.1 gower_0.2.2 pkgconfig_2.0.3 coin_1.4-1 foreign_0.8-76
[116] spatstat.sparse_2.0-0 recipes_0.1.16 xml2_1.3.2 prodlim_2019.11.13 rvest_1.0.1
[121] digest_0.6.27 httpcode_0.3.0 cellranger_1.1.0 fastmatch_1.1-3 nortest_1.0-4
[126] gld_2.6.2 curl_4.3.2 modeltools_0.2-23 lifecycle_1.0.0 jsonlite_1.7.2
[131] carData_3.0-4 askpass_1.1 fansi_0.5.0 pillar_1.6.2 DEoptimR_1.0-9
[136] fastmap_1.1.0 httr_1.4.2 glue_1.4.2 zip_2.2.0 prabclus_2.3-2
[141] png_0.1-7 class_7.3-19 stringi_1.7.4 stars_0.5-3 


#### At submit an issue, please attached the following information of your `rgee` session:

- [ X ] You can find the credentials file on your system:

ee_path <- path.expand("~/.config/earthengine/credentials") file.exists(ee_path) [1] TRUE

- [ X ] You can run a simple EE command from R: 

```r
# Initialize the Earth Engine module.
ee_Initialize(drive=TRUE)
── rgee 1.1.0 ─────────────────────────────────────────────────────────────────────────────────────── earthengine-api 0.1.280 ── 
 ✓ user: not_defined
 ✓ Google Drive credentials:  FOUND
 ✓ Initializing Google Earth Engine:  DONE!
 ✓ Earth Engine account: users/combateSF 
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Attach your Python (reticulate) configuration:

py_config()
python:         /home/combate/.virtualenvs/rgee/bin/python
libpython:      /home/combate/.local/share/r-miniconda/envs/r-reticulate/lib/libpython3.6m.so
pythonhome:     /home/combate/.virtualenvs/rgee:/home/combate/.virtualenvs/rgee
version:        3.6.13 | packaged by conda-forge | (default, Feb 19 2021, 05:36:01)  [GCC 9.3.0]
numpy:          /home/combate/.virtualenvs/rgee/lib/python3.6/site-packages/numpy
numpy_version:  1.19.5
ee:             /home/combate/.virtualenvs/rgee/lib/python3.6/site-packages/ee

NOTE: Python version was forced by RETICULATE_PYTHON

Description

Despite the credentials looks like OK, I have an unexpected behaviour during images download. All the S2 images during 2 years in the first ROI in are corrected downloaded, but the next ROI is in a loop just only the first image is downloaded and I have the error:

Reading layer `Mahal_AGROPECUARIAHB_35' from data source `/home/combate/Combate_SF_VM/extent/Mahal_AGROPECUARIAHB_35.shp' using driver `ESRI Shapefile'
Simple feature collection with 1 feature and 1 field
Geometry type: POLYGON
Dimension:     XY
Bounding box:  xmin: -51.46446 ymin: -19.76885 xmax: -51.46021 ymax: -19.76285
CRS:           4326
───────────────────────────────────────────────────────────────────────────────────── Downloading ImageCollection - via drive ──- region parameters
 sfg      : POLYGON ((-51.46446 -19.768 .... 76285, -51.46446 -19.76885)) 
 CRS      : GEOGCS["WGS 84",
    DATUM["World Geodetic System 1984",
        SPHEROID["WGS 84",6378137.0,298.257223563, ..... 
 geodesic : TRUE 
 evenOdd  : TRUE 

Auto-refreshing stale OAuth token._SF_VM/cal_imagery/Mahal_AGROPECUARIAHB_35_20190123_T22KDD.tif
File downloaded:
• Mahal_AGROPECUARIAHB_35_20190123_T22KDD_2021_08_27_00_40_49.tif <id: 1ybNEKkrFgibyN2glXVBCgir1qLZC4YGP>

...

Error: Don't know how to coerce an object of class <try-error> into a <dribble>.6_T22KCD.tif
Run `rlang::last_error()` to see where the error occurred.

I find a similar error a cause of not check the 3 boxes in the Google Drive permissions to tidyverse API (https://github.com/r-spatial/rgee/issues/175), but is not my case. I did it.

What I Did

My code used:

# Initialize the Earth Engine session --------------------------------------------
ee_Initialize(drive = TRUE)

# Function for remove cloud and shadows ------------------------------------------
getQABits <- function(image, qa) {
  # Convert decimal (character) to decimal (little endian)
  qa <- sum(2^(which(rev(unlist(strsplit(as.character(qa), "")) == 1))-1))
  # Return a single band image of the extracted QA bits, giving the qa value.
  image$bitwiseAnd(qa)$lt(1)
}
s2_clean <- function(img) {
  # Select only band of interest, for instance, B2,B3,B4,B8
  img_band_selected <- img$select("B[2-4|8]")

  # quality band
  ndvi_qa <- img$select("QA60")

  # Select pixels to mask
  quality_mask <- getQABits(ndvi_qa, "110000000000")

  # Mask pixels with value zero.
  img_band_selected$updateMask(quality_mask)
}

# Extent of the stands 
setwd("~/Combate_SF_VM/extent")
dir2 <- "~/Combate_SF_VM/extent"
all.shapes.target <- list.files(dir2, pattern="\\.shp$", full.names=TRUE)
all.shapes.target <- gsub('/home/combate/Combate_SF_VM/extent/','', all.shapes.target) 

# Download S2 BOA imagery from GEE
for (l in 1:length(all.shapes.target)){
setwd("~/Combate_SF_VM/extent")

# Selection stand by stand
roi <- st_read(all.shapes.target[l]) %>% 
  st_bbox() %>% 
  st_as_sfc() %>% 
  sf_as_ee()

# Sentinel-2 MSI dataset into the Earth Engine’s public data archive ------------              
s2 <- ee$ImageCollection("COPERNICUS/S2_SR")

# Select S2 images ---------------------------------------------------------------
s2_roi  <- s2$
  filterBounds(roi)$
  filter(ee$Filter$lte("CLOUDY_PIXEL_PERCENTAGE", 1))$
  filter(ee$Filter$date('2019-01-01', '2021-08-01'))$
  map(s2_clean)

s2_roi_add_area <- s2_roi$map(
  function(img) {
    img$set("area", img$clip(roi)$geometry()$area())
  }
)

# Get the dates and IDs of the selected images ------------------------------------
area <- floor(ee_utils_py_to_r(roi$area(maxError=1)$getInfo()))
ic_date_gt_area <- s2_roi_add_area$filterMetadata("area", "greater_than", area)

nimages <- ic_date_gt_area$size()$getInfo()
nimages 

ic_date <- ee_get_date_ic(s2_roi_add_area)
ic_date

# Function for the last 6 characters
substrRight <- function(x, n){
  substr(x, nchar(x)-n+1, nchar(x))
}

# Download the results
s2_ic_local <- ee_imagecollection_to_local(
  ic = s2_roi_add_area,
  scale = 10,
  region = roi,
  via = "drive",
  add_metadata = FALSE,
  dsn = paste0("/home/combate/Combate_SF_VM/cal_imagery/",gsub(".shp","",all.shapes.target[l]),"_",stringr::str_extract(gsub("COPERNICUS/S2_SR/","",ic_date$id),
"20\\d{2}\\d{2}\\d{2}"),"_",substrRight(ic_date$id,6))
)
}
#
csaybar commented 3 years ago

Hi it seems this problem is similar to https://github.com/r-spatial/rgee/issues/175#issuecomment-905611278 Please can you upgrade rgee to v.1.1.1 and tell us if the problem is fixed? Thanks!

remotes::install_github("r-spatial/rgee")
Leprechault commented 2 years ago

@csaybar problem not solved yet with rgee_1.1.2. In my case:

File downloaded:me/combate/Combate_SF_VM/cal_imagery/Novo_Oeste_LOBO_238_20200813_T22KCD.tif
• Novo_Oeste_LOBO_238_20200813_T22KCD_2021_09_20_14_26_07.tif <id: 1Dn0BSFz2KIBCT0eUoFsaHkwwBRniPXOB>
Saved locally as:
• /home/combate/Combate_SF_VM/cal_imagery/Novo_Oeste_LOBO_238_20200813_T22KCD.tif
Auto-refreshing stale OAuth token._SF_VM/cal_imagery/Mahal_BARRADOBOAVISTA_229_20190824_T22KDC.tif
Error: Don't know how to coerce an object of class <try-error> into a <dribble>.
Run `rlang::last_error()` to see where the error occurred.
 #
 rlang::last_error()
<error/rlang_error>
Don't know how to coerce an object of class <try-error> into a <dribble>.
Backtrace:
  1. rgee::ee_imagecollection_to_local(...)
  2. rgee::ee_as_stars(...)
  3. rgee:::to_evaluate()
  4. rgee:::ee_image_local(...)
  5. rgee:::ee_image_local_drive(task, dsn, metadata, public, quiet)
  6. rgee::ee_drive_to_local(...)
 10. googledrive::drive_share_anyone(file = files_gd)
 11. googledrive::drive_share(file = file, role = "reader", type = "anyone")
 13. googledrive:::as_dribble.default(file)
 14. googledrive:::drive_abort("\n    Don't know how to coerce an object of class {.cls {class(x)}} into \\\n    a {.cls dribble}.")
 15. cli::cli_abort(message = message, ..., .envir = .envir)
Run `rlang::last_trace()` to see the full context.

I try to force OAuth token using a Google storage API together in the code:

#Google storage API 
path<-paste0("/home/combate","XXXX.json")
googledrive::drive_auth(path =path)
1
#

But doesn't work. My session info:

R version 3.6.3 (2020-02-29)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.5 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8        LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8   
 [6] LC_MESSAGES=C.UTF-8    LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C           LC_TELEPHONE=C        
[11] LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

attached base packages:
[1] parallel  stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] remotes_2.4.0             kernlab_0.9-29            fpc_2.2-9                 magrittr_2.0.1           
 [5] ppclust_1.1.0             fclust_2.1.1              sf_1.0-2                  ggspatial_1.1.5          
 [9] rgee_1.1.2                lubridate_1.7.10          pls_2.8-0                 ggpubr_0.4.0             
[13] data.table_1.14.0         rcompanion_2.4.1          multcompView_0.1-8        keras_2.6.0              
[17] plyr_1.8.6                sen2r_1.5.0               multcomp_1.4-17           TH.data_1.0-10           
[21] MASS_7.3-54               survival_3.2-11           mvtnorm_1.1-2             e1071_1.7-9              
[25] rgeos_0.5-7               vegan_2.5-7               permute_0.9-5             doParallel_1.0.16        
[29] iterators_1.0.13          foreach_1.5.1             forcats_0.5.1             stringr_1.4.0            
[33] purrr_0.3.4               readr_2.0.1               tidyr_1.1.3               tibble_3.1.4             
[37] tidyverse_1.3.1           GGally_2.1.2              caret_6.0-88              lattice_0.20-44          
[41] mlbench_2.1-3             factoextra_1.0.7          spatstat_2.2-0            spatstat.linnet_2.3-0    
[45] spatstat.core_2.3-0       rpart_4.1-15              nlme_3.1-152              spatstat.geom_2.2-2      
[49] spatstat.data_2.1-0       fastDummies_1.6.3         mlr_2.19.0                ParamHelpers_1.14        
[53] cluster_2.1.2             ggplot2_3.3.5             randomForest_4.6-14       dplyr_1.0.7              
[57] raster_3.4-13             maptools_1.1-2            rgdal_1.5-27              sp_1.4-5                 
[61] googleCloudStorageR_0.6.0 googledrive_2.0.0        

loaded via a namespace (and not attached):
  [1] rappdirs_0.3.3        prabclus_2.3-2        ModelMetrics_1.2.2.2  jqr_1.2.1             generics_0.1.0       
  [6] leaflet_2.0.4.1       proxy_0.4-26          future_1.22.1         tensorflow_2.6.0      tzdb_0.1.2           
 [11] xml2_1.3.2            assertthat_0.2.1      gargle_1.2.0          gower_0.2.2           hms_1.1.0            
 [16] DEoptimR_1.0-9        fansi_0.5.0           dbplyr_2.1.1          readxl_1.3.1          DBI_1.1.1            
 [21] htmlwidgets_1.5.4     reshape_0.8.8         stats4_3.6.3          ellipsis_0.3.2        crosstalk_1.1.1      
 [26] backports_1.2.1       V8_3.4.2              libcoin_1.0-8         deldir_0.2-10         vctrs_0.3.8          
 [31] abind_1.4-5           cachem_1.0.6          withr_2.4.2           geojsonsf_2.0.1       robustbase_0.93-8    
 [36] checkmate_2.0.0       mclust_5.4.7          goftest_1.2-2         lazyeval_0.2.2        crayon_1.4.1         
 [41] crul_1.1.0            recipes_0.1.16        pkgconfig_2.0.3       units_0.7-2           nnet_7.3-16          
 [46] rlang_0.4.11          globals_0.14.0        diptest_0.76-0        lifecycle_1.0.0       sandwich_3.0-1       
 [51] httpcode_0.3.0        modelr_0.1.8          cellranger_1.1.0      polyclip_1.10-0       matrixStats_0.61.0   
 [56] lmtest_0.9-38         Matrix_1.3-4          carData_3.0-4         boot_1.3-28           zoo_1.8-9            
 [61] geojsonio_0.9.4       reprex_2.0.1          RcppTOML_0.1.7        base64enc_0.1-3       whisker_0.4          
 [66] processx_3.5.2        png_0.1-7             rootSolve_1.8.2.2     KernSmooth_2.23-20    pROC_1.18.0          
 [71] geojson_0.3.4         classInt_0.4-3        coin_1.4-1            parallelly_1.28.1     rstatix_0.7.0        
 [76] ggsignif_0.6.3        scales_1.1.1          memoise_2.0.0         inaparc_1.1.0         compiler_3.6.3       
 [81] RColorBrewer_1.1-2    cli_3.0.1             listenv_0.8.0         ps_1.6.0              mgcv_1.8-36          
 [86] tidyselect_1.1.1      stringi_1.7.4         kpeaks_1.1.0          yaml_2.2.1            askpass_1.1          
 [91] ggrepel_0.9.1         grid_3.6.3            fastmatch_1.1-3       tools_3.6.3           lmom_2.8             
 [96] future.apply_1.8.1    rio_0.5.27            rstudioapi_0.13       foreign_0.8-76        gld_2.6.2            
[101] prodlim_2019.11.13    stars_0.5-3           digest_0.6.27         lava_1.6.10           nortest_1.0-4        
[106] Rcpp_1.0.7            car_3.0-11            broom_0.7.9           lwgeom_0.2-7          httr_1.4.2           
[111] colorspace_2.0-2      rvest_1.0.1           XML_3.99-0.3          fs_1.5.0              tensor_1.5           
[116] reticulate_1.22       splines_3.6.3         expm_0.999-6          spatstat.utils_2.2-0  Exact_3.0            
[121] flexmix_2.3-17        jsonlite_1.7.2        BBmisc_1.11           timeDate_3043.102     zeallot_0.1.0        
[126] modeltools_0.2-23     ipred_0.9-12          R6_2.5.1              lhs_1.1.3             pillar_1.6.2         
[131] htmltools_0.5.2       glue_1.4.2            fastmap_1.1.0         class_7.3-19          codetools_0.2-18     
[136] utf8_1.2.2            spatstat.sparse_2.0-0 curl_4.3.2            tfruns_1.5.0          DescTools_0.99.43    
[141] zip_2.2.0             openxlsx_4.2.4        openssl_1.4.5         parallelMap_1.5.1     googleAuthR_1.4.0    
[146] munsell_0.5.0         haven_2.4.3           reshape2_1.4.4        gtable_0.3.0
csaybar commented 2 years ago

Hi @Leprechault can u share with me a fully reproducible example.

image

I don't have access to your shapefiles

Leprechault commented 2 years ago

@csaybar I find the problem. Is the Google server that's breaking the http2 connection.

I make before my code:

install.packages("https://github.com/jeroen/curl/archive/master.tar.gz", repos = NULL)
httr::set_config(httr::config(http_version = 0))

and problem solved!