search

r-spatial / rgee

Google Earth Engine for R
https://r-spatial.github.io/rgee/
Other
677 stars 146 forks source link

ee_extract() becomes extremely slow in a loop after several runs #185

Closed gaozitianhaha closed 3 years ago

gaozitianhaha commented 3 years ago

At submit an issue, please attached the following information of your rgee session:

ee_path <- path.expand("~/.config/earthengine/credentials") file.exists(ee_path) [1] TRUE

- [ ] You can run a simple EE command from R: 

```r
library(rgee)

# Initialize the Earth Engine module.
ee_Initialize()

# Print metadata for a DEM dataset.
print(ee$Image('USGS/SRTMGL1_003')$getInfo())

run with no problem

Attach your Python (reticulate) configuration:

library(reticulate)
py_config()

> py_config()
python:         C:/Users/zitiang/AppData/Local/r-miniconda/envs/rgee/python.exe
libpython:      C:/Users/zitiang/AppData/Local/r-miniconda/envs/rgee/python39.dll
pythonhome:     C:/Users/zitiang/AppData/Local/r-miniconda/envs/rgee
version:        3.9.7 | packaged by conda-forge | (default, Sep 14 2021, 01:11:01) [MSC v.1916 64 bit (AMD64)]
Architecture:   64bit
numpy:          C:/Users/zitiang/AppData/Local/r-miniconda/envs/rgee/Lib/site-packages/numpy
numpy_version:  1.21.2
ee:             C:\Users\zitiang\AppData\Local\r-miniconda\envs\rgee\lib\site-packages\ee\__init__.p

NOTE: Python version was forced by RETICULATE_PYTHON

Description

Dear rgee team, I have a script to extract NDVI time series from Landsat 8 based on individual polygons. This script is run without problem (and quite sufficient) in the past few months. However, I realize 3 days ago that the script starts to have a problem in the for loop - it becomes very slow ain ee_extract() after a couple of runs and the R studio system just stuck there. Then I have to restart R but the problem is not solved.

I attach my codes here: library(rgee) library(raster) library(sf)

ee_Initialize(user="myemail",drive=T)

L8=ee$ImageCollection('LANDSAT/LC08/C01/T1_SR');

this is a cloud mask maskL8sr <- function(image) { cloudShadowBitMask <- bitwShiftL(1, 3) cloudsBitMask <- bitwShiftL(1, 5) qa <- image$select('pixel_qa') mask <- qa$bitwiseAnd(cloudShadowBitMask)$eq(0)$ And(qa$bitwiseAnd(cloudsBitMask)$eq(0))

image$updateMask(mask)$ divide(10000)$ select("B[0-9]*")$ copyProperties(image, list("system:time_start")) }

this is a function to calculate satellite index L8_metrics <-function(image){ evi = image$expression( expression = '2.5 ((NIR - RED) / (NIR + 6 RED - 7.5 * BLUE + 1))', opt_map = list( 'NIR' = image$select('B5'), 'RED' = image$select('B4'), 'BLUE' = image$select('B2') ) ) gcvi =image$expression( expression = 'NIR/GREEN-1', opt_map = list( 'NIR' = image$select('B5'), 'GREEN' = image$select('B3') ) )

SAVI = ((Band 5 - Band 4) / (Band 5 + Band 4 + 0.5)) * (1.5)

SAVI = image$expression( expression = '((NIR-RED)/(NIR+RED+0.5))*(1.5)', opt_map = list( 'NIR' = image$select('B5'), 'RED' = image$select('B4') ) ) return( image$ addBands(image$normalizedDifference(c("B5","B4"))$ rename("NDVI"))$ addBands(image$normalizedDifference(c("B5","B6"))$ rename("NDWI"))$ addBands(image$addBands(evi$rename('EVI')))$ addBands(image$addBands(SAVI$rename('SAVI')))$ addBands(image$addBands(gcvi$rename('GCVI'))) ) }

this is my study area myarea<- ee$Geometry$Polygon(list(c(145.68996067298272, -34.60456363095119), c(145.6885873819671, -35.07457644753154), c(146.44939060462335, -35.072328600706506), c(146.44527073157647, -34.58478026104608), c(145.68996067298272, -34.60456363095119)))

_this is a satellite image that is under CRS arguments: +proj=longlat +datum=WGS84 +nodefs system NDVI_demo_iamge = raster(".../image.tif")

nc is my shapefile polygon, it contains 4000+ polygons so I run each polygon in a loop nc <- st_read( dsn = "C:/Users/zitiang/OneDrive - The University of Melbourne/dataset/AWAP/data/field boundary/field_boundary_coly.shp", stringsAsFactors = FALSE, quiet = TRUE)

I test the first 50 polygons, the loop stuck in around 10... temp_lst=list() for(i in 1:50){ roi = st_transform(nc[i,],crs(NDVI_image)) NDVI_img = L8$filterBounds(myarea)$map(maskL8sr)$ filterDate('2015-08-01','2016-06-30')$map(L8_metrics)$select("NDVI") ee_ndvi <- ee_extract(NDVI_img, roi,scale = 30, fun = ee$Reducer$mean(), sf = F) temp_lst[[i]] = ee_ndvi print(i) }

What I Did

All of those efforts still not solve this problem. In fact I don't quite understand why because previously I run the code with no problem, and I didn't change the codes. Is there any changes in rgee so that I have to change ee_extract() arguments correspondingly?

Any ideas would be very appreciated!

Best regards, Zitian

gaozitianhaha commented 3 years ago

I also notice that if I don't stop the loop, I will receive the errors:

Error in py_call_impl(callable, dots$args, dots$keywords) : EEException: Computation timed out. In addition: Warning messages: 1: In ee_utils_py_to_r(.) : restarting interrupted promise evaluation 2: In ee_utils_py_to_r(.) : restarting interrupted promise evaluation 3: In ee_utils_py_to_r(.) : restarting interrupted promise evaluation 4: In ee_utils_py_to_r(.) : Error in py_call_impl(callable, dots$args, dots$keywords) : EEException: Computation timed out.

Cheers, Zitian

csaybar commented 3 years ago

Hi @gaozitianhaha I think it is an error from the server-side. I can help u to optimize your code but please give me access to your shapefile.

gaozitianhaha commented 3 years ago

Hi Csaybar, Thank you so much for your kind helps and prompt reply. I figure out that if I upload results to google drive it seems working well. I agree this might be the problem of server not the package. I can accept the results to be saved in drive for now (and I change my codes accordingly). If I still find a problem in the future I will update here.

Thanks again for your help!

Best regards, Zitian

csaybar commented 3 years ago

great! :D thanks for letting us know!