search

r-spatial / rgee

Google Earth Engine for R
https://r-spatial.github.io/rgee/
Other
677 stars 146 forks source link

Error sf_as_ee GCS: ERROR in Earth Engine servers: Could not unzip file #154

Closed lalgonzales closed 2 years ago

lalgonzales commented 3 years ago

Description

I am trying to run sf_to_ee, and I get the following results:

The code was:

zona_sf <- st_read("myShape.shp") 
zona_ee <- sf_as_ee(zona_sf, via = "gcs_to_asset", 
                    assetId = "users/myUser/assetId",
                    overwrite = T,
                    bucket = "myBucket")

The code results was:

  1. Converting sf object to a compressed ZIP shapefile ... saving in /tmp
  2. From local to GCS
  3. Creating the manifest
  4. From GCS to Earth Engine Running the OS command: earthengine upload table --manifest C:/Users/user/AppData/Local/Temp/RtmpKcUkmI/file23c8534160a4_manifest.json Started upload task with ID: DT6P3MMAHG2S5RDKRBAU76FW

Polling for task <id: DT6P3MMAHG2S5RDKRBAU76FW, time: 0s>. Polling for task <id: DT6P3MMAHG2S5RDKRBAU76FW, time: 5s>. State: FAILED ERROR in Earth Engine servers: Could not unzip file. Error in ee_monitoring() : ee_monitoring was forced to stop before getting results

Thanks for your answers

csaybar commented 3 years ago

Hi @lalgonzales

It seems to be a bug on the server-side. It was reported in GoogleEarthEngine devs group: https://groups.google.com/g/google-earth-engine-developers/c/ybvaS_pwgTg

lalgonzales commented 3 years ago

Thanks, @csaybar, i will follow the discusion, it seems the bug continue.

csaybar commented 3 years ago

Hi @lalgonzales it seems the bug was solved. Feel free to open the issue again if you consider it necessary.

lalgonzales commented 3 years ago

Hello, @csaybar I tried to run te code now and I still have the same problem

csaybar commented 3 years ago

Hi @lalgonzales can u share with me your shapefile?

Thanks!

lalgonzales commented 3 years ago

Hi @lalgonzales can u share with me your shapefile?

Thanks!

shape.zip

csaybar commented 3 years ago

Hi! I don't have any problem. image

Can you share with us your some details about your R session?

sessionInfo()
lalgonzales commented 3 years ago

Hi! I don't have any problem. image

Can you share with us your some details about your R session?

sessionInfo()

image

csaybar commented 3 years ago

Hi @lalgonzales when u use sf_as_ee(via="gcs_to_asset") several messages appear on the console:

1. Converting sf object to a compressed ZIP shapefile ... saving in /tmp
2. From local to GCS
2021-08-09 01:16:59 -- File size detected as 2.5 Mb
3. Creating the manifest
4. From GCS to Earth Engine
Running the OS command: earthengine upload table --manifest /tmp/Rtmp9x87dI/filea81315dc53b_manifest.json
Started upload task with ID: E55APQFBEGBLG7VDWBUYJ2FB

It seems to me that for some reason, your zipfile is empty. Can you share with us that the zipfile storage in the /tmp folder '/tmp/Rtmp9x87dI/' if it exists? (see point 4). You can also find it in the manisfest.json.

{
  "name": "projects/earthengine-legacy/assets/users/csaybar/toy_poly_gcs2",
  "sources": [{ "uris": ["gs://rgee_dev/filea81355e8233.zip"] }],
  "start_time": { "seconds": 0.0 },
  "end_time": { "seconds": 0.0 }
}
lalgonzales commented 3 years ago

Hi @csaybar, i share a image of the folder, there is no zipfile image

MichaelTD83 commented 3 years ago

Since this is still open I hope it is okay to add to this because I believe I have the same or very similar issue and wanted to try to get some help with it.

TestShps.zip

toy_poly <- read_sf("G:/School/NAU/Forestry/Masters/MSF_Project/Michaels_Analysis/R/TestShps/df_2016-AZCOF-001378.shp")
assetID <- sprintf("%s/fires2/%s", ee_get_assethome(), 'toy_poly_gcs')

eex <- sf_as_ee(
  x = toy_poly, 
  via = "gcs_to_asset", 
  assetId = assetID, 
  monitoring = TRUE, 
  bucket = "sw_fires", 
  overwrite = TRUE
)

Code result

1. Converting sf object to a compressed ZIP shapefile ... saving in /tmp
2. From local to GCS
2021-09-16 14:55:23 -- File size detected as 2.6 Kb
i 2021-09-16 14:55:25 > Request Status Code:  400
2021-09-16 14:55:25 -- File size detected as 2.6 Kb
i 2021-09-16 14:55:25 > Request Status Code:  400
2021-09-16 14:55:25 -- File size detected as 2.6 Kb
i 2021-09-16 14:55:25 > Request Status Code:  400
2021-09-16 14:55:25 -- File size detected as 2.6 Kb
i 2021-09-16 14:55:25 > Request Status Code:  400
2021-09-16 14:55:25 -- File size detected as 2.6 Kb
i 2021-09-16 14:55:25 > Request Status Code:  400
3. Creating the manifest
4. From GCS to Earth Engine
Running the OS command: earthengine upload table --manifest C:/Users/mtd25/AppData/Local/Temp/RtmponVcjt/file6050791f19e1_manifest.json
Started upload task with ID: F7JVRHBF2CLTAUICLACPKGUV

Polling for task <id: F7JVRHBF2CLTAUICLACPKGUV, time: 0s>.
Polling for task <id: F7JVRHBF2CLTAUICLACPKGUV, time: 5s>.
Polling for task <id: F7JVRHBF2CLTAUICLACPKGUV, time: 10s>.
State: FAILED
ERROR in Earth Engine servers: Could not unzip file.
Error in ee_monitoring()  :
  ee_monitoring was forced to stop before getting results

I do have all of these files show up in the Temp folder, including a zip file. I have also checked this .zip file and there is a shapefile in it that matches the one in the TestShp file above.

Tmp

file60507e654d9d.zip

Finally, the session info

R version 4.1.1 (2021-08-10)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 18362)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] sf_1.0-2                  googleCloudStorageR_0.6.0 googledrive_2.0.0         rgee_1.1.1               

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.7         lubridate_1.7.10   lattice_0.20-44    class_7.3-19       png_0.1-7         
 [6] ps_1.6.0           assertthat_0.2.1   digest_0.6.27      utf8_1.2.2         mime_0.11         
[11] R6_2.5.1           e1071_1.7-8        httr_1.4.2         pillar_1.6.2       rlang_0.4.11      
[16] curl_4.3.2         rstudioapi_0.13    raster_3.4-13      googleAuthR_1.4.0  Matrix_1.3-4      
[21] reticulate_1.20    stringr_1.4.0      htmlwidgets_1.5.4  proxy_0.4-26       compiler_4.1.1    
[26] httpuv_1.6.2       janitor_2.1.0      xfun_0.25          base64enc_0.1-3    pkgconfig_2.0.3   
[31] askpass_1.1        htmltools_0.5.2    openssl_1.4.5      tidyselect_1.1.1   tibble_3.1.4      
[36] codetools_0.2-18   fansi_0.5.0        crayon_1.4.1       dplyr_1.0.7        withr_2.4.2       
[41] later_1.3.0        grid_4.1.1         jsonlite_1.7.2     lifecycle_1.0.0    DBI_1.1.1         
[46] magrittr_2.0.1     units_0.7-2        KernSmooth_2.23-20 zip_2.2.0          cli_3.0.1         
[51] stringi_1.7.4      cachem_1.0.6       fs_1.5.0           promises_1.2.0.1   remotes_2.4.0     
[56] sp_1.4-5           leaflet_2.0.4.1    snakecase_0.11.0   ellipsis_0.3.2     generics_0.1.0    
[61] vctrs_0.3.8        tools_4.1.1        leafem_0.1.6       glue_1.4.2         purrr_0.3.4       
[66] crosstalk_1.1.1    processx_3.5.2     fastmap_1.1.0      yaml_2.2.1         gargle_1.2.0      
[71] classInt_0.4-3     memoise_2.0.0      knitr_1.33