ee_Initialize() tries to authenticate even if already done?

MatthieuStigler commented 2 years ago

rgee version: 1.1.5
R version: 4.2.0
Operating System: Ubuntu 22.04 server

At submit an issue, please attached the following information of your `rgee` session:

[x] You have the Python API installed (from terminal):
[x] You can find the credentials file on your system:
[ ] You can run a simple EE command from R: NO

Attach your Python (reticulate) configuration:

library(reticulate)
py_config()
python:         /home/user/anaconda3/envs/ee/bin/python
libpython:      /home/user/anaconda3/envs/ee/lib/libpython3.8.so
pythonhome:     /home/user/anaconda3/envs/ee:/home/user/anaconda3/envs/ee
version:        3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:10)  [GCC 10.3.0]
numpy:           [NOT FOUND]
ee:             /home/user/anaconda3/envs/ee/lib/python3.8/site-packages/ee

Description

When running ee_Initialize() rgee tries to authenticate the session even if my session is already authenticate. In theory this should not be a big issue as I could go to the authentication steps again, but as I am running on a server, this makes it much more complicated.

How is rgee detecting whether the sessions is already authenticated or not?

ee_install_set_pyenv(py_path = "~/anaconda3/envs/ee/bin/python", py_env = "ee")
system("~/anaconda3/envs/ee/bin/earthengine ls") ## works
ee_Initialize(user = 'user@gmail.com', gcs = FALSE, drive = FALSE)
── rgee 1.1.5 ───────────────────────────────────────────────────────────────────────────────────── earthengine-api 0.1.317 ── 
 ✔ user: user@gmail.com 
 ✔ Initializing Google Earth Engine:Your browser has been opened to visit:

    https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=517222506229-**************************

treetexan commented 2 years ago

I would like to +1 this issue--I had some bugs and got Google's help in setting up my authentificate, but rgee is bouncing and unable to run it the way they specified. Would be a huge help if this could be disabled.

treetexan commented 2 years ago

also, not sure if this is a good idea, but with a working python earthengine authenticate, I was able to circumvent ee_Initialize() and just run rgee::ee$Initialize() which it is a wrapper for.

MatthieuStigler commented 2 years ago

oh nice, thanks! So you are just running rgee::ee$Initialize(), without any option/argument, right? Do you have a sense of what this means regarding the gcs or drive arguments in ee_Initialize?

csaybar commented 2 years ago

Hi @MatthieuStigler and @treetexan, sorry for the inconvenience!

did you try ee_Authenticate? #293

I'm working on upgrading the docker container (if it works there, it must work in a server too). I will come back here when I solve it.

MatthieuStigler commented 2 years ago

Unfortunately, the new ee_Authenticate with auth_quiet will not solve it. I am in a situation where:

system("~/anaconda3/envs/ee/bin/earthengine ls"): :heavy_check_mark: works, meaning I did the authentication
rgee::ee$Initialize(): :heavy_check_mark: works
ee_Initialize(user = xxx): :x: does NOT work, and tries to do the authentication again

So it looks like rgee is looking somewhere for credentials, and doesn't find them? How can we check that?

Below are my env variables:

Sys.getenv("EARTHENGINE_PYTHON")
[1] "~/anaconda3/envs/ee/bin/python"
> Sys.getenv("EARTHENGINE_ENV")
[1] "ee"

MatthieuStigler commented 2 years ago

Mmh, maybe the problem is the user to authenticate? I was doing rgee::ee_Initialize(auth_quiet = TRUE) but then trying rgee::ee_Initialize(user="username") in my scripts, which would not work.

Trying now rgee::ee_Initialize(user="username", auth_quiet = TRUE) it works, but it seems unable to save the conf file? I see the warning:

In file.create(to[okay]) : cannot create file '~/.config/earthengine//username/credentials', reason 'Not a directory'

As a consequence, running just after rgee::ee_Initialize(user = 'username@gmail.com') will try again to authenticate :-(

rgee::ee_Initialize(user = 'username@gmail.com', auth_quiet = TRUE) 
── rgee 1.1.5 ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── earthengine-api 0.1.317 ── 
 ✔ user: username@gmail.com 
 ✔ Initializing Google Earth Engine:Fetching credentials using gcloud
You are authorizing client libraries without access to a web browser. ...

...

Credentials saved to file: [~/.config/gcloud/application_default_credentials.json]

These credentials will be used by any library that requests Application Default Credentials (ADC).

Successfully saved authorization token.
 ✔ Initializing Google Earth Engine:  DONE!
 ✔ Earth Engine account: users/username 
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── 
Warning message:
In file.create(to[okay]) :
  cannot create file '~/.config/earthengine//username/credentials', reason 'Not a directory'

csaybar commented 2 years ago

r u on server side isn't it?

MatthieuStigler commented 2 years ago

yes, all commands are run on server side (except for the two-step deferred gcloud auth)

csaybar commented 2 years ago

this is weird...

In file.create(to[okay]) : cannot create file '~/.config/earthengine//username/credentials', reason 'Not a directory' do

do you have permission to write? maybe creating the folder before could solve the problem, btw I can't reproduce the error.

MatthieuStigler commented 2 years ago

ok, I see what happened: ~/.config/earthengine//username was actually a file, not a directory, which triggered the error above.

Not sure how this happened, that file was a private-key I believe (see pseudo-output). So maybe I did a mistake once and wrongly put this file under this name...

Anyway, removing that file now works, rgee::ee_Initialize(user = 'username@gmail.com', auth_quiet = TRUE) is now permanent. So the issue can be closed, though we might want to wait for @treetexan to confirm the problem is also solved on their side?

THanks again @csaybar for your kind help!

treetexan commented 2 years ago

Hi all

Thanks for hammering away at this! I am watching my young kids all weekend but will reply tmrw with more.

In my setup the .config file is read only, and I can’t undo that (tried but it reverts). But R had no problem creating an empty username folder. Thing is, the credentials folder is stored in /.config/eathengine when you create the credentials file via the auth_mode=notebook route. There is no username subfolder. Copying the credentials into a username subfolder I create before did not work.

Thanks Matt

On Sun, Oct 2, 2022 at 7:43 AM Cesar Aybar @.***> wrote:

this is weird...

In file.create(to[okay]) : cannot create file '~/.config/earthengine//username/credentials', reason 'Not a directory' do

do you have permission to write? maybe creating the folder before must solve the problem, I can't reproduce the error.

— Reply to this email directly, view it on GitHub https://github.com/r-spatial/rgee/issues/291#issuecomment-1264623199, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJAJL5EEYJPD3TXEDNAIFV3WBFYNNANCNFSM6AAAAAAQPMARLY . You are receiving this because you were mentioned.Message ID: @.***>

treetexan commented 2 years ago

this is what I got--what the solution i tested (third line of code) what you were asking me to do?

library(rgee) options(error = traceback) rgee::ee_Initialize(user = @.', auth_quiet = TRUE) ── rgee 1.1.5 ──────────────────────────────────── earthengine-api 0.1.324 ── ✔ user: @. ✔ Initializing Google Earth Engine:Error in (function (authorization_code = NULL, code_verifier = NULL, auth_mode = NULL, : unused argument (auth_quiet = TRUE) No traceback available db <- 'CGIAR/SRTM90_V4' image <- ee$Image(db) Error in py_call_impl(callable, dots$args, dots$keywords) : ee.ee_exception.EEException: Earth Engine client library not initialized. Run ee.Initialize() image$bandNames()$getInfo() Error in image$bandNames : object of type 'closure' is not subsettable rgee::ee$Initialize() db <- 'CGIAR/SRTM90_V4' image <- ee$Image(db) image$bandNames()$getInfo() [1] "elevation"

On Sun, Oct 2, 2022 at 8:57 AM Matthew F @.***> wrote:

Hi all

Thanks for hammering away at this! I am watching my young kids all weekend but will reply tmrw with more.

In my setup the .config file is read only, and I can’t undo that (tried but it reverts). But R had no problem creating an empty username folder. Thing is, the credentials folder is stored in /.config/eathengine when you create the credentials file via the auth_mode=notebook route. There is no username subfolder. Copying the credentials into a username subfolder I create before did not work.

Thanks Matt

On Sun, Oct 2, 2022 at 7:43 AM Cesar Aybar @.***> wrote:

this is weird...

In file.create(to[okay]) : cannot create file '~/.config/earthengine//username/credentials', reason 'Not a directory' do

do you have permission to write? maybe creating the folder before must solve the problem, I can't reproduce the error.

— Reply to this email directly, view it on GitHub https://github.com/r-spatial/rgee/issues/291#issuecomment-1264623199, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJAJL5EEYJPD3TXEDNAIFV3WBFYNNANCNFSM6AAAAAAQPMARLY . You are receiving this because you were mentioned.Message ID: @.***>

treetexan commented 2 years ago

Wrote too soon, got it fixed! Same code, but I first copied my credentials file into the @.*** subfolder, and now it runs. Note that rgee created that subfolder in the last set of code I shared (it didn't previously exist and wasn't created by the credentials authorization process in Python), so this required code failing first to work.

Thank you both! I am very happy to not be fighting this bug anymore.

library(rgee) rgee::ee_Initialize(user = @.', auth_quiet = TRUE) ── rgee 1.1.5 ──────────────────────────────────── earthengine-api 0.1.324 ── ✔ user: @. ✔ Initializing Google Earth Engine: ✔ Initializing Google Earth Engine: DONE!

✔ Earth Engine account: users/mfagan ─────────────────────────────────────────────────────────────────────────────

db <- 'CGIAR/SRTM90_V4' image <- ee$Image(db) image$bandNames()$getInfo() [1] "elevation"

On Mon, Oct 3, 2022 at 3:09 PM Matthew F @.***> wrote:

this is what I got--what the solution i tested (third line of code) what you were asking me to do?

library(rgee) options(error = traceback) rgee::ee_Initialize(user = @.', auth_quiet = TRUE) ── rgee 1.1.5 ──────────────────────────────────── earthengine-api 0.1.324 ── ✔ user: @. ✔ Initializing Google Earth Engine:Error in (function (authorization_code = NULL, code_verifier = NULL, auth_mode = NULL, : unused argument (auth_quiet = TRUE) No traceback available db <- 'CGIAR/SRTM90_V4' image <- ee$Image(db) Error in py_call_impl(callable, dots$args, dots$keywords) : ee.ee_exception.EEException: Earth Engine client library not initialized. Run ee.Initialize() image$bandNames()$getInfo() Error in image$bandNames : object of type 'closure' is not subsettable rgee::ee$Initialize() db <- 'CGIAR/SRTM90_V4' image <- ee$Image(db) image$bandNames()$getInfo() [1] "elevation"

On Sun, Oct 2, 2022 at 8:57 AM Matthew F @.***> wrote:

Hi all

Thanks for hammering away at this! I am watching my young kids all weekend but will reply tmrw with more.

In my setup the .config file is read only, and I can’t undo that (tried but it reverts). But R had no problem creating an empty username folder. Thing is, the credentials folder is stored in /.config/eathengine when you create the credentials file via the auth_mode=notebook route. There is no username subfolder. Copying the credentials into a username subfolder I create before did not work.

Thanks Matt

On Sun, Oct 2, 2022 at 7:43 AM Cesar Aybar @.***> wrote:

this is weird...

In file.create(to[okay]) : cannot create file '~/.config/earthengine//username/credentials', reason 'Not a directory' do

do you have permission to write? maybe creating the folder before must solve the problem, I can't reproduce the error.

— Reply to this email directly, view it on GitHub https://github.com/r-spatial/rgee/issues/291#issuecomment-1264623199, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJAJL5EEYJPD3TXEDNAIFV3WBFYNNANCNFSM6AAAAAAQPMARLY . You are receiving this because you were mentioned.Message ID: @.***>

MatthieuStigler commented 2 years ago

great, so it seems the issue was solved for both of us, I'll close it then

r-spatial / rgee