update earthengine python api fork ?

[12rambau]

Describe the bug We are using a custom fork of earthengine in SEPAL (0.270) and the curent version have changed a bit. I start to face deprecation notice and more problematic, I'm not able to use some of the documenation examples any more.

To Reproduce

import ee 

help(ee.Authenticate)

>>>Help on function Authenticate in module ee:
>>>Authenticate(authorization_code=None, quiet=None, code_verifier=None)
>>>Prompts the user to authorize access to Earth Engine via OAuth2.
>>>
>>>Args:
>>>authorization_code: An optional authorization code.
>>>quiet: If true, do not require interactive prompts.
>>>code_verifier: PKCE verifier to prevent auth code stealing.

Expected behavior According to the documentation there is now 4th parameter: auth_mode that is essential to connect to using the gcloud auth.

[12rambau]

@dfguerrerom I tried to run earthengine authenticate from my SEPAL terminal and I got the following error :

Traceback (most recent call last):
  File "/usr/local/bin/earthengine", line 5, in <module>
    from ee.cli.eecli import main
  File "/usr/local/lib/python3.8/dist-packages/ee/cli/eecli.py", line 17, in <module>
    from ee.cli import commands
  File "/usr/local/lib/python3.8/dist-packages/ee/cli/commands.py", line 34, in <module>
    import tensorflow.compat.v1 as tf
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/__init__.py", line 55, in <module>
    from ._api.v2 import compat
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/_api/v2/compat/__init__.py", line 39, in <module>
    from . import v1
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/_api/v2/compat/v1/__init__.py", line 34, in <module>
    from . import compat
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/_api/v2/compat/v1/compat/__init__.py", line 40, in <module>
    from . import v2
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/_api/v2/compat/v1/compat/v2/__init__.py", line 32, in <module>
    from tensorflow._api.v2.compat.v2 import __internal__
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/_api/v2/compat/v2/__init__.py", line 37, in <module>
    from . import compat
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/_api/v2/compat/v2/compat/__init__.py", line 40, in <module>
    from . import v2
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/_api/v2/compat/v2/compat/v2/__init__.py", line 331, in <module>
    from tensorboard.summary._tf import summary
  File "/usr/local/lib/python3.8/dist-packages/tensorboard/summary/__init__.py", line 22, in <module>
    from tensorboard.summary import v1  # noqa: F401
  File "/usr/local/lib/python3.8/dist-packages/tensorboard/summary/v1.py", line 23, in <module>
    from tensorboard.plugins.histogram import summary as _histogram_summary
  File "/usr/local/lib/python3.8/dist-packages/tensorboard/plugins/histogram/summary.py", line 35, in <module>
    from tensorboard.plugins.histogram import summary_v2
  File "/usr/local/lib/python3.8/dist-packages/tensorboard/plugins/histogram/summary_v2.py", line 35, in <module>
    from tensorboard.util import tensor_util
  File "/usr/local/lib/python3.8/dist-packages/tensorboard/util/tensor_util.py", line 20, in <module>
    from tensorboard.compat.tensorflow_stub import dtypes, compat, tensor_shape
  File "/usr/local/lib/python3.8/dist-packages/tensorboard/compat/tensorflow_stub/__init__.py", line 22, in <module>
    from .dtypes import as_dtype  # noqa
  File "/usr/local/lib/python3.8/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py", line 19, in <module>
    from . import pywrap_tensorflow
  File "/usr/local/lib/python3.8/dist-packages/tensorboard/compat/tensorflow_stub/pywrap_tensorflow.py", line 22, in <module>
    from .io import gfile
  File "/usr/local/lib/python3.8/dist-packages/tensorboard/compat/tensorflow_stub/io/__init__.py", line 17, in <module>
    from . import gfile  # noqa
  File "/usr/local/lib/python3.8/dist-packages/tensorboard/compat/tensorflow_stub/io/gfile.py", line 40, in <module>
    import fsspec
  File "/home/sepal-user/.local/lib/python3.8/site-packages/fsspec/__init__.py", line 12, in <module>
    from .compression import available_compressions
  File "/home/sepal-user/.local/lib/python3.8/site-packages/fsspec/compression.py", line 149, in <module>
    register_compression("lz4", lz4.frame.open, "lz4")
AttributeError: module 'lz4.frame' has no attribute 'open'

Could you confirm you get something of the same type (meaining compatibility issues in the EE API dependencies) ?

@cdanielw Is there any chance that the OpenForis fork of earthengine get updated ? the current version on main is 1.276

[dfguerrerom]

@12rambau I just tested and I do not have that error, it creates the link. However, when I paste it on browser, it says that the access is locked (below in Spanish):

Could that relate to the deletion of my .local?

[12rambau]

that's the one I get when I create a fresh env so no I think it's related to the fact our fork is deprecated

[cdanielw]

We have been running v0.1.319 since August, and I've updated our fork to latest v0.1.332, so that can get deployed. There obviously have been changes on Google sides and ee.Authenticate() in v.0.1.270 provides an incorrect URL. Using v0.1.332 or newer solves this. Unfortunately, now ee.Initialize() breaks. We've finally reached the point where there been enough changes so authentication in our fork is broken. I'm investigating how to solve this.

[cdanielw]

I've updated the fork - this should work now. Let me know.

git+https://github.com/openforis/earthengine-api.git@v0.1.332#egg=earthengine-api&subdirectory=python

[12rambau]

Is it already available on prod or only on test ?

[cdanielw]

It's not deployed to any server yet as the system library, but it doesn't have to be in order to use it inside a venv or as a user library.

[12rambau]

I installed it on my SEPAL account and I still get an error.

If I copy the link I still get the same pop-up from earthengine stating: "Erreur 400 : redirect_uri_mismatch"

The new error is that it's not waiting for user input before failing. Here is the complete displayed traceback:

Enter authorization code: ERROR: There was a problem with web authentication.
ERROR: (gcloud.auth.application-default.login) Please supply either code or authorization_response parameters.
Traceback (most recent call last):
  File "/home/sepal-user/.local/lib/python3.8/site-packages/ee/oauth.py", line 293, in _load_app_default_credentials
    subprocess.run(command, check=True)
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['gcloud', 'auth', 'application-default', 'login', '--scopes=https://www.googleapis.com/auth/earthengine,https://www.googleapis.com/auth/devstorage.full_control', '--client-id-file=/home/sepal-user/.config/earthengine/credentials-client-id.json']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/sepal-user/.local/bin/earthengine", line 8, in <module>
    sys.exit(main())
  File "/home/sepal-user/.local/lib/python3.8/site-packages/ee/cli/eecli.py", line 80, in main
    tf_module.app.run(_run_command, argv=sys.argv[:1])
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/usr/local/lib/python3.8/dist-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.8/dist-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
  File "/home/sepal-user/.local/lib/python3.8/site-packages/ee/cli/eecli.py", line 61, in _run_command
    dispatcher.run(args, config)
  File "/home/sepal-user/.local/lib/python3.8/site-packages/ee/cli/commands.py", line 355, in run
    self.command_dict[vars(args)[self.dest]].run(args, config)
  File "/home/sepal-user/.local/lib/python3.8/site-packages/ee/cli/commands.py", line 397, in run
    ee.Authenticate(**args_auth)
  File "/home/sepal-user/.local/lib/python3.8/site-packages/ee/__init__.py", line 104, in Authenticate
    return oauth.authenticate(authorization_code, quiet, code_verifier, auth_mode,
  File "/home/sepal-user/.local/lib/python3.8/site-packages/ee/oauth.py", line 394, in authenticate
    _load_app_default_credentials(auth_mode == 'gcloud', scopes, quiet)
  File "/home/sepal-user/.local/lib/python3.8/site-packages/ee/oauth.py", line 303, in _load_app_default_credentials
    raise Exception('gcloud failed. ' + tip) from e
Exception: gcloud failed. Please check for any errors above.
*Possible fixes: If you loaded a page with a "redirect_uri_mismatch" error, run earthengine authenticate with the --quiet flag; if the error page says "invalid_request", be sure to run the entire gcloud auth command that is shown.
More information: https://developers.google.com/earth-engine/guides/python_install

Question: is gcloud installed on SEPAL ? because this is how the latest earth engine authenticate (at least on my local computer