Open-EO / openeo-python-client

Python client API for OpenEO
https://open-eo.github.io/openeo-python-client/
Apache License 2.0
151 stars 40 forks source link

Problem with mask in OpenEO local python library #532

Closed automataIA closed 8 months ago

automataIA commented 8 months ago

I wanted to load a cube, using the example at link, and process it locally (the classic method for cloud removal via SCL band). the code I used to create the cube is:

from openeo.local import LocalConnection
import matplotlib.pyplot as plt  
import xarray   
import numpy as np
import pprint

url = "https://earth-search.aws.element84.com/v1/collections/sentinel-2-l2a"
spatial_extent = {"west": west, "south": south, "east": east, "north": north}

temporal_extent = ["2022-01-01", "2022-02-01"]

bands = ["red","green","blue", "scl"]

properties = {"eo:cloud_cover": dict(lt=100)}

s2_cube = local_conn.load_stac(
    url=url,
    spatial_extent=spatial_extent,
    temporal_extent=temporal_extent,
    bands=bands,
    properties=properties,
) 

s2_cube.execute()

And it doesn't give me any errors. Instead, when I go to do the various necessary processes:

scl_band = s2_cube.band("scl")

cloud_mask = (scl_band == 3) | (scl_band == 7) | (scl_band == 8) | (scl_band == 9) | (scl_band == 10) 

cloud_mask = cloud_mask.resample_cube_spatial(s2_cube)

cube_masked = s2_cube.mask(cloud_mask)

composite_masked = cube_masked.min_time()

composite_masked.execute()

It gives me this error:

Unexpected exception formatting exception. Falling back to standard exception
Traceback (most recent call last):
  File "/home/dio/.pyenv/versions/3.10.13/envs/venv/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3553, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "/tmp/ipykernel_507/1291644292.py", line 14, in <module>
    composite_masked.execute()
  File "/home/dio/.pyenv/versions/3.10.13/envs/venv/lib/python3.10/site-packages/openeo/rest/datacube.py", line 2311, in execute
    return self._connection.execute(self.flat_graph(), validate=validate)
  File "/home/dio/.pyenv/versions/3.10.13/envs/venv/lib/python3.10/site-packages/openeo/local/connection.py", line 275, in execute
    return OpenEOProcessGraph(process_graph).to_callable(PROCESS_REGISTRY)()
  File "/home/dio/.pyenv/versions/3.10.13/envs/venv/lib/python3.10/site-packages/openeo_pg_parser_networkx/graph.py", line 318, in to_callable
  File "/home/dio/.pyenv/versions/3.10.13/envs/venv/lib/python3.10/site-packages/openeo_pg_parser_networkx/graph.py", line 347, in _map_node_to_callable
  File "/home/dio/.pyenv/versions/3.10.13/envs/venv/lib/python3.10/site-packages/openeo_pg_parser_networkx/graph.py", line 340, in _map_node_to_callable
    prebaked_process_impl = partial(
  File "/home/dio/.pyenv/versions/3.10.13/envs/venv/lib/python3.10/site-packages/openeo_pg_parser_networkx/process_registry.py", line 49, in __getitem__
    if process.implementation is not None:
KeyError: 'Process mask not found in namespace predefined!'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dio/.pyenv/versions/3.10.13/envs/venv/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 2144, in showtraceback
    stb = self.InteractiveTB.structured_traceback(
  File "/home/dio/.pyenv/versions/3.10.13/envs/venv/lib/python3.10/site-packages/IPython/core/ultratb.py", line 1435, in structured_traceback
    return FormattedTB.structured_traceback(
  File "/home/dio/.pyenv/versions/3.10.13/envs/venv/lib/python3.10/site-packages/IPython/core/ultratb.py", line 1326, in structured_traceback
    return VerboseTB.structured_traceback(
  File "/home/dio/.pyenv/versions/3.10.13/envs/venv/lib/python3.10/site-packages/IPython/core/ultratb.py", line 1173, in structured_traceback
    formatted_exception = self.format_exception_as_a_whole(etype, evalue, etb, number_of_lines_of_context,
  File "/home/dio/.pyenv/versions/3.10.13/envs/venv/lib/python3.10/site-packages/IPython/core/ultratb.py", line 1088, in format_exception_as_a_whole
    frames.append(self.format_record(record))
  File "/home/dio/.pyenv/versions/3.10.13/envs/venv/lib/python3.10/site-packages/IPython/core/ultratb.py", line 970, in format_record
    frame_info.lines, Colors, self.has_colors, lvals
  File "/home/dio/.pyenv/versions/3.10.13/envs/venv/lib/python3.10/site-packages/IPython/core/ultratb.py", line 792, in lines
    return self._sd.lines
  File "/home/dio/.pyenv/versions/3.10.13/envs/venv/lib/python3.10/site-packages/stack_data/utils.py", line 145, in cached_property_wrapper
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/home/dio/.pyenv/versions/3.10.13/envs/venv/lib/python3.10/site-packages/stack_data/core.py", line 734, in lines
    pieces = self.included_pieces
  File "/home/dio/.pyenv/versions/3.10.13/envs/venv/lib/python3.10/site-packages/stack_data/utils.py", line 145, in cached_property_wrapper
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/home/dio/.pyenv/versions/3.10.13/envs/venv/lib/python3.10/site-packages/stack_data/core.py", line 681, in included_pieces
    pos = scope_pieces.index(self.executing_piece)
  File "/home/dio/.pyenv/versions/3.10.13/envs/venv/lib/python3.10/site-packages/stack_data/utils.py", line 145, in cached_property_wrapper
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/home/dio/.pyenv/versions/3.10.13/envs/venv/lib/python3.10/site-packages/stack_data/core.py", line 660, in executing_piece
    return only(
  File "/home/dio/.pyenv/versions/3.10.13/envs/venv/lib/python3.10/site-packages/executing/executing.py", line 116, in only
    raise NotOneValueFound('Expected one value, found 0')
executing.executing.NotOneValueFound: Expected one value, found 0

Is this a library error? How can I solve this problem to be able to do the various processes locally? Win 11, wsl2, vscode, ubuntu, pyenv.

soxofaan commented 8 months ago

This is a problem in the openeo-processes-dask library. The mask process was only added recently there: https://github.com/Open-EO/openeo-processes-dask/pull/165

what version of the openeo-processes-dask library is currently installed in your environment?

If I understand correctly, mask was added in version 2024.1.1

automataIA commented 8 months ago

This is a problem in the openeo-processes-dask library. The mask process was only added recently there: Open-EO/openeo-processes-dask#165

what version of the openeo-processes-dask library is currently installed in your environment?

If I understand correctly, mask was added in version 2024.1.1

Ok. Now I have a meta-question. I'm trying to build an openeo back-end(python based) which uses a stac catalog address (example: https://earth-search.aws.element84.com/v1/collections/sentinel-2-l2a).

Where(link, forum, community) can I ask these questions and other relevant questions for this topic?

soxofaan commented 8 months ago

Where(link, forum, community) can I ask these questions and other relevant questions for this topic?

The openeo python client repo is indeed not the ideal place to discuss that.

The best option currently I think is the openEO Platform forum at https://discuss.eodc.eu/c/openeo-platform/5

soxofaan commented 8 months ago

Assuming the original problem is solved, I'm going to close this. Feel free to reopen