EO Patches Conversion Bug

[spaceie08]

I have bought the Batch processing api to replicate this workflow. The data is downloaded to S3 bucket and now I am trying to convert the tiffs into EO Patches through the following code and getting this error:

Any support would be much appreciated

[devisperessutti]

Hi @spaceie08

Can you confirm config is a dictionary with the AWS details? It might be that they are not set as they are supposed to.

(No need to post what config is, especially as it has secret credentials that you don't want to share)

[spaceie08]

which configuration dictionary are you referring to?, i.e. download_config or tiff_to_eo_patches_config? As the data can be observed downloaded in the offered bucket, the AWS details are configured in the downloading stage. Only the conversation configuration for the EO patches is producing the problem.

[spaceie08]

it's working now but taking a lot of time to execute, the eo patches are being written in s3 bucket but the execution process is still not complete.

What could be the reason for that?

[devisperessutti]

It depends on how large your area of interest is, and the length of time interval.

It is an async process and generally processes very large areas in less than a few hours.

On Fri, 11 Nov 2022, 11:26 spaceie08, @.***> wrote:

it's working now but taking a lot of time to execute, the eo patches are being written in s3 bucket but the execution process is still not complete.

What could be the reason for that?

— Reply to this email directly, view it on GitHub https://github.com/sentinel-hub/field-delineation/issues/19#issuecomment-1311513938, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADMTBWAPUHSLTEKNLFCXB7DWHYNN5ANCNFSM6AAAAAAR4TEVQQ . You are receiving this because you commented.Message ID: @.***>

[spaceie08]

the length of time interval is not more than 10 days, and hence there are only 4 tiles in the s3 bucket. The AOI is 1/50th in area of a single sentinel-2 tile. This is the reason I am worried about it. Had I run it on bigger areas, I don't know how much it will enquire.

[devisperessutti]

maybe the following questions would help us understand whether there is any issue with batch processing, but no disruptions to service were reported

• how many tiles did the analysis show?
• how many tiles are downloaded and how long did it take?
• is processing stuck or tiles are being added/the job finished?

[spaceie08]

As I said earlier that there's no issue with the batch processing API because the tiles are successfully downloaded to the S3 bucket. The issue persists when the code tries to load these tiles from AWS to run the eo_patches_conversion process. This process takes a long time and even when the code-logs stops, the code execution keeps running.

what I tried:

1. I interrupted the execution of eo_patch_convesion step after I saw that the EO patches were written in the provided S3 bucket folder (P.S: I don't know if they were completely written)
2. Then I started the rasterize_training_data process which loads the EO patches from the S3 bucket to write the training data in them (acc. to the workflow given in the source code). This process also takes a long time and does not complete even after 30 mins.

P.S: I am trying to test the complete workflow on a very small area (not more than 30KMs) and on a very short period of time (10 days)

[spaceie08]

There are only 4 tiles downloaded in s3 bucket as expected for 10 days time span

[devisperessutti]

I see, sorry I misunderstood your issue.

This is odd, as both conversion and rasterization are supposed to be very fast. And if there would be some issue with the configuration you wouldn't get any eopatch processed.

Can you post where the logs are stopping at? Can you also post the versions of sh-py and eo-learn?

On Sat, 12 Nov 2022, 09:40 spaceie08, @.***> wrote:

There are only 4 tiles downloaded in s3 bucket as expected for 10 days time span

— Reply to this email directly, view it on GitHub https://github.com/sentinel-hub/field-delineation/issues/19#issuecomment-1312413766, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADMTBWANLFJTAEHZLR3R743WH5JWXANCNFSM6AAAAAAR4TEVQQ . You are receiving this because you commented.Message ID: @.***>

[spaceie08]

No problem. Please find below the requested

P.S I did not install the sh-py because it was not required (not in the requirements.txt either)

Could this lag be due to the bucket region? My bucket region is eu-central.

[devisperessutti]

Sorry, I meant sentinelhub-py, which gets installed through eo-learn

It looks like urlib3 is returning a bad request 400 error if I understand the logs, but again it's weird because some requests go through. Are you using AWS instances or running locally? if the first, could you try from one located in eu-central-1?

I don't think it should give such long lags, but might be worth checking. Also you wouldn't incur in data transfer costs between bucket regions.

If you check the size on bucket of eopatches, it is probably few hundred MB. I would suggest you set up an AWS profile using awscli with the credentials you use in the notebook, and try to download one eopatch locally to test the connection and see how long it takes.

[spaceie08]

I have tried it running on EC2 instance as well but the issue remains the same.

As you mentioned to download it using awscli, I tried it and the download was successful. Please see the below attached screenshots. It did not take more than 5 seconds.

Regarding the region, my bucket is in eu-central-1 but when I read the logs it recommends to me have the bucket in us‑east‑1 region. Could this be an issue?

[spaceie08]

Now tried to run the code on Jupyter notebook, found out the following error:

Traceback (most recent call last):
  File "c:\users\hf\anaconda3\lib\site-packages\eolearn\core\eotask.py", line 72, in _execute_handling
    return_value = self.execute(*eopatches, **kwargs)
  File "c:\users\hf\anaconda3\lib\site-packages\eolearn\core\core_tasks.py", line 162, in execute
    return EOPatch.load(path, filesystem=self.filesystem, **self.kwargs)
  File "c:\users\hf\anaconda3\lib\site-packages\eolearn\core\eodata.py", line 573, in load
    return load_eopatch(EOPatch(), filesystem, path, features=features, lazy_loading=lazy_loading)
  File "c:\users\hf\anaconda3\lib\site-packages\eolearn\core\eodata_io.py", line 100, in load_eopatch
    for (ftype, fname, _), value in zip(features, loading_data):
  File "C:\Users\HF\anaconda3\lib\concurrent\futures\_base.py", line 609, in result_iterator
    yield fs.pop().result()
  File "C:\Users\HF\anaconda3\lib\concurrent\futures\_base.py", line 439, in result
    return self.__get_result()
  File "C:\Users\HF\anaconda3\lib\concurrent\futures\_base.py", line 391, in __get_result
    raise self._exception
  File "C:\Users\HF\anaconda3\lib\concurrent\futures\thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "c:\users\hf\anaconda3\lib\site-packages\eolearn\core\eodata_io.py", line 98, in <lambda>
    loading_data = executor.map(lambda loader: loader.load(), loading_data)
  File "c:\users\hf\anaconda3\lib\site-packages\eolearn\core\eodata_io.py", line 256, in load
    self.loaded_value = self._decode(file_handle, self.path)
  File "c:\users\hf\anaconda3\lib\site-packages\eolearn\core\eodata_io.py", line 298, in _decode
    data = pickle.load(file)
  File "c:\users\hf\anaconda3\lib\site-packages\fs_s3fs\_s3fs.py", line 154, in readinto
    return self._f.readinto()
  File "C:\Users\HF\anaconda3\lib\tempfile.py", line 478, in func_wrapper
    return func(*args, **kwargs)
TypeError: BufferedRandom.readinto() takes exactly one argument (0 given)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\hf\anaconda3\lib\site-packages\eolearn\core\eoexecution.py", line 199, in _execute_workflow
    results = workflow.execute(input_args, monitor=True)
  File "c:\users\hf\anaconda3\lib\site-packages\eolearn\core\eoworkflow.py", line 172, in execute
    results = WorkflowResults(self._execute_tasks(input_args=input_args, out_degs=out_degs, monitor=monitor))
  File "c:\users\hf\anaconda3\lib\site-packages\eolearn\core\eoworkflow.py", line 207, in _execute_tasks
    result = self._execute_task(dependency=dep,
  File "c:\users\hf\anaconda3\lib\site-packages\eolearn\core\eoworkflow.py", line 245, in _execute_task
    return task(*task_args, **task_kwargs, monitor=monitor)
  File "c:\users\hf\anaconda3\lib\site-packages\eolearn\core\eotask.py", line 59, in __call__
    return self._execute_handling(*eopatches, **kwargs)
  File "c:\users\hf\anaconda3\lib\site-packages\eolearn\core\eotask.py", line 85, in _execute_handling
    raise extended_exception.with_traceback(traceback)
  File "c:\users\hf\anaconda3\lib\site-packages\eolearn\core\eotask.py", line 72, in _execute_handling
    return_value = self.execute(*eopatches, **kwargs)
  File "c:\users\hf\anaconda3\lib\site-packages\eolearn\core\core_tasks.py", line 162, in execute
    return EOPatch.load(path, filesystem=self.filesystem, **self.kwargs)
  File "c:\users\hf\anaconda3\lib\site-packages\eolearn\core\eodata.py", line 573, in load
    return load_eopatch(EOPatch(), filesystem, path, features=features, lazy_loading=lazy_loading)
  File "c:\users\hf\anaconda3\lib\site-packages\eolearn\core\eodata_io.py", line 100, in load_eopatch
    for (ftype, fname, _), value in zip(features, loading_data):
  File "C:\Users\HF\anaconda3\lib\concurrent\futures\_base.py", line 609, in result_iterator
    yield fs.pop().result()
  File "C:\Users\HF\anaconda3\lib\concurrent\futures\_base.py", line 439, in result
    return self.__get_result()
  File "C:\Users\HF\anaconda3\lib\concurrent\futures\_base.py", line 391, in __get_result
    raise self._exception
  File "C:\Users\HF\anaconda3\lib\concurrent\futures\thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "c:\users\hf\anaconda3\lib\site-packages\eolearn\core\eodata_io.py", line 98, in <lambda>
    loading_data = executor.map(lambda loader: loader.load(), loading_data)
  File "c:\users\hf\anaconda3\lib\site-packages\eolearn\core\eodata_io.py", line 256, in load
    self.loaded_value = self._decode(file_handle, self.path)
  File "c:\users\hf\anaconda3\lib\site-packages\eolearn\core\eodata_io.py", line 298, in _decode
    data = pickle.load(file)
  File "c:\users\hf\anaconda3\lib\site-packages\fs_s3fs\_s3fs.py", line 154, in readinto
    return self._f.readinto()
  File "C:\Users\HF\anaconda3\lib\tempfile.py", line 478, in func_wrapper
    return func(*args, **kwargs)
TypeError: During execution of task LoadTask: BufferedRandom.readinto() takes exactly one argument (0 given)

[spaceie08]

when tried loading the EO patches It threw:

---------------------------------------------------------------------------
ClientError                               Traceback (most recent call last)
c:\users\hf\anaconda3\lib\site-packages\fs_s3fs\_s3fs.py in s3errors(path)
    172     try:
--> 173         yield
    174     except ClientError as error:

c:\users\hf\anaconda3\lib\site-packages\fs_s3fs\_s3fs.py in _get_object(self, path, key)
    345                 obj = self.s3.Object(self._bucket_name, _key)
--> 346                 obj.load()
    347         except errors.ResourceNotFound:

c:\users\hf\anaconda3\lib\site-packages\boto3\resources\factory.py in do_action(self, *args, **kwargs)
    563             def do_action(self, *args, **kwargs):
--> 564                 response = action(self, *args, **kwargs)
    565                 self.meta.data = response

c:\users\hf\anaconda3\lib\site-packages\boto3\resources\action.py in __call__(self, parent, *args, **kwargs)
     87 
---> 88         response = getattr(parent.meta.client, operation_name)(*args, **params)
     89 

c:\users\hf\anaconda3\lib\site-packages\botocore\client.py in _api_call(self, *args, **kwargs)
    507             # The "self" in this scope is referring to the BaseClient.
--> 508             return self._make_api_call(operation_name, kwargs)
    509 

c:\users\hf\anaconda3\lib\site-packages\botocore\client.py in _make_api_call(self, operation_name, api_params)
    914             error_class = self.exceptions.from_code(error_code)
--> 915             raise error_class(parsed_response, operation_name)
    916         else:

ClientError: An error occurred (404) when calling the HeadObject operation: Not Found

During handling of the above exception, another exception occurred:

ResourceNotFound                          Traceback (most recent call last)
c:\users\hf\anaconda3\lib\site-packages\fs_s3fs\_s3fs.py in _get_object(self, path, key)
    345                 obj = self.s3.Object(self._bucket_name, _key)
--> 346                 obj.load()
    347         except errors.ResourceNotFound:

~\anaconda3\lib\contextlib.py in __exit__(self, typ, value, traceback)
    136             try:
--> 137                 self.gen.throw(typ, value, traceback)
    138             except StopIteration as exc:

c:\users\hf\anaconda3\lib\site-packages\fs_s3fs\_s3fs.py in s3errors(path)
    182         if http_status == 404:
--> 183             raise errors.ResourceNotFound(path)
    184         elif http_status == 403:

ResourceNotFound: resource '/eoPatches\42RWS_7_7' not found

During handling of the above exception, another exception occurred:

ClientError                               Traceback (most recent call last)
c:\users\hf\anaconda3\lib\site-packages\fs_s3fs\_s3fs.py in s3errors(path)
    172     try:
--> 173         yield
    174     except ClientError as error:

c:\users\hf\anaconda3\lib\site-packages\fs_s3fs\_s3fs.py in _get_object(self, path, key)
    349                 obj = self.s3.Object(self._bucket_name, _key + self.delimiter)
--> 350                 obj.load()
    351                 return obj

c:\users\hf\anaconda3\lib\site-packages\boto3\resources\factory.py in do_action(self, *args, **kwargs)
    563             def do_action(self, *args, **kwargs):
--> 564                 response = action(self, *args, **kwargs)
    565                 self.meta.data = response

c:\users\hf\anaconda3\lib\site-packages\boto3\resources\action.py in __call__(self, parent, *args, **kwargs)
     87 
---> 88         response = getattr(parent.meta.client, operation_name)(*args, **params)
     89 

c:\users\hf\anaconda3\lib\site-packages\botocore\client.py in _api_call(self, *args, **kwargs)
    507             # The "self" in this scope is referring to the BaseClient.
--> 508             return self._make_api_call(operation_name, kwargs)
    509 

c:\users\hf\anaconda3\lib\site-packages\botocore\client.py in _make_api_call(self, operation_name, api_params)
    914             error_class = self.exceptions.from_code(error_code)
--> 915             raise error_class(parsed_response, operation_name)
    916         else:

ClientError: An error occurred (404) when calling the HeadObject operation: Not Found

During handling of the above exception, another exception occurred:

ResourceNotFound                          Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_6772\2847823639.py in <module>
----> 1 eop = EOPatch.load(os.path.join(EOPATCHES_FOLDER, eops[1]), filesystem=filesystem)

c:\users\hf\anaconda3\lib\site-packages\eolearn\core\eodata.py in load(path, features, lazy_loading, filesystem)
    571             path = '/'
    572 
--> 573         return load_eopatch(EOPatch(), filesystem, path, features=features, lazy_loading=lazy_loading)
    574 
    575     def merge(self, *eopatches, features=..., time_dependent_op=None, timeless_op=None):

c:\users\hf\anaconda3\lib\site-packages\eolearn\core\eodata_io.py in load_eopatch(eopatch, filesystem, patch_location, features, lazy_loading)
     91     """ A utility function used by EOPatch.load method
     92     """
---> 93     features = list(walk_filesystem(filesystem, patch_location, features))
     94     loading_data = [FeatureIO(filesystem, path) for _, _, path in features]
     95 

c:\users\hf\anaconda3\lib\site-packages\eolearn\core\eodata_io.py in walk_filesystem(filesystem, patch_location, features)
    108     """
    109     existing_features = defaultdict(dict)
--> 110     for ftype, fname, path in walk_main_folder(filesystem, patch_location):
    111         existing_features[ftype][fname] = path
    112 

c:\users\hf\anaconda3\lib\site-packages\eolearn\core\eodata_io.py in walk_main_folder(filesystem, folder_path)
    152     them as well, which allows `walk_filesystem` to skip such subfolders from further searches.
    153     """
--> 154     for path in filesystem.listdir(folder_path):
    155         raw_path = path.split('.')[0].strip('/')
    156 

c:\users\hf\anaconda3\lib\site-packages\fs_s3fs\_s3fs.py in listdir(self, path)
    494 
    495         if not _directory:
--> 496             if not self.getinfo(_path).is_dir:
    497                 raise errors.DirectoryExpected(path)
    498 

c:\users\hf\anaconda3\lib\site-packages\fs_s3fs\_s3fs.py in getinfo(self, path, namespaces)
    449             )
    450 
--> 451         obj = self._get_object(path, _key)
    452         info = self._info_from_object(obj, namespaces)
    453         return Info(info)

c:\users\hf\anaconda3\lib\site-packages\fs_s3fs\_s3fs.py in _get_object(self, path, key)
    349                 obj = self.s3.Object(self._bucket_name, _key + self.delimiter)
    350                 obj.load()
--> 351                 return obj
    352         else:
    353             return obj

~\anaconda3\lib\contextlib.py in __exit__(self, typ, value, traceback)
    135                 value = typ()
    136             try:
--> 137                 self.gen.throw(typ, value, traceback)
    138             except StopIteration as exc:
    139                 # Suppress StopIteration *unless* it's the same exception that

c:\users\hf\anaconda3\lib\site-packages\fs_s3fs\_s3fs.py in s3errors(path)
    181             raise errors.ResourceError(path, exc=error, msg=error_msg)
    182         if http_status == 404:
--> 183             raise errors.ResourceNotFound(path)
    184         elif http_status == 403:
    185             raise errors.PermissionDenied(path=path, msg=error_msg)

ResourceNotFound: resource '/eoPatches\42RWS_7_7' not found

[devisperessutti]

mhh, but eopatch exists on bucket right?

This is what you were trying right?

eop = EOPatch.load(os.path.join(EOPATCHES_FOLDER, eops[1]), filesystem=filesystem)

Could you try when EOPATCHES_FOLDER = "./eopatches"? I would also advise to use fs library instead of os, as it might mix Windows paths and Linux paths, which could be a source of issues (we develop in Linux, and don't test the code thoroughly on Windows).

[spaceie08]

tried with fs as well. The path issue is resolved I guess. But there's an other issue now: code: eop =EOPatch.load(fs.path.combine(EOPATCHES_FOLDER, eops[0]), filesystem=filesystem) ` error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_7144\854617404.py in <module>
----> 1 eop = EOPatch.load(fs.path.combine(EOPATCHES_FOLDER, eops[0]), filesystem=filesystem)

c:\users\hf\anaconda3\lib\site-packages\eolearn\core\eodata.py in load(path, features, lazy_loading, filesystem)
    571             path = '/'
    572 
--> 573         return load_eopatch(EOPatch(), filesystem, path, features=features, lazy_loading=lazy_loading)
    574 
    575     def merge(self, *eopatches, features=..., time_dependent_op=None, timeless_op=None):

c:\users\hf\anaconda3\lib\site-packages\eolearn\core\eodata_io.py in load_eopatch(eopatch, filesystem, patch_location, features, lazy_loading)
     98             loading_data = executor.map(lambda loader: loader.load(), loading_data)
     99 
--> 100     for (ftype, fname, _), value in zip(features, loading_data):
    101         eopatch[(ftype, fname)] = value
    102 

~\anaconda3\lib\concurrent\futures\_base.py in result_iterator()
    607                     # Careful not to keep a reference to the popped future
    608                     if timeout is None:
--> 609                         yield fs.pop().result()
    610                     else:
    611                         yield fs.pop().result(end_time - time.monotonic())

~\anaconda3\lib\concurrent\futures\_base.py in result(self, timeout)
    437                     raise CancelledError()
    438                 elif self._state == FINISHED:
--> 439                     return self.__get_result()
    440 
    441                 self._condition.wait(timeout)

~\anaconda3\lib\concurrent\futures\_base.py in __get_result(self)
    389         if self._exception:
    390             try:
--> 391                 raise self._exception
    392             finally:
    393                 # Break a reference cycle with the exception in self._exception

~\anaconda3\lib\concurrent\futures\thread.py in run(self)
     56 
     57         try:
---> 58             result = self.fn(*self.args, **self.kwargs)
     59         except BaseException as exc:
     60             self.future.set_exception(exc)

c:\users\hf\anaconda3\lib\site-packages\eolearn\core\eodata_io.py in <lambda>(loader)
     96     if not lazy_loading:
     97         with concurrent.futures.ThreadPoolExecutor() as executor:
---> 98             loading_data = executor.map(lambda loader: loader.load(), loading_data)
     99 
    100     for (ftype, fname, _), value in zip(features, loading_data):

c:\users\hf\anaconda3\lib\site-packages\eolearn\core\eodata_io.py in load(self)
    254                     self.loaded_value = self._decode(gzip_fp, self.path)
    255             else:
--> 256                 self.loaded_value = self._decode(file_handle, self.path)
    257 
    258         return self.loaded_value

c:\users\hf\anaconda3\lib\site-packages\eolearn\core\eodata_io.py in _decode(file, path)
    296         """
    297         if FileFormat.PICKLE.extension() in path:
--> 298             data = pickle.load(file)
    299 
    300             # There seems to be an issue in geopandas==0.8.1 where unpickling GeoDataFrames, which were saved with an

c:\users\hf\anaconda3\lib\site-packages\fs_s3fs\_s3fs.py in readinto(self, b)
    152 
    153     def readinto(self, b):
--> 154         return self._f.readinto()
    155 
    156     def write(self, b):

~\anaconda3\lib\tempfile.py in func_wrapper(*args, **kwargs)
    476             @_functools.wraps(func)
    477             def func_wrapper(*args, **kwargs):
--> 478                 return func(*args, **kwargs)
    479             # Avoid closing the file as long as the wrapper is alive,
    480             # see issue #18879.

TypeError: BufferedRandom.readinto() takes exactly one argument (0 given)

[spaceie08]

Could you please test the code on windows? I have bought the Enterprise account just to test the model.

[devisperessutti]

You mentioned you tried the code on an ec2 instance, was it also Windows? Could you try to run the code on a linux-based instance in eu-central-1?

We are a bit busy with projects and testing this on our side on Windows would take some time.

[spaceie08]

when converting the tiff into eopatches why is it giving the error that: " None is not a valid CRS"

[devisperessutti]

This is a more familiar error, which points to a newer version of geopandas. Can you try downgrading to version 0.9.0 or 0.8.2 and try it again?

Can you list (conda list or pip list) the versions of the following packages: numpy, pandas, geopandas, sentinelhub-py, eo-learn-*, rasterio

I will try to set up an env with working packages when I have some time, but if you can test beforehand it would be easier.

[spaceie08]

eo-flow                        1.2.0
eo-learn                       0.10.2
eo-learn-core                  0.10.2
eo-learn-coregistration        0.10.2
eo-learn-features              0.10.2
eo-learn-geometry              0.10.2
eo-learn-io                    0.10.2
eo-learn-mask                  0.10.2
eo-learn-ml-tools              0.10.2
eo-learn-visualization         0.10.2
geopandas                      0.12.1
pandas                         1.5.1
rasterio                       1.3.3
sentinelhub                    3.8.0
numpy                          1.23.4

[batic]

Closing issue, the examples should now work with fixed package versions.