OpenEoApiError: [400] JobNotFinished: batch job has not finished computing the results yet

[gemyasser]

I have developed this function to download the SCL band, It works correctly without any error and there have been a lot of images for several days downloaded successfully, But suddenly yesterday the code gave me the error below.

def download_Scene_Classification(startDate, endDate, AOI = None, Directory = None)-> None:
    """
    Downloads scene classification band (SCL)by querying based on specified criteria..

    Parameters:
        startDate: The start date for the temporal range to query.
        endDate: The end date for the temporal range to query.
        AOI: The spatial area of interest to query as a GeoJSON. Default is cloud_data_location.
        Directory: The directory to download results to. Default is cloud_data_location.

    Returns:
        None
    """
    cube = openeo.connect("openeo.dataspace.copernicus.eu")
    cube.authenticate_oidc()

    if AOI is None:
        AOI = footprint
    if Directory is None:
        Directory = cloud_data_location

    print('Download Scene Classification Data...')  
    cube = cube.load_collection(
        "SENTINEL2_L2A",
        temporal_extent= (startDate, endDate),
        spatial_extent= AOI,
        bands=["SCL"]
        )

    if not cube:
        print("No data available for the specified date range.")
    else:
        result = cube.save_result(format = "GTiff", options = {"filename_prefix": "SCL"})

        job = result.create_job(out_format = "GTiff")
        results = job.get_results()
        results.download_files(Directory)

Authenticated using refresh token. Download Scene Classification Data... Traceback (most recent call last): File "D:\Project_Automation\codeV2.2.1\main.py", line 70, in download_Scene_Classification(preDate, nexDate) File "D:\Project_Automation\codeV2.2.1\M1_dataDownloading.py", line 62, in download_Scene_Classification results.download_files(Directory) File "C:\Users\mohamed.yasser\AppData\Local\Programs\Python\Python311\Lib\site-packages\openeo\rest\job.py", line 493, in download_files downloaded = [a.download(target) for a in self.get_assets()] ^^^^^^^^^^^^^^^^^ File "C:\Users\mohamed.yasser\AppData\Local\Programs\Python\Python311\Lib\site-packages\openeo\rest\job.py", line 430, in get_assets
metadata = self.get_metadata() ^^^^^^^^^^^^^^^^^^^ File "C:\Users\mohamed.yasser\AppData\Local\Programs\Python\Python311\Lib\site-packages\openeo\rest\job.py", line 418, in get_metadata self._results = self._job.connection.get( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\mohamed.yasser\AppData\Local\Programs\Python\Python311\Lib\site-packages\openeo\rest\connection.py", line 207, in get return self.request("get", path=path, stream=stream, auth=auth, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\mohamed.yasser\AppData\Local\Programs\Python\Python311\Lib\site-packages\openeo\rest\connection.py", line 756, in request return _request() ^^^^^^^^^^ File "C:\Users\mohamed.yasser\AppData\Local\Programs\Python\Python311\Lib\site-packages\openeo\rest\connection.py", line 749, in _request return super(Connection, self).request( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\mohamed.yasser\AppData\Local\Programs\Python\Python311\Lib\site-packages\openeo\rest\connection.py", line 164, in request self._raise_api_error(resp) File "C:\Users\mohamed.yasser\AppData\Local\Programs\Python\Python311\Lib\site-packages\openeo\rest\connection.py", line 196, in _raise_api_error raise exception openeo.rest.OpenEoApiError: [400] JobNotFinished: Batch job has not finished computing the results yet. Please try again later or contact our support. (ref: r-23112242243f41a19a981d3a4c2766d6) PS D:\Project_Automation\codeV2.2.1>

[bossie]

This is to be expected because you create the batch job, but never actually start it.

To quote the docs for create_job:

Note that the batch job will just be created at the back-end, it still needs to be started and tracked explicitly.

As suggested further on, the Python client has a convenience method called execute_batch that will create and start the job for you; this method will also wait until the job has finished so you can download the results:

Use execute_batch() instead to have the openEO Python client take care of that job management.

An alternative approach, if you have already created the job, is to call start_and_wait on it.

[bossie]

For reference: the job ID is j-23112262ea234a12bf875f212d01bbc7.

[gemyasser]

I did this, but still have same error...

def download_Scene_Classification(startDate, endDate, AOI = None, Directory = None)-> None:
    """
    Downloads scene classification band (SCL)by querying based on specified criteria..

    Parameters:
        startDate: The start date for the temporal range to query.
        endDate: The end date for the temporal range to query.
        AOI: The spatial area of interest to query as a GeoJSON. Default is cloud_data_location.
        Directory: The directory to download results to. Default is cloud_data_location.

    Returns:
        None
    """
    cube = openeo.connect("openeo.dataspace.copernicus.eu")
    cube.authenticate_oidc()

    if AOI is None:
        AOI = footprint
    if Directory is None:
        Directory = cloud_data_location

    print('Download Scene Classification Data...')  
    cube = cube.load_collection(
        "SENTINEL2_L2A",
        temporal_extent= (startDate, endDate),
        spatial_extent= AOI,
        bands=["SCL"]
        )
    result = cube.save_result(format = "GTiff", options = {"filename_prefix": "SCL"})

    job = result.create_job(out_format = "GTiff")
    job.start_and_wait()
    results = job.get_results()
    results.download_files(Directory)

PS D:\Project_Automation\codeV2.2.2> python .\main.py 2021-12-11 00:00:00 2021-12-16 00:00:00 Authenticated using refresh token. Download Scene Classification Data... 0:00:00 Job 'j-231126c2f07f467ca89e194d9eddc038': send 'start' 0:00:12 Job 'j-231126c2f07f467ca89e194d9eddc038': running (progress N/A) 0:00:18 Job 'j-231126c2f07f467ca89e194d9eddc038': running (progress N/A) 0:00:24 Job 'j-231126c2f07f467ca89e194d9eddc038': running (progress N/A) 0:00:32 Job 'j-231126c2f07f467ca89e194d9eddc038': running (progress N/A) 0:00:42 Job 'j-231126c2f07f467ca89e194d9eddc038': running (progress N/A) 0:00:55 Job 'j-231126c2f07f467ca89e194d9eddc038': running (progress N/A) 0:01:10 Job 'j-231126c2f07f467ca89e194d9eddc038': error (progress N/A) Your batch job 'j-231126c2f07f467ca89e194d9eddc038' failed. Error logs: [{'id': '[1701008260389, 33278]', 'time': '2023-11-26T14:17:40.389Z', 'level': 'error', 'message': 'create_pyramid failed: Could not find data for your load_collection request with catalog ID "Sentinel2". The catalog query had correlation ID "j-231126c2f07f467ca89e194d9eddc038" and returned 0 results.'}, {'id': '[1701008261900, 59474]', 'time': '2023-11-26T14:17:41.900Z', 'level': 'error', 'message': 'OpenEO batch job failed: OpenEOApiException(status_code=400, code=\'NoDataAvailable\', message=\'There is no data available for the given extents. Could not find data for your load_collection request with catalog ID "Sentinel2". The catalog query had correlation ID "j-231126c2f07f467ca89e194d9eddc038" and returned 0 results.\', id=\'no-request\')'}] Full logs can be inspected in an openEO (web) editor or with connection.job('j-231126c2f07f467ca89e194d9eddc038').logs(). Traceback (most recent call last): File "D:\Project_Automation\codeV2.2.2\main.py", line 70, in download_Scene_Classification(preDate, nexDate) File "D:\Project_Automation\codeV2.2.2\M1_dataDownloading.py", line 61, in download_Scene_Classification job.start_and_wait() File "C:\Users\mohamed.yasser\AppData\Local\Programs\Python\Python311\Lib\site-packages\openeo\rest\job.py", line 315, in start_and_wait raise JobFailedException( openeo.rest.JobFailedException: Batch job 'j-231126c2f07f467ca89e194d9eddc038' didn't finish successfully. Status: error (after 0:01:11). PS D:\Project_Automation\codeV2.2.2>

[jdries]

Indeed, this is caused by an issue in the Cloudferro catalog. They logged a ticket internally, and will hopefully come with a solution very soon. I will raise the priority to speed things up.

Reference to internal ticket: https://jira.cloudferro.com/browse/CDSE-613

[soxofaan]

Hi @mo7yasser

I'm curious about this usage pattern:

    cube = cube.load_collection(...)

    if not cube:
        print("No data available for the specified date range.")
    else:

Did you copy-paste this from somewhere? if not cube: print... will never be triggered: load_collection will always return something, or raise an exception if something is wrong. That if construct is useless basically.

Also:

    result = cube.save_result(format = "GTiff", options = {"filename_prefix": "SCL"})
    job = result.create_job(out_format = "GTiff")

You don't need the out_format = "GTiff" in create_job if you already have this in save_result

[gemyasser]

thnx @soxofaan for ur reply, for if not cube: print.. I tried because I wasn't completely aware of openeo, because I was using sentinelsat. Whatever, I caught the error reason. I found that the error was raised because there was no data available in the date range, so I incorporated a try and except block, along with a while loop to dynamically extend the date range by 5 days in each iteration. This way, the code attempts to fetch data in progressively larger date ranges until it successfully retrieves the required data or reaches a specified limit.

[jdries]

Hi @mo7yasser, I had a more detailed look at your query, by simply looking for products in that area in the Copernicus Browser at dataspace.copernicus.eu. It seems to confirm that there simply are no products. This makes sense because your date interval is fairly limited. If you want to always find products, better to use a larger date interval, depends a bit on what you are exactly after.

This screenshot shows there's indeed a gap for that date range, which is normal:

Please let us know if you have further questions. Note that the best place for questions like this is the openEO forum at copernicus.dataspace.eu This issue tracker is mostly for confirmed issues with the openeo python client.