ecmwf-projects / cads-api-client

CADS API Python client for developing and testing
Apache License 2.0
2 stars 2 forks source link

improve debug logs and error messages #46

Closed malmans2 closed 4 months ago

malmans2 commented 4 months ago
import cdsapi
client = cdsapi.Client(
    key="00000000-0000-4000-a000-000000000000",
    url="https://cds-stable-bopen.copernicus-climate.eu/api",
    debug=True,
)
client.retrieve('test-adaptor-dummy', {}, 'test.grib')
2024-05-10 09:11:50,250 DEBUG CDSAPI {'url': 'https://cds-stable-bopen.copernicus-climate.eu/api', 'key': '00000000-0000-4000-a000-000000000000', 'quiet': False, 'timeout': 60, 'sleep_max': 120, 'retry_max': 500}
2024-05-10 09:11:50,251 DEBUG POST https://cds-stable-bopen.copernicus-climate.eu/api/retrieve/v1/processes/test-adaptor-dummy/execute {}
2024-05-10 09:11:50,608 DEBUG REPLY {"processID":"test-adaptor-dummy","type":"process","jobID":"f6eed10f-bcd6-41b6-94ef-c006334b96e4","status":"accepted","created":"2024-05-10T07:11:50.554374","updated":"2024-05-10T07:11:50.554374","links":[{"href":"https://cds-stable-bopen.copernicus-climate.eu/api/retrieve/v1/processes/test-adaptor-dummy/execute","rel":"self"},{"href":"https://cds-stable-bopen.copernicus-climate.eu/api/retrieve/v1/jobs/f6eed10f-bcd6-41b6-94ef-c006334b96e4","rel":"monitor","type":"application/json","title":"job status info"}],"metadata":{"datasetMetadata":{"messages":[{"date":"2022-11-16T14:29:32","severity":"info","content":"The `test-adaptor-dummy` is our new nice dataset to be used!"},{"date":"2023-02-28T10:15:00","severity":"warning","content":"The `test-adaptor-dummy` is **just a proof-of-concept** for testing features!"},{"date":"2023-07-25T16:19:32","severity":"info","content":"You can commit changes to this dataset freely."}]}}}
2024-05-10 09:11:50,608 INFO [2022-11-16T14:29:32] The `test-adaptor-dummy` is our new nice dataset to be used!
2024-05-10 09:11:50,609 WARNING [2023-02-28T10:15:00] The `test-adaptor-dummy` is **just a proof-of-concept** for testing features!
2024-05-10 09:11:50,609 INFO [2023-07-25T16:19:32] You can commit changes to this dataset freely.
2024-05-10 09:11:50,609 DEBUG Request UID is f6eed10f-bcd6-41b6-94ef-c006334b96e4
2024-05-10 09:11:50,609 DEBUG GET https://cds-stable-bopen.copernicus-climate.eu/api/retrieve/v1/jobs/f6eed10f-bcd6-41b6-94ef-c006334b96e4
2024-05-10 09:11:50,726 DEBUG REPLY {"processID":"test-adaptor-dummy","type":"process","jobID":"f6eed10f-bcd6-41b6-94ef-c006334b96e4","status":"accepted","created":"2024-05-10T07:11:50.554374","updated":"2024-05-10T07:11:50.554374","links":[{"href":"https://cds-stable-bopen.copernicus-climate.eu/api/retrieve/v1/jobs/f6eed10f-bcd6-41b6-94ef-c006334b96e4","rel":"self","type":"application/json"}],"metadata":{"log":[]}}
2024-05-10 09:11:50,726 INFO status has been updated to accepted
2024-05-10 09:11:50,726 DEBUG result not ready, waiting for 1.5 seconds
2024-05-10 09:11:52,231 DEBUG GET https://cds-stable-bopen.copernicus-climate.eu/api/retrieve/v1/jobs/f6eed10f-bcd6-41b6-94ef-c006334b96e4
2024-05-10 09:11:52,318 DEBUG REPLY {"processID":"test-adaptor-dummy","type":"process","jobID":"f6eed10f-bcd6-41b6-94ef-c006334b96e4","status":"running","created":"2024-05-10T07:11:50.554374","started":"2024-05-10T07:11:50.846682","updated":"2024-05-10T07:11:50.846682","links":[{"href":"https://cds-stable-bopen.copernicus-climate.eu/api/retrieve/v1/jobs/f6eed10f-bcd6-41b6-94ef-c006334b96e4","rel":"self","type":"application/json"}],"metadata":{"log":[]}}
2024-05-10 09:11:52,318 INFO status has been updated to running
2024-05-10 09:11:52,318 DEBUG result not ready, waiting for 2.25 seconds
2024-05-10 09:11:54,572 DEBUG GET https://cds-stable-bopen.copernicus-climate.eu/api/retrieve/v1/jobs/f6eed10f-bcd6-41b6-94ef-c006334b96e4
2024-05-10 09:11:54,699 DEBUG REPLY {"processID":"test-adaptor-dummy","type":"process","jobID":"f6eed10f-bcd6-41b6-94ef-c006334b96e4","status":"successful","created":"2024-05-10T07:11:50.554374","started":"2024-05-10T07:11:50.846682","finished":"2024-05-10T07:11:52.505647","updated":"2024-05-10T07:11:52.505647","links":[{"href":"https://cds-stable-bopen.copernicus-climate.eu/api/retrieve/v1/jobs/f6eed10f-bcd6-41b6-94ef-c006334b96e4","rel":"self","type":"application/json"},{"href":"https://cds-stable-bopen.copernicus-climate.eu/api/retrieve/v1/jobs/f6eed10f-bcd6-41b6-94ef-c006334b96e4/results","rel":"results"}],"metadata":{"log":[]}}
2024-05-10 09:11:54,700 INFO status has been updated to successful
2024-05-10 09:11:55,702 DEBUG GET https://cds-stable-bopen.copernicus-climate.eu/api/retrieve/v1/jobs/f6eed10f-bcd6-41b6-94ef-c006334b96e4
2024-05-10 09:11:55,916 DEBUG REPLY {"processID":"test-adaptor-dummy","type":"process","jobID":"f6eed10f-bcd6-41b6-94ef-c006334b96e4","status":"successful","created":"2024-05-10T07:11:50.554374","started":"2024-05-10T07:11:50.846682","finished":"2024-05-10T07:11:52.505647","updated":"2024-05-10T07:11:52.505647","links":[{"href":"https://cds-stable-bopen.copernicus-climate.eu/api/retrieve/v1/jobs/f6eed10f-bcd6-41b6-94ef-c006334b96e4","rel":"self","type":"application/json"},{"href":"https://cds-stable-bopen.copernicus-climate.eu/api/retrieve/v1/jobs/f6eed10f-bcd6-41b6-94ef-c006334b96e4/results","rel":"results"}],"metadata":{"log":[]}}
2024-05-10 09:11:55,916 DEBUG GET https://cds-stable-bopen.copernicus-climate.eu/api/retrieve/v1/jobs/f6eed10f-bcd6-41b6-94ef-c006334b96e4
2024-05-10 09:11:56,000 DEBUG REPLY {"processID":"test-adaptor-dummy","type":"process","jobID":"f6eed10f-bcd6-41b6-94ef-c006334b96e4","status":"successful","created":"2024-05-10T07:11:50.554374","started":"2024-05-10T07:11:50.846682","finished":"2024-05-10T07:11:52.505647","updated":"2024-05-10T07:11:52.505647","links":[{"href":"https://cds-stable-bopen.copernicus-climate.eu/api/retrieve/v1/jobs/f6eed10f-bcd6-41b6-94ef-c006334b96e4","rel":"self","type":"application/json"},{"href":"https://cds-stable-bopen.copernicus-climate.eu/api/retrieve/v1/jobs/f6eed10f-bcd6-41b6-94ef-c006334b96e4/results","rel":"results"}]}
2024-05-10 09:11:56,000 DEBUG GET https://cds-stable-bopen.copernicus-climate.eu/api/retrieve/v1/jobs/f6eed10f-bcd6-41b6-94ef-c006334b96e4/results {}
2024-05-10 09:11:56,166 DEBUG REPLY {"asset":{"value":{"type":"application/x-grib","href":"https://object-store.os-api.cci2.ecmwf.int:443/bopen-cds2-stable-cache/5fa49b9d7192f25d8d6f484fe4e9d9e8.grib","file:checksum":"d41d8cd98f00b204e9800998ecf8427e","file:size":0,"file:local_path":"s3://bopen-cds2-stable-cache/5fa49b9d7192f25d8d6f484fe4e9d9e8.grib"}}}
malmans2 commented 4 months ago
import cdsapi

client = cdsapi.Client(
    key="00000000-0000-4000-a000-000000000000",
    url="https://cds-stable-bopen.copernicus-climate.eu/api",
)
collection_id = "reanalysis-era5-single-levels"
request = {
    "variable": "2t",
    "product_type": "reanalysis",
    "date": "2012-12-01",
    "time": "12:00",
    "foo": "bar",
}
client.retrieve(collection_id, request, "test.grib")
---------------------------------------------------------------------------
ProcessingFailedError                     Traceback (most recent call last)
Cell In[5], line 15
      7 collection_id = "reanalysis-era5-single-levels"
      8 request = {
      9     "variable": "2t",
     10     "product_type": "reanalysis",
   (...)
     13     "foo": "bar",
     14 }
---> 15 client.retrieve(collection_id, request, "test.grib")

File ~/MyGit/CADS/cads-api-client/cads_api_client/legacy_api_client.py:149, in LegacyApiClient.retrieve(self, name, request, target)
    146 def retrieve(
    147     self, name: str, request: dict[str, Any], target: str | None = None
    148 ) -> str | processing.Results:
--> 149     result = self.logging_decorator(self.client.submit_and_wait_on_result)(
    150         collection_id=name,
    151         retry_options=self.retry_options,
    152         **request,
    153     )
    154     partial_download: Callable[..., str] = functools.partial(
    155         result.download,
    156         timeout=self.timeout,
    157         retry_options=self.retry_options,
    158     )
    159     result.download = self.logging_decorator(partial_download)  # type: ignore[method-assign]

File ~/MyGit/CADS/cads-api-client/cads_api_client/legacy_api_client.py:134, in LegacyApiClient.logging_decorator.<locals>.wrapper(*args, **kwargs)
    129 @functools.wraps(func)
    130 def wrapper(*args: Any, **kwargs: Any) -> Any:
    131     with LoggingContext(
    132         logger=processing.logger, quiet=self.quiet, debug=self._debug
    133     ):
--> 134         return func(*args, **kwargs)

File ~/MyGit/CADS/cads-api-client/cads_api_client/api_client.py:77, in ApiClient.submit_and_wait_on_result(self, collection_id, retry_options, **request)
     74 def submit_and_wait_on_result(
     75     self, collection_id: str, retry_options: Dict[str, Any] = {}, **request: Any
     76 ) -> processing.Results:
---> 77     return self.retrieve_api.submit_and_wait_on_result(
     78         collection_id, retry_options=retry_options, **request
     79     )

File ~/MyGit/CADS/cads-api-client/cads_api_client/processing.py:456, in Processing.submit_and_wait_on_result(self, collection_id, retry_options, **request)
    452 def submit_and_wait_on_result(
    453     self, collection_id: str, retry_options: Dict[str, Any] = {}, **request: Any
    454 ) -> Results:
    455     remote = self.submit(collection_id, retry_options=retry_options, **request)
--> 456     remote.wait_on_result(retry_options=retry_options)
    457     return remote.make_results()

File ~/MyGit/CADS/cads-api-client/cads_api_client/processing.py:246, in Remote.wait_on_result(self, retry_options)
    244     time.sleep(1)
    245     results = multiurl.robust(self.make_results, **retry_options)(self.url)
--> 246     raise ProcessingFailedError(error_json_to_message(results.json))
    247 elif status in ("accepted", "running"):
    248     sleep *= 1.5

ProcessingFailedError: job failed
MARS has returned an error, please check your selection.
Exception: 400 Client Error: Bad Request for url: http://mars-worker-2011.shared.compute.cci2.ecmwf.int:9000/
Request submitted to the MARS server:
[{'date': ['2012-12-01'], 'foo': 'bar', 'dataset': 'reanalysis', 'time': '12:00:00', 'param': '2t', 'class': ['ea'], 'expect': ['any'], 'number': ['all'], 'levtype': ['sfc']}]

Trace ID is cb49634c-2486-4f89-a6dd-a41fc554eedc