Microphone data is not stored in data-gateway-processed-data bucket.

time-trader commented 2 years ago

Bug report

What is the current behavior?

Microphone data still doesn't reach data-gateway-processed-data bucket. Possibly due to #45 ?

This causes cloud function error as it cannot locate microphone files:

Traceback (most recent call last):
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 2073, in wsgi_app
    response = self.full_dispatch_request()
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1518, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1516, in full_dispatch_request
    rv = self.dispatch_request()
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1502, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/functions_framework/__init__.py", line 171, in view_func
    function(data, context)
  File "/workspace/main.py", line 41, in upload_window
    window_handler.persist_window(unix_timestamped_window["sensor_data"], window_metadata)
  File "/workspace/window_handler.py", line 92, in persist_window
    self._store_microphone_data(
  File "/workspace/window_handler.py", line 141, in _store_microphone_data
    with datafile.open("w") as f:
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/octue/resources/datafile.py", line 688, in __enter__
    os.makedirs(os.path.split(self.datafile.local_path)[0], exist_ok=True)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/octue/resources/datafile.py", line 454, in local_path
    return self.download()
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/octue/resources/datafile.py", line 440, in download
    raise e
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/octue/resources/datafile.py", line 433, in download
    GoogleCloudStorageClient(project_name=self.project_name).download_to_file(
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/octue/cloud/storage/client.py", line 173, in download_to_file
    blob.download_to_filename(local_path, timeout=timeout)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/storage/blob.py", line 1281, in download_to_filename
    client.download_blob_to_file(
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/storage/client.py", line 1152, in download_blob_to_file
    _raise_from_invalid_response(exc)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/storage/blob.py", line 4466, in _raise_from_invalid_response
    raise exceptions.from_http_status(response.status_code, message, response=response)
google.api_core.exceptions.NotFound: 404 GET https://storage.googleapis.com/download/storage/v1/b/data-gateway-processed-data/o/microphone%2F790791%2Fwindow-343.hdf5?alt=media: No such object: data-gateway-processed-data/microphone/790791/window-343.hdf5: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)

And surely goes hand in hand with another new error:

InvalidResponse: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)

 Traceback (most recent call last):
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/storage/client.py", line 1139, in download_blob_to_file
    blob_or_uri._do_download(
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/storage/blob.py", line 1001, in _do_download
    response = download.consume(transport, timeout=timeout)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/resumable_media/requests/download.py", line 214, in consume
    return _request_helpers.wait_and_retry(
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/resumable_media/requests/_request_helpers.py", line 147, in wait_and_retry
    response = func()
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/resumable_media/requests/download.py", line 207, in retriable_request
    self._process_response(result)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/resumable_media/_download.py", line 188, in _process_response
    _helpers.require_status_code(
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/resumable_media/_helpers.py", line 105, in require_status_code
    raise common.InvalidResponse(
google.resumable_media.common.InvalidResponse: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)

Your environment

Gateway Version: 0.11.5
Platform Linux

Other information

Please give as much detail as you can, like:

[x] detailed explanation
[x] stacktraces
[x] related issues
[ ] suggestions how to fix
[ ] links for us to have context, eg. stackoverflow, gitter, etc

cortadocodes commented 2 years ago

This is caused by a conceptual debt in the SDK (see https://github.com/octue/octue-sdk-python/issues/407) - I'll fix this and get it into the gateway.

cortadocodes commented 2 years ago

This may be caused more by the fact that an old version of the SDK is being used for the cloud functions. I think microphone data upload might be untested too, so I'll upgrade and test.

aerosense-ai / data-gateway