search

enram / vptstools

Python library to transfer and convert vertical profile time series data
https://enram.github.io/vptstools/
MIT License
3 stars 1 forks source link

Keyerror directory on `vph5_tp_vpts --path-s3-folder` #75

Closed PietrH closed 5 months ago

PietrH commented 5 months ago

In the v0.4.0-alpha.1 release

When running the command: 'vph5_to_vpts --path-s3-folder baltrad/hdf5/bejab/2018/05/18

We are getting errors: KeyError: 'directory'

It seems to me we are trying to locate a key in a dictionary that doesn't exist, however, the path does definitely exist: https://aloftdata.eu/browse/?prefix=baltrad/hdf5/bejab/2018/05/18/

@stijnvanhoey What could be going wrong?

Logs

Locally

(also no bucket id set)

Applying the vpts conversion to all files within baltrad/hdf5/bejab/2018/05/18. Ignoring the modified date of the files.
Create 0 daily VPTS files.
CLI routine 'vph5_to_vpts --path-s3-folder baltrad/hdf5/bejab/2018/05/18' failed raising error: '<class 'KeyError'>: 'directory''.
Traceback (most recent call last):
  File "/home/pieter/.local/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3802, in get_loc
    return self._engine.get_loc(casted_key)
  File "index.pyx", line 153, in pandas._libs.index.IndexEngine.get_loc
  File "index.pyx", line 182, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 7081, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 7089, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'directory'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/pieter/.local/lib/python3.10/site-packages/vptstools/bin/click_exception.py", line 40, in invoke
    return super(Cls, self).invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/pieter/.local/lib/python3.10/site-packages/vptstools/bin/vph5_to_vpts.py", line 155, in cli
    for j, daily_vpts in enumerate(days_to_create_vpts["directory"]):
  File "/home/pieter/.local/lib/python3.10/site-packages/pandas/core/frame.py", line 4090, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/pieter/.local/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3809, in get_loc
    raise KeyError(key) from err
KeyError: 'directory'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pieter/.local/bin/vph5_to_vpts", line 8, in <module>
    sys.exit(cli())
  File "/usr/lib/python3/dist-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/pieter/.local/lib/python3.10/site-packages/vptstools/bin/click_exception.py", line 47, in invoke
    handler(self, ctx.info_name, exc)
  File "/home/pieter/.local/lib/python3.10/site-packages/vptstools/bin/click_exception.py", line 96, in report_click_exception_to_sns
    report_message_to_sns(subject, sns_message, aws_sns_topic,
  File "/home/pieter/.local/lib/python3.10/site-packages/vptstools/bin/click_exception.py", line 74, in report_message_to_sns
    sns_client.publish(TopicArn=aws_sns_topic,
  File "/home/pieter/.local/lib/python3.10/site-packages/botocore/client.py", line 553, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/pieter/.local/lib/python3.10/site-packages/botocore/client.py", line 962, in _make_api_call
    request_dict = self._convert_to_request_dict(
  File "/home/pieter/.local/lib/python3.10/site-packages/botocore/client.py", line 1036, in _convert_to_request_dict
    request_dict = self._serializer.serialize_to_request(
  File "/home/pieter/.local/lib/python3.10/site-packages/botocore/validate.py", line 381, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter TopicArn, value: None, type: <class 'NoneType'>, valid types: <class 'str'>

AWS Log

05 March 2024 at 09:59 (UTC+1:00)   raise KeyError(key) from err    sync
05 March 2024 at 09:59 (UTC+1:00)   KeyError: 'directory'   sync
05 March 2024 at 09:59 (UTC+1:00)   indexer = self.columns.get_loc(key) sync
05 March 2024 at 09:59 (UTC+1:00)   File "/usr/local/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3654, in get_loc    sync
05 March 2024 at 09:59 (UTC+1:00)   return ctx.invoke(self.callback, **ctx.params)  sync
05 March 2024 at 09:59 (UTC+1:00)   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 760, in invoke    sync
05 March 2024 at 09:59 (UTC+1:00)   return __callback(*args, **kwargs)  sync
05 March 2024 at 09:59 (UTC+1:00)   File "/usr/local/lib/python3.9/site-packages/vptstools/bin/vph5_to_vpts.py", line 155, in cli   sync
05 March 2024 at 09:59 (UTC+1:00)   for j, daily_vpts in enumerate(days_to_create_vpts["directory"]):   sync
05 March 2024 at 09:59 (UTC+1:00)   File "/usr/local/lib/python3.9/site-packages/pandas/core/frame.py", line 3761, in __getitem__   sync
05 March 2024 at 09:59 (UTC+1:00)   KeyError: 'directory'   sync
05 March 2024 at 09:59 (UTC+1:00)   The above exception was the direct cause of the following exception:    sync
05 March 2024 at 09:59 (UTC+1:00)   Traceback (most recent call last):  sync
05 March 2024 at 09:59 (UTC+1:00)   File "/usr/local/bin/vph5_to_vpts", line 8, in <module> sync
05 March 2024 at 09:59 (UTC+1:00)   sys.exit(cli()) sync
05 March 2024 at 09:59 (UTC+1:00)   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__ sync
05 March 2024 at 09:59 (UTC+1:00)   return self.main(*args, **kwargs)   sync
05 March 2024 at 09:59 (UTC+1:00)   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1055, in main sync
05 March 2024 at 09:59 (UTC+1:00)   rv = self.invoke(ctx)   sync
05 March 2024 at 09:59 (UTC+1:00)   File "/usr/local/lib/python3.9/site-packages/vptstools/bin/click_exception.py", line 40, in invoke  sync
05 March 2024 at 09:59 (UTC+1:00)   return super(Cls, self).invoke(ctx) sync
05 March 2024 at 09:59 (UTC+1:00)   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke   sync
05 March 2024 at 09:59 (UTC+1:00)   return self._engine.get_loc(casted_key) sync
05 March 2024 at 09:59 (UTC+1:00)   File "pandas/_libs/index.pyx", line 147, in pandas._libs.index.IndexEngine.get_loc  sync
05 March 2024 at 09:59 (UTC+1:00)   File "pandas/_libs/index.pyx", line 176, in pandas._libs.index.IndexEngine.get_loc  sync
05 March 2024 at 09:59 (UTC+1:00)   File "pandas/_libs/hashtable_class_helper.pxi", line 7080, in pandas._libs.hashtable.PyObjectHashTable.get_item sync
05 March 2024 at 09:59 (UTC+1:00)   File "pandas/_libs/hashtable_class_helper.pxi", line 7088, in pandas._libs.hashtable.PyObjectHashTable.get_item sync
05 March 2024 at 09:59 (UTC+1:00)   Traceback (most recent call last):  sync
05 March 2024 at 09:59 (UTC+1:00)   File "/usr/local/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3652, in get_loc    sync
05 March 2024 at 09:59 (UTC+1:00)   Sent error message to SNS topic.    sync
05 March 2024 at 09:59 (UTC+1:00)   CLI routine 'vph5_to_vpts --path-s3-folder baltrad/hdf5/bejab/2018/05/18' failed raising error: '<class 'KeyError'>: 'directory''.  sync
05 March 2024 at 09:59 (UTC+1:00)   Create 0 daily VPTS files.  sync
05 March 2024 at 09:59 (UTC+1:00)   Applying the vpts conversion to all files within baltrad/hdf5/bejab/2018/05/18. Ignoring the modified date of the files.    sync
stijnvanhoey commented 5 months ago

@PietrH see https://github.com/enram/vptstools/pull/76