Applying the vpts conversion to all files within baltrad/hdf5/bejab/2018/05/18. Ignoring the modified date of the files.
Create 0 daily VPTS files.
CLI routine 'vph5_to_vpts --path-s3-folder baltrad/hdf5/bejab/2018/05/18' failed raising error: '<class 'KeyError'>: 'directory''.
Traceback (most recent call last):
File "/home/pieter/.local/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3802, in get_loc
return self._engine.get_loc(casted_key)
File "index.pyx", line 153, in pandas._libs.index.IndexEngine.get_loc
File "index.pyx", line 182, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 7081, in pandas._libs.hashtable.PyObjectHashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 7089, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'directory'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/pieter/.local/lib/python3.10/site-packages/vptstools/bin/click_exception.py", line 40, in invoke
return super(Cls, self).invoke(ctx)
File "/usr/lib/python3/dist-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python3/dist-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/pieter/.local/lib/python3.10/site-packages/vptstools/bin/vph5_to_vpts.py", line 155, in cli
for j, daily_vpts in enumerate(days_to_create_vpts["directory"]):
File "/home/pieter/.local/lib/python3.10/site-packages/pandas/core/frame.py", line 4090, in __getitem__
indexer = self.columns.get_loc(key)
File "/home/pieter/.local/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3809, in get_loc
raise KeyError(key) from err
KeyError: 'directory'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/pieter/.local/bin/vph5_to_vpts", line 8, in <module>
sys.exit(cli())
File "/usr/lib/python3/dist-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/usr/lib/python3/dist-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/home/pieter/.local/lib/python3.10/site-packages/vptstools/bin/click_exception.py", line 47, in invoke
handler(self, ctx.info_name, exc)
File "/home/pieter/.local/lib/python3.10/site-packages/vptstools/bin/click_exception.py", line 96, in report_click_exception_to_sns
report_message_to_sns(subject, sns_message, aws_sns_topic,
File "/home/pieter/.local/lib/python3.10/site-packages/vptstools/bin/click_exception.py", line 74, in report_message_to_sns
sns_client.publish(TopicArn=aws_sns_topic,
File "/home/pieter/.local/lib/python3.10/site-packages/botocore/client.py", line 553, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/pieter/.local/lib/python3.10/site-packages/botocore/client.py", line 962, in _make_api_call
request_dict = self._convert_to_request_dict(
File "/home/pieter/.local/lib/python3.10/site-packages/botocore/client.py", line 1036, in _convert_to_request_dict
request_dict = self._serializer.serialize_to_request(
File "/home/pieter/.local/lib/python3.10/site-packages/botocore/validate.py", line 381, in serialize_to_request
raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter TopicArn, value: None, type: <class 'NoneType'>, valid types: <class 'str'>
AWS Log
05 March 2024 at 09:59 (UTC+1:00) raise KeyError(key) from err sync
05 March 2024 at 09:59 (UTC+1:00) KeyError: 'directory' sync
05 March 2024 at 09:59 (UTC+1:00) indexer = self.columns.get_loc(key) sync
05 March 2024 at 09:59 (UTC+1:00) File "/usr/local/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3654, in get_loc sync
05 March 2024 at 09:59 (UTC+1:00) return ctx.invoke(self.callback, **ctx.params) sync
05 March 2024 at 09:59 (UTC+1:00) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 760, in invoke sync
05 March 2024 at 09:59 (UTC+1:00) return __callback(*args, **kwargs) sync
05 March 2024 at 09:59 (UTC+1:00) File "/usr/local/lib/python3.9/site-packages/vptstools/bin/vph5_to_vpts.py", line 155, in cli sync
05 March 2024 at 09:59 (UTC+1:00) for j, daily_vpts in enumerate(days_to_create_vpts["directory"]): sync
05 March 2024 at 09:59 (UTC+1:00) File "/usr/local/lib/python3.9/site-packages/pandas/core/frame.py", line 3761, in __getitem__ sync
05 March 2024 at 09:59 (UTC+1:00) KeyError: 'directory' sync
05 March 2024 at 09:59 (UTC+1:00) The above exception was the direct cause of the following exception: sync
05 March 2024 at 09:59 (UTC+1:00) Traceback (most recent call last): sync
05 March 2024 at 09:59 (UTC+1:00) File "/usr/local/bin/vph5_to_vpts", line 8, in <module> sync
05 March 2024 at 09:59 (UTC+1:00) sys.exit(cli()) sync
05 March 2024 at 09:59 (UTC+1:00) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__ sync
05 March 2024 at 09:59 (UTC+1:00) return self.main(*args, **kwargs) sync
05 March 2024 at 09:59 (UTC+1:00) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1055, in main sync
05 March 2024 at 09:59 (UTC+1:00) rv = self.invoke(ctx) sync
05 March 2024 at 09:59 (UTC+1:00) File "/usr/local/lib/python3.9/site-packages/vptstools/bin/click_exception.py", line 40, in invoke sync
05 March 2024 at 09:59 (UTC+1:00) return super(Cls, self).invoke(ctx) sync
05 March 2024 at 09:59 (UTC+1:00) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke sync
05 March 2024 at 09:59 (UTC+1:00) return self._engine.get_loc(casted_key) sync
05 March 2024 at 09:59 (UTC+1:00) File "pandas/_libs/index.pyx", line 147, in pandas._libs.index.IndexEngine.get_loc sync
05 March 2024 at 09:59 (UTC+1:00) File "pandas/_libs/index.pyx", line 176, in pandas._libs.index.IndexEngine.get_loc sync
05 March 2024 at 09:59 (UTC+1:00) File "pandas/_libs/hashtable_class_helper.pxi", line 7080, in pandas._libs.hashtable.PyObjectHashTable.get_item sync
05 March 2024 at 09:59 (UTC+1:00) File "pandas/_libs/hashtable_class_helper.pxi", line 7088, in pandas._libs.hashtable.PyObjectHashTable.get_item sync
05 March 2024 at 09:59 (UTC+1:00) Traceback (most recent call last): sync
05 March 2024 at 09:59 (UTC+1:00) File "/usr/local/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3652, in get_loc sync
05 March 2024 at 09:59 (UTC+1:00) Sent error message to SNS topic. sync
05 March 2024 at 09:59 (UTC+1:00) CLI routine 'vph5_to_vpts --path-s3-folder baltrad/hdf5/bejab/2018/05/18' failed raising error: '<class 'KeyError'>: 'directory''. sync
05 March 2024 at 09:59 (UTC+1:00) Create 0 daily VPTS files. sync
05 March 2024 at 09:59 (UTC+1:00) Applying the vpts conversion to all files within baltrad/hdf5/bejab/2018/05/18. Ignoring the modified date of the files. sync
In the v0.4.0-alpha.1 release
When running the command:
'vph5_to_vpts --path-s3-folder baltrad/hdf5/bejab/2018/05/18
We are getting errors:
KeyError: 'directory'
It seems to me we are trying to locate a key in a dictionary that doesn't exist, however, the path does definitely exist: https://aloftdata.eu/browse/?prefix=baltrad/hdf5/bejab/2018/05/18/
@stijnvanhoey What could be going wrong?
Logs
Locally
(also no bucket id set)
AWS Log