search

enram / vptstools

Python library to transfer and convert vertical profile time series data
https://enram.github.io/vptstools/
MIT License
3 stars 1 forks source link

AWS error references deleted file #62

Closed peterdesmet closed 1 year ago

peterdesmet commented 1 year ago

I got an AWS notification email today:

CLI routine 'vph5_to_vpts --modified-days-ago 2' failed raising error: '<class 'ValueError'>: File name uva/hdf5/dbl/2008/02/17/nldbl_vp_20080217t0000_nl50_v0-3-20.h5 is not a valid ODIM h5 file.'.

It now includes the name of the file 👍. The mentioned file is however no longer in the repository. It was deleted August 21 or 22 (2 or 1 days ago). I'll see tomorrow if the issue resolves itself, i.e. the inventory is updated and the file is no longer listed there and no notification is generated.

@stijnvanhoey @TheJenne18 can deleted files linger in the inventory? Should this resolve itself automatically? Not sure we considered this when designing the architecture.

Note this error did not stop the creation of the daily and monthly files 👍

stijnvanhoey commented 1 year ago

Glad the logging adjustments are working.

Up to now I did not consider this scenario of files no longer existing in the S3 bucket. Not sure what the inventory is doing with removed files and if these willl be no longer enlisted in a next inventory. @TheJenne18 ?

TheJenne18 commented 1 year ago

The inventory system in Amazon S3 provides a snapshot of the objects within a bucket at a specific point in time. When a file is deleted, it will not appear in the inventories generated after the deletion.

However, if the file was deleted recently (in this case on August 21 or 22), and the inventory with the CLI routine 'vph5_to_vpts --modified-days-ago 2' was performed during that time frame, it might still include the now-deleted file.

peterdesmet commented 1 year ago

Note: I got the another notification for the same file tonight. Will see if I get it tomorrow.

peterdesmet commented 1 year ago

Ok, mystery solved: those notifications were send from the UAT environment, where those files are not deleted.