Move and adjust the s3 inventory

enram / vptstools

Python library to transfer and convert vertical profile time series data

https://enram.github.io/vptstools/

MIT License

4 stars 1 forks source link

Move and adjust the s3 inventory #25

Open stijnvanhoey opened 1 year ago

stijnvanhoey commented 1 year ago

[x] create new bucket "aloft-inventory" and move inventory their
[x] use the CSV inventory (not the parquet one)
[ ] add the functionality to remove old inventory files (from manifest -> remove files)
[x] inventory need to be applied for all 'sources' (baltrad, ecol,...)
[x] the file name is used as single source of truth and the file name <-> s3-path logic is a function that can be passed (injected) to the routine.
[x] assign ec2 engine rights to access aloft-inventory bucket

stijnvanhoey commented 1 year ago

New bucket is created and inventory configuration switched.

inventory need to be applied for all 'sources' (baltrad, ecol,...)

The setup is done that it looks for */hdf5 as prefix of the path; need to verify that this effectively works.

peterdesmet commented 1 year ago

@stijnvanhoey there isn't necessarily an issue to include daily and monthly in that inventory. Those could also be included in the coverage file. It's fairly easy for the user to filter them out

stijnvanhoey commented 1 year ago

As the inventory is not working with the *-wildcard, w"ll manage without the daily/monthly and filter them on the data routine.

stijnvanhoey commented 1 year ago

add the functionality to remove old inventory files (from manifest -> remove files)

Will be tackled by @TheJenne18 as part of the updated deployment