search

enram / vptstools

Python library to transfer and convert vertical profile time series data
https://enram.github.io/vptstools/
MIT License
3 stars 1 forks source link

Where to drop csv files? #13

Closed peterdesmet closed 1 year ago

peterdesmet commented 1 year ago

The directory consensus for files is (#https://github.com/enram/data-repository/issues/65#issuecomment-1108679530) is source/format/radar/yyyy/

I suggest:

# source data
baltrad/hdf5/radar/yyyy/mm/dd/ file.h5

# daily unzipped csv
baltrad/daily/radar/yyyy/ file.csv

# monthly gzipped csv
baltrad/monthly/radar/yyyy/ file.csv.gz
stijnvanhoey commented 1 year ago

@peterdesmet to abstract away the logic of the path, while keeping it all rely on the filename an additional class is added:

https://github.com/enram/vptstools/blob/d6ec680af9b29e86e75622cfda176f53e272df01/src/vptstools/vpts.py#L148-L232

It contains the logic for s3_folder_path_h5, s3_file_path_daily_vpts and s3_file_path_monthly_vpts as properties and the components are generated from the file name using the parse_file_name method (which can be used outside the context of the class as well to parse in other context as this is a staticmethod).

stijnvanhoey commented 1 year ago

Implemented in https://github.com/enram/vptstools/pull/19, closing