NOAA-PSL / observation-archive

Tools to assemble observational archive record
Apache License 2.0
0 stars 0 forks source link

merge files from multiple sources to a single destination. #8

Closed frolovsa closed 2 years ago

frolovsa commented 2 years ago

Closes #3 and #5

This PR implements a complete functionality: multiple sources are specified and copied over to the destination following the order in which they were specified. E.g. if the top choice is missing the next one is executed.

Log files are generated specifying exactly which files were copied over. In the follow up PR i will (optionally) upload log files to the destination bucket.

One needs to specify aws credentials in ~/.aws/credentials to enable the write to the destination.

For reviews, i chose to write this as a very linear script for two reasons (1) it is easy to understand and (2) this is a software with a limited shelf life and i decided not to over engineer it. The downside of this approach is that i can not implement proper unit tests. To confirm that things are working, i executed the script and i get this listing of the destination folder:

aws s3 --no-sign-request --region=us-east-1 ls s3://noaa-reanalyses-pds/observations/reanalysis-test/conv/gefs-prepbufr/1993/01/prepbufr/

2022-08-17 15:29:41    3545856 gdas.19930101.t00z.prepfur.nr
2022-08-17 15:29:41    2041624 gdas.19930101.t06z.prepfur.nr
2022-08-17 15:29:42    2948944 gdas.19930101.t12z.prepfur.nr
2022-08-17 15:29:43    2172824 gdas.19930101.t18z.prepfur.nr
2022-08-17 15:29:44    3334552 gdas.19930102.t00z.prepfur.nr

I am also attaching the log files from this run

attached are the log files: 20220817155331-2-cfsr_correction-missing.log 20220817155331-success.log 20220817155331-0-gefs_blend_with_error-missing.log 20220817155331-1-gefs_blend-missing.log

frolovsa commented 2 years ago

@HenryWinterbottom-NOAA thank you for your review. Sounds like there are a couple of things i can learn from you if you explain some of your reasoning a bit further. See my replies above.

frolovsa commented 2 years ago

Thank you for the discussion @HenryWinterbottom-NOAA