search

enram / vptstools

Python library to transfer and convert vertical profile time series data
https://enram.github.io/vptstools/
MIT License
3 stars 1 forks source link

Add CLI option to use the s3 file enlisting instead of the modified date to create daily/monthly files #70

Closed stijnvanhoey closed 7 months ago

stijnvanhoey commented 11 months ago

I could not fully test the concept due to access rights, but I could check the reading part of the public bucket. This PR adds an option path-s3-folder to recreate all daily/monthlmy files within a given S3 path, e.g. vph5_to_vpts --path-s3-folder uva/hdf5/nldhl/2019. It scans all folder recursively for h5 files and uses this as input to define the days to recreate vpts files.

@peterdesmet certainly open to a better name for the new CLI option :-)

peterdesmet commented 8 months ago

@stijnvanhoey Picking this up again 😅

  1. I don't understand why the tests in this PR are failing for the last commit (which is a typo correction). Dependency changes?
  2. I tried locally, and the test are even failing on the main branch (7 failed, 84 passed)
  3. Once tests pass, how can I install and test the functionality of this PR?
stijnvanhoey commented 8 months ago

@stijnvanhoey Picking this up again 😅

1. I don't understand why the tests in this PR are failing for the last commit (which is a typo correction). Dependency changes?

2. I tried locally, and the test are even failing on the `main` branch (7 failed, 84 passed)

Is probably due to adjustment in one of the dependencies as it seems to fail on the pytest fixture I setup to mock AWS calls, see https://github.com/enram/vptstools/blob/main/tests/conftest.py#L71-L98 as all failing tests define

self = <[AttributeError("'MockHttpClientResponse' object has no attribute '_cache'") raised in repr()] MockHttpClientResponse object at 0x7fd75aaf0e50>

or

 E       AttributeError: 'MockHttpClientResponse' object has no attribute '_loop'

When testing with aiobotocore=2.5.0 (via s3fs 2023.5.0), these tests pass. I fixed the dependency to check in CI

stijnvanhoey commented 8 months ago 
3. Once tests pass, how can I install and test the functionality of this PR?

afaik you can do custom runs of the vph5_to_vpts command in the uat/prd versions of the AWS deployment and pass the command to run. So instead of the default command that runs daily (vph5_to_vpts --modified-days-ago 3), you pass the command with the S3-path to run the conversion, e.g. vph5_to_vpts --path-s3-folder uva/hdf5/nldhl/2019 (see also vph5_to_vpts --help)