Closed stijnvanhoey closed 7 months ago
@stijnvanhoey Picking this up again 😅
main
branch (7 failed, 84 passed)@stijnvanhoey Picking this up again 😅
1. I don't understand why the tests in this PR are failing for the last commit (which is a typo correction). Dependency changes? 2. I tried locally, and the test are even failing on the `main` branch (7 failed, 84 passed)
Is probably due to adjustment in one of the dependencies as it seems to fail on the pytest fixture I setup to mock AWS calls, see https://github.com/enram/vptstools/blob/main/tests/conftest.py#L71-L98 as all failing tests define
self = <[AttributeError("'MockHttpClientResponse' object has no attribute '_cache'") raised in repr()] MockHttpClientResponse object at 0x7fd75aaf0e50>
or
E AttributeError: 'MockHttpClientResponse' object has no attribute '_loop'
When testing with aiobotocore=2.5.0 (via s3fs 2023.5.0), these tests pass. I fixed the dependency to check in CI
3. Once tests pass, how can I install and test the functionality of this PR?
afaik you can do custom runs of the vph5_to_vpts
command in the uat/prd versions of the AWS deployment and pass the command to run. So instead of the default command that runs daily (vph5_to_vpts --modified-days-ago 3
), you pass the command with the S3-path to run the conversion, e.g. vph5_to_vpts --path-s3-folder uva/hdf5/nldhl/2019
(see also vph5_to_vpts --help
)
I could not fully test the concept due to access rights, but I could check the reading part of the public bucket. This PR adds an option
path-s3-folder
to recreate all daily/monthlmy files within a given S3 path, e.g.vph5_to_vpts --path-s3-folder uva/hdf5/nldhl/2019
. It scans all folder recursively for h5 files and uses this as input to define the days to recreate vpts files.@peterdesmet certainly open to a better name for the new CLI option :-)