Closed stijnvanhoey closed 5 months ago
@peterdesmet the unit tests are failing as the patch of https://github.com/enram/vptstools/pull/74 need to be included first for the mock-s3 library.
tests on pinned dependencies failed on merging #74: https://github.com/enram/vptstools/actions/runs/8201007562/job/22428917910 because cannot import name 'mock_s3' from 'moto'
What happens when a single H5 file is invalid in a vph5_to_vpts --path-s3-folder
? Does the whole folder fail?
Or would they always all be invalid as soon as one is invalid?
What happens when a single H5 file is invalid in a
vph5_to_vpts --path-s3-folder
? Does the whole folder fail?Or would they always all be invalid as soon as one is invalid?
When a single h5 file is invalid, the daily-file of the vpts-file that h5 file is part of will fail, but other daily files will still be generated. A Warning will be sent to the logs to inform about the failure, see https://github.com/enram/vptstools/blob/main/src/vptstools/bin/vph5_to_vpts.py#L193-L195
tests on pinned dependencies failed on merging #74: https://github.com/enram/vptstools/actions/runs/8201007562/job/22428917910 because
cannot import name 'mock_s3' from 'moto'
What is this test for The pinned version is to specifically check with the versions of the dependencies as they are releases/deployed on AWS.
Why does it run on release, not on PR? We only deploy releases to AWS, so this step is a safeguard to make sure a release is not deployed to AWS when this check fails.
Is this a blocker for the pre-release Not for a pre-release. But i do think https://github.com/enram/vptstools/pull/76/commits/8d127c1a3119a032c2c0ce59ec42d3ffc006a508 will fix it
8d127c1a3119a032c2c0ce59ec42d3ffc006a508 didn't fix it, and instead of using a workflow trigger, I merged. Sorry.
test on pinned is still failing, /home/runner/work/_temp/6d4f39ee-94eb-4c5d-808c-5b737e23657e.sh: line 4: 5.0: No such file or directory
I believe it's failing on: python -m pip install pytest moto[s3]<5.0 pytest-cov
Should I revert?
I think the solution might be python -m pip install pytest 'moto[s3]<5.0' pytest-cov
As reported in https://github.com/enram/vptstools/issues/75 files in the current filepath enlisted are not found (the glob of s3fs behaves a bit different than a regular glob). This PR adds the current files to the file generator and provides a more useful information message when the routine fails on absence of files.
Note: the specific example of the issue
vph5_to_vpts --path-s3-folder baltrad/hdf5/bejab/2018/05/18
still fails, but this is due to the fact that the H5 files are not according the expected format as defined in theVptsCsvV1
(v1) mapping. The files are missing certain fields. For example the file 'bejab_vp_20180518T101500Z_0x5.h5':{'WMO': '06410', 'RAD': 'BX42', 'PLC': 'Jabbeke'}
If I run the operation on a more recent folder, e.g.
vph5_to_vpts --path-s3-folder baltrad/hdf5/bejab/2022/10/02
, the conversion itself does run correctly.Remark that you probably rather run it on a higher/nested level anyway, e.g.
vph5_to_vpts --path-s3-folder baltrad/hdf5/bejab/2022/10
will create all the daily files of the month Ocotber 2022 in a single command orvph5_to_vpts --path-s3-folder baltrad/hdf5/bejab/2022
all of the year 2022. This is more convenient than running it day by day yoruself.