Add files in current path also to filelist

[stijnvanhoey]

As reported in https://github.com/enram/vptstools/issues/75 files in the current filepath enlisted are not found (the glob of s3fs behaves a bit different than a regular glob). This PR adds the current files to the file generator and provides a more useful information message when the routine fails on absence of files.

Note: the specific example of the issue vph5_to_vpts --path-s3-folder baltrad/hdf5/bejab/2018/05/18 still fails, but this is due to the fact that the H5 files are not according the expected format as defined in the VptsCsvV1 (v1) mapping. The files are missing certain fields. For example the file 'bejab_vp_20180518T101500Z_0x5.h5':

• The h5 file 'root_source' does not contain an "NOD" key - {'WMO': '06410', 'RAD': 'BX42', 'PLC': 'Jabbeke'}
• The h5 file 'how' does not contain a "vcp" key -

  {'beamwidth': 1.0,
  'clutterMap': '',
  'comment': '',
  'dealiased': 0,
  'enddate': '20180518',
  'endtime': '101910',
  'filename_pvol': '/data/baltrad/projects/vol2bird/data/out/merged/bejab_pvol_20180518T101500Z_0x5.h5',
  'filename_vp': '/data/baltrad/projects/vol2bird/data/out/bejab_vp_20180518T101500Z_0x5.h5',
  'maxazim': 360.0,
  'maxrange': 25.0,
  'minazim': 0.0,
  'minrange': 5.0,
  'rcs_bird': 11.0,
  'sd_vvp_thresh': 2.0,
  'startdate': '20180518',
  'starttime': '101505',
  'task': 'vol2bird',
  'task_args':  ...<LONG_SECTION>,
  'task_version': '0.3.18',
  'wavelength': 5.333000183105469}

If I run the operation on a more recent folder, e.g. vph5_to_vpts --path-s3-folder baltrad/hdf5/bejab/2022/10/02, the conversion itself does run correctly.

Remark that you probably rather run it on a higher/nested level anyway, e.g. vph5_to_vpts --path-s3-folder baltrad/hdf5/bejab/2022/10 will create all the daily files of the month Ocotber 2022 in a single command or vph5_to_vpts --path-s3-folder baltrad/hdf5/bejab/2022 all of the year 2022. This is more convenient than running it day by day yoruself.

[stijnvanhoey]

@peterdesmet the unit tests are failing as the patch of https://github.com/enram/vptstools/pull/74 need to be included first for the mock-s3 library.

[PietrH]

tests on pinned dependencies failed on merging #74: https://github.com/enram/vptstools/actions/runs/8201007562/job/22428917910 because cannot import name 'mock_s3' from 'moto'

• What is this test for
• Why does it run on release, not on PR?
• Is this a blocker for the pre-release

[PietrH]

What happens when a single H5 file is invalid in a vph5_to_vpts --path-s3-folder? Does the whole folder fail?

Or would they always all be invalid as soon as one is invalid?

[stijnvanhoey]

What happens when a single H5 file is invalid in a vph5_to_vpts --path-s3-folder? Does the whole folder fail?

Or would they always all be invalid as soon as one is invalid?

When a single h5 file is invalid, the daily-file of the vpts-file that h5 file is part of will fail, but other daily files will still be generated. A Warning will be sent to the logs to inform about the failure, see https://github.com/enram/vptstools/blob/main/src/vptstools/bin/vph5_to_vpts.py#L193-L195

[stijnvanhoey]

tests on pinned dependencies failed on merging #74: https://github.com/enram/vptstools/actions/runs/8201007562/job/22428917910 because cannot import name 'mock_s3' from 'moto'

• What is this test for The pinned version is to specifically check with the versions of the dependencies as they are releases/deployed on AWS.

• Why does it run on release, not on PR? We only deploy releases to AWS, so this step is a safeguard to make sure a release is not deployed to AWS when this check fails.

• Is this a blocker for the pre-release Not for a pre-release. But i do think https://github.com/enram/vptstools/pull/76/commits/8d127c1a3119a032c2c0ce59ec42d3ffc006a508 will fix it

[PietrH]

8d127c1a3119a032c2c0ce59ec42d3ffc006a508 didn't fix it, and instead of using a workflow trigger, I merged. Sorry.

test on pinned is still failing, /home/runner/work/_temp/6d4f39ee-94eb-4c5d-808c-5b737e23657e.sh: line 4: 5.0: No such file or directory

I believe it's failing on: python -m pip install pytest moto[s3]<5.0 pytest-cov

Should I revert?

[PietrH]

I think the solution might be python -m pip install pytest 'moto[s3]<5.0' pytest-cov