'viirs_sdr' filename `end_time` not handled properly around midnight

[arturimgw]

Describe the bug Although there are files in the directory that meet the given time criteria the find_files_and_readers return ValueError: No supported files found. I observed it only for some times close to midnight, e.g. 23:51

files:

GDNBO_j01_d20220619_t2351438_e0003050_b23759_c20220620001612399860_cspp_dev.h5 GIMGO_j01_d20220619_t2351438_e0003050_b23759_c20220620001613080841_cspp_dev.h5 GITCO_j01_d20220619_t2351438_e0003050_b23759_c20220620001612026996_cspp_dev.h5 GMODO_j01_d20220619_t2351438_e0003050_b23759_c20220620001618513659_cspp_dev.h5 GMTCO_j01_d20220619_t2351438_e0003050_b23759_c20220620001612762736_cspp_dev.h5 IVCDB_j01_d20220619_t2351438_e0003050_b23759_c20220620001625276661_cspp_dev.h5 SVDNB_j01_d20220619_t2351438_e0003050_b23759_c20220620001625489783_cspp_dev.h5 SVI01_j01_d20220619_t2351438_e0003050_b23759_c20220620001623557681_cspp_dev.h5 SVI02_j01_d20220619_t2351438_e0003050_b23759_c20220620001624553445_cspp_dev.h5 SVI03_j01_d20220619_t2351438_e0003050_b23759_c20220620001623558110_cspp_dev.h5 SVI04_j01_d20220619_t2351438_e0003050_b23759_c20220620001624553257_cspp_dev.h5 SVI05_j01_d20220619_t2351438_e0003050_b23759_c20220620001623559782_cspp_dev.h5 SVM01_j01_d20220619_t2351438_e0003050_b23759_c20220620001624604441_cspp_dev.h5 SVM02_j01_d20220619_t2351438_e0003050_b23759_c20220620001623551455_cspp_dev.h5 SVM03_j01_d20220619_t2351438_e0003050_b23759_c20220620001623653962_cspp_dev.h5 SVM04_j01_d20220619_t2351438_e0003050_b23759_c20220620001624555074_cspp_dev.h5 SVM05_j01_d20220619_t2351438_e0003050_b23759_c20220620001624661236_cspp_dev.h5 SVM06_j01_d20220619_t2351438_e0003050_b23759_c20220620001624706478_cspp_dev.h5 SVM07_j01_d20220619_t2351438_e0003050_b23759_c20220620001624801815_cspp_dev.h5 SVM08_j01_d20220619_t2351438_e0003050_b23759_c20220620001624764398_cspp_dev.h5 SVM09_j01_d20220619_t2351438_e0003050_b23759_c20220620001625014005_cspp_dev.h5 SVM10_j01_d20220619_t2351438_e0003050_b23759_c20220620001624770840_cspp_dev.h5 SVM11_j01_d20220619_t2351438_e0003050_b23759_c20220620001625003619_cspp_dev.h5 SVM12_j01_d20220619_t2351438_e0003050_b23759_c20220620001625082861_cspp_dev.h5 SVM13_j01_d20220619_t2351438_e0003050_b23759_c20220620001625334705_cspp_dev.h5 SVM14_j01_d20220619_t2351438_e0003050_b23759_c20220620001625116759_cspp_dev.h5 SVM15_j01_d20220619_t2351438_e0003050_b23759_c20220620001625368220_cspp_dev.h5 SVM16_j01_d20220619_t2351438_e0003050_b23759_c20220620001625252645_cspp_dev.h5

To Reproduce

from satpy import find_files_and_readers
from datetime import datetime
reader='viirs_sdr'
files = find_files_and_readers(base_dir='/home/troll/scripts/test_data', reader=reader,start_time=datetime(2022,6,19,23,50), end_time=datetime(2022,6,19,23,52))

after changing the start_time=datetime(2022,6,19,23,50) parameter to start_time=datetime(2022,6,19,0,3) or earlier files are found.

Expected behavior Files should be found.

Actual results Text output of actual results or error messages including full tracebacks if applicable.

Traceback (most recent call last):

  Input In [57] in <cell line: 1>
    runfile('/home/troll/scripts/test_data/test_time_viirs2.py', wdir='/home/troll/scripts/test_data')

  File ~/anaconda3/envs/pytroll/lib/python3.9/site-packages/debugpy/_vendored/pydevd/_pydev_bundle/pydev_umd.py:175 in runfile
    execfile(filename, namespace)

  File ~/anaconda3/envs/pytroll/lib/python3.9/site-packages/debugpy/_vendored/pydevd/_pydev_bundle/_pydev_execfile.py:25 in execfile
    exec(compile(contents + "\n", file, 'exec'), glob, loc)

  File ~/scripts/test_data/test_time_viirs2.py:12 in <module>
    files = find_files_and_readers(base_dir='/home/troll/scripts/test_data', reader=reader,start_time=datetime(2022,6,19,23,50), end_time=datetime(2022,6,19,23,52))

  File ~/anaconda3/envs/pytroll/lib/python3.9/site-packages/satpy/readers/__init__.py:483 in find_files_and_readers
    raise ValueError("No supported files found")

ValueError: No supported files found

Screenshots If applicable, add screenshots to help explain your problem.

Environment Info:

• OS: Ubuntu 20.04 LTS
• Satpy Version: 0.36.0]
• PyResample Version: 1.23.0
• Readers and writers dependencies (when relevant): [run from satpy.utils import check_satpy; check_satpy()]

Additional context Add any other context about the problem here.

[djhoese]

Thanks for filing this issue. As mentioned on slack, the main issue here is that VIIRS SDR filenames don't technically include a date for the end of the file, only a time:

d20220619_t2351438_e0003050

So the start time is 2022-06-19 23:51:43.8 and the end time is ????-??-?? 00:03:05.0. To get around this, Satpy currently does this:

https://github.com/pytroll/satpy/blob/bafe54e1d3c598f1d70b616e8d27bb2b25c62190/satpy/readers/yaml_reader.py#L563-L567

Meaning, take the date from the start time and use that as the date for the end time as well. This doesn't work when the end of the file is in the next day. It should be a relatively simple fix on the code side and it is definitely a bug.

@TAlonglong had mentioned that @mraspaud had a fix for this in another pytroll package that handles viirs files. Perhaps when he is back from holiday he can copy and paste the code here so we can include it in satpy.

[TAlonglong]

https://github.com/pytroll/pytroll-collectors/blob/main/pytroll_collectors/region_collector.py#L299-L307

[djhoese]

@TAlonglong or @arturimgw either of you want to try your hand at a pull request for this?