Swell couldn't handle missing observation data files during fetching.

[gmao-wgu]

I purposely removed the IASI Metop-B and Metop-C observation data files from the R2D2 location, and it crashed during fetching. The error message in the job.err file within the GetObservations-geos_atmosphere directory is as follows:

File "/discover/nobackup/projects/gmao/advda/JediOpt/core/r2d2/sles12_skylab7/lib/python3.10/site-packages/r2d2/queries/fetch.py", line 23, in action database.fetch_data(source, target, request, archive_side_request) File "/discover/nobackup/projects/gmao/advda/JediOpt/core/r2d2/sles12_skylab7/lib/python3.10/site-packages/r2d2/database/db_manager.py", line 87, in fetch_data raise FileNotFoundError(f'Could not find {source} in {" or ".join(self.fetch_order)}') FileNotFoundError: Could not find ncdiag.x0048v2.ob.PT6H.iasi_metop-b.2021-12-11T21:00:00Z.nc4 in wgu or gmao-shared 2024-11-20T16:51:24Z CRITICAL - failed/ERR

[Dooruk]

Missing files handling has multiple steps: 1) r2d2 fetch should ignore missing obs, 2) get_observations.py task should handle missing observations as it serves multiple purposes beyond fetching obs such as combining obs if needed, varBC, and aircraftBC cycling. Hence, this multi-purpose design requires careful modification, 3) UFO filters (which there are multiple PRs in SWELL currently)

I'm working on number 1 (this one is handled) and 2 (it broke atmosphere side) in the following PR as ODAS has lot of missing obs (due to temporal data sparsity):

https://github.com/GEOS-ESM/swell/pull/444

FYI @rtodling @mranst @ashiklom for visibility

[gmao-wgu]

@Dooruk While working on number 1, is it possible for a zero-length NC4 file to be created if an observation file is missing? this zero-length file could then be fetched and tested in UFO filters.