Best way to open RHI Cfradial file

[aladinor]

• xradar version: 0.4.0
• Python version: 3.10
• Operating System: ubuntu

Hi everyone! I am trying to open a RHI file from DOW8 using xradar but I got a StopIteration error.

import xradar as xd
from open_radar_data import DATASETS

def main():
    filepath = DATASETS.fetch('cfrad.20211011_201733.023_to_20211011_201745.299_DOW8_RHI.nc')
    rhi_dt = xd.io.open_cfradial1_datatree(filepath)
    print(1)

if __name__ == "__main__":
    main()

and this is the error I get when running this short script

Traceback (most recent call last):
  File "/snap/pycharm-community/352/plugins/python-ce/helpers/pydev/pydevd.py", line 1500, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/snap/pycharm-community/352/plugins/python-ce/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/media/alfonso/drive/Alfonso/python/raw2zarr/src/del.py", line 12, in <module>
    main()
  File "/media/alfonso/drive/Alfonso/python/raw2zarr/src/del.py", line 7, in main
    rhi_dt = xd.io.open_cfradial1_datatree(filepath)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alfonso/mambaforge/envs/atmoscol2023/lib/python3.11/site-packages/xradar/io/backends/cfradial1.py", line 344, in open_cfradial1_datatree
    calib = _get_radar_calibration(ds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alfonso/mambaforge/envs/atmoscol2023/lib/python3.11/site-packages/xradar/io/backends/cfradial1.py", line 280, in _get_radar_calibration
    item = next(
           ^^^^^
StopIteration

Please let me know your thoughts.

Cheers,

Alfonso.

[aladinor]

I think the problem is when creating the Datatree object. If we open the file using

ds = xr.open_dataset(filepath, engine='netcdf4')

it works

[mgrover1]

I wonder if it has something to do with the sweep groups within the file; I just cut a new release of open-radar-data https://github.com/openradar/open-radar-data with the dataset. We should test this and find out a solution. I have tested other RHI files with xradar which function properly.

[aladinor]

I tried with the PPI file and the error is the same cfrad.20211011_201711.345_to_20211011_201732.860_DOW8_PPI.nc. Seems like something is happening when creating datatree using cfradial files

[kmuehlbauer]

The error is raised while extracting calibration subgroup items. The StopIteration is raised, when the iterator is exhausted. So something is wrong with the calibration subgroup items.

[kmuehlbauer]

@aladinor A quick workaround to at least read the file as datatree is wrapping the next in try/except and continue the for loop if that item isn't found:

https://github.com/openradar/xradar/blob/19487ead4813cd57e3a39d9dfcfe049077b447f5/xradar/io/backends/cfradial1.py#L278-L283

        for name in subgroup.data_vars:
            try:
                item = next(
                    filter(lambda x: x[0] in name, radar_calibration_subgroup.items())
                )
            except StopIteration:
                continue
            item = item[1] if item[1] else item[0]
            calib_vars[name] = item

Need to check, if that could be a permanent fix or if we have to be a bit smarter.

[kmuehlbauer]

@aladinor The root cause is missing items in xradar.model.radar_calibration_subgroup. Good news is, the missing items have already been added in #132 by @syedhamidali.

So, please update xradar to at least 0.4.1 (better 0.4.2).

Nevertheless, we should implement some fix along my above suggestion. That way we can at least warn the user that there are mismatched/new items in his file which do not get imported. I'd really only import items which are referenced in the CfRadial1 standard and not trying to import everything.

[syedhamidali]

@aladinor can you share this RHI file? We don’t have to use try and except but to add the missing metadata to the model/backends.

[aladinor]

Sure!

@syedhamidali can get these files from this google drive folder or by grabbing them directly from open-radar-data package.

from open_radar_data import DATASETS

filepath = DATASETS.fetch('cfrad.20211011_201733.023_to_20211011_201745.299_DOW8_RHI.nc')
rhi_dt = xd.io.open_cfradial1_datatree(filepath)

[kmuehlbauer]

@syedhamidali @aladinor this runs for me without the try/except using xradar 0.4.2. As I said @syedhamidali added the missing parts in #132.

[aladinor]

As @kmuehlbauer said, after updating Xradar to 0.4.2, it worked!

[syedhamidali]

@kmuehlbauer I was trying to reproduce the error, and I couldn’t, it is working as expected. Thanks for pointing it out.

[aladinor]

I think I will close this issue since it is now working. Thanks @kmuehlbauer @mgrover1 @syedhamidali for your time and help