search

openradar / xradar

A tool to work in weather radar data in xarray
https://docs.openradarscience.org/projects/xradar
MIT License
85 stars 17 forks source link

Range inaccuracies make it unable to resolve 'if unique' statement in xradar/model.py, causing crash #150

Closed martinpaule closed 4 months ago

martinpaule commented 4 months ago

Description

When exporting a dtree into ODIM HDF5 and reading it back, there are certain inaccuracies introduced to float numbers. Presumably due to these inaccuracies, combined with an improperly defined if statement, the program crashes when reading back the exported file.

Below is a description of the float inaccuracies, as well as the core of this issue, the crash itself.

Observed errors: For example, in our case, we used a distance_step of 600.6 metres and 333 bins.

During exporting, the following errors were introduced already: /opt/env/lib/python3.10/site-packages/xradar/io/export/odim.py/to_odim rstart: -5.684341886080802e-14 instead of 0 rscale: 600.6000000000001 instead of 600.6

even if rounded/set to the correct number, output of: /opt/env/lib/python3.10/site-packages/xradar/io/backends/odim.py/_get_range produces array with slight errors, in our case:

[ 300.3 900.9 1501.5 2102.1 2702.7002 3303.3003 3903.9001 4504.5 5105.1 5705.7 6306.3003 6906.9004 ... 194894.7 195495.31 196095.9 196696.52 197297.11 197897.7 198498.31 199098.9 199699.52 ] ^ should all just be 1 decimal place. This step produces errors of up to 0.02

Crash when reading back exported file:

More than just inaccuracy, these errors pose an issue for _the get_rangeattrs function in /opt/env/lib/python3.10/site-packages/xradar/model.py.

at line line 635, the condition 'if unique' cannot be resolved, since it contains an array of multiple unique values. In our case it was: unique: [600.59375 600.59766 600.5996 600.59985 600.6 600.60004 600.6001 600.6006 600.60156 600.6094 ]

Proposed solution: Retaining the logic of the if statements, the line 'if unique:' may be changed to 'if len(unique) == 1:'

Error output

File "/opt/env/lib/python3.10/site-packages/xradar/model.py", line 635, in get_range_attrs
    if unique:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
kmuehlbauer commented 4 months ago

Thanks @martinpaule for the thorough description. Looking at the code your suggestion makes absolutely sense. PR welcome to solve the unique issue.

The issues on write look like float32/float64 issues. What is your source data in? Did you read it from ODIM or some other format. Or did you create the data from scratch? Anyway if there is any chance to solve this in xradar we are happy to help.

martinpaule commented 4 months ago

Thank You @kmuehlbauer for the prompt reply.

The data comes from our radar's custom data format (made to adhere to ODIM HDF5 specifications). We read this data into a dtree and exported it into ODIM using xradar. The issue arose when opening said exported file.

On our side, we were able to fix it by rounding numbers to 0.25 number increments (meaning 600.6 would be 600.5).

The slight inaccuracy isn't a big issue as it is very small and gets lost in further processing/visualization.

The problem seems to mainly lie with the unique issue, as it would introduce the mentioned crash not only in this case, but also in the case of working with a radar with pre-set varying distances between gates.

May we prepare and submit a PR request addressing this 'unique' issue?

kmuehlbauer commented 4 months ago

@martinpaule Thanks for the additional info. Yes, I'd be happy to review an according PR.