search

TUW-GEO / ascat

Read and visualize data from the Advanced Scatterometer (ASCAT) on-board the series of Metop satellites
https://ascat.readthedocs.io/
MIT License
23 stars 16 forks source link

OverflowError: Python integer -128 out of bounds for uint8 #65

Closed claytharrison closed 1 month ago

claytharrison commented 2 months ago

On current numpy versions for Python > 3.8, lines 1329 and 1647 in eps_native.py raise OverflowError: Python integer -128 out of bounds for uint8 when setting nan_val on data["swath_indicator"], because data["swath_indicator"] is of dtype np.uint8 at this point, while the nan_val being set on it is int8_nan (-128).

https://github.com/TUW-GEO/ascat/blob/e5c02d4ccc08c59ca9057bfc148a40f0a098f792/src/ascat/read_native/eps_native.py#L1322C1-L1329C34

https://github.com/TUW-GEO/ascat/blob/e5c02d4ccc08c59ca9057bfc148a40f0a098f792/src/ascat/read_native/eps_native.py#L1623-L1647

In current numpy for Python <= 3.8 this operation casts nan_val to uint8 (changing its value from -128 to 128) and produces a warning, but this warning is apparently suppressed somewhere in the script when reading from eps files. This behavior is now deprecated.

If it is possible for swath_indicator to have any nan values before it reaches this line, this silent casting may cause some incorrect data, but I think it's not possible, so I don't think it's an issue in that sense.

claytharrison commented 2 months ago

This problem also exists for some other variables:

"f_kp", "f_usable": https://github.com/TUW-GEO/ascat/blob/e5c02d4ccc08c59ca9057bfc148a40f0a098f792/src/ascat/read_native/eps_native.py#L1331-L1340

https://github.com/TUW-GEO/ascat/blob/e5c02d4ccc08c59ca9057bfc148a40f0a098f792/src/ascat/read_native/eps_native.py#L1416-L1426

https://github.com/TUW-GEO/ascat/blob/e5c02d4ccc08c59ca9057bfc148a40f0a098f792/src/ascat/read_native/eps_native.py#L1862-L1870

and "flagfield_gen2": https://github.com/TUW-GEO/ascat/blob/e5c02d4ccc08c59ca9057bfc148a40f0a098f792/src/ascat/read_native/eps_native.py#L1546-L1558