Default fill value of -9999 is incompatible with numpy 2.0 after integration of NEP 50 - Githubissues

pyoceans / pocean-core

🐍 + 🌊 A python framework for working with met-ocean data

https://pyoceans.github.io/pocean-core/

MIT License

22 stars 16 forks source link

Default fill value of -9999 is incompatible with numpy 2.0 after integration of NEP 50 #98

Open lukecampbell opened 3 weeks ago

lukecampbell commented 3 weeks ago

https://numpy.org/neps/nep-0050-scalar-promotion.html#nep50

https://github.com/pyoceans/pocean-core/blob/acd977bb3b513fd7338e33aefb7937c6ab47b653/pocean/utils.py#L305

https://github.com/pyoceans/pocean-core/blob/acd977bb3b513fd7338e33aefb7937c6ab47b653/pocean/cf.py#L14

With the release of numpy 2.0 adopting NEP 50 by default, type promotion has changed and any attempts to coerce python scalar values into data types that can't represent that value now raise an OverflowError.

Before, if you had a dtype of uint8 the fill value would wind up being 241 after overflowing.

I recommend creating a more sophisticated mapping of appropriate fill values based on dtypes:

uint8 - 255
int8 - -128
...
float32 - nan
float64 - nan
datetime64 - nan (depends on context... numpy supported contexts are NaT, float-only contexts would be NaN)

I'll try to take a look at a PR if I can find the time.

ocefpaf commented 3 weeks ago

You can use np._set_promotion_state("legacy") for now until this is properly fixed, in case you have broken workflows.