search

inbo / niche_vlaanderen

Python package to run the NICHE Vlaanderen model
https://inbo.github.io/niche_vlaanderen/
MIT License
5 stars 0 forks source link

Harmonize nodata usage #375

Open johanvdw opened 2 months ago

johanvdw commented 2 months ago

There are too many different nodata definitions in use in niche:

I think we should consider moving all data types to float32. Areas are relatively small, and we compress tif files anyway. This would also mean we can use np.nan for no data, which propagates properly without special tweaks.

masked arrays are another option to remove this low level code, but fixing the data type is probably even better.

Note that currently the code seems to be working well, but it is rather complex internally, leading eg to #335 .

cecileherr commented 1 month ago

A side note: in the past (with niche 1.2, Win 10, 8Go RAM but hardly any free hard disk < 10 Go) I have had problems with memory issues with some projects (example: for a project with resolution of 5*5 m

MemoryError: unable to allocate 43.8 MiB for an array with shape (2760, 4160) and data type float32

).

I suspect changing the data type to float/solving issue #335 might lead to more memory problems (?). Would it be possible to test this/give an idea of the impact on memory/speed? Thx!