NOAA-OWP / gval

A high-level Python framework to evaluate the skill of geospatial datasets by comparing candidates to benchmark maps producing agreement maps and metrics.
https://noaa-owp.github.io/gval/
Other
22 stars 0 forks source link

Tabulation performance #162

Closed GregoryPetrochenkov-NOAA closed 11 months ago

GregoryPetrochenkov-NOAA commented 11 months ago

As of now crosstab from xarray spatial is too slow and the workflow in incurring extra memory overhead. The new workflow should create the crosstab dataframe using flox groupby on the agreement map which contains the pairing dictionary as an attribute.

According to tests this should boost the speed up by more than 40%.