manaakiwhenua / vector2dggs

DGGS indexer for vector data
https://pypi.org/project/vector2dggs/
GNU Lesser General Public License v3.0
6 stars 0 forks source link

Out-of-memory error when processing large data #14

Closed alpha-beta-soup closed 1 year ago

alpha-beta-soup commented 1 year ago

The recent change to repartition the intermediate output on H3 parent seems to load a lot of data into memory; after some time it exhausts my available RAM and the processes are killled. The method should be reviewed to prevent this.