search

wildlife-dynamics / ecoscope-workflows

An extensible task specification and compiler for local and distributed workflows.
https://ecoscope-workflows.readthedocs.org/
BSD 3-Clause "New" or "Revised" License
7 stars 3 forks source link

For calculate_time_density, allow configuring pixel_size to avoid OOM for large trajectories #437

Closed cisaacstern closed 1 week ago

cisaacstern commented 2 weeks ago

When the trajectories passed into calculate_time_density are very large, the default pixel size of 250.0 can cause a huge memory explosion, crashing the python process it's running in.

To mitigate this, @walljcg suggests we dynamically set pixel_size to based on the scale of the trajectories passed to this task. Quoting Jake:

The geographic expanse of those tracks is probably huge since they've been flying all over the place. And the cell size is 250 m so we might need to add some logic that will scale back the cell size to something larger if it's trying to generate a global grid to calculate etd.

maybe we can decide some threshold based on the diagonal distance of the bounding envelope of the tracks versus the grid cell size. When that is exceeded we choose a cell size that brings the calc within the threshold

cisaacstern commented 2 weeks ago

An alternative which is a faster path for right now, also per Jake's suggestion, would be to expose pixel_size as a "basic config" option in the RJSF.

cisaacstern commented 1 week ago

I've split off https://github.com/wildlife-dynamics/ecoscope-workflows/issues/444 to track the continuation of this work. Going to call this first pass closed based on #441.