Parallelize unwrapping in space

isce-framework / spurt

Spatial and Temporal phase Unwrapping for InSAR time-series

https://spurt.readthedocs.io/

Apache License 2.0

22 stars 4 forks source link

Parallelize unwrapping in space #26

Closed scottstanie closed 4 months ago

scottstanie commented 4 months ago

The other edits are

Returning the filenames for ease of use within python
Passing through geo metadata when writing out geotiffs

scottstanie commented 4 months ago

The main question I had for @piyushrpt / @gmgunter : do we think it makes sense to use self.settings.worker_count for both the temporal and spatial unwrapping parallelism? Maybe it doesn't make a big difference, I was just thinking of how temporal unwrapping ~millions of tiny problems, but spatial is ~dozens of large ones

piyushrpt commented 4 months ago

I was thinking the same thing. Maybe we need another setting for number of spatial workers - memory usage also goes up with number of threads. I would cap the number of workers to 2-4 at best - we mostly do 1 vcpu / 4GB or 2 vcpu / 8GB instances and the current worker_count is ok for these but you might be working on larger instances.

piyushrpt commented 4 months ago

Launching the test run now