search

AlexeyPechnikov / pygmtsar

PyGMTSAR (Python InSAR): Powerful and Accessible Satellite Interferometry
http://insar.dev/
BSD 3-Clause "New" or "Revised" License
364 stars 84 forks source link

[Help]: Gaussian detrending out of memory for large stacks #87

Open SteffanDavies opened 4 months ago

SteffanDavies commented 4 months ago

Is Gaussian detrending being performed for pixels across the entire time stack? I'm running out of memory and have tried reducing chunksize to 128 with the same problem. My system has 128GB RAM.

Here is the unwrap phase stack:

image

image

Jan 22 22:14:33 steffan systemd-oomd[664]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-93055cd3-b868-4ce5-82f7-fadf47971679.scope due to memory pressure for /user.slice/user-1000.slice/user@1000.service being 69.24% > 50.00% for > 20s with reclaim activity
Jan 22 22:14:33 steffan systemd[2141]: vte-spawn-93055cd3-b868-4ce5-82f7-fadf47971679.scope: systemd-oomd killed 118 process(es) in this unit.
AlexeyPechnikov commented 4 months ago

The error on the plot is related to the Tornado web interface communication and is not a code issue; we can ignore it. Your OOM killer message notifies that 'memory pressure for ... being 69.24% > 50.00% for > 20s with reclaim activity,' and it means only 64GB RAM is available due to your system settings. Probably, the processing notebook uses more than 64GB RAM from the 128 GB available, and it's normal behavior to utilize available RAM for 80-90%. ou can configure the Dask scheduler to use only 64GB of available RAM if you prefer to limit your RAM usage so strong.

AlexeyPechnikov commented 4 months ago

I've simulated a large stack, and it works well on my Apple Silicon host with 16 GB of RAM:

image
SteffanDavies commented 4 months ago

I've simulated a large stack, and it works well on my Apple Silicon host with 16 GB of RAM:

image

I see you split the stack in 10. I was willing to do something similar but wasn't sure if the Gaussian function worked across the time dimension. But I guess it works independently on each pair.

AlexeyPechnikov commented 4 months ago

Yes, it works on 2D grids. In fact, we can speed up all the code by making the 'pair' dimension mandatory for stacks. Currently, we always need to check whether the input is a single 2D grid or a 3D stack. This approach is more flexible but comes at a cost.