[Help]: Cannot complete unwrapping - Terminated / semlock + semaphore errors

[SteffanDavies]

System: Python 3.10.6 Ubuntu 22.04 RAM 240 GB Processor 16 vCores

My code:

import platform, sys, os

from pygmtsar import SBAS

import xarray as xr import numpy as np import pandas as pd

supress numpy warnings

import warnings warnings.filterwarnings('ignore')

sbas = SBAS('raw_orig', dem_filename='topo/DEM_WGS84.nc', basedir='raw_stack', landmask_filename='landmask/landmask.nc').set_master('2021-06-04') sbas.download_orbits()

About 16GB RAM for this stack, 4 minutes (12)

sbas.stack_parallel()

baseline_pairs = sbas.baseline_pairs(days=50, meters=100)

Under 1 min

sbas.topo_ra_parallel() topo_ra = sbas.get_topo_ra()

pairs = baseline_pairs[['ref_date', 'rep_date']]

About 27 min (34)

sbas.intf_parallel(pairs, wavelength=100)

Fix merging bug

sbas.to_dataframe().sort_values(by=["date","subswath"],inplace=True)

About 14GB RAM, 3.5 min (34 ss) (72 topo)

sbas.merge_parallel(pairs)

phasefilts = sbas.open_grids(pairs, 'phasefilt') phasefilts_ll = sbas.open_grids(pairs, 'phasefilt', geocode=True) corrs = sbas.open_grids(pairs, 'corr')

landmask_ll = sbas.get_landmask() landmask_ra = sbas.get_landmask(inverse_geocode=True) filter = lambda corr, unwrap: xr.where(corr_stack>=0.1, unwrap, np.nan) snaphuconf = sbas.snaphu_config(defomax=0, NTILEROW=4, NTILECOL=4, ROWOVRLP=400, COLOVRLP=400)

About 38GB RAM (4x4 tiles, 16 procs, 1 job)

sbas.unwrap_parallel(pairs, n_jobs=1, func=filter, conf=snaphuconf, mask=landmask_ra)

[SteffanDavies]

I am now trying unwrapping with only 1 subswath (F2 only, deleted F1 + F3 from raw_data and no merging) and get a different error:

FileNotFoundError: [Errno 2] No such file or directory: 'raw_stack/F2_20210511_20210523_unwrap.out'

Workdir:

Snaphu tiledir:

[SteffanDavies]

I noticed sbas.snaphu_config lists TILEDIR as master filename. Is this a problem for SBAS stack of multiple pairs? Or does your code change the TILEDIR on the fly?

After looking at your code, do I have to create a separate snaphu conf file for each interferogram with the correct master? How do you use tilling on multiple parallel interferograms unwrapping (ex: 4 tiles per IFG while running 2 unwraps for 8 CPU total)?

[AlexeyPechnikov]

I noticed sbas.snaphu_config lists TILEDIR as master filename. Is this a problem for SBAS stack of multiple pairs? Or does your code change the TILEDIR on the fly?

Yes, for parallel unwrapping the TILEDIR path is different for the every ref-rep pair, see the code below:

    def unwrap_parallel(self, pairs, n_jobs=-1, **kwargs):
...
        def unwrap(subswath, pair, **kwargs):
            # define unique tiledir name for parallel processing
            if 'conf' in kwargs:
                dirname = f'F{subswath}_{"_".join(pair).replace("-","")}_snaphu_tiledir'
                dirpath = os.path.join(self.basedir, dirname)
                kwargs['conf'] += f'    TILEDIR {dirpath}'
            return self.unwrap(subswath, pair, **kwargs)
...

TILEDIR as master filename is applied for 'unwrap' function designed to process a single grid to check the parameters. Please try to unwrap a single interferogram in 'debug' mode to see SNAPHU logs and errors (/dev/stdout and /dev/stderr):

unwrap, conn = sbas.unwrap(sbas.get_subswath(), pairs.values[0],
                           func=filter, conf=snaphuconf, mask=landmask_ra, conncomp=True, debug=True)

[SteffanDavies]

Yes, I read your code a few hours ago and TILEDIR is correctly created on the fly.

I am currently unwrapping parallel in debug mode.

[AlexeyPechnikov]

@SteffanDavies Does it work for you now?