stackSentinel.py: chunk processing: not all *zip files are staged for unpacking for small latitude range

[falkamelung]

missing_subswath_KokoxiliBig35SenAT143.template A big dataset is downloaded (11 degree) but stackSentinel (lat range 35.00:36.00) does not stage all imagery for unpacking and the processing fails (one subswath is missing for 20170321 . However a large

• Download for latitude range 28.9 40.0

cd SLC
ssara_federated_query.bash --platform=SENTINEL-1A,SENTINEL-1B --relativeOrbit=143 --intersectsWith='Polygon((88.00 29.80, 88.00 40.00, 98.00 40.00, 98.00 29.80, 88.00 29.80))' --start=2017-02-25 --end=2017-03-23 --parallel=5 --print --download

• run create_runfiles.py or stackSentinel.py for 35.0, 36.0 latitude range:

  ` stackSentinel.py --slc_directory ./SLC --orbit_directory /home1/05861/tg851601/insarlab/S1orbits --aux_directory /home1/05861/tg851601/insarlab/S1aux --working_directory ./ --dem DEM/demLat_N33_N38_Lon_E087_E099.dem.wgs84 --num_connections 5 --num_overlap_connections 3 --swath_num "1 2 3" --bbox "35.0 36.0 88.0 98.0" --exclude_dates 20150401,20150706 --azimuth_looks 21 --range_looks 57 --filter_strength 0.2 --esd_coherence_threshold 0.85 --snr_misreg_threshold 10 --unw_method snaphu --polarization vv --coregistration NESD --workflow interferogram --num-process 8

  There were only two *zip files found for 20170321:

  //login3/scratch/05861/tg851601/missing_subswath_KokoxiliBig35SenAT143[1093] cat configs/config_secondary_20170321
  [Common]
  ##########################
  ##########################
  [Function-1]
  Sentinel1_TOPS : 
  dirname : /scratch/05861/tg851601/missing_subswath_KokoxiliBig35SenAT143/SLC/S1A_IW_SLC__1SDV_20170321T115031_20170321T115059_015790_01A007_402B.zip /scratch/05861/tg851601/missing_subswath_KokoxiliBig35SenAT143/SLC/S1A_IW_SLC__1SDV_20170321T115121_20170321T115148_015790_01A007_533B.zip
  swaths : 1 2 3
  orbit : /scratch/05861/tg851601/missing_subswath_KokoxiliBig35SenAT143/orbits/20170321/S1A_OPER_AUX_POEORB_OPOD_20170410T121606_V20170320T225942_20170322T005942.EOF
  outdir : /scratch/05861/tg851601/missing_subswath_KokoxiliBig35SenAT143/secondarys/20170321
  auxdir : /home1/05861/tg851601/insarlab/S1aux
  bbox : 35.0 36.0 88.0 98.0
  pol : vv
  ##########################

As a result, orbit 20170321 is missing subswath IW2.

 ll secondarys/*
secondarys/20170309:
total 896
drwxrwx--- 2 tg851601 G-820134   4096 Dec 20 23:49 IW1
-rw-rw---- 1 tg851601 G-820134 285922 Dec 20 23:49 IW1.xml
drwxrwx--- 2 tg851601 G-820134   4096 Dec 20 23:49 IW2
-rw-rw---- 1 tg851601 G-820134 323569 Dec 20 23:49 IW2.xml
drwxrwx--- 2 tg851601 G-820134   4096 Dec 20 23:49 IW3
-rw-rw---- 1 tg851601 G-820134 294439 Dec 20 23:49 IW3.xml

secondarys/20170321:
total 88
drwxrwx--- 2 tg851601 G-820134  4096 Dec 20 23:49 IW1
-rw-rw---- 1 tg851601 G-820134 39812 Dec 20 23:49 IW1.xml
drwxrwx--- 2 tg851601 G-820134  4096 Dec 20 23:49 IW3
-rw-rw---- 1 tg851601 G-820134 39818 Dec 20 23:49 IW3.xml

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX When I run a larger lat-range (34.5 to 36.5)

stackSentinel.py --slc_directory ./SLC --orbit_directory /home1/05861/tg851601/insarlab/S1orbits --aux_directory /home1/05861/tg851601/insarlab/S1aux --working_directory ./ --dem DEM/demLat_N33_N38_Lon_E087_E099.dem.wgs84 --num_connections 5 --num_overlap_connections 3 --swath_num "1 2 3" --bbox "34.5 36.5 88.0 98.0" --exclude_dates 20150401,20150706 --azimuth_looks 21 --range_looks 57 --filter_strength 0.2 --esd_coherence_threshold 0.85 --snr_misreg_threshold 10 --unw_method snaphu --polarization vv --coregistration NESD --workflow interferogram --num-process 8

there are more zip files and processing works well:

cat configs/config_secondary_20170321
[Common]
##########################
##########################
[Function-1]
Sentinel1_TOPS : 
dirname : /scratch/05861/tg851601/missing_subswath_KokoxiliBig35SenAT143/SLC/S1A_IW_SLC__1SDV_20170321T115056_20170321T115124_015790_01A007_F4BB.zip /scratch/05861/tg851601/missing_subswath_KokoxiliBig35SenAT143/SLC/S1A_IW_SLC__1SDV_20170321T115031_20170321T115059_015790_01A007_402B.zip /scratch/05861/tg851601/missing_subswath_KokoxiliBig35SenAT143/SLC/S1A_IW_SLC__1SDV_20170321T115121_20170321T115148_015790_01A007_533B.zip
swaths : 1 2 3
orbit : /scratch/05861/tg851601/missing_subswath_KokoxiliBig35SenAT143/orbits/20170321/S1A_OPER_AUX_POEORB_OPOD_20170410T121606_V20170320T225942_20170322T005942.EOF
outdir : /scratch/05861/tg851601/missing_subswath_KokoxiliBig35SenAT143/secondarys/20170321
auxdir : /home1/05861/tg851601/insarlab/S1aux
bbox : 34.5 36.5 88.0 98.0
pol : vv
##########################

[falkamelung]

solved using geopandas