search

geodesymiami / rsmas_insar

RSMAS InSAR code
https://rsmas-insar.readthedocs.io/
GNU General Public License v3.0
62 stars 23 forks source link

Troubleshooting Sentinel: Bus error within launcher run #392

Closed falkamelung closed 3 years ago

falkamelung commented 4 years ago

The baseline calculation /baselines/20141018_20160129 was missing. It turns out there was a bus error within the launcher run. Everything continued. This is different to other bus errors which happened un the tmux shell. This is first time I see this.

I think we need to check for Bus error in run*.e and rerun or stop if found.

ManyiSenAT12[172] cat out_run_03_average_baseline.e
#########################
#### /scratch/05861/tg851601/ManyiSenAT12/run_files/run_03_average_baseline_0_6098047.e 
#########################
using /tmp/launcher.6098047.hostlist.O8EGNozP to get hosts
starting job on c476-014
/opt/apps/launcher/launcher-3.7/launcher: line 93: 364062 Bus error               SentinelWrapper.py -c /scratch/05861/tg851601/ManyiSenAT12/configs/config_baseline_20160129 > /scratch/05861/tg851601/ManyiSenAT12/run_files/run_03_average_baseline_0_$LAUNCHER_JID.o 2> /scratch/05861/tg851601/ManyiSenAT12/run_files/run_03_average_baseline_0_$LAUNCHER_JID.e
#########################
#### /scratch/05861/tg851601/ManyiSenAT12/run_files/run_03_average_baseline_1_6098048.e 
#########################
using /tmp/launcher.6098048.hostlist.6OhuOHBv to get hosts
starting job on c476-042
#########################
#### /scratch/05861/tg851601/ManyiSenAT12/run_files/run_03_average_baseline_2_6098049.e 
#########################
using /tmp/launcher.6098049.hostlist.VYfEHYQc to get hosts
starting job on c476-053
falkamelung commented 4 years ago

It happened again. Same thing, The job was not run and there was not even a run_03_average_baseline_5_*_.o file

out_run_03_average_baseline.e:/opt/apps/launcher/launcher-3.7/launcher: line 93: 317550 Bus error               SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_baseline_20200127 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_03_average_baseline_5_$LAUNCHER_JID.o 2> /scratch/05861/tg851601/BarnesSenDT26/run_files/run_03_average_baseline_5_$LAUNCHER_JID.e
falkamelung commented 4 years ago

Bus error in launcher jobs on Stampede

Hi TACC team I am getting occasionally a Bus error (see below for 2 examples). It is not reproducible. When I run the same job again it generally runs fine but sometimes I get the same error in a different task. Do you have any suggestions on how to avoid this?

Do I need to give -n 48 even tough the job has only 43 tasks?

We actually have 253 tasks but instead of running them as one launcher jobs we have split it into 5 jobs with 43, 43, 43, 43, 38 tasks each so that walltime and wait times are shorter.

Thank you Falk

/opt/apps/launcher/launcher-3.7/launcher: line 93: 10274 Bus error               SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20171214 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2> /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
#! /bin/bash
#SBATCH -J run_02_unpack_slave_slc_2
#SBATCH -A TG-EAR180014
#SBATCH --mail-user=famelung@rsmas.miami.edu
#SBATCH --mail-type=fail
#SBATCH -N 1
#SBATCH -n 43
#SBATCH -o /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_%J.o
#SBATCH -e /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_%J.e
#SBATCH -p skx-normal
#SBATCH -t 0:11:30

module load launcher
export OMP_NUM_THREADS=2
export PATH=/home1/05861/tg851601/test/operations/rsmas_insar/sources/isce2/contrib/stack/topsStack:$PATH
export LAUNCHER_WORKDIR=/scratch/05861/tg851601/BarnesSenDT26/run_files
export LAUNCHER_JOB_FILE=/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2
export LD_PRELOAD=/home1/apps/tacc-patches/python_cacher/myopen.so

$LAUNCHER_DIR/paramrun
wc -l run_02_unpack_slave_slc_2
43 run_02_unpack_slave_slc_2
//login3/scratch/05861/tg851601/BarnesSenDT26/run_files/stdout_run_02_unpack_slave_slc[1018] cat run_02_unpack_slave_slc_2
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20171003 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20171009 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20171015 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20171021 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20171027 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20171102 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20171108 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20171114 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20171120 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20171126 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20171202 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20171208 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20171214 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20171220 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20171226 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20180101 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20180107 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20180113 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20180119 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20180125 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20180131 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20180206 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20180212 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20180218 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20180224 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20180302 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20180308 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20180314 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20180320 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20180326 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20180401 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20180407 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20180413 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20180419 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20180425 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20180501 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20180507 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20180519 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20180525 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20180531 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20180606 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20180612 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_slave_20180618 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_02_unpack_slave_slc_2_$LAUNCHER_JID.e

Second example:

/opt/apps/launcher/launcher-3.7/launcher: line 93: 281345 Bus error               SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_fullBurst_geo2rdr_20170223 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.o 2> /scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.e
! /bin/bash
#SBATCH -J run_09_fullBurst_geo2rdr_2
#SBATCH -A TG-EAR180014
#SBATCH --mail-user=famelung@rsmas.miami.edu
#SBATCH --mail-type=fail
#SBATCH -N 1
#SBATCH -n 23
#SBATCH -o /scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_%J.o
#SBATCH -e /scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_%J.e
#SBATCH -p skx-normal
#SBATCH -t 1:09:00

module load launcher
export OMP_NUM_THREADS=4
export PATH=/home1/05861/tg851601/test/operations/rsmas_insar/sources/isce2/contrib/stack/topsStack:$PATH
export LAUNCHER_WORKDIR=/scratch/05861/tg851601/BarnesSenDT26/run_files
export LAUNCHER_JOB_FILE=/scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2
export LD_PRELOAD=/home1/apps/tacc-patches/python_cacher/myopen.so

$LAUNCHER_DIR/paramrun
wc -l run_09_fullBurst_geo2rdr_2
23 run_09_fullBurst_geo2rdr_2
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_fullBurst_geo2rdr_20170130 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_fullBurst_geo2rdr_20170205 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_fullBurst_geo2rdr_20170211 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_fullBurst_geo2rdr_20170217 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_fullBurst_geo2rdr_20170223 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_fullBurst_geo2rdr_20170301 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_fullBurst_geo2rdr_20170307 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_fullBurst_geo2rdr_20170313 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_fullBurst_geo2rdr_20170319 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_fullBurst_geo2rdr_20170325 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_fullBurst_geo2rdr_20170331 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_fullBurst_geo2rdr_20170406 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_fullBurst_geo2rdr_20170412 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_fullBurst_geo2rdr_20170418 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_fullBurst_geo2rdr_20170424 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_fullBurst_geo2rdr_20170430 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_fullBurst_geo2rdr_20170506 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_fullBurst_geo2rdr_20170512 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_fullBurst_geo2rdr_20170518 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_fullBurst_geo2rdr_20170524 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_fullBurst_geo2rdr_20170530 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_fullBurst_geo2rdr_20170605 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.e
SentinelWrapper.py -c /scratch/05861/tg851601/BarnesSenDT26/configs/config_fullBurst_geo2rdr_20170611 > /scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.o 2>/scratch/05861/tg851601/BarnesSenDT26/run_files/run_09_fullBurst_geo2rdr_2_$LAUNCHER_JID.e