search

geodesymiami / rsmas_insar

RSMAS InSAR code
https://rsmas-insar.readthedocs.io/
GNU General Public License v3.0
58 stars 22 forks source link

minsarApp.bash terminating with `Hangup` message #505

Open falkamelung opened 2 years ago

falkamelung commented 2 years ago

I ran 4 minsarApp.bash at the same time. They stopped with a Hangup message. They all exited at the same time. What could be the reason?

The bash scripts were submitted into the background using &. The exit time agrees (roughly?) with the time when I disconnected from the internet. However, I independently checked that disconnecting is not a problem. Is this a freak incident? I saw it once before: https://github.com/geodesymiami/rsmas_insar/issues/494

tail test1KokoxiliChunk31SenDT19/workflow.1.log 
test1KokoxiliChunk31SenDT19, run_08_generate_burst_igram, 82 jobs: 48 COMPLETED, 6 RUNNING , 28 PENDING, 0 WAITING   .
test1KokoxiliChunk31SenDT19, run_08_generate_burst_igram, 82 jobs: 48 COMPLETED, 6 RUNNING , 28 PENDING, 0 WAITING   .
test1KokoxiliChunk31SenDT19, run_08_generate_burst_igram, 82 jobs: 48 COMPLETED, 6 RUNNING , 28 PENDING, 0 WAITING   .
test1KokoxiliChunk31SenDT19, run_08_generate_burst_igram, 82 jobs: 48 COMPLETED, 6 RUNNING , 28 PENDING, 0 WAITING   .
test1KokoxiliChunk31SenDT19, run_08_generate_burst_igram, 82 jobs: 48 COMPLETED, 6 RUNNING , 28 PENDING, 0 WAITING   .
test1KokoxiliChunk31SenDT19, run_08_generate_burst_igram, 82 jobs: 48 COMPLETED, 6 RUNNING , 28 PENDING, 0 WAITING   .
test1KokoxiliChunk31SenDT19, run_08_generate_burst_igram, 82 jobs: 48 COMPLETED, 6 RUNNING , 28 PENDING, 0 WAITING   .
test1KokoxiliChunk31SenDT19, run_08_generate_burst_igram, 82 jobs: 51 COMPLETED, 5 RUNNING , 26 PENDING, 0 WAITING   .
/work2/05861/tg851601/stampede2/code/rsmas_insar/minsar/run_workflow.bash: line 1: 245463 Hangup                  tail -f $logfile_name
/work2/05861/tg851601/stampede2/code/rsmas_insar/minsar/run_workflow.bash: line 1: 166616 Hangup                  sleep $wait_time

The jobs were submitted using:

minsarApp.bash $SCRATCHDIR/test1KokoxiliBigSenDT19/chunk_templates/test1KokoxiliChunk31SenDT19.template --start dem --sleep 0 &
minsarApp.bash $SCRATCHDIR/test1KokoxiliBigSenDT19/chunk_templates/test1KokoxiliChunk32SenDT19.template --start dem --sleep 180 &
minsarApp.bash $SCRATCHDIR/test1KokoxiliBigSenDT19/chunk_templates/test1KokoxiliChunk33SenDT19.template --start dem --sleep 360 &
minsarApp.bash $SCRATCHDIR/test1KokoxiliBigSenDT19/chunk_templates/test1KokoxiliChunk34SenDT19.template --start dem --sleep 540 &