TACC meeting April 2022

Installation of code and setting environment

copy /home1/05861/tg851601/accounts to your /home area
install code (up to Orbits and aux files): https://github.com/geodesymiami/rsmas_insar/blob/master/docs/installation.md

add to your .bashrc (RSMASINSAR_HOME is the installation directory)

alias source_environment='cd $RSMASINSAR_HOME; export PATH=/bin; unset PYTHONPATH; source ~/accounts/platforms_defaults.bash; source setup/environment.bash; export PATH=$ISCE_STACK/topsStack:$PATH; s ~/accounts/alias.bash; s ~/accounts/login_alias.bash; cd -;'
alias s.bw2t='export RSMASINSAR_HOME=${WORK2%/*}/stampede2/codet/rsmas_insar; source_environment'

source environment using s.bw2t

Task: Our goal is to find out how many tasks (jobs) we can run simultaneously without copy-to-tmp of input files.

create in your $SCRATCHDIR a mkdir SouthWesternNAMChunk36SenAT64 directory.
Copy the file https://github.com/geodesymiami/infiles_famelung/blob/master/TEMPLATES/SouthWesternNAMChunk36SenAT64.template into this directory
The next step depends on whether you have read access during a run to my scratch dir. If you do , in /SouthWesternNAMChunk36SenAT64 run minsarApp.bash $PWD/SouthWesternNAMChunk36SenAT64.template --start jobfiles. If you don't somehow copy my data (cp -r /scratch2/05861/tg851601/SouthWesternNAMChunk36SenAT64/SLC . (6 TB). Then run minsarApp.bash $PWD/SouthWesternNAMChunk36SenAT64.template --start dem

Once you have successfully ran lets deal with this:

cd /scratch2/05861/tg851601/SouthWesternNAMChunk36SenAT64/run_files_tmp

Current trouble-causing jobs:

sbatch run_08_generate_burst_igram_0.job
sbatch run_08_generate_burst_igram_1.job

(note to Falk: run_08_generate_burst_igram_3.job, etc got accidentally deleted)

New jobs (without copy-to-tmp of data):

sbatch  run_08_generate_burst_igram_32_fast10.job
sbatch  run_08_generate_burst_igram_36_fast10.job
sbatch  run_08_generate_burst_igram_40_fast10.job
sbatch  run_08_generate_burst_igram_44_fast10.job
sbatch  run_08_generate_burst_igram_48_fast10.job

Each job contains 40 tasks which are run using launcher. The question is how many tasks can I run simultaneously. Can I run the 5 jobs (50 tasks) simultaneously? Or can I run even more, e.g. 75 or 100 (change LAUNCHER_PPN to 15 or 20)?

We also could put a sleep 30 before each task to ensure that the bulk data reading of the previous tasks is completed before the next task starts,

geodesymiami / rsmas_insar