How to write plain bash code into python scripts

falkamelung commented 3 years ago

We are now writing a lot of bash code into our jobfiles. This is done in job_submission,py here:

https://github.com/geodesymiami/rsmas_insar/blob/master/minsar/job_submission.py#L712-L730

Is there any way to write in the python code exactly as it will show up when written? I mean without all this

job_file_lines.append(“"”  \n“””) syntax.

It would be nice if we just have a line feed or newline character at the end of each line.

Here how ist shows up in the job file.

#! /bin/bash
#SBATCH -J run_03_average_baseline_0
#SBATCH -A TG-EAR200012
#SBATCH --mail-user=tg851601@rsmas.miami.edu
#SBATCH --mail-type=fail
#SBATCH -N 1
#SBATCH -n 48
#SBATCH -o /scratch/05861/tg851601/BalotschistanSenDT49/run_files/run_03_average_baseline_0_%J.o
#SBATCH -e /scratch/05861/tg851601/BalotschistanSenDT49/run_files/run_03_average_baseline_0_%J.e
#SBATCH -p skx-normal
#SBATCH -t 0:10:00
################################################
#   install code on /tmp                       #
################################################
df -h /tmp
rm -rf /tmp/rsmas_insar
mkdir -p /tmp/rsmas_insar
cp -r $RSMASINSAR_HOME/minsar /tmp/rsmas_insar
cp -r $RSMASINSAR_HOME/setup  /tmp/rsmas_insar
mkdir -p /tmp/rsmas_insar/3rdparty ;
cp -r $RSMASINSAR_HOME/3rdparty/launcher /tmp/rsmas_insar/3rdparty 
cp $SCRATCH/miniconda3.tar /tmp
tar xf /tmp/miniconda3.tar -C /tmp/rsmas_insar/3rdparty
rm /tmp/miniconda3.tar
# set environment    
export RSMASINSAR_HOME=/tmp/rsmas_insar
cd $RSMASINSAR_HOME; source ~/accounts/platforms_defaults.bash; source setup/environment.bash; export PATH=$ISCE_STACK/topsStack:$PATH; cd -;
# remove /scratch and /work from PATH
export PATH=`echo ${PATH} | awk -v RS=: -v ORS=: '/scratch/ {next} {print}' | sed 's/:*$//'` 
export PATH=`echo ${PATH} | awk -v RS=: -v ORS=: '/work/ {next} {print}' | sed 's/:*$//'` 
export PATH=`echo ${PATH} | awk -v RS=: -v ORS=: '/home/ {next} {print}' | sed 's/:*$//'` 
export PYTHONPATH=`echo ${PYTHONPATH} | awk -v RS=: -v ORS=: '/scratch/ {next} {print}' | sed 's/:*$//'` 
export PYTHONPATH=`echo ${PYTHONPATH} | awk -v RS=: -v ORS=: '/home/ {next} {print}' | sed 's/:*$//'` 
export PYTHONPATH_RSMAS=`echo ${PYTHONPATH_RSMAS} | awk -v RS=: -v ORS=: '/scratch/ {next} {print}' | sed 's/:*$//'` 
export PYTHONPATH_RSMAS=`echo ${PYTHONPATH_RSMAS} | awk -v RS=: -v ORS=: '/home/ {next} {print}' | sed 's/:*$//'` 
################################################
# copy infiles to local /tmp and adjust *.xml  #
################################################
# reference
cp -r /scratch/05861/tg851601/BalotschistanSenDT49/reference /tmp
files="/tmp/reference/*.xml /tmp/reference/*/*.xml"
old=/scratch/05861/tg851601/BalotschistanSenDT49
sed -i "s|$old|/tmp|g" $files
# secondarys
date_list=( $(awk '{printf "%s\n",$3}' /scratch/05861/tg851601/BalotschistanSenDT49/run_files/run_03_average_baseline_0 | awk -F _ '{printf "%s\n",$3}' ) )
mkdir -p /tmp/secondarys
for date in "${date_list[@]}"; do
    cp -r /scratch/05861/tg851601/BalotschistanSenDT49/secondarys/$date /tmp/secondarys
done
files1="/tmp/secondarys/????????/*.xml"
files2="/tmp/secondarys/????????/*/*.xml"
old=/scratch/05861/tg851601/BalotschistanSenDT49
sed -i "s|$old|/tmp|g" $files1
sed -i "s|$old|/tmp|g" $files2
df -h /tmp

Ovec8hkin commented 3 years ago

You should be able to do something like this: https://stackoverflow.com/questions/16162383/how-to-easily-write-a-multi-line-file-with-variables-python-2-6/16162599

Replace all of the job_file_lines.append("...") with a triple quoted string formatted as you like. Then write that to the file using file.write().

falkamelung commented 3 years ago

Yes. this works. Thank you.

But it is quite a lot of work..... I may do later.

            job_file_lines.append("""
            # reference
            cp -r """ + self.out_dir + """/reference /tmp
            files="/tmp/reference/*.xml /tmp/reference/*/*.xml"
            old=""" + self.out_dir + """ 
            sed -i "s|$old|/tmp|g" $files
            # geom_reference
            cp -r """ + self.out_dir + """/geom_reference /tmp
            files="/tmp/geom_reference/*/*.xml"
            old=""" + self.out_dir + """
            sed -i "s|$old|/tmp|g" $files
            # secondarys
            date_list=( $(awk '{printf "%s\\n",$3}' """ + batch_file +  """ | awk -F _ '{printf "%s\\n",$NF}' ) )
            mkdir -p /tmp/secondarys
            for date in "${date_list[@]}"; do
                cp -r """ + self.out_dir + """/secondarys/$date /tmp/secondarys
            done
            files1="/tmp/secondarys/????????/*.xml"
            files2="/tmp/secondarys/????????/*/*.xml"
            old=""" + self.out_dir + """
            sed -i "s|$old|/tmp|g" $files1
            sed -i "s|$old|/tmp|g" $files2
            """)

After this we have to split according to '\n', similar to:

tmp1 = job_file_lines.split('\n')
        tmp2 = []
        for line in tmp:
            #print(line, flush=True)
            #print(line.strip(), flush=True)
            tmp2.append(line.strip())
        job_file_lines = tmp2

geodesymiami / rsmas_insar

How to write plain bash code into python scripts #471