workshop 7: Automating your analyses and executing long-running analyses on remote computers

[ctb]

Thursday August 19 from 9 am - 11:30 PDT

Instructors: Titus, Marisa Moderator: Abhijna Helpers: Saranya Canchi

Zoom link:

Description:

This two hour workshop will show attendees how to automate their analyses using shell scripts, as well as run and manage software that takes minutes, hours, or days to execute. We’ll also show you how to disconnect from and resume running processes using the ‘screen’ command.

no draft lesson

owner: ???

CTB: maybe switch this with Structuring your projects for current and future you?

potential topics:

• automating things by putting together lists of commands in text file
• set -x, set -e
• long-running jobs at the command-line: recovering their output

forward links:

• scripting,
• slurm

[ctb]

• topics/learning objectives
  • automating things by putting stuff in a text file
  • running it with 'bash' explicitly
  • putting a header in and changing permissions; executing via shell prompt
  • what happens on failure; set -e and set -x
  • an intro to conditions and for loops
  • using echo
• prerequisites
• assessments

[marisalim]

interested in co-teaching this one

[marisalim]

we have a section on using screen in the AWS lesson - https://training.nih-cfde.org/en/latest/Cloud-Platforms/Introduction_to_Amazon_Web_Services/introtoaws5_Screen.html

or maybe we want to teach tmux instead of screen?

[ctb]

will probably start where workshop #1 and #6 leave off.

[ctb]

maybe with salt from https://github.com/ngs-docs/2021-july-remote-computing/issues/4.

[ctb]

pamela and nick cannot attend.

[marisalim]

idea for an exercise to demo set -e - run bash script that loops through a command 3 times and has an error (i.e., wrong input file path) with (will fail/exit after 1st iteration) & without (will fail 3 times for each iteration) set -e in script.

[marisalim]

notes and lesson development - https://hackmd.io/VpMBdsq8T1WhvkEiU32OHQ

[marisalim]

talk about the for loop material from workshop 1 notes (we saved it for later)

https://ngs-docs.github.io/2021-august-remote-computing/introduction-to-the-unix-command-line.html#renaming-a-bunch-of-files

[marisalim]

add some material from https://github.com/ngs-docs/2021-GGG298/tree/latest/Week8-project_organization_and_UNIX_shell#file-manipulation

if statements, explanation of $, [ ]; script parameters

[marisalim]

possible assessment questions - feel free to edit!

• i know how to write a bash for loop.
• i can execute scripts from the shell.
• i know how to start a screen or tmux session.

[marisalim]

lesson layout:

marisa

• intro
• what are scripts, demo echo statement in script, bash headers
• for loops, subsetting, variables

titus

• troubleshooting with set -e, set -x
• if statements, exercise of running if statement script in a for loop

marisa

• screen and tmux
• wrap up

[marisalim]

slides for Thursday - @ctb feel free to edit! 2021-remote-computing-7.pptx

[jeremywalter]

Pre-Survey: https://forms.gle/8DrR4RfwUDNiu5T48 Post-Survey: https://forms.gle/zhwW9zTP3uwxqLt56

[marisalim]

reminder to fix subset example extension

for i in *.fq; do echo "head -400 $i > subset/$i"; newname=$(basename $i .fq)subset.fq; echo mv subset/$i subset/$newname; done

[marisalim]

lessons always need xkcd :D

add - https://xkcd.com/1205/

[jeremywalter]

7.4.2 : As a 'student' I found the variables section confusing and without context 7.5.1 : needs explicit instructions on where conda instructions are (could refer to specific lesson and section) --> add ANSWERS to QUESTIONS?

[marisalim]

add note that you don't actually have to use indentation for the bash for loops! the indentations are for us, not the computer

to if statement section, add putting it in a script

cd ~/
nano if-for.sh

# copy in
for i in *
do
   if [ -f $i ]; then
      echo $i is a file
   elif [ -d $i ]; then
      echo $i is a directory
   fi
done

# run
bash if-for.sh

to if.sh

echo running $0
echo a will be $1
echo b will be $2
a=$1
b=$2

[ctb]

from saranya:

My suggestion here would be to try and connect workshop 6 and
workshop 7 with the same examples. In particular, the commands from 4.8 amd 4.9 can be updated to work with files from MISeq folder only.