12 June 2019, Charlottesville: Bioinformatics in a High-Performance Computing Environment

[Daniel-Mietchen]

as per https://education.cadre.virginia.edu/#/workshop/1596 and https://workshops.somrc.virginia.edu/lesson/hpc-bioinformatics/

[Daniel-Mietchen]

Good mix of people in the room, including some people from iGEM Virginia with whom I was trying to get in touch anyway, following the synthetic biology symposium last week (cf. https://github.com/Daniel-Mietchen/events/issues/568 ).

In the afternoon, there will be room to discuss what software is/ should be running on the UVA HPC cluster Rivanna (see my separate notes on that). I'd like to use a MyBinder/ Docker/ Wikibase combo, and I'd like to learn more about the different file formats used in sequence analysis as well as about dark taxa.

[Daniel-Mietchen]

Another thing that's on my mind in this context is how things like http://wikigenomes.org/ can be integrated into sequence analysis workflows, or https://wikidp.org/ into software curation workflows.

[Daniel-Mietchen]

• popular sequence alignment software:

  • BWA
  • bowtie2

• key sequence alignment algorithms:

  • Burrows - Wheeler transform (used by the two tools mentioned above)

[Daniel-Mietchen]

I asked about how the environmental footprint of Rivanna is managed, and the VP for IT was mentioned as overseeing that. Will contact them.

[Daniel-Mietchen]

as allocation, users can use "rivanna-training" up to certain limits.

[Daniel-Mietchen]

For large file transfers, use Globus.

[Daniel-Mietchen]

For the hands-on, I am using the Terminal under rivanna-portal, along with instructions from https://workshops.somrc.virginia.edu/lesson/ngs-seq-alignment/ .

[Daniel-Mietchen]

The hostname command gives udc-ba35-36 for me right now. Gonna avoid that for running my jobs.

[Daniel-Mietchen]

after launching the ijob, hostname gives udc-ba26-28a.

[Daniel-Mietchen]

to get a list of all of a user's Rivanna jobs on that host machine, type squeue -u UVACOMPUTEID, where UVACOMPUTEID is their compute id

[Daniel-Mietchen]

to get a list of most recently run jobs, type sacct, or sacct -S 2017-01-30 to get all the jobs run since a given start date. Output format can be specified using things like sacct --format=jobid,jobname,start,elapsed,end -S 2017-01-30. See sacct -h for details.

[Daniel-Mietchen]

echo $SLURM_CPUS_PER_TASK gives the number of cores requested by the ijob command

[Daniel-Mietchen]

If there is no scratch directory, then instead of cd /scratch/dm7gn/ngs-aln-workshop/ref/, one can do cd /sfs/lustre/scratch/dm7gn/ngs-aln-workshop/ref/.

[Daniel-Mietchen]

met Sarah French. will send email to follow up.

under /project/genomes, there is a list of genomes for which indexes for alignment have already been built.

[Daniel-Mietchen]

Both the iGEM team and Sara are working on styrofoam.