s3 bucket compatibility

[rlancaster96]

• pull files from s3 bucket
• perform work in compute space (gscratch or mnt/scratch), with option to clean up and delete work dir upon successful completion
• output files back into s3 bucket (pared down version: see comment of all files included in output)
  • no longer outputs individual fastqc results; outputs multiqc only

[rlancaster96]

Will output the following files back into the bucket in this structure:

aligned/
    |- unsorted
            |- SAMPLE_G.bam
            |- SAMPLE_T.bam
    |- duplicate_marked
            |- SAMPLE_G_sorted_marked_duplicates_sorted.bam
            |- SAMPLE_G_sorted_marked_duplicates_sorted.bam.bai
            |- SAMPLE_T_sorted_marked_duplicates_sorted.bam
            |- SAMPLE_T_sorted_marked_duplicates_sorted.bam.bai
annotated/
    |- SAMPLE_T_annotated.vcf
    |- SAMPLE_T_annotated_PASSED.vcf
multiqc/
    |- multiqc_report.html
summaries/
    |- SAMPLE_G_sorted_marked_duplicates_sorted.getpileupsummaries.table
    |- SAMPLE_T_sorted_marked_duplicates_sorted.getpileupsummaries.table
svc/
    |- SAMPLE_T_filtered.vcf
    |- SAMPLE_T_read-orientation-model.tar.gz
    |- SAMPLE_T_unfiltered.vcf.all.stats
    |- SAMPLE_T_unfiltered.vcf.gz
    |- SAMPLE_T_unfiltered_normalized_sorted.vcf.gz
    |- SAMPLE_T_unfiltered_normalized_sorted.vcf.gz.tbi
tables/
    |- SAMPLE_T_sorted_marked_duplicates_sorted_contamination_table
    |- SAMPLE_T_sorted_marked_duplicates_sorted_segmentation_table

This is compared to the previous output file directory, which looks like this:

├── aligned
│   ├── YOURSAMPLEID_G.bam
│   ├── YOURSAMPLEID_T.bam
│   ├── markduplicates
│   │   ├── YOURSAMPLEID_G_sorted_marked_duplicates.bam
│   │   ├── YOURSAMPLEID_G_sorted_marked_duplicates_metrics.txt
│   │   ├── YOURSAMPLEID_T_sorted_marked_duplicates.bam
│   │   ├── YOURSAMPLEID_T_sorted_marked_duplicates_metrics.txt
│   │   └── sorted
│   │       ├── YOURSAMPLEID_G_sorted_marked_duplicates_sorted.bam
│   │       ├── YOURSAMPLEID_G_sorted_marked_duplicates_sorted.bam.bai
│   │       ├── YOURSAMPLEID_T_sorted_marked_duplicates_sorted.bam
│   │       └── YOURSAMPLEID_T_sorted_marked_duplicates_sorted.bam.bai
│   └── sort_index
│       ├── YOURSAMPLEID_G_sorted.bam
│       └── YOURSAMPLEID_T_sorted.bam
├── fastqc
│   ├── YOURSAMPLEID_G_R1_1P_fastqc.html
│   ├── YOURSAMPLEID_G_R1_1P_fastqc.zip
│   ├── YOURSAMPLEID_G_R1_fastqc.html
│   ├── YOURSAMPLEID_G_R1_fastqc.zip
│   ├── YOURSAMPLEID_G_R2_2P_fastqc.html
│   ├── YOURSAMPLEID_G_R2_2P_fastqc.zip
│   ├── YOURSAMPLEID_G_R2_fastqc.html
│   ├── YOURSAMPLEID_G_R2_fastqc.zip
│   ├── YOURSAMPLEID_T_R1_1P_fastqc.html
│   ├── YOURSAMPLEID_T_R1_1P_fastqc.zip
│   ├── YOURSAMPLEID_T_R1_fastqc.html
│   ├── YOURSAMPLEID_T_R1_fastqc.zip
│   ├── YOURSAMPLEID_T_R2_2P_fastqc.html
│   ├── YOURSAMPLEID_T_R2_2P_fastqc.zip
│   ├── YOURSAMPLEID_T_R2_fastqc.html
│   └── YOURSAMPLEID_T_R2_fastqc.zip
├── filtered
│   └── YOURSAMPLEID_filtered.vcf
├── multiqc
│   └── multiqc_report.html
├── summaries
│   ├── YOURSAMPLEID_G_sorted_marked_duplicates_sorted.getpileupsummaries.table
│   └── YOURSAMPLEID_T_sorted_marked_duplicates_sorted.getpileupsummaries.table
├── svc
│   ├── annotated_variants
│   │   └── YOURSAMPLEID_annotated_variants.vcf
│   ├── YOURSAMPLEID_chr10_unfiltered.vcf.gz
│   ├── YOURSAMPLEID_chr10_unfiltered.vcf.gz.tbi
│   ├── YOURSAMPLEID_chr11_unfiltered.vcf.gz
│   ├── YOURSAMPLEID_chr11_unfiltered.vcf.gz.tbi
│   ├── YOURSAMPLEID_chr12_unfiltered.vcf.gz
│   ├── YOURSAMPLEID_chr12_unfiltered.vcf.gz.tbi
│   ├── YOURSAMPLEID_chr13_unfiltered.vcf.gz
│   ├── YOURSAMPLEID_chr13_unfiltered.vcf.gz.tbi
│   ├── YOURSAMPLEID_chr14_unfiltered.vcf.gz
│   ├── YOURSAMPLEID_chr14_unfiltered.vcf.gz.tbi
│   ├── YOURSAMPLEID_chr15_unfiltered.vcf.gz
│   ├── YOURSAMPLEID_chr15_unfiltered.vcf.gz.tbi
│   ├── YOURSAMPLEID_chr16_unfiltered.vcf.gz
│   ├── YOURSAMPLEID_chr16_unfiltered.vcf.gz.tbi
│   ├── YOURSAMPLEID_chr17_unfiltered.vcf.gz
│   ├── YOURSAMPLEID_chr17_unfiltered.vcf.gz.tbi
│   ├── YOURSAMPLEID_chr18_unfiltered.vcf.gz
│   ├── YOURSAMPLEID_chr18_unfiltered.vcf.gz.tbi
│   ├── YOURSAMPLEID_chr19_unfiltered.vcf.gz
│   ├── YOURSAMPLEID_chr19_unfiltered.vcf.gz.tbi
│   ├── YOURSAMPLEID_chr1_unfiltered.vcf
│   ├── YOURSAMPLEID_chr1_unfiltered.vcf.gz
│   ├── YOURSAMPLEID_chr1_unfiltered.vcf.gz.tbi
│   ├── YOURSAMPLEID_chr20_unfiltered.vcf.gz
│   ├── YOURSAMPLEID_chr20_unfiltered.vcf.gz.tbi
│   ├── YOURSAMPLEID_chr21_unfiltered.vcf.gz
│   ├── YOURSAMPLEID_chr21_unfiltered.vcf.gz.tbi
│   ├── YOURSAMPLEID_chr22_unfiltered.vcf.gz
│   ├── YOURSAMPLEID_chr22_unfiltered.vcf.gz.tbi
│   ├── YOURSAMPLEID_chr2_unfiltered.vcf.gz
│   ├── YOURSAMPLEID_chr2_unfiltered.vcf.gz.tbi
│   ├── YOURSAMPLEID_chr3_unfiltered.vcf.gz
│   ├── YOURSAMPLEID_chr3_unfiltered.vcf.gz.tbi
│   ├── YOURSAMPLEID_chr4_unfiltered.vcf.gz
│   ├── YOURSAMPLEID_chr4_unfiltered.vcf.gz.tbi
│   ├── YOURSAMPLEID_chr5_unfiltered.vcf.gz
│   ├── YOURSAMPLEID_chr5_unfiltered.vcf.gz.tbi
│   ├── YOURSAMPLEID_chr6_unfiltered.vcf.gz
│   ├── YOURSAMPLEID_chr6_unfiltered.vcf.gz.tbi
│   ├── YOURSAMPLEID_chr7_unfiltered.vcf.gz
│   ├── YOURSAMPLEID_chr7_unfiltered.vcf.gz.tbi
│   ├── YOURSAMPLEID_chr8_unfiltered.vcf.gz
│   ├── YOURSAMPLEID_chr8_unfiltered.vcf.gz.tbi
│   ├── YOURSAMPLEID_chr9_unfiltered.vcf.gz
│   ├── YOURSAMPLEID_chr9_unfiltered.vcf.gz.tbi
│   ├── YOURSAMPLEID_chrX_unfiltered.vcf.gz
│   ├── YOURSAMPLEID_chrX_unfiltered.vcf.gz.tbi
│   ├── f1r2files
│   │   └── YOURSAMPLEID_read-orientation-model.tar.gz
│   └── sort_index
│       ├── YOURSAMPLEID_normalized.vcf.gz
│       ├── YOURSAMPLEID_normalized.vcf.gz.tbi
│       ├── YOURSAMPLEID_unfiltered_sorted.vcf.gz
│       ├── YOURSAMPLEID_unfiltered_sorted.vcf.gz.tbi
│       ├── YOURSAMPLEID_unfiltered.vcf.all.stats
│       └── YOURSAMPLEID_unfiltered.vcf.gz
├── tables
│   ├── YOURSAMPLEID_T_sorted_marked_duplicates_sorted_contamination_table
│   └── YOURSAMPLEID_T_sorted_marked_duplicates_sorted_segmentation_table
└── trim_reads
    ├── YOURSAMPLEID_G_R1_1P.fastq.gz
    ├── YOURSAMPLEID_G_R1_1U.fastq.gz
    ├── YOURSAMPLEID_G_R2_2P.fastq.gz
    ├── YOURSAMPLEID_G_R2_2U.fastq.gz
    ├── YOURSAMPLEID_T_R1_1P.fastq.gz
    ├── YOURSAMPLEID_T_R1_1U.fastq.gz
    ├── YOURSAMPLEID_T_R2_2P.fastq.gz
    └── YOURSAMPLEID_T_R2_2U.fastq.gz

Would appreciate some eyes on this to double check if there are any files missing that should have been included!