Update output bucket and refactor pipeline stacks

[scwatts]

Change summary

• Updated output S3 bucket for production, now uses data account bucket
• Adjusted output/intermediate filepath prefixes accordingly
• Move responsibility for determining output directory to run.sh
• Remove option to manually specify output directory for now
• Greatly reduce repetition shared between pipeline stacks

Testing

Deployed and tested in dev account

• Hartwig mini COLO829 BAMs
• SBJ000910 (see commands below)
  • WTS (GDS inputs)
  • WGS (GDS inputs)
  • WGTS (S3 inputs, previously staged above)
  • WGTS with existing WGS (S3 inputs, previously staged above)

Batch submit commands for test jobs (click to show)

*WTS (GDS inputs)* ```bash aws batch submit-job \ --job-name SBJ00910_star_align \ --job-queue nextflow-pipeline-ondemand \ --job-definition JobDefinitionstaralignn-f443abcd4432c59 \ --container-overrides '{ "resourceRequirements": [ { "type": "MEMORY", "value": "15000" }, { "type": "VCPU", "value": "2" } ], "command": ["bash", "-c", "./assets/run.sh --subject_id SBJ00910 --sample_id MDX210177 --fastq_fwd gds://development/primary_data/210708_A00130_0166_AH7KTJDSX2/202203122acad425/WTS_NebRNA/MDX210177_L2100732_S1_L001_R1_001.fastq.gz --fastq_rev gds://development/primary_data/210708_A00130_0166_AH7KTJDSX2/202203122acad425/WTS_NebRNA/MDX210177_L2100732_S1_L001_R2_001.fastq.gz"] }' ``` *WGS (GDS inputs)* ```bash aws batch submit-job \ --job-name SBJ00910_oncoanalyser_wgs \ --job-queue nextflow-pipeline-ondemand \ --job-definition JobDefinitiononcoanalys-be222ad3a1290f5 \ --container-overrides '{ "resourceRequirements": [ { "type": "MEMORY", "value": "15000" }, { "type": "VCPU", "value": "2" } ], "command": ["bash", "-c", "./assets/run.sh --mode wgs --subject_id SBJ00910 --tumor_wgs_id MDX210176 --normal_wgs_id MDX210175 --tumor_wgs_bam gds://development/analysis_data/SBJ00910/wgs_tumor_normal/202203129d57935d/L2100746_L2100745_dragen/MDX210176_tumor.bam --normal_wgs_bam gds://development/analysis_data/SBJ00910/wgs_tumor_normal/202203129d57935d/L2100746_L2100745_dragen/MDX210175_normal.bam"] }' ``` *WGTS (S3 inputs, previously staged above)* ```bash aws batch submit-job \ --job-name SBJ00910_oncoanalyser_wgts \ --job-queue nextflow-pipeline-ondemand \ --job-definition JobDefinitiononcoanalys-be222ad3a1290f5 \ --container-overrides '{ "resourceRequirements": [ { "type": "MEMORY", "value": "15000" }, { "type": "VCPU", "value": "2" } ], "command": ["bash", "-c", "./assets/run.sh --mode wgts --subject_id SBJ00910 --tumor_wgs_id MDX210176 --normal_wgs_id MDX210175 --tumor_wts_id MDX210177 --tumor_wgs_bam s3://umccr-temp-dev/temp_data/SBJ00910/oncoanalyser/202305227b6be8e7/staging/MDX210176_tumor.bam --normal_wgs_bam s3://umccr-temp-dev/temp_data/SBJ00910/oncoanalyser/202305227b6be8e7/staging/MDX210175_normal.bam --tumor_wts_bam s3://umccr-temp-dev/analysis_data/SBJ00910/star-align-nf/202305227e302487/MDX210177/MDX210177/MDX210177.md.bam"] }' ``` *WGTS with existing WGS (S3 inputs, previously staged above)* ```bash aws batch submit-job \ --job-name SBJ00910_oncoanalyser_wgts_existing_wgs \ --job-queue nextflow-pipeline-ondemand \ --job-definition JobDefinitiononcoanalys-be222ad3a1290f5 \ --container-overrides '{ "resourceRequirements": [ { "type": "MEMORY", "value": "15000" }, { "type": "VCPU", "value": "2" } ], "command": ["bash", "-c", "./assets/run.sh --mode wgts_existing_wgs --subject_id SBJ00910 --tumor_wgs_id MDX210176 --normal_wgs_id MDX210175 --tumor_wts_id MDX210177 --tumor_wts_bam s3://umccr-temp-dev/analysis_data/SBJ00910/star-align-nf/202305227e302487/MDX210177/MDX210177/MDX210177.md.bam --previous_run_dir s3://umccr-temp-dev/analysis_data/SBJ00910/oncoanalyser/202305227b6be8e7/wgs/MDX210176__MDX210175/SBJ00910_MDX210176/"] }' ```

[scwatts]

Related

• https://github.com/umccr/infrastructure/pull/313
• https://github.com/umccr/infrastructure/pull/314

[victorskl]

oncoanalyser release: Decide output paths for AWS prod

[victorskl]

Reviewing in progress...