bcbio-vm running on cluster + setting cluster+ template issues

[mortunco]

Dear @chapmanb;

I tried to run tumor-normal paired tutorial as I was able to run bcbionextgen.py without you it would not happen so thanks again). Now I am trying to discover vm because I am really exited about clustering so as my PI. We would like to do this task efficient as possible.

1.

First, I tried to initiate template.yam as it was explained in the AWS setup.

This is my bucket structure.

[ec2-user@ip-172-31-59-88 tempa]$ aws s3 ls s3://tuncproject/bcbiovmrun/input
                           PRE input/
[ec2-user@ip-172-31-59-88 tempa]$ aws s3 ls s3://tuncproject/bcbiovmrun/input/
2016-03-28 13:28:41    3772179 NGv3.bed
2016-03-28 13:28:40       1767 cancer-dream-syn3-getdata.sh
2016-03-28 18:02:21       2742 cancer-dream-syn3.yaml
2016-03-28 18:02:44       2498 deneme.yaml
2016-03-28 18:03:35        584 project1.csv
2016-03-28 13:28:40     434674 refseq-merged.bed
2016-03-28 13:28:40 4102975791 synthetic_challenge_set3_normal_NGv3_1.fq.gz
2016-03-28 13:29:54 4220049669 synthetic_challenge_set3_normal_NGv3_2.fq.gz
2016-03-28 13:31:12     400627 synthetic_challenge_set3_tumor_20pctmasked_truth.tar.gz
2016-03-28 13:31:13     203295 synthetic_challenge_set3_tumor_20pctmasked_truth.vcf.gz
2016-03-28 13:31:13     145034 synthetic_challenge_set3_tumor_20pctmasked_truth.vcf.gz.tbi
2016-03-28 13:31:13      60371 synthetic_challenge_set3_tumor_20pctmasked_truth_regions.bed
2016-03-28 13:31:13      14759 synthetic_challenge_set3_tumor_20pctmasked_truth_sv_DEL.bed
2016-03-28 13:31:13      15566 synthetic_challenge_set3_tumor_20pctmasked_truth_sv_DUP.bed
2016-03-28 13:31:13      14319 synthetic_challenge_set3_tumor_20pctmasked_truth_sv_INS.bed
2016-03-28 13:31:13      15625 synthetic_challenge_set3_tumor_20pctmasked_truth_sv_INV.bed
2016-03-28 13:31:13 4101584178 synthetic_challenge_set3_tumor_NGv3_1.fq.gz
2016-03-28 13:32:29 4218223673 synthetic_challenge_set3_tumor_NGv3_2.fq.gz

Template try and stdout. There is something wrong but I dont know why.

[ec2-user@ip-172-31-59-88 ~]$ bcbio_vm.py template s3://tuncproject/bcbiovm/input/template.yaml s3://tuncproject/bcbiovmrun/input/project.csv
Traceback (most recent call last):
  File "/usr/local/bin/bcbio_vm.py", line 4, in <module>
    __import__('pkg_resources').run_script('bcbio-nextgen-vm==0.1.0a0', 'bcbio_vm.py')
  File "/home/ec2-user/install/bcbio-vm/anaconda/lib/python2.7/site-packages/setuptools-20.3-py2.7.egg/pkg_resources/__init__.py", line 726, in run_script

  File "/home/ec2-user/install/bcbio-vm/anaconda/lib/python2.7/site-packages/setuptools-20.3-py2.7.egg/pkg_resources/__init__.py", line 1491, in run_script

  File "/home/ec2-user/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio_nextgen_vm-0.1.0a0-py2.7.egg/EGG-INFO/scripts/bcbio_vm.py", line 307, in <module>

  File "/home/ec2-user/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/workflow/template.py", line 399, in setup
    template, template_txt = name_to_config(args.template)
  File "/home/ec2-user/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/workflow/template.py", line 128, in name_to_config
    with objectstore.open(template) as in_handle:
  File "/home/ec2-user/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/distributed/objectstore.py", line 636, in open
    return manager.open(fname)
  File "/home/ec2-user/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/distributed/objectstore.py", line 452, in open
    return S3Handle(s3_key)
  File "/home/ec2-user/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/distributed/objectstore.py", line 114, in __init__
    if self._key.name.endswith(".gz"):
AttributeError: 'NoneType' object has no attribute 'name'

## 2. Then, I modified the cancer-dream-syn3.yaml to set path to my s3://to-input-directory. Because I have already gotten a working configuration file from bcbionextgen_py

[ec2-user@ip-172-31-59-88 tempa]$ cat deneme.yaml
#Cancer tumor/normal calling evaluation using synthetic dataset 3
# from the ICGC-TCGA DREAM challenge:
# https://www.synapse.org/#!Synapse:syn312572/wiki/62018

---
details:
- algorithm:
    aligner: bwa
    align_split_size: 5000000
    nomap_split_targets: 100
    mark_duplicates: true
    recalibrate: true
    realign: true
    remove_lcr: true
    platform: illumina
    quality_format: standard
    variantcaller: [mutect, freebayes, vardict, varscan]
    indelcaller: false
    ensemble:
      numpass: 2
    variant_regions: s3://tuncproject/bcbiovmrun/input/NGv3.bed
    # svcaller: [cnvkit, lumpy, delly]
    # coverage_interval: amplicon
  analysis: variant2
  description: syn3-normal
  #files: ../input/synthetic.challenge.set3.normal.bam
  files:
    - s3://tuncproject/bcbiovmrun/input/synthetic_challenge_set3_normal_NGv3_1.fq.gz
    - s3://tuncproject/bcbiovmrun/input/synthetic_challenge_set3_normal_NGv3_2.fq.gz
  genome_build: GRCh37
  metadata:
    batch: syn3
    phenotype: normal
- algorithm:
    aligner: bwa
    align_split_size: 5000000
    nomap_split_targets: 100
    mark_duplicates: true
    recalibrate: true
    realign: true
    remove_lcr: true
    platform: illumina
    quality_format: standard
    variantcaller: [mutect, freebayes, vardict, varscan]
    indelcaller: false
    ensemble:
      numpass: 2
    variant_regions: s3://tuncproject/bcbiovmrun/input/NGv3.bed
    validate_regions: s3://tuncproject/bcbiovmrun/input/synthetic_challenge_set3_tumor_20pctmasked_truth.vcf.gz
    validate_regions: s3://tuncproject/bcbiovmrun/input/synthetic_challenge_set3_tumor_20pctmasked_truth_regions.bed
    # svcaller: [cnvkit, lumpy, delly]
    # coverage_interval: amplicon
  #   svvalidate:
  #     DEL: ../input/synthetic_challenge_set3_tumor_20pctmasked_truth_sv_DEL.bed
  #     DUP: ../input/synthetic_challenge_set3_tumor_20pctmasked_truth_sv_DUP.bed
  #     INS: ../input/synthetic_challenge_set3_tumor_20pctmasked_truth_sv_INS.bed
  #     INV: ../input/synthetic_challenge_set3_tumor_20pctmasked_truth_sv_INV.bed
  analysis: variant2
  description: syn3-tumor
  #files: ../input/synthetic.challenge.set3.tumor.bam
  files:
    - s3://tuncproject/bcbiovmrun/input/synthetic_challenge_set3_tumor_NGv3_1.fq.gz
    - s3://tuncproject/bcbiovmrun/input/synthetic_challenge_set3_tumor_NGv3_2.fq.gz
  genome_build: GRCh37
  metadata:
    batch: syn3
    phenotype: tumor
fc_date: '2014-08-13'
fc_name: dream-syn3
upload:
  dir: s3://tuncproject/bcbiovmrun/final/

I continued to process to set up cluster. Error 2: Setting Cluster up.

Problem with bcbiovm_py aws vpc command Referring to my closed issue I faced with the same problem. I believe I solved it again(maybe luck). I posted this because, you might want to be informed by this problem. I am not sure if this enough for going on.

[ec2-user@ip-172-31-59-88 ~]$ bcbio_vm.py aws vpc
ERROR:gc3.elasticluster:Ignoring cluster `bcbio`: section cloud/amazon-us-east-1: length of value must be at least 1 for dictionary value @ data['ec2_access_key'], section cloud/amazon-us-east-1: length of value must be at least 1 for dictionary value @ data['ec2_secret_key']
Traceback (most recent call last):
  File "/usr/local/bin/bcbio_vm.py", line 4, in <module>
    __import__('pkg_resources').run_script('bcbio-nextgen-vm==0.1.0a0', 'bcbio_vm.py')
  File "/home/ec2-user/install/bcbio-vm/anaconda/lib/python2.7/site-packages/setuptools-20.3-py2.7.egg/pkg_resources/__init__.py", line 726, in run_script

  File "/home/ec2-user/install/bcbio-vm/anaconda/lib/python2.7/site-packages/setuptools-20.3-py2.7.egg/pkg_resources/__init__.py", line 1491, in run_script

  File "/home/ec2-user/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio_nextgen_vm-0.1.0a0-py2.7.egg/EGG-INFO/scripts/bcbio_vm.py", line 307, in <module>

  File "build/bdist.linux-x86_64/egg/bcbiovm/aws/vpc.py", line 14, in bootstrap
  File "build/bdist.linux-x86_64/egg/bcbiovm/aws/vpc.py", line 48, in _setup_vpc
  File "build/bdist.linux-x86_64/egg/bcbiovm/aws/common.py", line 97, in ecluster_config
  File "/home/ec2-user/install/bcbio-vm/anaconda/lib/python2.7/site-packages/elasticluster/conf.py", line 135, in fromConfig
    conf = config_reader.read_config()
  File "/home/ec2-user/install/bcbio-vm/anaconda/lib/python2.7/site-packages/elasticluster/conf.py", line 707, in read_config
    raise errors
voluptuous.MultipleInvalid: section cloud/amazon-us-east-1: length of value must be at least 1 for dictionary value @ data['ec2_access_key']
[ec2-user@ip-172-31-59-88 ~]$ bcbio_vm.py aws iam --recreate
User credentials for bcbio:
 access_key_id: ######3
 secret_access_key: #######
Instance profile: bcbio_full_s3_access

Wrote elasticluster config file at: /home/ec2-user/.bcbio/elasticluster/config
[ec2-user@ip-172-31-59-88 ~]$ bcbio_vm.py aws vpc
VPC bcbio already exists. Skipping. Use --recreate to re-create if needed.
Placement group bcbio_cluster_pg already exists. Skipping

I intiated cluster succesfully. Without getting spesific errors.

[ec2-user@ip-172-31-59-88 tempa]$ bcbio_vm.py aws info
Available clusters: bcbio

Configuration for cluster 'bcbio':
 Frontend: c3.large with 200Gb NFS storage
 Cluster: 2 c3.4xlarge machines

AWS setup:
 OK: expected IAM user 'bcbio' exists.
 OK: expected security group 'bcbio_cluster_sg' exists.
 OK: VPC 'bcbio' exists.

Instances in VPC 'bcbio':
    bcbio-compute001 (c3.4xlarge, running) at 54.175.97.37 in us-east-1a
    bcbio-frontend001 (c3.large, running) at 54.84.196.164 in us-east-1a
    bcbio-compute002 (c3.4xlarge, running) at 52.91.36.245 in us-east-1a

Running analysis.

I am almost sure about why I get this error. Its about the configuration file. If you may guide me to create a proper file I believe I will get this through. I am also sharing the slurm output.

One more thing, my compute nodes add up to 32 core. Is this a problem ? Also, I prefer not to use lustre. Does that benefit siginifcantly more ?

If I can solve my configuration and s3: file structure I think I will be 95% done. Rest is fine tuning. I am ready to do everything you ask.

Thank you very very much for your help and patience even reading all this issue.

Best regards,

Tunc.

SLURM-2.out

[2016-03-29T20:02Z] compute001: Resource requests: bwa, sambamba, samtools; memory: 1.75, 1.75; cores: 15, 15, 15
[2016-03-29T20:02Z] compute001: Configuring 2 jobs to run, using 13 cores each with 23.48203125g of memory reserved for each job
[ProfileCreate] Generating default config file: u'/encrypted/tuncproject/work/log/ipython/ipython_config.py'
[ProfileCreate] Generating default config file: u'/encrypted/tuncproject/work/log/ipython/ipython_kernel_config.py'
[ProfileCreate] Generating default config file: u'/encrypted/tuncproject/work/log/ipython/ipcontroller_config.py'
[ProfileCreate] Generating default config file: u'/encrypted/tuncproject/work/log/ipython/ipengine_config.py'
[ProfileCreate] Generating default config file: u'/encrypted/tuncproject/work/log/ipython/ipcluster_config.py'
2016-03-29 20:02:15.387 [IPClusterStart] Using existing profile dir: u'/encrypted/tuncproject/work/log/ipython'
2016-03-29 20:02:15.388 [IPClusterStart] Searching path [u'/encrypted/tuncproject/work', u'/encrypted/tuncproject/work/log/ipython', '/usr/local/etc/ipython', '/etc/ipython'] for config files
2016-03-29 20:02:15.388 [IPClusterStart] Attempting to load config file: ipython_config.py
2016-03-29 20:02:15.388 [IPClusterStart] Looking for ipython_config in /etc/ipython
2016-03-29 20:02:15.388 [IPClusterStart] Looking for ipython_config in /usr/local/etc/ipython
2016-03-29 20:02:15.388 [IPClusterStart] Looking for ipython_config in /encrypted/tuncproject/work/log/ipython
2016-03-29 20:02:15.389 [IPClusterStart] Loaded config file: /encrypted/tuncproject/work/log/ipython/ipython_config.py
2016-03-29 20:02:15.389 [IPClusterStart] Looking for ipython_config in /encrypted/tuncproject/work
2016-03-29 20:02:15.390 [IPClusterStart] Attempting to load config file: ipcluster_c5176466_33e3_46b1_9f45_d70324e25667_config.py
2016-03-29 20:02:15.390 [IPClusterStart] Looking for ipcontroller_config in /etc/ipython
2016-03-29 20:02:15.390 [IPClusterStart] Looking for ipcontroller_config in /usr/local/etc/ipython
2016-03-29 20:02:15.390 [IPClusterStart] Looking for ipcontroller_config in /encrypted/tuncproject/work/log/ipython
2016-03-29 20:02:15.391 [IPClusterStart] Loaded config file: /encrypted/tuncproject/work/log/ipython/ipcontroller_config.py
2016-03-29 20:02:15.391 [IPClusterStart] Looking for ipcontroller_config in /encrypted/tuncproject/work
2016-03-29 20:02:15.391 [IPClusterStart] Attempting to load config file: ipcluster_c5176466_33e3_46b1_9f45_d70324e25667_config.py
2016-03-29 20:02:15.391 [IPClusterStart] Looking for ipengine_config in /etc/ipython
2016-03-29 20:02:15.391 [IPClusterStart] Looking for ipengine_config in /usr/local/etc/ipython
2016-03-29 20:02:15.391 [IPClusterStart] Looking for ipengine_config in /encrypted/tuncproject/work/log/ipython
2016-03-29 20:02:15.392 [IPClusterStart] Loaded config file: /encrypted/tuncproject/work/log/ipython/ipengine_config.py
2016-03-29 20:02:15.392 [IPClusterStart] Looking for ipengine_config in /encrypted/tuncproject/work
2016-03-29 20:02:15.392 [IPClusterStart] Attempting to load config file: ipcluster_c5176466_33e3_46b1_9f45_d70324e25667_config.py
2016-03-29 20:02:15.392 [IPClusterStart] Looking for ipcluster_config in /etc/ipython
2016-03-29 20:02:15.392 [IPClusterStart] Looking for ipcluster_config in /usr/local/etc/ipython
2016-03-29 20:02:15.393 [IPClusterStart] Looking for ipcluster_config in /encrypted/tuncproject/work/log/ipython
2016-03-29 20:02:15.393 [IPClusterStart] Loaded config file: /encrypted/tuncproject/work/log/ipython/ipcluster_config.py
2016-03-29 20:02:15.393 [IPClusterStart] Looking for ipcluster_config in /encrypted/tuncproject/work
[2016-03-29T20:02Z] compute001: Timing: organize samples
/home/ubuntu/install/bcbio-vm/data/anaconda/lib/python2.7/site-packages/IPython/parallel.py:13: ShimWarning: The `IPython.parallel` package has been deprecated. You should import from ipyparallel instead.
  "You should import from ipyparallel instead.", ShimWarning)
[2016-03-29T20:02Z] compute001: ipython: organize_samples
[2016-03-29T20:02Z] compute002: Running in docker container: b0e4525a887b8c16ab5b3594bab8a2479b89b889f56bc70826da704c706cd822
1 Engines running
[2016-03-29T20:02Z] compute002: Using input YAML configuration: /encrypted/tuncproject/work/deneme-ready.yaml
duration: 310.650228ms

[mortunco]

Dear @chapmanb any news from issues ?

Best, Tunc.

[chapmanb]

Tunc; Apologies about the problems and delay in working through this.

The error appears to be caused by the template file not existing in S3. You've specified s3://tuncproject/bcbiovm/input/template.yaml but the S3 bucket/folder you listed is s3://tuncproject/bcbiovmrun/input/ and also does not contain a template.yaml. I believe it's looking up the path you gave and not finding it, leading the confusing error message. Could you double check that template file exists in S3?

For your IAM and VPC errors, I'm a bit confused about the process leading up to this. Are you exporting the AWS_ keys prior to running the iam and vpc creation commands the first time? This error appears to be from not having them in place on the first run, but then re-creating them with it in place writes the correct configuration.

As far as I can see the other parts are in place and the cluster startup seems to be working okay so hopefully things will work smoother for your analysis with that in place. Hope this helps.

[mortunco]

Dear @chapmanb,

Thank you for the response. As far as I remember, i have seen that in the configuration file the aws key and aws secret key were missing and I filled those spaces. About the template let me try it and report you back again. But I understood that, template should create a configuration file ? Doesn't it ? Is the documentation updated?

Regarding this question, I felt I might have made a mistake and re-run it. The good thing is I got results in couple tries.

Bcbiovm is great! Since I was able to run with the tutorial. I will run for my data.

I have couple questions regarding to my data.

• Could you teach me how to calculate/estimate the amount of storage required for x Gb of data. In my case my bam files add up to ~300 Gb. (i spared 500 gb of space to single cluster one with bcbionextgen. it gave me error related with going out of memory)?
• Do I have to use bcbiovm over +60 cores ?
• How can I get an alert or some kind of notification if process is aborted somehow ? As we know sometimes we get unexpected errors even we run analysis with the same configuration file.
• Lastly, this is more like a using your experience, Since my aim is to call mutations of 214 patients of ICGC. How should I set a pipeline. For example, instead of downloading ICGC data in S3, can I get around with that problem ?

As I go deeper with bcbio I realise what a great job your team has done. I appreciate the fact that you gave me in terms of my master project's fate. Moreover, you are answering all the questions with a polite and caring way.

Best regards,

Tunc.

[chapmanb]

Tunc; Thanks for all the additional questions. I'll try to tackle each of them:

• For the configuration file generation, I'm not sure what you're asking in this question. It's a bit difficult to offer advice now not understanding the exact steps you went through earlier. If you have a working configuration now for iam and vpc, there is no need to regenerate those.
• I estimate between 3x and 4x the size of the initial input files for temporary space. You'll need to have space to support the inputs reads, the alignment BAM files, plus all of the associated variant calls. If you do recalibration and realignment you'll have another set of BAM files so will require 4-5x.
• You'd need to use bcbio_vm if you want to scale on Amazon beyond a single machine. If you're comfortable running on a single larger machine, using standard bcbio_nextgen.py is the suggested way to go.
• We don't have alerts built in. Our current work is on migrating bcbio to have a generic workflow representation so it can run inside other platforms which would have this functionality.
• I'd suggest investigating the NCI cancer cloud (http://www.cancergenomicscloud.org/) which hosts TCGA and ICGC data on Amazon S3. If you can access data through their platform this can save you the work of moving it to S3.

Hope this all helps.