search

bcbio / bcbio-nextgen

Validated, scalable, community developed variant calling, RNA-seq and small RNA analysis
https://bcbio-nextgen.readthedocs.io
MIT License
986 stars 353 forks source link

Chr6 Reproducilibty Problem: ``IOError: Failed to check fastq file sizes with: `` #1383

Closed mortunco closed 8 years ago

mortunco commented 8 years ago

Hi,

I am trying to validate if bcbio is set right in our HPC. For this, we need a small data that can be optimised easily. That is why I would like to reproduce chr 6 data. When I run the process, this error comes up from the server. Honestly, I have seen many errors from bcbio, but this is the new one.

Also, given configuration files requires svvalidate files to run. I had to command out related lines in the chr6 configuration file. Can it be related with .gz ? I think, ( if the source is true) the only problem that can cause this problem is gz making cat reading fastq file problematic?

Thank you very much for your help,

Best,

T.

My newest error. 

[tmorova15@yunus work]$ cat yunus_example_error 
[2016-05-06T09:24Z] Resource requests: bwa, sambamba, samtools; memory: 2.00, 2.00; cores: 16, 16, 16
[2016-05-06T09:24Z] Configuring 1 jobs to run, using 16 cores each with 32.1g of memory reserved for each job
[2016-05-06T09:24Z] Timing: organize samples
[2016-05-06T09:24Z] multiprocessing: organize_samples
[2016-05-06T09:24Z] Using input YAML configuration: /mnt/kufs/scratch/tmorova15/contol_data/cancer-syn3-chr6/config/cancer-syn3-chr6.yaml
[2016-05-06T09:24Z] Checking sample YAML configuration: /mnt/kufs/scratch/tmorova15/contol_data/cancer-syn3-chr6/config/cancer-syn3-chr6.yaml
[2016-05-06T09:24Z] Testing minimum versions of installed programs
[2016-05-06T09:24Z] Timing: alignment preparation
[2016-05-06T09:24Z] multiprocessing: prep_align_inputs
[2016-05-06T09:24Z] bgzip input file
[2016-05-06T09:24Z] bgzip input file
[2016-05-06T09:24Z] Resource requests: ; memory: 1.00; cores: 1
[2016-05-06T09:24Z] Configuring 2 jobs to run, using 1 cores each with 1.00g of memory reserved for each job
[2016-05-06T09:24Z] Index input with grabix: cancer-syn3-chr6-normal-1.fq.gz
[2016-05-06T09:24Z] Index input with grabix: cancer-syn3-chr6-normal-2.fq.gz
[2016-05-06T09:24Z] bgzip input file
[2016-05-06T09:24Z] bgzip input file
[2016-05-06T09:25Z] Resource requests: ; memory: 1.00; cores: 1
[2016-05-06T09:25Z] Configuring 2 jobs to run, using 1 cores each with 1.00g of memory reserved for each job
[2016-05-06T09:25Z] Index input with grabix: cancer-syn3-chr6-tumor-1.fq.gz
[2016-05-06T09:25Z] Index input with grabix: cancer-syn3-chr6-tumor-2.fq.gz
[2016-05-06T09:25Z] multiprocessing: disambiguate_split
[2016-05-06T09:25Z] Timing: alignment
[2016-05-06T09:25Z] multiprocessing: process_alignment
[2016-05-06T09:25Z] Aligning lane 1_2015-11-18_syn3-cshl with bwa aligner
Traceback (most recent call last):
  File "/mnt/kufs/scratch/tmorova15/bcbio/bin/bcbio_nextgen.py", line 4, in <module>
    __import__('pkg_resources').run_script('bcbio-nextgen==0.9.7', 'bcbio_nextgen.py')
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/setuptools-20.7.0-py2.7.egg/pkg_resources/__init__.py", line 719, in run_script

  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/setuptools-20.7.0-py2.7.egg/pkg_resources/__init__.py", line 1504, in run_script

  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio_nextgen-0.9.7-py2.7.egg-info/scripts/bcbio_nextgen.py", line 226, in <module>
    main(**kwargs)
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio_nextgen-0.9.7-py2.7.egg-info/scripts/bcbio_nextgen.py", line 43, in main
    run_main(**kwargs)
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 39, in run_main
    fc_dir, run_info_yaml)
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 82, in _run_toplevel
    for xs in pipeline(config, run_info_yaml, parallel, dirs, samples):
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 123, in variant2pipeline
    samples = run_parallel("process_alignment", samples)
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/multi.py", line 28, in run_parallel
    return run_multicore(fn, items, config, parallel=parallel)
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/multi.py", line 86, in run_multicore
    for data in joblib.Parallel(parallel["num_jobs"])(joblib.delayed(fn)(x) for x in items):
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 800, in __call__
    while self.dispatch_one_batch(iterator):
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 658, in dispatch_one_batch
    self._dispatch(tasks)
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 566, in _dispatch
    job = ImmediateComputeBatch(batch)
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 180, in __init__
    self.results = batch()
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 72, in __call__
    return [func(*args, **kwargs) for func, args, kwargs in self.items]
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/utils.py", line 51, in wrapper
    return apply(f, *args, **kwargs)
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/multitasks.py", line 67, in process_alignment
    return sample.process_alignment(*args)
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/sample.py", line 110, in process_alignment
    data = align_to_sort_bam(fastq1, fastq2, aligner, data)
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/alignment.py", line 68, in align_to_sort_bam
    names, align_dir, data)
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/alignment.py", line 103, in _align_from_fastq
    out = align_fn(fastq1, fastq2, align_ref, names, align_dir, data)
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/ngsalign/bwa.py", line 142, in align_pipe
    ("bwa-mem" in dd.get_tools_off(data) or not _can_use_mem(fastq_file, data, min_size))):
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/ngsalign/bwa.py", line 109, in _can_use_mem
    raise IOError("Failed to check fastq file sizes with: %s" % cmd.format(**locals()))
IOError: Failed to check fastq file sizes with: cat <(grabix grab /mnt/kufs/scratch/tmorova15/contol_data/cancer-syn3-chr6/work/align_prep/cancer-syn3-chr6-normal-1.fq.gz 1 8840152) | head -n 8000000 | /mnt/kufs/scratch/tmorova15/bcbio/galaxy/../anaconda/bin/seqtk sample -s42 - 5000 | awk '{if(NR%4==2) print length($1)}' | sort | uniq -c

my configuration file.

[tmorova15@yunus config]$ cat cancer-syn3-chr6.yaml 
# Cancer tumor/normal calling evaluation using synthetic dataset 3
# from the ICGC-TCGA DREAM challenge:
# https://www.synapse.org/#!Synapse:syn312572/wiki/62018
# Subset to chromosome 6 for teaching purposes

---
details:
- algorithm:
    aligner: bwa
    align_split_size: 5000000
    recalibrate: false
    realign: false
    remove_lcr: true
    variant_regions: ../input/NGv3-chr6-hg38.bed
    min_allele_fraction: 2
    variantcaller: [varscan ,vardict, mutect, freebayes]
    indelcaller: scalpel
    ensemble:
      numpass: 2
    #svcaller: [lumpy, manta]
  analysis: variant2
  description: syn3-normal
  files: [../input/cancer-syn3-chr6-normal-1.fq.gz, ../input/cancer-syn3-chr6-normal-2.fq.gz]
  genome_build: hg38
  metadata:
    batch: syn3
    phenotype: normal
    sex: female
- algorithm:
    aligner: bwa
    align_split_size: 5000000
    recalibrate: false
    realign: false
    variant_regions: ../input/NGv3-chr6-hg38.bed
    min_allele_fraction: 2
    variantcaller: [varscan, vardict, mutect, freebayes]
    indelcaller: scalpel
    ensemble:
      numpass: 2
    #svcaller: [lumpy, manta]
    #svprioritize: cancer/civic
    hlacaller: optitype
    remove_lcr: true
    #validate: dream-syn3-crossmap/truth_small_variants.vcf.gz
    #validate_regions: dream-syn3-crossmap/truth_regions.bed
    #svvalidate:
     # DEL: dream-syn3-crossmap/truth_DEL.bed
      #DUP: dream-syn3-crossmap/truth_DUP.bed
      #INV: dream-syn3-crossmap/truth_INV.bed
  analysis: variant2
  description: syn3-tumor
  files: [../input/cancer-syn3-chr6-tumor-1.fq.gz, ../input/cancer-syn3-chr6-tumor-2.fq.gz]
  genome_build: hg38
  metadata:
    batch: syn3
    phenotype: tumor
    sex: female
fc_date: '2015-11-18'
fc_name: syn3-cshl
upload:
  dir: ../final

Input Folder ls

[tmorova15@yunus config]$ ls -la  ../input/
total 733124
drwxrwxr-x 2 tmorova15 domainusers      4096 Nov 16 19:01 .
drwxr-xr-x 6 tmorova15 domainusers      4096 May  6 12:18 ..
-rw-rw-r-- 1 tmorova15 domainusers 184826842 Nov 16 04:08 cancer-syn3-chr6-normal-1.fq.gz
-rw-rw-r-- 1 tmorova15 domainusers 190582035 Nov 16 04:08 cancer-syn3-chr6-normal-2.fq.gz
-rw-rw-r-- 1 tmorova15 domainusers 184661452 Nov 16 04:08 cancer-syn3-chr6-tumor-1.fq.gz
-rw-rw-r-- 1 tmorova15 domainusers 190411241 Nov 16 04:08 cancer-syn3-chr6-tumor-2.fq.gz
-rw-rw-r-- 1 tmorova15 domainusers    212225 Nov 16 04:08 NGv3-chr6-hg38.bed
chapmanb commented 8 years ago

Tunc; Sorry about the issue. The error you're seeing is a sanity check in bcbio that you can read the indexed fastq file and retrieve size counts, but for some reason it's failing. If you run the command separately from bcbio:

cat <(grabix grab /mnt/kufs/scratch/tmorova15/contol_data/cancer-syn3-chr6/work/align_prep/cancer-syn3-chr6-normal-1.fq.gz 1 8840152) | head -n 8000000 | /mnt/kufs/scratch/tmorova15/bcbio/galaxy/../anaconda/bin/seqtk sample -s42 - 5000 | awk '{if(NR%4==2) print length($1)}' | sort | uniq -c

Do you get any errors or other output that might help us track down the problem? Thanks for the help debugging.

mortunco commented 8 years ago

Of course,

I did it. Here is the output. 

[tmorova15@yunus input]$ cat <(grabix grab /mnt/kufs/scratch/tmorova15/contol_data/cancer-syn3-chr6/work/align_prep/cancer-syn3-chr6-normal-1.fq.gz 1 8840152) | head -n 8000000 | /mnt/kufs/scratch/tmorova15/bcbio/galaxy/../anaconda/bin/seqtk sample -s42 - 5000 | awk '{if(NR%4==2) print length($1)}' | sort | uniq -c

   5000 101
chapmanb commented 8 years ago

Tunc; Thank you, that all looks right so I'm not sure why bcbio is failing the sanity check. Does it always fail consistently at this point on re-runs, or is it intermittent? If the later, then there might be some kind of shared filesystem problems with retrieving the file. If it happens consistently then I'm a bit stuck and will have to think up other ideas about why it might fail.

mortunco commented 8 years ago

Brad;

Unfortunately it happened again. Could you guide me in a way that how can we check it in our side. I will ask our HPC - technician about the file system.

Also, I am not 100% trusting our installation so could you please tell me a command that at least can say if the pathings were ok? ( even though i 100% followed documentation, I swear.)

[tmorova15@yunus work]$ cat yunus_example_error 
[2016-05-06T10:34Z] Resource requests: bwa, sambamba, samtools; memory: 2.00, 2.00; cores: 16, 16, 16
[2016-05-06T10:34Z] Configuring 1 jobs to run, using 16 cores each with 32.1g of memory reserved for each job
[2016-05-06T10:34Z] Timing: organize samples
[2016-05-06T10:34Z] multiprocessing: organize_samples
[2016-05-06T10:34Z] Using input YAML configuration: /mnt/kufs/scratch/tmorova15/contol_data/cancer-syn3-chr6/config/cancer-syn3-chr6.yaml
[2016-05-06T10:34Z] Checking sample YAML configuration: /mnt/kufs/scratch/tmorova15/contol_data/cancer-syn3-chr6/config/cancer-syn3-chr6.yaml
[2016-05-06T10:34Z] Testing minimum versions of installed programs
[2016-05-06T10:34Z] Timing: alignment preparation
[2016-05-06T10:34Z] multiprocessing: prep_align_inputs
[2016-05-06T10:34Z] bgzip input file
[2016-05-06T10:34Z] bgzip input file
[2016-05-06T10:34Z] Resource requests: ; memory: 1.00; cores: 1
[2016-05-06T10:34Z] Configuring 2 jobs to run, using 1 cores each with 1.00g of memory reserved for each job
[2016-05-06T10:34Z] Index input with grabix: cancer-syn3-chr6-normal-1.fq.gz
[2016-05-06T10:34Z] Index input with grabix: cancer-syn3-chr6-normal-2.fq.gz
[2016-05-06T10:34Z] bgzip input file
[2016-05-06T10:34Z] bgzip input file
[2016-05-06T10:34Z] Resource requests: ; memory: 1.00; cores: 1
[2016-05-06T10:34Z] Configuring 2 jobs to run, using 1 cores each with 1.00g of memory reserved for each job
[2016-05-06T10:34Z] Index input with grabix: cancer-syn3-chr6-tumor-1.fq.gz
[2016-05-06T10:34Z] Index input with grabix: cancer-syn3-chr6-tumor-2.fq.gz
[2016-05-06T10:34Z] multiprocessing: disambiguate_split
[2016-05-06T10:34Z] Timing: alignment
[2016-05-06T10:34Z] multiprocessing: process_alignment
[2016-05-06T10:34Z] Aligning lane 1_2015-11-18_syn3-cshl with bwa aligner
Traceback (most recent call last):
  File "/mnt/kufs/scratch/tmorova15/bcbio/bin/bcbio_nextgen.py", line 4, in <module>
    __import__('pkg_resources').run_script('bcbio-nextgen==0.9.7', 'bcbio_nextgen.py')
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/setuptools-20.7.0-py2.7.egg/pkg_resources/__init__.py", line 719, in run_script

  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/setuptools-20.7.0-py2.7.egg/pkg_resources/__init__.py", line 1504, in run_script

  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio_nextgen-0.9.7-py2.7.egg-info/scripts/bcbio_nextgen.py", line 226, in <module>
    main(**kwargs)
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio_nextgen-0.9.7-py2.7.egg-info/scripts/bcbio_nextgen.py", line 43, in main
    run_main(**kwargs)
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 39, in run_main
    fc_dir, run_info_yaml)
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 82, in _run_toplevel
    for xs in pipeline(config, run_info_yaml, parallel, dirs, samples):
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 123, in variant2pipeline
    samples = run_parallel("process_alignment", samples)
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/multi.py", line 28, in run_parallel
    return run_multicore(fn, items, config, parallel=parallel)
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/multi.py", line 86, in run_multicore
    for data in joblib.Parallel(parallel["num_jobs"])(joblib.delayed(fn)(x) for x in items):
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 800, in __call__
    while self.dispatch_one_batch(iterator):
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 658, in dispatch_one_batch
    self._dispatch(tasks)
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 566, in _dispatch
    job = ImmediateComputeBatch(batch)
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 180, in __init__
    self.results = batch()
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 72, in __call__
    return [func(*args, **kwargs) for func, args, kwargs in self.items]
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/utils.py", line 51, in wrapper
    return apply(f, *args, **kwargs)
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/multitasks.py", line 67, in process_alignment
    return sample.process_alignment(*args)
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/sample.py", line 110, in process_alignment
    data = align_to_sort_bam(fastq1, fastq2, aligner, data)
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/alignment.py", line 68, in align_to_sort_bam
    names, align_dir, data)
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/alignment.py", line 103, in _align_from_fastq
    out = align_fn(fastq1, fastq2, align_ref, names, align_dir, data)
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/ngsalign/bwa.py", line 142, in align_pipe
    ("bwa-mem" in dd.get_tools_off(data) or not _can_use_mem(fastq_file, data, min_size))):
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/ngsalign/bwa.py", line 109, in _can_use_mem
    raise IOError("Failed to check fastq file sizes with: %s" % cmd.format(**locals()))
IOError: Failed to check fastq file sizes with: cat <(grabix grab /mnt/kufs/scratch/tmorova15/contol_data/cancer-syn3-chr6/work/align_prep/cancer-syn3-chr6-normal-1.fq.gz 1 8840152) | head -n 8000000 | /mnt/kufs/scratch/tmorova15/bcbio/galaxy/../anaconda/bin/seqtk sample -s42 - 5000 | awk '{if(NR%4==2) print length($1)}' | sort | uniq -c
chapmanb commented 8 years ago

Tunc; Sorry that the problem is consistent. My best guess now is that something is wrong with the paths or tools when run on a compute node that causes this to fail. Unfortunately we swallow the standard error here because it can give some non-problematic warnings that were confusing people earlier (the head command truncates the previous command) but I think it's better to know what is going on so reverted this. If you update to the latest development:

bcbio_nextgen.py upgrade -u development

and re-run hopefully it'll give a useful error message about why the command fails. Hope this will help identify the issue.

mortunco commented 8 years ago

Hi Brad:

I installed the development version and ran the process. The output is right below. Also, I would like to point out an interesting thing(for me at least). In amazon, i used to run my processes with screen and the stdout of the process and bcbio-nextgen-debug.log were used to be same. But here, ( I usually copy/paste the jobid.error file to you ), they are not same. Can it be the clue of our problem. Forgive my ignorance about the systems. My aim is to learn everyday and be like you at the end.

Best regards,

Tunc.

This is the output of my final run with your most recent advice.

[tmorova15@yunus work]$ cat 532867.error 
[2016-05-07T13:31Z] System YAML configuration: /mnt/kufs/scratch/tmorova15/bcbio/galaxy/bcbio_system.yaml
[2016-05-07T13:31Z] Resource requests: bwa, sambamba, samtools; memory: 2.00, 2.00; cores: 16, 16, 16
[2016-05-07T13:31Z] Configuring 1 jobs to run, using 16 cores each with 32.1g of memory reserved for each job
[2016-05-07T13:31Z] Timing: organize samples
[2016-05-07T13:31Z] multiprocessing: organize_samples
[2016-05-07T13:31Z] Using input YAML configuration: /mnt/kufs/scratch/tmorova15/contol_data/cancer-syn3-chr6/config/cancer-syn3-chr6.yaml
[2016-05-07T13:31Z] Checking sample YAML configuration: /mnt/kufs/scratch/tmorova15/contol_data/cancer-syn3-chr6/config/cancer-syn3-chr6.yaml
[2016-05-07T13:31Z] Testing minimum versions of installed programs
[2016-05-07T13:31Z] Timing: alignment preparation
[2016-05-07T13:31Z] multiprocessing: prep_align_inputs
[2016-05-07T13:31Z] bgzip input file
[2016-05-07T13:32Z] bgzip input file
[2016-05-07T13:32Z] Resource requests: ; memory: 1.00; cores: 1
[2016-05-07T13:32Z] Configuring 2 jobs to run, using 1 cores each with 1.00g of memory reserved for each job
[2016-05-07T13:32Z] Index input with grabix: cancer-syn3-chr6-normal-1.fq.gz
[2016-05-07T13:32Z] Index input with grabix: cancer-syn3-chr6-normal-2.fq.gz
[2016-05-07T13:32Z] bgzip input file
[2016-05-07T13:32Z] bgzip input file
[2016-05-07T13:32Z] writer fwrite: bug encountered
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 21, in run
    _do_run(cmd, checks, log_stdout)
  File "/mnt/kufs/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 95, in _do_run
    raise subprocess.CalledProcessError(exitcode, error_msg)
subprocess.CalledProcessError: Command 'set -o pipefail; gunzip -c /mnt/kufs/scratch/tmorova15/contol_data/cancer-syn3-chr6/input/cancer-syn3-chr6-tumor-2.fq.gz | /mnt/kufs/scratch/tmorova15/bcbio/galaxy/../anaconda/bin/pbgzip -n 16  -c /dev/stdin > /mnt/kufs/scratch/tmorova15/contol_data/cancer-syn3-chr6/work/align_prep/tx/tmpVmLizD/cancer-syn3-chr6-tumor-2.fq.gz
writer fwrite: bug encountered
writer writer_write_block: bug encountered
' returned non-zero exit status 1

This is my bcbio-debug.log output.

[tmorova15@yunus log]$ cat *debug*
[2016-05-07T13:31Z] System YAML configuration: /mnt/kufs/scratch/tmorova15/bcbio/galaxy/bcbio_system.yaml
[2016-05-07T13:31Z] Resource requests: bwa, sambamba, samtools; memory: 2.00, 2.00; cores: 16, 16, 16
[2016-05-07T13:31Z] Configuring 1 jobs to run, using 16 cores each with 32.1g of memory reserved for each job
[2016-05-07T13:31Z] Timing: organize samples
[2016-05-07T13:31Z] multiprocessing: organize_samples
[2016-05-07T13:31Z] Using input YAML configuration: /mnt/kufs/scratch/tmorova15/contol_data/cancer-syn3-chr6/config/cancer-syn3-chr6.yaml
[2016-05-07T13:31Z] Checking sample YAML configuration: /mnt/kufs/scratch/tmorova15/contol_data/cancer-syn3-chr6/config/cancer-syn3-chr6.yaml
[2016-05-07T13:31Z] Testing minimum versions of installed programs
[2016-05-07T13:31Z] Timing: alignment preparation
[2016-05-07T13:31Z] multiprocessing: prep_align_inputs
[2016-05-07T13:31Z] bgzip input file
[2016-05-07T13:32Z] bgzip input file
[2016-05-07T13:32Z] Resource requests: ; memory: 1.00; cores: 1
[2016-05-07T13:32Z] Configuring 2 jobs to run, using 1 cores each with 1.00g of memory reserved for each job
[2016-05-07T13:32Z] Index input with grabix: cancer-syn3-chr6-normal-1.fq.gz
[2016-05-07T13:32Z] Index input with grabix: cancer-syn3-chr6-normal-2.fq.gz
[2016-05-07T13:32Z] bgzip input file
[2016-05-07T13:32Z] bgzip input file

This is just to prove that I installed the development version right.

[tmorova15@yunus bcbio]$ bcbio_nextgen.py upgrade -u development
Fetching package metadata: ......
Solving package specifications: .........
Warning: 2 possible package resolutions (only showing differing packages):
  - sailfish-0.9.0-1.tar.bz2
  - sailfish-0.9.0-boost1.60_1.tar.bz2

Package plan for installation in environment /mnt/kufs/scratch/tmorova15/bcbio/anaconda:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    yamllint-1.2.1             |           py27_0          66 KB
    bcbio-nextgen-0.9.8a0      |           py27_4         682 KB
    ------------------------------------------------------------
                                           Total:         747 KB

The following NEW packages will be INSTALLED:

    yamllint:      1.2.1-py27_0  

The following packages will be UPDATED:

    bcbio-nextgen: 0.9.7-py27_0 --> 0.9.8a0-py27_4

Upgrading bcbio-nextgen to latest development version
Collecting bcbio-nextgen from git+https://github.com/chapmanb/bcbio-nextgen.git#egg=bcbio-nextgen
  Cloning https://github.com/chapmanb/bcbio-nextgen.git to /tmp/pip-build-jM5_Ly/bcbio-nextgen
Installing collected packages: bcbio-nextgen
  Found existing installation: bcbio-nextgen 0.9.8a0
    Uninstalling bcbio-nextgen-0.9.8a0:
      Successfully uninstalled bcbio-nextgen-0.9.8a0
  Running setup.py install for bcbio-nextgen ... done
Successfully installed bcbio-nextgen-0.9.8a0
Upgrade of bcbio-nextgen development code complete.
Upgrading third party tools to latest versions
--2016-05-07 16:26:20--  https://github.com/chapmanb/cloudbiolinux/archive/master.tar.gz
Resolving github.com... 192.30.252.120
Connecting to github.com|192.30.252.120|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/chapmanb/cloudbiolinux/tar.gz/master [following]
--2016-05-07 16:26:20--  https://codeload.github.com/chapmanb/cloudbiolinux/tar.gz/master
Resolving codeload.github.com... 192.30.252.163
Connecting to codeload.github.com|192.30.252.163|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5151216 (4.9M) [application/x-gzip]
Saving to: `STDOUT'

100%[=================================================================================================================================>] 5,151,216   1.91M/s   in 2.6s    

2016-05-07 16:26:24 (1.91 MB/s) - `-' saved [5151216/5151216]

Setting up virtual machine
INFO: <cloudbio.flavor.Flavor instance at 0x2ae91b32ea28>
INFO: This is a ngs_pipeline_minimal flavor
INFO: Reading default fabricrc.txt
DBG [config.py]: Using config file /mnt/kufs/scratch/tmorova15/bcbio/tmpbcbio-install/cloudbiolinux/cloudbio/../config/fabricrc.txt
INFO: Distribution __auto__
INFO: Get local environment
INFO: CentOS setup
DBG [distribution.py]: NixPkgs: Ignored
[localhost] local: echo $HOME
[localhost] local: uname -m
INFO: Now, testing connection to host...
INFO: Connection to host appears to work!
DBG [utils.py]: Expand paths
DBG [config.py]: Using config file /mnt/kufs/scratch/tmorova15/bcbio/tmpbcbio-install/cloudbiolinux/cloudbio/../contrib/flavor/ngs_pipeline_minimal/main.yaml
INFO: Meta-package information from /mnt/kufs/scratch/tmorova15/bcbio/tmpbcbio-install/cloudbiolinux/cloudbio/../contrib/flavor/ngs_pipeline_minimal/main.yaml
- Packages: minimal,libraries,perl,python,java,r,ruby,bio_nextgen
- Libraries: r-libs
[localhost] local: whoami
[localhost] local: echo $PATH
DBG [config.py]: Using config file /mnt/kufs/scratch/tmorova15/bcbio/tmpbcbio-install/cloudbiolinux/cloudbio/../contrib/flavor/ngs_pipeline_minimal/custom.yaml
INFO: Reading /mnt/kufs/scratch/tmorova15/bcbio/tmpbcbio-install/cloudbiolinux/cloudbio/../contrib/flavor/ngs_pipeline_minimal/custom.yaml
DBG [shared.py]: Packages to install: 
DBG [config.py]: Using config file /mnt/kufs/scratch/tmorova15/bcbio/tmpbcbio-install/cloudbiolinux/cloudbio/../config/chef_recipes.yaml
INFO: Reading /mnt/kufs/scratch/tmorova15/bcbio/tmpbcbio-install/cloudbiolinux/cloudbio/../config/chef_recipes.yaml
DBG [shared.py]: Packages to install: 
DBG [config.py]: Using config file /mnt/kufs/scratch/tmorova15/bcbio/tmpbcbio-install/cloudbiolinux/cloudbio/../config/puppet_classes.yaml
INFO: Reading /mnt/kufs/scratch/tmorova15/bcbio/tmpbcbio-install/cloudbiolinux/cloudbio/../config/puppet_classes.yaml
DBG [shared.py]: Packages to install: 
DBG [config.py]: Using config file /mnt/kufs/scratch/tmorova15/bcbio/tmpbcbio-install/cloudbiolinux/cloudbio/../contrib/flavor/ngs_pipeline_minimal/main.yaml
INFO: Meta-package information from /mnt/kufs/scratch/tmorova15/bcbio/tmpbcbio-install/cloudbiolinux/cloudbio/../contrib/flavor/ngs_pipeline_minimal/main.yaml
- Packages: minimal,libraries,perl,python,java,r,ruby,bio_nextgen
- Libraries: r-libs
DBG [config.py]: Using config file /mnt/kufs/scratch/tmorova15/bcbio/tmpbcbio-install/cloudbiolinux/cloudbio/../contrib/flavor/ngs_pipeline_minimal/packages-homebrew.yaml
INFO: Reading /mnt/kufs/scratch/tmorova15/bcbio/tmpbcbio-install/cloudbiolinux/cloudbio/../contrib/flavor/ngs_pipeline_minimal/packages-homebrew.yaml
DBG [shared.py]: Packages to install: 
DBG [config.py]: Using config file /mnt/kufs/scratch/tmorova15/bcbio/tmpbcbio-install/cloudbiolinux/cloudbio/../contrib/flavor/ngs_pipeline_minimal/main.yaml
INFO: Meta-package information from /mnt/kufs/scratch/tmorova15/bcbio/tmpbcbio-install/cloudbiolinux/cloudbio/../contrib/flavor/ngs_pipeline_minimal/main.yaml
- Packages: minimal,libraries,perl,python,java,r,ruby,bio_nextgen
- Libraries: r-libs
DBG [config.py]: Using config file /mnt/kufs/scratch/tmorova15/bcbio/tmpbcbio-install/cloudbiolinux/cloudbio/../contrib/flavor/ngs_pipeline_minimal/packages-conda.yaml
INFO: Reading /mnt/kufs/scratch/tmorova15/bcbio/tmpbcbio-install/cloudbiolinux/cloudbio/../contrib/flavor/ngs_pipeline_minimal/packages-conda.yaml
DBG [shared.py]: Packages to install: age-metasv,bamtools,bamutil,bcbio-prioritize,bcbio-rnaseq,bcbio-variation,bcbio-variation-recall,bcbreport,bcftools,bedtools=2.24.0,bio-vcf,biobambam,bowtie,bowtie2,bwa,bwakit,cage,cancerit-allelecount,cnvkit,cpat,cramtools,crossmap,cufflinks,cutadapt,delly,express,fastqc,freebayes=1.0.2,gatk-framework,gemini,geneimpacts,glpk=4.57,grabix,gsort,hall-lab-svtools,hap.py,hisat2,hmmlearn,htseq,htslib,impute2,kraken,lofreq,lumpy-sv,macs2,manta,mbuffer,metasv,mirdeep2,moreutils,multiqc,multiqc-bcbio,novoalign,oncofuse,optitype,pandoc,parallel,pbgzip,perl-sanger-cgp-battenberg,phylowgs,picard,pindel,platypus-variant,pyloh,pytabix,pythonpy,qsignature,qualimap,rapmap,razers3,rtg-tools,sailfish,salmon,sambamba,samblaster,samtools,scalpel,seqbuster,seqtk,sickle-trim,simple_sv_annotation,snap-aligner,snpeff,spades,star,stringtie,subread,svtyper,tdrmapper,theta2,tophat,ucsc-bedgraphtobigwig,ucsc-bedtobigbed,ucsc-bigbedinfo,ucsc-bigbedsummary,ucsc-bigbedtobed,ucsc-bigwiginfo,ucsc-bigwigsummary,ucsc-bigwigtobedgraph,ucsc-bigwigtowig,ucsc-fatotwobit,ucsc-gtftogenepred,ucsc-liftover,ucsc-wigtobigwig,vardict,vardict-java,variant-effect-predictor,variantbam,varscan,vawk,vcf2db,vcfanno,vcflib,viennarna,vt,wham,awscli,p7zip,s3gof3r,perl-app-cpanminus,perl-archive-extract,perl-archive-zip,perl-bio-db-sam,perl-cgi,perl-dbi,perl-encode-locale,perl-file-fetch,perl-file-sharedir,perl-file-sharedir-install,perl-ipc-system-simple,perl-lwp-protocol-https,perl-lwp-simple,perl-statistics-descriptive,perl-time-hires,perl-vcftools-vcf,bioconductor-annotate,bioconductor-biocinstaller,bioconductor-biocstyle,bioconductor-biostrings,bioconductor-biovizbase,bioconductor-bubbletree,bioconductor-cn.mops,bioconductor-degreport,bioconductor-deseq2,bioconductor-dexseq,bioconductor-dnacopy,bioconductor-genomicranges,bioconductor-iranges,bioconductor-limma,bioconductor-rtracklayer,r,r-cghflasso,r-chbutils,r-devtools,r-dplyr,r-dt,r-ggdendro,r-ggplot2,r-gplots,r-gsalib,r-knitr,r-pheatmap,r-plyr,r-pscbs,r-reshape,r-rmarkdown,r-rsqlite,r-sleuth,r-snow,r-stringi,r-wasabi
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda info --json
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda install -y -c bioconda -c r -c conda-forge age-metasv bamtools bamutil bcbio-prioritize bcbio-rnaseq bcbio-variation bcbio-variation-recall bcbreport bcftools bedtools=2.24.0 bio-vcf biobambam bowtie bowtie2 bwa bwakit cage cancerit-allelecount cnvkit cpat cramtools crossmap cufflinks cutadapt delly express fastqc freebayes=1.0.2 gatk-framework gemini geneimpacts glpk=4.57 grabix gsort hall-lab-svtools hap.py hisat2 hmmlearn htseq htslib impute2 kraken lofreq lumpy-sv macs2 manta mbuffer metasv mirdeep2 moreutils multiqc multiqc-bcbio novoalign oncofuse optitype pandoc parallel pbgzip perl-sanger-cgp-battenberg phylowgs picard pindel platypus-variant pyloh pytabix pythonpy qsignature qualimap rapmap razers3 rtg-tools sailfish salmon sambamba samblaster samtools scalpel seqbuster seqtk sickle-trim simple_sv_annotation snap-aligner snpeff spades star stringtie subread svtyper tdrmapper theta2 tophat ucsc-bedgraphtobigwig ucsc-bedtobigbed ucsc-bigbedinfo ucsc-bigbedsummary ucsc-bigbedtobed ucsc-bigwiginfo ucsc-bigwigsummary ucsc-bigwigtobedgraph ucsc-bigwigtowig ucsc-fatotwobit ucsc-gtftogenepred ucsc-liftover ucsc-wigtobigwig vardict vardict-java variant-effect-predictor variantbam varscan vawk vcf2db vcfanno vcflib viennarna vt wham awscli p7zip s3gof3r perl-app-cpanminus perl-archive-extract perl-archive-zip perl-bio-db-sam perl-cgi perl-dbi perl-encode-locale perl-file-fetch perl-file-sharedir perl-file-sharedir-install perl-ipc-system-simple perl-lwp-protocol-https perl-lwp-simple perl-statistics-descriptive perl-time-hires perl-vcftools-vcf bioconductor-annotate bioconductor-biocinstaller bioconductor-biocstyle bioconductor-biostrings bioconductor-biovizbase bioconductor-bubbletree bioconductor-cn.mops bioconductor-degreport bioconductor-deseq2 bioconductor-dexseq bioconductor-dnacopy bioconductor-genomicranges bioconductor-iranges bioconductor-limma bioconductor-rtracklayer r r-cghflasso r-chbutils r-devtools r-dplyr r-dt r-ggdendro r-ggplot2 r-gplots r-gsalib r-knitr r-pheatmap r-plyr r-pscbs r-reshape r-rmarkdown r-rsqlite r-sleuth r-snow r-stringi r-wasabi
Fetching package metadata: ..........
Solving package specifications: .........
Warning: 2 possible package resolutions (only showing differing packages):
  - sailfish-0.9.0-1.tar.bz2
  - sailfish-0.9.0-boost1.60_1.tar.bz2

# All requested packages already installed.
# packages in environment at /mnt/kufs/scratch/tmorova15/bcbio/anaconda:
#
age-metasv                2015.01.29.3                  0    bioconda
awscli                    1.8.3                    py27_0    bioconda
bamtools                  2.4.0                         1    bioconda
bamutil                   1.0.14                        1    bioconda
bcbio-prioritize          0.0.7                         0    bioconda
bcbio-rnaseq              1.1.1                  r3.2.2_1    bioconda
bcbio-variation           0.2.6                         0    bioconda
bcbio-variation-recall    0.1.6                         0    bioconda
bcbreport                 0.99.28                  py27_0    bioconda
bcftools                  1.3                           0    bioconda
bedtools                  2.24.0                        0    bioconda
bio-vcf                   0.9.2                         0    bioconda
biobambam                 2.0.42                        0    bioconda
bioconductor-annotate     1.48.0                        1    bioconda
bioconductor-biocinstaller 1.20.1                        0    bioconda
bioconductor-biocstyle    1.8.0                  r3.2.2_0    bioconda
bioconductor-biostrings   2.38.4                        0    bioconda
bioconductor-biovizbase   1.18.0                        1    bioconda
bioconductor-bubbletree   2.1.5                         0    bioconda
bioconductor-cn.mops      1.16.2                        0    bioconda
bioconductor-degreport    1.7.2                  r3.2.2_1    bioconda
bioconductor-deseq2       1.10.1                 r3.2.2_0    bioconda
bioconductor-dexseq       1.16.6                 r3.2.2_0    bioconda
bioconductor-dnacopy      1.44.0                 r3.2.2_0    bioconda
bioconductor-genomicranges 1.22.4                        0    bioconda
bioconductor-iranges      2.4.8                         0    bioconda
bioconductor-limma        3.27.4                        1    bioconda
bioconductor-rtracklayer  1.30.4                        0    bioconda
bowtie                    1.1.2                    py27_1    bioconda
bowtie2                   2.2.8                    py27_2    bioconda
bwa                       0.7.13                        0    bioconda
bwakit                    0.7.12                        0    bioconda
cage                      2016.01.24               py27_0    bioconda
cancerit-allelecount      2.1.2                         1    bioconda
cnvkit                    0.7.11                   py27_0    bioconda
cpat                      1.2.2                    py27_2    bioconda
cramtools                 3.0.b127                      0    bioconda
crossmap                  0.2.2                    py27_2    bioconda
cufflinks                 2.2.1                    py27_1    bioconda
cutadapt                  1.9.1                    py27_0    bioconda
delly                     0.7.2                         1    bioconda
express                   1.5.1                         0    bioconda
fastqc                    0.11.5                        1    bioconda
freebayes                 1.0.2                         0    bioconda
gatk-framework            3.5.21                        0    bioconda
gemini                    0.18.3                   py27_1    bioconda
geneimpacts               0.1.1                    py27_0    bioconda
glpk                      4.57                          0    bioconda
grabix                    0.1.6                         0    bioconda
gsort                     0.0.2                         0    bioconda
hall-lab-svtools          0.1.1                    py27_0    bioconda
hap.py                    0.2.9                    py27_1    bioconda
hisat2                    2.0.3beta                py27_0    bioconda
hmmlearn                  20151031                 py27_0    bioconda
htseq                     0.6.1p1                  py27_0    bioconda
htslib                    1.3                           0    bioconda
impute2                   2.3.2                         0    bioconda
kraken                    0.10.6_eaf8fb68               2    bioconda
lofreq                    2.1.2                    py27_0    bioconda
lumpy-sv                  0.2.12                   py27_1    bioconda
macs2                     2.1.0                         0    bioconda
manta                     0.29.3                   py27_0    bioconda
mbuffer                   20160228                      0    bioconda
metasv                    0.4.0                    py27_4    bioconda
mirdeep2                  2.0.0.7                       6    bioconda
moreutils                 0.5.7                         1    bioconda
multiqc                   0.6                      py27_0    bioconda
multiqc-bcbio             0.1.1                    py27_1    bioconda
novoalign                 3.04.04                       0    bioconda
oncofuse                  1.1.0                         1    bioconda
optitype                  2015.10.20               py27_1    bioconda
p7zip                     15.09                         3    bioconda
pandoc                    1.17.0.2                      0    conda-forge
parallel                  20150922                      0    bioconda
pbgzip                    2015.10.28                    0    bioconda
perl-app-cpanminus        1.7039                        2    bioconda
perl-archive-extract      0.76                          3    bioconda
perl-archive-zip          1.55                          3    bioconda
perl-bio-db-sam           1.41                          2    bioconda
perl-cgi                  4.22                          3    bioconda
perl-dbi                  1.634                         2    bioconda
perl-encode-locale        1.05                          3    bioconda
perl-file-fetch           0.48                          3    bioconda
perl-file-sharedir        1.102                         3    bioconda
perl-file-sharedir-install 0.10                          3    bioconda
perl-ipc-system-simple    1.25                          3    bioconda
perl-lwp-protocol-https   6.06                          2    bioconda
perl-lwp-simple           6.15                          3    bioconda
perl-sanger-cgp-battenberg 1.4.1                         3    bioconda
perl-statistics-descriptive 3.0609                        3    bioconda
perl-time-hires           1.9728                        2    bioconda
perl-vcftools-vcf         0.953                         3    bioconda
phylowgs                  20150714                 py27_1    bioconda
picard                    1.141                         3    bioconda
pindel                    0.2.5b8                       0    bioconda
platypus-variant          0.8.1                    py27_0    bioconda
pyloh                     1.4.1                    py27_1    bioconda
pytabix                   0.0.2                    py27_1    bioconda
pythonpy                  0.4.2                    py27_0    bioconda
qsignature                0.1pre                        2    bioconda
qualimap                  2.1.3                         1    bioconda
r                         3.2.2                         0    r
r-cghflasso               0.2_1                  r3.2.2_0    bioconda
r-chbutils                0.1_2015_12_21         r3.2.2_0    bioconda
r-devtools                1.9.1                 r3.2.2_0a    r
r-dplyr                   0.4.3                 r3.2.2_0a    r
r-dt                      0.1                   r3.2.2_0a    r
r-ggdendro                0.1_17                 r3.2.2_0    bioconda
r-ggplot2                 2.1.0                  r3.2.2_0    bioconda
r-gplots                  2.17.0                r3.2.2_0a    r
r-gsalib                  2.1                    r3.2.2_0    bioconda
r-knitr                   1.11                  r3.2.2_0a    r
r-pheatmap                1.0.8                  r3.2.2_0    bioconda
r-plyr                    1.8.3                 r3.2.2_0a    r
r-pscbs                   0.60.0                 r3.2.2_0    bioconda
r-reshape                 0.8.5                 r3.2.2_1a    r
r-rmarkdown               0.8.1                 r3.2.2_0a    r
r-rsqlite                 1.0.0                 r3.2.2_1a    r
r-sleuth                  0.28.0                 r3.2.2_0    bioconda
r-snow                    0.4_1                  r3.2.2_0    bioconda
r-stringi                 1.0_1                 r3.2.2_0a    r
r-wasabi                  0.0.1                  r3.2.2_0    bioconda
rapmap                    v0.2.1                        1    bioconda
razers3                   3.5.0                         0    bioconda
rtg-tools                 3.6                           0    bioconda
s3gof3r                   0.5.0                         0    bioconda
sailfish                  0.9.0                         1    bioconda
salmon                    0.6.0               boost1.60_2    bioconda
sambamba                  0.5.9                         1    bioconda
samblaster                0.1.22                        0    bioconda
samtools                  1.3.1                         0    bioconda
scalpel                   0.5.1                         3    bioconda
seqbuster                 2.3                           3    bioconda
seqtk                     r82                           0    bioconda
sickle-trim               1.33                          0    bioconda
simple_sv_annotation      2015.11.24               py27_0    bioconda
snap-aligner              1.0beta.23                    0    bioconda
snpeff                    4.2                           0    bioconda
spades                    3.7.0                         0    bioconda
star                      2.5.1b                        0    bioconda
stringtie                 1.2.2                         1    bioconda
subread                   1.5.0                         0    bioconda
svtyper                   0.0.2                    py27_4    bioconda
tdrmapper                 1.0                           1    bioconda
theta2                    0.7                      py27_0    bioconda
tophat                    2.1.0                    py27_0    bioconda
ucsc-bedgraphtobigwig     324                           0    bioconda
ucsc-bedtobigbed          324                           0    bioconda
ucsc-bigbedinfo           324                           0    bioconda
ucsc-bigbedsummary        324                           0    bioconda
ucsc-bigbedtobed          324                           0    bioconda
ucsc-bigwiginfo           324                           0    bioconda
ucsc-bigwigsummary        324                           0    bioconda
ucsc-bigwigtobedgraph     324                           0    bioconda
ucsc-bigwigtowig          324                           0    bioconda
ucsc-fatotwobit           324                           0    bioconda
ucsc-gtftogenepred        324                           0    bioconda
ucsc-liftover             324                           0    bioconda
ucsc-wigtobigwig          324                           0    bioconda
vardict                   2016.02.19                    0    bioconda
vardict-java              1.4.5                         1    bioconda
variant-effect-predictor  83                            3    bioconda
variantbam                1.2.1_2015.01.08               0    bioconda
varscan                   2.4.1                         0    bioconda
vawk                      0.0.2                    py27_0    bioconda
vcf2db                    2016.04.29               py27_0    bioconda
vcfanno                   0.0.11                        0    bioconda
vcflib                    1.0.0_rc0                     0    bioconda
viennarna                 2.2.5                         2    bioconda
vt                        2015.11.10                    0    bioconda
wham                      1.7.0.162                     0    bioconda
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f age-metasv
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bamtools
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bamutil
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bcbio-prioritize
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bcbio-rnaseq
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bcbio-variation
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bcbio-variation-recall
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bcbreport
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bcftools
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bedtools
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bio-vcf
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f biobambam
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bowtie
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bowtie2
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bwa
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bwakit
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f cage
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f cancerit-allelecount
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f cnvkit
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f cpat
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f cramtools
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f crossmap
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f cufflinks
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f cutadapt
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f delly
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f express
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f fastqc
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f freebayes
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f gatk-framework
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f gemini
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f geneimpacts
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f glpk
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f grabix
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f gsort
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f hall-lab-svtools
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f hap.py
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f hisat2
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f hmmlearn
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f htseq
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f htslib
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f impute2
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f kraken
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f lofreq
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f lumpy-sv
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f macs2
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f manta
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f mbuffer
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f metasv
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f mirdeep2
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f moreutils
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f multiqc
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f multiqc-bcbio
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f novoalign
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f oncofuse
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f optitype
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f pandoc
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f parallel
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f pbgzip
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f perl-sanger-cgp-battenberg
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f phylowgs
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f picard
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f pindel
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f platypus-variant
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f pyloh
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f pytabix
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f pythonpy
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f qsignature
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f qualimap
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f rapmap
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f razers3
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f rtg-tools
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f sailfish
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f salmon
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f sambamba
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f samblaster
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f samtools
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f scalpel
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f seqbuster
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f seqtk
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f sickle-trim
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f simple_sv_annotation
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f snap-aligner
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f snpeff
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f spades
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f star
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f stringtie
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f subread
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f svtyper
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f tdrmapper
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f theta2
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f tophat
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f ucsc-bedgraphtobigwig
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f ucsc-bedtobigbed
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f ucsc-bigbedinfo
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f ucsc-bigbedsummary
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f ucsc-bigbedtobed
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f ucsc-bigwiginfo
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f ucsc-bigwigsummary
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f ucsc-bigwigtobedgraph
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f ucsc-bigwigtowig
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f ucsc-fatotwobit
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f ucsc-gtftogenepred
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f ucsc-liftover
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f ucsc-wigtobigwig
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f vardict
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f vardict-java
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f variant-effect-predictor
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f variantbam
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f varscan
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f vawk
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f vcf2db
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f vcfanno
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f vcflib
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f viennarna
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f vt
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f wham
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f awscli
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f p7zip
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f s3gof3r
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f perl-app-cpanminus
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f perl-archive-extract
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f perl-archive-zip
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f perl-bio-db-sam
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f perl-cgi
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f perl-dbi
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f perl-encode-locale
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f perl-file-fetch
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f perl-file-sharedir
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f perl-file-sharedir-install
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f perl-ipc-system-simple
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f perl-lwp-protocol-https
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f perl-lwp-simple
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f perl-statistics-descriptive
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f perl-time-hires
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f perl-vcftools-vcf
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bioconductor-annotate
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bioconductor-biocinstaller
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bioconductor-biocstyle
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bioconductor-biostrings
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bioconductor-biovizbase
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bioconductor-bubbletree
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bioconductor-cn.mops
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bioconductor-degreport
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bioconductor-deseq2
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bioconductor-dexseq
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bioconductor-dnacopy
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bioconductor-genomicranges
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bioconductor-iranges
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bioconductor-limma
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f bioconductor-rtracklayer
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f r
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f r-cghflasso
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f r-chbutils
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f r-devtools
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f r-dplyr
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f r-dt
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f r-ggdendro
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f r-ggplot2
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f r-gplots
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f r-gsalib
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f r-knitr
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f r-pheatmap
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f r-plyr
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f r-pscbs
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f r-reshape
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f r-rmarkdown
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f r-rsqlite
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f r-sleuth
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f r-snow
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f r-stringi
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f r-wasabi
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda update -y -c r ncurses
Fetching package metadata: ......
# All requested packages already installed.
# packages in environment at /mnt/kufs/scratch/tmorova15/bcbio/anaconda:
#
ncurses                   5.9                           4    r
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f python
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f conda
[localhost] local: /mnt/kufs/scratch/tmorova15/bcbio/anaconda/bin/conda list --json -f pip
DBG [config.py]: Using config file /mnt/kufs/scratch/tmorova15/bcbio/tmpbcbio-install/cloudbiolinux/cloudbio/../contrib/flavor/ngs_pipeline_minimal/r-libs.yaml
Creating manifest of installed packages in /mnt/kufs/scratch/tmorova15/bcbio/manifest
Third party tools upgrade complete.
Upgrade completed successfully.
chapmanb commented 8 years ago

Tunc; Thanks for the detailed report. This is turning up an error is a different place in a pipeline (during bgzip preparation of the fastq files) and again points to a filesystem error:

writer fwrite: bug encountered
writer writer_write_block: bug encountered

My guess is that you're running into problems reading/writing to your shared filesystem. I don't think this has to do with anything about how you're running the job.

Shared filesystem issues are pretty difficult to debug remotely, so your best bet might be to ask the team that manages your cluster and see if they have suggestions. Sorry to not have been ideas but hope this helps.

mortunco commented 8 years ago

Brad;

Thank you for effort to debug the problem. Even though our team is here very strong in server related errors, they have just started using bcbio with my project actually. In order to seek the source of the problem could you give some definitions to me to tell them where is the problem so they can manage maybe write/read permissions or filesystem specifications.

As I understood from our previous conversations with our team here, the server here distributes our processes to 6 drives, therefore there might be complication during this distribution.

To sum up, I understood that the problem is in our side but could you just point me some topics that at least I can tell them the problem might be related with that of specific problem.

Best regards,

Tunc.

chapmanb commented 8 years ago

Tunc; From this thread, it sounds like bcbio fails in multiple places. My best suggestion would be to run a few times, try to identify command lines that fail regularly and see if you can submit that to them for debugging. The failing command is in the tracebacks from bcbio and also in log/bcbio-nextgen-commands.log. Hopefully you could reproduce the problem outside of bcbio in a shell script so they could debug from there. Hope this helps.

mortunco commented 8 years ago

Dear Brad;

I asked about the errors. I dont know why but people in IT told me that this occurred because I have reached the maximum storage (I was downloading my raw data behind). When they increased the error, it seemed to be getting normal bcbio errors. I will be posting them.

Thank you for your help.