search

PacificBiosciences / pb-falcon-phase

FALCON-Phase integrates PacBio long-read assemblies with Phase Genomics Hi-C data to create phased, diploid, chromosome-scale scaffolds
Other
5 stars 3 forks source link

'unzip' output_format doesn't work #4

Closed ruthfa closed 5 years ago

ruthfa commented 5 years ago

I have done a falcon phasing with the output format 'pseudohap' and everything was ok. But when I try to do the phasing for the same data with the output format 'unzip', I obtain an error in the output folder This is the stderr file for the output generation

executable=${PYPEFLOW_JOB_START_SCRIPT}
+ executable=/gpfs/project/projects/qggp/Potatodenovo/experiments/PacBio_Agriadiploid/FALCON_new_reads/5-phase/output/run-P561a3e23a9a402.bash
timeout=${PYPEFLOW_JOB_START_TIMEOUT:-60} # wait 60s by default
+ timeout=60

# Wait up to timeout seconds for the executable to become "executable",
# then exec.
#timeleft = int(timeout)
while [[ ! -x "${executable}" ]]; do
    if [[ "${timeout}" == "0" ]]; then
        echo "timed out waiting for (${executable})"
        exit 77
    fi
    echo "not executable: '${executable}', waiting ${timeout}s"
    sleep 1
    timeout=$((timeout-1))
done
+ [[ ! -x /gpfs/project/projects/qggp/Potatodenovo/experiments/PacBio_Agriadiploid/FALCON_new_reads/5-phase/output/run-P561a3e23a9a402.bash ]]

/bin/bash ${executable}
+ /bin/bash /gpfs/project/projects/qggp/Potatodenovo/experiments/PacBio_Agriadiploid/FALCON_new_reads/5-phase/output/run-P561a3e23a9a402.bash
+ '[' '!' -d /gpfs/project/projects/qggp/Potatodenovo/experiments/PacBio_Agriadiploid/FALCON_new_reads/5-phase/output ']'
+ cd /gpfs/project/projects/qggp/Potatodenovo/experiments/PacBio_Agriadiploid/FALCON_new_reads/5-phase/output
+ eval '/bin/bash run.sh'
++ /bin/bash run.sh
export PATH=$PATH:/bin
+ export PATH=/home/freireal/.conda/envs/phase/bin:/software/conda/3/bin:/software/conda/3/condabin:/software/conda/3/bin:/usr/pbs/bin:/usr/lpp/mmfs/bin:/usr/pbs/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/opt/pbs/bin:/usr/local/bin:/usr/X11R6/bin:/bin
+ PATH=/home/freireal/.conda/envs/phase/bin:/software/conda/3/bin:/software/conda/3/condabin:/software/conda/3/bin:/usr/pbs/bin:/usr/lpp/mmfs/bin:/usr/pbs/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/opt/pbs/bin:/usr/local/bin:/usr/X11R6/bin:/bin
cd /gpfs/project/projects/qggp/Potatodenovo/experiments/PacBio_Agriadiploid/FALCON_new_reads/5-phase/output
+ cd /gpfs/project/projects/qggp/Potatodenovo/experiments/PacBio_Agriadiploid/FALCON_new_reads/5-phase/output
/bin/bash task.sh
+ /bin/bash task.sh
pypeflow 2.1.1
2019-09-11 11:06:58,248 - root - DEBUG - Running "/home/freireal/.conda/envs/phase/lib/python2.7/site-packages/pypeflow/do_task.py /gpfs/project/projects/qggp/Potatodenovo/experiments/PacBio_Agriadiploid/FALCON_new_reads/5-phase/output/task.json"
2019-09-11 11:06:58,250 - root - DEBUG - Checking existence of '/gpfs/project/projects/qggp/Potatodenovo/experiments/PacBio_Agriadiploid/FALCON_new_reads/5-phase/output/task.json' with timeout=30
2019-09-11 11:06:58,250 - root - DEBUG - Loading JSON from '/gpfs/project/projects/qggp/Potatodenovo/experiments/PacBio_Agriadiploid/FALCON_new_reads/5-phase/output/task.json'
2019-09-11 11:06:58,251 - root - DEBUG - {u'bash_template_fn': u'template.sh',
 u'inputs': {u'BC_bed': u'../prep-emit/BC.bed',
             u'p_h_ctg_fasta': u'../prep-emit/p_h_ctg.fa',
             u'phased': u'../phasing/phased.txt'},
 u'outputs': {u'B0': u'phased.0.bed',
              u'B1': u'phased.1.bed',
              u'F0': u'phased.0.fasta',
              u'F1': u'phased.1.fasta'},
 u'parameters': {u'output_format': u'unzip',
                 u'pypeflow_mb': 4000,
                 u'pypeflow_nproc': 26,
                 u'topdir': u'../..'}}
2019-09-11 11:06:58,252 - root - WARNING - CD: '/gpfs/project/projects/qggp/Potatodenovo/experiments/PacBio_Agriadiploid/FALCON_new_reads/5-phase/output' <- '/gpfs/project/projects/qggp/Potatodenovo/experiments/PacBio_Agriadiploid/FALCON_new_reads/5-phase/output'
2019-09-11 11:06:58,252 - root - DEBUG - Checking existence of u'../prep-emit/p_h_ctg.fa' with timeout=30
2019-09-11 11:06:58,252 - root - DEBUG - Checking existence of u'../prep-emit/BC.bed' with timeout=30
2019-09-11 11:06:58,252 - root - DEBUG - Checking existence of u'../phasing/phased.txt' with timeout=30
2019-09-11 11:06:58,252 - root - DEBUG - Checking existence of u'template.sh' with timeout=30
2019-09-11 11:06:58,252 - root - WARNING - CD: '/gpfs/project/projects/qggp/Potatodenovo/experiments/PacBio_Agriadiploid/FALCON_new_reads/5-phase/output' <- '/gpfs/project/projects/qggp/Potatodenovo/experiments/PacBio_Agriadiploid/FALCON_new_reads/5-phase/output'
2019-09-11 11:06:58,258 - root - INFO - $('/bin/bash user_script.sh')
hostname
+ hostname
pwd
+ pwd
date
+ date
# Substitution will be similar to snakemake "shell".
fc_emit_haplotigs.py ../phasing/phased.txt ../prep-emit/BC.bed ../prep-emit/p_h_ctg.fa bedtools unzip
+ fc_emit_haplotigs.py ../phasing/phased.txt ../prep-emit/BC.bed ../prep-emit/p_h_ctg.fa bedtools unzip
Traceback (most recent call last):
  File "/home/freireal/.conda/envs/phase/bin/fc_emit_haplotigs.py", line 290, in <module>
    main(*sys.argv)
  File "/home/freireal/.conda/envs/phase/bin/fc_emit_haplotigs.py", line 94, in main
    makeBED(BC_bed_file, output_format);
  File "/home/freireal/.conda/envs/phase/bin/fc_emit_haplotigs.py", line 281, in makeBED
    func(BC, ZERO, ONE)
  File "/home/freireal/.conda/envs/phase/bin/fc_emit_haplotigs.py", line 230, in _makeBED_unzip
    array0[3], array1[3] = array1[3], array0[3]
IndexError: list index out of range
2019-09-11 11:06:58,392 - root - WARNING - Call '/bin/bash user_script.sh' returned 256.
2019-09-11 11:06:58,392 - root - WARNING - CD: '/gpfs/project/projects/qggp/Potatodenovo/experiments/PacBio_Agriadiploid/FALCON_new_reads/5-phase/output' -> '/gpfs/project/projects/qggp/Potatodenovo/experiments/PacBio_Agriadiploid/FALCON_new_reads/5-phase/output'
2019-09-11 11:06:58,392 - root - WARNING - CD: '/gpfs/project/projects/qggp/Potatodenovo/experiments/PacBio_Agriadiploid/FALCON_new_reads/5-phase/output' -> '/gpfs/project/projects/qggp/Potatodenovo/experiments/PacBio_Agriadiploid/FALCON_new_reads/5-phase/output'
2019-09-11 11:06:58,393 - root - CRITICAL - Error in /home/freireal/.conda/envs/phase/lib/python2.7/site-packages/pypeflow/do_task.py with args="{'json_fn': '/gpfs/project/projects/qggp/Potatodenovo/experiments/PacBio_Agriadiploid/FALCON_new_reads/5-phase/output/task.json',\n 'timeout': 30,\n 'tmpdir': None}"
Traceback (most recent call last):
  File "/home/freireal/.conda/envs/phase/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/freireal/.conda/envs/phase/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/freireal/.conda/envs/phase/lib/python2.7/site-packages/pypeflow/do_task.py", line 267, in <module>
    main()
  File "/home/freireal/.conda/envs/phase/lib/python2.7/site-packages/pypeflow/do_task.py", line 259, in main
    run(**vars(parsed_args))
  File "/home/freireal/.conda/envs/phase/lib/python2.7/site-packages/pypeflow/do_task.py", line 253, in run
    run_cfg_in_tmpdir(cfg, tmpdir, '.')
  File "/home/freireal/.conda/envs/phase/lib/python2.7/site-packages/pypeflow/do_task.py", line 228, in run_cfg_in_tmpdir
    run_bash(bash_template, myinputs, myoutputs, parameters)
  File "/home/freireal/.conda/envs/phase/lib/python2.7/site-packages/pypeflow/do_task.py", line 187, in run_bash
    util.system(cmd)
  File "/home/freireal/.conda/envs/phase/lib/python2.7/site-packages/pypeflow/io.py", line 29, in syscall
    raise Exception(msg)
Exception: Call '/bin/bash user_script.sh' returned 256.
+++ pwd
++ echo 'FAILURE. Running top in /gpfs/project/projects/qggp/Potatodenovo/experiments/PacBio_Agriadiploid/FALCON_new_reads/5-phase/output (If you see -terminal database is inaccessible- you are using the python bin-wrapper, so you will not get diagnostic info. No big deal. This process is crashing anyway.)'
++ rm -f top.txt
++ which python
++ which top
++ env -u LD_LIBRARY_PATH top -b -n 1
++ env -u LD_LIBRARY_PATH top -b -n 1
++ pstree -apl

real    0m1.118s
user    0m0.410s
sys 0m0.351s
+ finish
+ echo 'finish code: 1'
pb-cdunn commented 5 years ago

Could you post the output of conda list? You are using phase/bin/fc_emit_haplotigs.py, but we would expect to use python -m falcon_phase.mains.emit_haplotigs, i.e. falcon_phase/mains/emit_haplotigs.py. Current falcon-phase calls

falcon_phase/tasks/tasks_falcon_phase.py:%(python)s -m falcon_phase.mains.emit_haplotigs {input.phased} {input.BC_bed} ...

So I think you are out of date. How did you run falcon-phase? I mean the top-level call.

... yeah, I just double-checked current bioconda. It's possible that you are using a python2 env, which will prevent you from getting our updates. We deploy for python3 only now.

ruthfa commented 5 years ago

This is my conda list

# packages in environment at /home/freireal/.conda/envs/phase:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
asn1crypto                0.24.0                py27_1003    conda-forge
avro-python2              1.9.0                      py_0    bioconda
bcftools                  1.8                  h4da6232_3    bioconda
bedtools                  2.27.1               he860b03_3    bioconda
blasr                     5.3.2                hac9d22c_3    bioconda
blasr_libcpp              5.3.1                hac9d22c_2    bioconda
bwa                       0.7.17               hed695b0_6    bioconda
bzip2                     1.0.8                h516909a_0    conda-forge
ca-certificates           2019.6.16            hecc5488_0    conda-forge
certifi                   2019.6.16                py27_1    conda-forge
cffi                      1.12.3           py27h8022711_0    conda-forge
chardet                   3.0.4                 py27_1003    conda-forge
cryptography              2.5              py27hb7f436b_1    conda-forge
curl                      7.64.0               h646f8bb_0    conda-forge
cython                    0.29.13          py27he1b5a44_0    conda-forge
decorator                 4.4.0                      py_0    conda-forge
enum34                    1.1.6                 py27_1001    conda-forge
falcon-kit                1.2.5                    pypi_0    pypi
falcon-phase              1.0.0                    pypi_0    pypi
falcon-unzip              1.1.5                    pypi_0    pypi
future                    0.17.1                py27_1000    conda-forge
genomicconsensus          2.3.3                    py27_0    bioconda
h5py                      2.8.0           py27h3010b51_1003    conda-forge
hdf5                      1.10.2               hc401514_3    conda-forge
htslib                    1.7                           0    bioconda
idna                      2.8                   py27_1000    conda-forge
ipaddress                 1.0.22                     py_1    conda-forge
iso8601                   0.1.12                     py_1    conda-forge
krb5                      1.16.3            hc83ff2d_1000    conda-forge
libblas                   3.8.0               12_openblas    conda-forge
libcblas                  3.8.0               12_openblas    conda-forge
libcurl                   7.64.0               h01ee5af_0    conda-forge
libdeflate                1.0                  h14c3975_1    bioconda
libedit                   3.1.20170329      hf8c457e_1001    conda-forge
libffi                    3.2.1             he1b5a44_1006    conda-forge
libgcc                    7.2.0                h69d50b8_2    conda-forge
libgcc-ng                 9.1.0                hdf63c60_0  
libgfortran               3.0.0                         1    conda-forge
libgfortran-ng            7.3.0                hdf63c60_0  
liblapack                 3.8.0               12_openblas    conda-forge
libopenblas               0.3.7                h6e990d7_1    conda-forge
libssh2                   1.8.0             h1ad7b7a_1003    conda-forge
libstdcxx-ng              9.1.0                hdf63c60_0  
linecache2                1.0.0                      py_1    conda-forge
minimap2                  2.17                 h84994c4_0    bioconda
mummer4                   4.0.0beta2      pl526hf484d3e_4    bioconda
ncurses                   6.1               hf484d3e_1002    conda-forge
networkx                  2.2                        py_1    conda-forge
nim-falcon                0.0.0                         0    bioconda
numpy                     1.16.4           py27h95a1406_0    conda-forge
openssl                   1.0.2r               h14c3975_0    conda-forge
pb-assembly               0.0.4                    py27_3    bioconda
pb-dazzler                0.0.1                h516909a_0    bioconda
pb-falcon                 0.2.5            py27ha92aebf_1    bioconda
pbalign                   0.3.1                    py27_2    bioconda
pbbam                     0.18.0               h1310cd9_1    bioconda
pbcommand                 1.1.1                    py27_2    bioconda
pbcopper                  0.4.2                h3f2a3d2_1    bioconda
pbcore                    1.6.5                    py27_0    bioconda
pbgcpp                    0.0.1                         0    bioconda
pbmm2                     0.8.1                ha87ae23_0    bioconda
pcre                      8.41              hf484d3e_1003    conda-forge
perl                      5.26.2            h516909a_1006    conda-forge
pip                       19.2.3                   py27_0    conda-forge
pycparser                 2.19                     py27_1    conda-forge
pyopenssl                 19.0.0                   py27_0    conda-forge
pysam                     0.14.1           py27hae42fb6_1    bioconda
pysocks                   1.7.0                    py27_0    conda-forge
python                    2.7.15            h938d71a_1006    conda-forge
python-consensuscore      1.1.1            py27h3f2a3d2_3    bioconda
python-consensuscore2     3.4.1                    py27_0    bioconda
python-edlib              1.3.4            py27hc9558a2_0    bioconda
python-intervaltree       3.0.2                      py_0    bioconda
python-msgpack            0.6.1            py27h6bb024c_0    bioconda
python-sortedcontainers   2.1.0                      py_0    bioconda
pytz                      2019.2                     py_0    conda-forge
racon                     1.4.3                he513fc3_0    bioconda
readline                  7.0               hf8c457e_1001    conda-forge
requests                  2.22.0                   py27_1    conda-forge
samtools                  1.9                  h43f6869_9    bioconda
setuptools                41.2.0                   py27_0    conda-forge
six                       1.12.0                py27_1000    conda-forge
sqlite                    3.28.0               h8b20d00_0    conda-forge
tk                        8.6.9             hed695b0_1002    conda-forge
traceback2                1.4.0                    py27_0    conda-forge
unittest2                 1.1.0                      py_0    conda-forge
urllib3                   1.25.3                   py27_0    conda-forge
wheel                     0.33.6                   py27_0    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
zlib                      1.2.11            h516909a_1005    conda-forge

And, yes, I think I am working on a python2 env.

pb-cdunn commented 5 years ago

We don't support py2 anymore, nor py3.6. Please upgrade to py3.7.