search

PacificBiosciences / FALCON

FALCON: experimental PacBio diploid assembler -- Out-of-date -- Please use a binary release: https://github.com/PacificBiosciences/FALCON_unzip/wiki/Binaries
https://github.com/PacificBiosciences/FALCON_unzip/wiki/Binaries
Other
205 stars 102 forks source link

Falcon error Task Fred{'URL': 'task://localhost/build_pdb'} failed with exit-code=256 #469

Closed danshu closed 8 years ago

danshu commented 8 years ago

I have successfully installed FALCON-integrate and I can fun "make test" successfully. However, when I fun on my own genome, I have the following error: "Task Fred{'URL': 'task://localhost/build_pdb'} failed with exit-code=256 [INFO]Failure ('fail'). Joining 'task://localhost/build_pdb'... [CRITICAL]Any exception caught in RefreshTargets() indicates an unrecoverable error. Shutting down..." "Traceback (most recent call last): File "/tools/assembler/FALCON-integrate/pypeFLOW/pwatcher/fs_based.py", line 553, in get_status os.remove(heartbeat_path) OSError: [Errno 2] No such file or directory: 'falcon/t7_rerun/mypwatcher/heartbeats/heartbeat-Jfaddbf7fb358bccb296e270b8ef986acc5e127096a87b6ea0562d156dded7878'

pb-cdunn commented 8 years ago

That error is reported but trapped. Not real:

To see the real error, go to the run-dir for build_pdb. (I think it's 0-raw_reads/.) Look for pwatcher.dir/stderr.

danshu commented 8 years ago

Here is what's in "pwatcher.dir/stderr":

cwd:'/data/falcon/san_falcon_t7_rerun' hostname=** heartbeat_fn='/data/falcon/san_falcon_t7_rerun/mypwatcher/heartbeats/heartbeat-J14bb1d26bf6adf0f9de22d9f51f85edd13c6840843f9b23b051dc5e4a16476ba' exit_fn='/data/falcon/san_falcon_t7_rerun/mypwatcher/exits/exit-J14bb1d26bf6adf0f9de22d9f51f85edd13c6840843f9b23b051dc5e4a16476ba' sleep_s=10.0 before setpgid: pid=61336 pgid=61321 after setpgid: pid=61336 pgid=61336 In cwd: /data/falcon/san_falcon_t7_rerun, Blocking call: '/bin/bash noop.sh' returned: 0

danshu commented 8 years ago

Actually I have got falcon installed and running successfully previously. But the problem is, I can not run falcon again after a fewer weeks using the same command. I'm not familiar with python and linux and every time I reinstall falcon and struggled to make it work by trial and error. Thus I can not exactly find which step works and can not repeat the solution.
Actually somehow I suspect that the problem is caused by python. The "PYTHONUSERBASE" is set to be the fc_env, so is python supposed to only use sidepackages in this directory? I noticed this message when "make -j all", which seems that some reauirements are in /usr/local/lib/python2.7/site-packages. Source in /tools/assembler/FALCON-integrate/pypeFLOW has version 0.1.1, which satisfies requirement pypeflow==0.1.1 from file:///tools/assembler/FALCON-integrate/pypeFLOW Requirement already satisfied (use --upgrade to upgrade): rdflib==3.4.0 in /usr/local/lib/python2.7/site-packages (from pypeflow==0.1.1) Requirement already satisfied (use --upgrade to upgrade): rdfextras>=0.1 in /usr/local/lib/python2.7/site-packages (from pypeflow==0.1.1) And in the "../FALCON-integrate/fc_env/lib/python2.7/site-packages/easy-install.path" file ,there are only two lines: "/tools/assembler/FALCON-integrate/pypeFLOW /tools/assembler/FALCON-integrate/FALCON"

Alll I want is to understand this error and than solve it. Sorry that some of my questions may be naive and thanks in advance for any help!

danshu commented 8 years ago

In "/usr/local/lib/python2.7/site-packages/easy-install.pth", I find: "... /tools/assembler/FALCON-integrate/FALCON/.eggs/networkx-1.10-py2.7.egg ./pypeflow-0.1.1-py2.7.egg"

danshu commented 8 years ago

This error was not solved after fresh installation following "The easy way (in a new directory)" (https://github.com/PacificBiosciences/FALCON-integrate/wiki/Installation)

danshu commented 8 years ago

Actually I was trying to reproduce a previous facon fun, because I previously deleted the alignments files produced by falcon which are needed to run falcon_unzip. I tried running falcon using raw reads and failed. After numerous installation, tesing and uninstallation, I realized that I don't need to rerun falcon from raw reads and then after a fresh installation I set in the config file "input_type = preads" to use previously corrected preads for assembly. I failed again and have the error message listed above. Then I tried "sudo python setup.py install" under /tools/assembler/FALCON-integrate/FALCON and failed again. Then I set "input_type = raw" in the config file and it seems that falcon is running now!

This is how I get falcon running and everytime I feel like I'm playing magic...

pb-cdunn commented 8 years ago

This error was not solved after fresh installation following "The easy way (in a new directory)"

Did our test fail as part of the installation process? Or was installation successful, and only your own run failed later? That's a big difference.

I tried a couple days ago, and it worked for me, in a new shell on a Linux machine. Also, we use TravisCI. It definitely works. You would need to post your exact shell-commands and the results.

My best guess is that you are not setting the environment variables from env.sh. Are you familiar with "shell environments" in Linux? If so, please post the values which you have copied into your shell before running on your own data.

This is how I get falcon running and everytime I feel like I'm playing magic...

Use job_type=local the first time, to separate network problems from integration problems.

danshu commented 8 years ago

Did our test fail as part of the installation process? Or was installation successful, and only your own run failed later? That's a big difference. Installation and test were successful. My own run failed later.

My best guess is that you are not setting the environment variables from env.sh. Are you familiar with "shell environments" in Linux? If so, please post the values which you have copied into your shell before running on your own data. I followed "the easy way "(https://github.com/PacificBiosciences/FALCON-integrate/wiki/Installation" exactly and I think "source env.sh" during installation has already set environment variables. Here is what's in the stdout file: PYTHONUSERBASE=/tools/assembler/FALCON-integrate/fc_env FALCON_WORKSPACE=/tools/assembler/FALCON-integrate FALCON_PREFIX=/tools/assembler/FALCON-integrate/fc_env PATH=/tools/assembler/FALCON-integrate/fc_env/bin:/tools/assembler/FALCON-integrate/fc_env/bin:/tools/assembler/FALCON-integrate/fc_env/bin:/tools/assembler/FALCON-integrate/fc_env/bin:/tools/assembler/FALCON-integrate/fc_env/bin:/tools/assembler/FALCON-integrate/fc_env/bin:/tools/assembler/FALCON-integrate/fc_env/bin:/tools/assembler/FALCON-integrate/fc_env/bin:/tools/assembler/FALCON-integrate/fc_env/bin:/tools/assembler/FALCON-integrate/fc_env/bin:/tools/assembler/FALCON-integrate/fc_env/bin:/tools/assembler/FALCON-integrate/fc_env/bin:/tools/assembler/FALCON-integrate/fc_env/bin:/tools/assembler/FALCON-integrate/fc_env/bin:/tools/assembler/FALCON-integrate/fc_env/bin:/tools/assembler/FALCON-integrate/fc_env/bin:/tools/assembler/FALCON-integrate/fc_env/bin:/tools/assembler/FALCON-integrate/fc_env/bin:/tools/assembler/FALCON-integrate/fc_env/bin:/tools/assembler/FALCON-integrate/fc_env/bin:/tools/assembler/FALCON-integrate/fc_env/bin:/tools/assembler/FALCON-integrate/fc_env/bin:/tools/assembler/FALCON-integrate/fc_env/bin:/tools/assembler/FALCON-integrate/fc_env/bin:/tools/assembler/FALCON-integrate-1.7.7/fc_env/bin:/tools/assembler/FALCON-integrate-1.7.7/fc_env/bin:...

Use job_type=local the first time, to separate network problems from integration problems. I was funning using job_type=local all the time.

pb-cdunn commented 8 years ago

Did our test fail as part of the installation process? Or was installation successful, and only your own run failed later? That's a big difference.

I still need to know that. There is a big difference between an integration problem and a configuration problem.

And you can try simply

fc_run.py
# or
fc_run

That will start with an error (missing arguments), but Python should import everything fine. Please post the result.

Your PATH is probably long because you have sourced env.sh many times into the same shell.

danshu commented 8 years ago

fc_run usage: fc_run [-h] config [logger] fc_run: error: too few arguments

Actually falcon is running after "sudo python setup.py install" under /tools/assembler/FALCON-integrate/FALCON. I suspect that this problem is caused by python but I can not understand why "sudo python setup.py install" solved the problem.

pb-cdunn commented 8 years ago

I strongly suggest the a "user" install. (Google it. That's what PYTHONUSERBASE is for.) With a system install, you will need to care on future updates to ensure that you are actually over-writing the current installation. But I'm glad you got it working.