FALCON-integrate installation problem(about the final test part)

I am new to bioinformatics. Recently, I tried to install FALCON-integrate on our centos linux server. The python version which I used is Python 2.7.12 :: Anaconda 4.2.0 (64-bit). The installation procedure is as follows.

git clone git://github.com/PacificBiosciences/FALCON-integrate.git
cd FALCON-integrate
git checkout master  # or whatever version you want
make init
source env.sh
make config-edit-user
make -j all
make test  # to run a simple one

Everything seems normal until I tried the final test step. And the log is as follows. It suggested that I failed to connect to www.dropbox.com port 443. Is there any other method to download the linked files and then try the test? I would really appreciate it if anyone can give me some suggestions.

make -C ./FALCON-make/ test make[1]: Entering directory /public/home/lijing/software/FALCON-integrate/FALCON-make' make -C /public/home/lijing/software/FALCON-integrate/FALCON-examples test make[2]: Entering directory/public/home/lijing/software/FALCON-integrate/FALCON-examples' python -c 'import pypeflow.common; print pypeflow.common' <module 'pypeflow.common' from '/public/home/lijing/software/FALCON- integrate/pypeFLOW/pypeflow/common.pyc'> python -c 'import falcon_kit; print falcon_kit.falcon' <CDLL '/public/home/lijing/software/FALCON-integrate/FALCON/ext_falcon.so', handle fd1310 at 2b41e0aef1d0> make run-synth0 make[3]: Entering directory /public/home/lijing/software/FALCON-integrate/FALCON-examples' git-sym update run/synth0 mkdir -p /public/home/lijing/software/FALCON-integrate/FALCON-examples/.git/git-sym-local/links os.symlink('.git/git-sym-local/links', '/public/home/lijing/software/FALCON-integrate/FALCON-examples/.git-sym') git: 'check-ignore' is not a git command. See 'git --help'. mkdir -p /public/home/lijing/software/FALCON-integrate/FALCON-examples/.git/git-sym-local/cache -> in dir 'run/synth0' <- back to dir '/public/home/lijing/software/FALCON-integrate/FALCON-examples' symlink: 'run/synth0/data/ref.fasta' symlink: 'run/synth0/data/synth0.fasta' 'run/synth0/data/ref.fasta' -> '../../../.git-sym/synth0.ref.fasta' does not exist 'run/synth0/data/synth0.fasta' -> '../../../.git-sym/synth0-circ-20.pb.fasta' does not exist -> in dir '/public/home/lijing/software/FALCON-integrate/FALCON-examples/.git/git-sym-local/links' -> in dir '/public/home/lijing/software/FALCON-integrate/FALCON-examples/.git/git-sym-local/cache' make -j -f /public/home/lijing/software/FALCON-integrate/FALCON-examples/git-sym.makefile 'synth0.ref.fasta' make[4]: Entering directory/public/home/lijing/software/FALCON-integrate/FALCON-examples/.git/git-sym-local/cache' curl -L https://www.dropbox.com/s/jz0m0n2a1b19xyd/from.fasta.gz | zcat > synth0.ref.fasta % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- 0:01:03 --:--:-- 0curl: (7) Failed to connect to www.dropbox.com port 443: Connection timed out

gzip: stdin: unexpected end of file make[4]: * [synth0.ref.fasta] Error 1 make[4]: Leaving directory `/public/home/lijing/software/FALCON-integrate/FALCON-examples/.git/git-sym-local/cache' Traceback (most recent call last): File "/public/home/lijing/software/FALCON-integrate/fc_env/bin/git-sym", line 455, in main cmd_table[cmd](args) File "/public/home/lijing/software/FALCON-integrate/fc_env/bin/git-sym", line 357, in git_sym_update retrieve(needed) File "/public/home/lijing/software/FALCON-integrate/fc_env/bin/git-sym", line 284, in retrieve retrieve_using_make(makefilename, paths) File "/public/home/lijing/software/FALCON-integrate/fc_env/bin/git-sym", line 277, in retrieve_using_make system(cmd) File "/public/home/lijing/software/FALCON-integrate/fc_env/bin/git-sym", line 81, in system raise Exception('%d <- %r' %(rc, cmd)) Exception: 512 <- "make -j -f /public/home/lijing/software/FALCON-integrate/FALCON-examples/git-sym.makefile 'synth0.ref.fasta'"

make[3]: [setup-synth0] Error 1 make[3]: Leaving directory `/public/home/lijing/software/FALCON-integrate/FALCON-examples' make[2]: [test] Error 2 make[2]: Leaving directory /public/home/lijing/software/FALCON-integrate/FALCON-examples' make[1]: *** [test] Error 2 make[1]: Leaving directory/public/home/lijing/software/FALCON-integrate/FALCON-make' make: *** [test] Error 2

And then I tried to run fc_run.py, It appears:

Traceback (most recent call last): File "/public/home/lijing/software/FALCON-integrate/fc_env/bin/fc_run.py", line 4, in import('pkg_resources').require('falcon-kit==0.7+git.91823bfc08f2f3f6c3e7e99be1dfc325c38556ec') File "/public/home/lijing/software/anaconda2/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/init.py", line 2985, in File "/public/home/lijing/software/anaconda2/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/init.py", line 2971, in _call_aside File "/public/home/lijing/software/anaconda2/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/init.py", line 2998, in _initialize_master_working_set File "/public/home/lijing/software/anaconda2/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/init.py", line 660, in _build_master File "/public/home/lijing/software/anaconda2/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/init.py", line 968, in require File "/public/home/lijing/software/anaconda2/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/init.py", line 854, in resolve pkg_resources.DistributionNotFound: The 'falcon-kit==0.7+git.91823bfc08f2f3f6c3e7e99be1dfc325c38556ec' distribution was not found and is required by the application

Can anybody tell me what is the standard that the FALCON-integrate has been completely installed? Thanks a lot.

git checkout master, you might want to check out a tagged version
curl -L https://www.dropbox.com/s/jz0m0n2a1b19xyd/from.fasta.gz | zcat > synth0.ref.fasta problem, I just tested it, I could fetch it. Your institute might block port 443 or dropbox for some reason. Please contact your system admin to see what is going on. If your institute blocks it, then you need to find a way to workaround it, e.g., VPN or something else.

Your institute might block port 443 or dropbox for some reason.

Fortunately, you need that only to obtain synth0*.fasta, which is a little test-case that we use. Without that, you will need to find your own way to validate your installation, but it looks fine to me.

One solution is to do all this on your home network first. Then you can send the data/ directory to yourself via email or something.

I just upload some files and updated FALCON-examples. You should be able to do something like this:

cd FALCON-examples
git pull origin master
cd run/synth0
git-sym update .
ls -lh data/synth5k/

-rw-r--r-- 1 cdunn Domain Users   414 Nov  2 11:46 README
-r--r--r-- 1 cdunn Domain Users  5021 Nov  2 11:44 ref.fasta
-r--r--r-- 1 cdunn Domain Users 25665 Nov  2 11:47 subreads.dexta
-rw-r--r-- 1 cdunn Domain Users 35341 Nov  2 11:44 synth5k.bam
-rw-r--r-- 1 cdunn Domain Users    16 Nov  2 11:44 synth5k.bam.bai
-rw-r--r-- 1 cdunn Domain Users   346 Nov  2 11:44 synth5k.bam.pbi
-rw-r--r-- 1 cdunn Domain Users  1377 Nov  2 11:44 synth5k.xml

You don't need the BAM files, but we might use them someday.

@pb-cdunn @pb-jchin,really thanks a lot. I have some PacBio reads in hand needed to assemble right now. And I know that FALCON is one of the greatest PacBio assembly softwares. So I want to try it first.

I have successfully updated the FALCON-examples. And now I can see all the files you listed. Then I tried to run make test. But something wrong happened again. It appeared as follows:

make -C ./FALCON-make/ test make[1]: Entering directory /public/home/lijing/software/FALCON-integrate/FALCON-make' make -C /public/home/lijing/software/FALCON-integrate/FALCON-examples test make[2]: Entering directory/public/home/lijing/software/FALCON-integrate/FALCON-examples' python -c 'import pypeflow.common; print pypeflow.common'

python -c 'import falcon_kit; print falcon_kit.falcon'

make run-synth0 make[3]: Entering directory `/public/home/lijing/software/FALCON-integrate/FALCON-examples' git-sym update run/synth0 git: 'check-ignore' is not a git command. See 'git --help'. -> in dir 'run/synth0' <- back to dir '/public/home/lijing/software/FALCON-integrate/FALCON-examples' symlink: 'run/synth0/data/synth5k' -> in dir '/public/home/lijing/software/FALCON-integrate/FALCON-examples/.git/git-sym-local/links' <- back to dir '/public/home/lijing/software/FALCON-integrate/FALCON-examples' git-sym show run/synth0 git: 'check-ignore' is not a git command. See 'git --help'. -> in dir 'run/synth0' <- back to dir '/public/home/lijing/software/FALCON-integrate/FALCON-examples' symlink: 'run/synth0/data/synth5k' / run/synth0/data/synth5k .git-sym/synth5k.2016-11-02 git-sym check run/synth0 git: 'check-ignore' is not a git command. See 'git --help'. -> in dir 'run/synth0' <- back to dir '/public/home/lijing/software/FALCON-integrate/FALCON-examples' symlink: 'run/synth0/data/synth5k' cd run/synth0; fc_run.py fc_run.cfg logging.ini 2016-11-03 02:09:27,679[INFO] Setup logging from file "logging.ini". 2016-11-03 02:09:27,680[INFO] fc_run started with configuration fc_run.cfg 2016-11-03 02:09:27,681[INFO] No target specified, assuming "assembly" as target 2016-11-03 02:09:28,252[INFO] # of tasks in complete graph: 1 2016-11-03 02:09:28,252[INFO] tick: 1, #updatedTasks: 0, sleep_time=0.000000 2016-11-03 02:09:28,253[INFO] Skipping already done task: task:///public/home/lijing/software/FALCON-integrate/FALCON/falcon_kit/mains/run1.py/task_make_fofn_abs_raw 2016-11-03 02:09:28,253[INFO] _refreshTargets() finished with no thread running and no new job to submit 2016-11-03 02:09:28,265[INFO] # of tasks in complete graph: 2 2016-11-03 02:09:28,266[INFO] tick: 1, #updatedTasks: 0, sleep_time=0.000000 2016-11-03 02:09:28,272[INFO] tick: 2, #updatedTasks: 0, sleep_time=0.100000 2016-11-03 02:09:28,272[INFO] Running task from function task_build_rdb() 2016-11-03 02:09:28,275[INFO] script_fn:'/public/home/lijing/software/FALCON-integrate/FALCON-examples/run/synth0/0-rawreads/prepare_rdb.sh' 2016-11-03 02:09:28,276[INFO] jobid=J69725f8fd020b21a32318bb32a2c27abbc47985f58e0118199175adb33212705 2016-11-03 02:09:28,276[INFO] starting job Job(jobid='J69725f8fd020b21a32318bb32a2c27abbc47985f58e0118199175adb33212705', cmd='/bin/bash prepare_rdb.sh', rundir='/public/home/lijing/software/FALCON-integrate/FALCON-examples/run/synth0/0-rawreads', options={'job_queue': 'production', 'sge_option': '-pe smp 8 -q production', 'job_type': 'local'}) 2016-11-03 02:09:28,285[INFO] Submitted backgroundjob=MetaJobLocal(MetaJob(job=Job(jobid='J69725f8fd020b21a32318bb32a2c27abbc47985f58e0118199175adb33212705', cmd='/bin/bash prepare_rdb.sh', rundir='/public/home/lijing/software/FALCON-integrate/FALCON-examples/run/synth0/0-rawreads', options={'job_queue': 'production', 'sge_option': '-pe smp 8 -q production', 'job_type': 'local'}), lang_exe='/bin/bash')) 2016-11-03 02:09:28,287[ERROR] Task Fred{'URL': 'task:///public/home/lijing/software/FALCON-integrate/FALCON/falcon_kit/mains/run1.py/task_build_rdb'} failed with exit-code=32512 2016-11-03 02:09:28,389[INFO] Queued 'task:///public/home/lijing/software/FALCON-integrate/FALCON/falcon_kit/mains/run1.py/task_build_rdb' ... 2016-11-03 02:09:28,389[INFO] Failure ('fail'). Joining 'task:///public/home/lijing/software/FALCON-integrate/FALCON/falcon_kit/mains/run1.py/task_build_rdb'... 2016-11-03 02:09:28,390[CRITICAL] Any exception caught in RefreshTargets() indicates an unrecoverable error. Shutting down... /public/home/lijing/software/FALCON-integrate/pypeFLOW/pypeflow/controller.py:537: UserWarning: "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" "! Please wait for all threads / processes to terminate !" "! Also, maybe use 'ps' or 'qstat' to check all threads,!" "! processes and/or jobs are terminated cleanly. !" "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"

warnings.warn(shutdown_msg) 2016-11-03 02:09:28,391[WARNING] #tasks=1, #alive=0 Traceback (most recent call last): File "/public/home/lijing/software/FALCON-integrate/fc_env/bin/fc_run.py", line 6, in exec(compile(open(file).read(), file, 'exec')) File "/public/home/lijing/software/FALCON-integrate/FALCON/src/py_scripts/fc_run.py", line 5, in main(sys.argv) File "/public/home/lijing/software/FALCON-integrate/FALCON/falcon_kit/mains/run1.py", line 662, in main main1(argv[0], args.config, args.logger) File "/public/home/lijing/software/FALCON-integrate/FALCON/falcon_kit/mains/run1.py", line 423, in main1 setNumThreadAllowed=PypeProcWatcherWorkflow.setNumThreadAllowed) File "/public/home/lijing/software/FALCON-integrate/FALCON/falcon_kit/mains/run1.py", line 482, in run wf.refreshTargets([rdb_build_done]) File "/public/home/lijing/software/FALCON-integrate/pypeFLOW/pypeflow/controller.py", line 548, in refreshTargets raise Exception('Caused by:\n' + tb) Exception: Caused by: Traceback (most recent call last): File "/public/home/lijing/software/FALCON-integrate/pypeFLOW/pypeflow/controller.py", line 523, in refreshTargets rtn = self._refreshTargets(task2thread, objs = objs, callback = callback, updateFreq = updateFreq, exitOnFailure = exitOnFailure) File "/public/home/lijing/software/FALCON-integrate/pypeFLOW/pypeflow/controller.py", line 740, in _refreshTargets raise TaskFailureError("Counted %d failure(s) with 0 successes so far." %failedJobCount) TaskFailureError: 'Counted 1 failure(s) with 0 successes so far.'

make[3]: * [run-synth0] Error 1 make[3]: Leaving directory /public/home/lijing/software/FALCON-integrate/FALCON-examples' make[2]: * [test] Error 2 make[2]: Leaving directory/public/home/lijing/software/FALCON-integrate/FALCON-examples' make[1]: * [test] Error 2 make[1]: Leaving directory `/public/home/lijing/software/FALCON-integrate/FALCON-make' make: * [test] Error 2

Then I tried to go into the FALCON-integrate/FALCON-examples/run/synth0 directory to run

fc_run.py fc_run.cfg

It appeared:

Traceback (most recent call last): File "/public/home/lijing/software/FALCON-integrate/fc_env/bin/fc_run.py", line 4, in import('pkg_resources').require('falcon-kit==0.7+git.91823bfc08f2f3f6c3e7e99be1dfc325c38556ec') File "/public/home/lijing/software/anaconda2/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/init.py", line 2985, in File "/public/home/lijing/software/anaconda2/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/init.py", line 2971, in _call_aside File "/public/home/lijing/software/anaconda2/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/init.py", line 2998, in _initialize_master_working_set File "/public/home/lijing/software/anaconda2/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/init.py", line 660, in _build_master File "/public/home/lijing/software/anaconda2/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/init.py", line 968, in require File "/public/home/lijing/software/anaconda2/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/init.py", line 854, in resolve pkg_resources.DistributionNotFound: The 'falcon-kit==0.7+git.91823bfc08f2f3f6c3e7e99be1dfc325c38556ec' distribution was not found and is required by the application

Then I went back to the /FALCON-integrate directory to run make test again. It appeared another mistake information:

make -C ./FALCON-make/ test make[1]: Entering directory /public/home/lijing/software/FALCON-integrate/FALCON-make' make -C /public/home/lijing/software/FALCON-integrate/FALCON-examples test make[2]: Entering directory/public/home/lijing/software/FALCON-integrate/FALCON-examples' python -c 'import pypeflow.common; print pypeflow.common' Traceback (most recent call last): File "", line 1, in ImportError: No module named pypeflow.common make[2]: * [test] Error 1 make[2]: Leaving directory `/public/home/lijing/software/FALCON-integrate/FALCON-examples' make[1]: * [test] Error 2 make[1]: Leaving directory`/public/home/lijing/software/FALCON-integrate/FALCON-make' make: *\ [test] Error 2

It seems that I did not correctly set the absolute path of pypeflow.common. But I have no idea of how to set these environment variables. Could you give me some suggestions? I would really appreciate you for any help.

the content of my env.sh is as follows:

FALCON_WORKSPACE=$(pwd) PYTHONUSERBASE=$(pwd)/fc_env FALCON_PREFIX=${PYTHONUSERBASE} PATH=${PYTHONUSERBASE}/bin:${FALCON_PREFIX}/bin:${PATH} export PYTHONUSERBASE export FALCON_WORKSPACE export FALCON_PREFIX export PATH mkdir -p ${FALCON_PREFIX}/include mkdir -p ${FALCON_PREFIX}/bin mkdir -p ${FALCON_PREFIX}/lib

And then I run:

git rev-parse HEAD

It appeared: 67dd8152b500d02ca9bd0b45d095866c6a201502

Hope this can help to do the troubleshooting.

First, you should update git. I'm not sure whether git-sym will work with your current version.

As for the integration problem, there is only so much we can do remotely. In theory, pypeflow.common is in your PYTHONUSERBASE. Search the web for python pip user to learn details on how that works. These "userbase" installs have been less problematic than virtualenv, but they still require you to understand how to install and run Python packages. Look around your userbase directory. You should be able to find pypeflow/common... somewhere. Most likely, you did not source env.sh into your current shell, so $PYTHONUSERBASE is not actually set.

If/when you learn the problem, please post that here.

You also need DEXTRACTOR for the new test-data, since we compressed it using dexta from thegenemyers. Try the new 1.8.0 pre-release.

But you definitely need to update your shell environment too.

Hi, @pb-cdunn Thank u for your help. I have successfully downloaded the FALCON-integrate on my linux centos now. I found that the main problem is that my server cannot get access to foreign network(because the great wall of china). Then I use VPN to install the FALON-integrate on my Mac first. Then transfer the FALCON package of Mac to server and change all the path of the linked files. After resourcing the software again on server. It worked. Then I followed the instruction of this website to test the running. https://github.com/PacificBiosciences/FALCON/wiki/Setup%3A-Running

The fc_run.cfg file is as follows:

[General]
# list of files of the initial subread fast files
input_fofn = input.fofn

input_type = raw
#input_type = reads

# The length cutoff used for seed reads used for initial mapping
length_cutoff = 12000

# The length cutoff used for seed reads usef for pre-assembly
length_cutoff_pr = 12000

# Cluster queue setting
sge_option_da = -pe smp 8 -q jobqueue
sge_option_la = -pe smp 2 -q jobqueue
sge_option_pda = -pe smp 8 -q jobqueue
sge_option_pla = -pe smp 2 -q jobqueue
sge_option_fc = -pe smp 24 -q jobqueue
sge_option_cns = -pe smp 8 -q jobqueue

# concurrency settgin
pa_concurrent_jobs = 32
cns_concurrent_jobs = 32
ovlp_concurrent_jobs = 32

# overlapping options for Daligner
pa_HPCdaligner_option =  -v -dal4 -t16 -e.70 -l1000 -s1000
ovlp_HPCdaligner_option = -v -dal4 -t32 -h60 -e.96 -l500 -s1000

pa_DBsplit_option = -x500 -s50
ovlp_DBsplit_option = -x500 -s50

# error correction consensus options
falcon_sense_option = --output_multi --min_idt 0.70 --min_cov 4 --local_match_count_threshold 2 --max_n_read 200 --n_core 6

# overlap filtering options
overlap_filtering_setting = --max_diff 100 --max_cov 100 --min_cov 20 --bestn 10

Then I run fc_run.py fc_run.cfg. It appeared:

[INFO]Setup logging from file "None".
[INFO]fc_run started with configuration fc_run.cfg
[WARNING]HPC.daligner option "-dal" has changed to "-B". Correcting this for you.
[WARNING]HPC.daligner option "-dal" has changed to "-B". Correcting this for you.
[ERROR]Failed to parse config "fc_run.cfg".
Traceback (most recent call last):
  File "/share/app/FALCON-integrate/FALCON/falcon_kit/mains/run1.py", line 422, in main1
    config = support.get_dict_from_old_falcon_cfg(support.parse_config(input_config_fn))
  File "/share/app/FALCON-integrate/FALCON/falcon_kit/run_support.py", line 237, in get_dict_from_old_falcon_cfg
    ' from "falcon_sense_option" in your cfg: %s' %repr(falcon_sense_option))
Exception: Please remove obsolete "--local_match_count_*" or "--output_dformat" from "falcon_sense_option" in your cfg: '--output_multi --min_idt 0.70 --min_cov 4 --local_match_count_threshold 2 --max_n_read 200 --n_core 6'
Traceback (most recent call last):
  File "/share/app/FALCON-integrate/fc_env/bin/fc_run.py", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/share/app/FALCON-integrate/FALCON/src/py_scripts/fc_run.py", line 5, in <module>
    main(sys.argv)
  File "/share/app/FALCON-integrate/FALCON/falcon_kit/mains/run1.py", line 668, in main
    main1(argv[0], args.config, args.logger)
  File "/share/app/FALCON-integrate/FALCON/falcon_kit/mains/run1.py", line 422, in main1
    config = support.get_dict_from_old_falcon_cfg(support.parse_config(input_config_fn))
  File "/share/app/FALCON-integrate/FALCON/falcon_kit/run_support.py", line 237, in get_dict_from_old_falcon_cfg
    ' from "falcon_sense_option" in your cfg: %s' %repr(falcon_sense_option))
Exception: Please remove obsolete "--local_match_count_*" or "--output_dformat" from "falcon_sense_option" in your cfg: '--output_multi --min_idt 0.70 --min_cov 4 --local_match_count_threshold 2 --max_n_read 200 --n_core 6'

Then I deleted --local_match_count_threshold 2 in file fc_run.cfg and run fc_run.py fc_run.cfg again. It appeared:

[INFO]Setup logging from file "None".
[INFO]fc_run started with configuration fc_run.cfg
[WARNING]HPC.daligner option "-dal" has changed to "-B". Correcting this for you.
[WARNING]HPC.daligner option "-dal" has changed to "-B". Correcting this for you.
[INFO] No target specified, assuming "assembly" as target
[WARNING]In simple_pwatcher_bridge, pwatcher_impl=<module 'pwatcher.fs_based' from '/share/app/FALCON-integrate/pypeFLOW/pwatcher/fs_based.pyc'>
[INFO]In simple_pwatcher_bridge, pwatcher_impl=<module 'pwatcher.fs_based' from '/share/app/FALCON-integrate/pypeFLOW/pwatcher/fs_based.pyc'>
[INFO]job_type='SGE', job_queue='default'
[INFO]Num unsatisfied: 1
[INFO]About to submit: Node(0-rawreads/raw-fofn-abs)
[INFO]starting job Job(jobid='Pab56a8beac8601', cmd='/bin/bash run.sh', rundir='/public/home/lijing/faclon_test/0-rawreads/raw-fofn-abs', options={'job_queue': 'default', 'sge_option': '-pe smp 8 -q jobqueue', 'job_type': 'SGE'})
[INFO]!qsub -N Pab56a8beac8601 -q default -pe smp 8 -V -cwd -o stdout -e stderr -S /bin/bash /public/home/lijing/faclon_test/mypwatcher/wrappers/run-Pab56a8beac8601.bash
Unable to run job: Job was rejected because job requests unknown queue "default".
Exiting.
[ERROR]In pwatcher.fs_based.cmd_run(), failed to submit background-job:
MetaJobSge(MetaJob(job=Job(jobid='Pab56a8beac8601', cmd='/bin/bash run.sh', rundir='/public/home/lijing/faclon_test/0-rawreads/raw-fofn-abs', options={'job_queue': 'default', 'sge_option': '-pe smp 8 -q jobqueue', 'job_type': 'SGE'}), lang_exe='/bin/bash'))
Traceback (most recent call last):
  File "/share/app/FALCON-integrate/pypeFLOW/pwatcher/fs_based.py", line 520, in cmd_run
    state.submit_background(job)
  File "/share/app/FALCON-integrate/pypeFLOW/pwatcher/fs_based.py", line 117, in submit_background
    bjob.submit(self, exe, script_fn) # Can raise
  File "/share/app/FALCON-integrate/pypeFLOW/pwatcher/fs_based.py", line 296, in submit
    system(sge_cmd, checked=True) # TODO: Capture q-jobid
  File "/share/app/FALCON-integrate/pypeFLOW/pwatcher/fs_based.py", line 539, in system
    raise Exception('{} <- {!r}'.format(rc, call))
Exception: 256 <- 'qsub -N Pab56a8beac8601 -q default -pe smp 8 -V -cwd -o stdout -e stderr -S /bin/bash /public/home/lijing/faclon_test/mypwatcher/wrappers/run-Pab56a8beac8601.bash'
[ERROR]Failed to enqueue 1 of 1 jobs: set([Node(0-rawreads/raw-fofn-abs)])
[WARNING]Nothing is happening, and we had 0 failures. Should we quit? Instead, we will just sleep.
[INFO]sleep 0.1
[INFO]Num unsatisfied: 2
[INFO]About to submit: Node(0-rawreads/raw-fofn-abs)
[INFO]starting job Job(jobid='Pab56a8beac8601', cmd='/bin/bash run.sh', rundir='/public/home/lijing/faclon_test/0-rawreads/raw-fofn-abs', options={'job_queue': 'default', 'sge_option': '-pe smp 8 -q jobqueue', 'job_type': 'SGE'})
[INFO]!qsub -N Pab56a8beac8601 -q default -pe smp 8 -V -cwd -o stdout -e stderr -S /bin/bash /public/home/lijing/faclon_test/mypwatcher/wrappers/run-Pab56a8beac8601.bash
Unable to run job: Job was rejected because job requests unknown queue "default".
Exiting.

I have no idea about this mistake information. Could you give me some help. Thanks a lot.

As mentioned in another Issue, you need job_type = XXX, not the default.

Also, I guess our wiki needs to be updated for fc_run.cfg. Feel free! Let us know if you need access.

Yea. I think the manual is not very easy for a green hand to get on. If there is a clear example, such us how to set each parameter of fc_run.cfg, how to parse the result et al, would be great help to new comers of PacBio assembly.

@pd-cdumn Hi, I used the FALCON-examples/run/synth0/fc_run.cfg to test E.coli reads. The fc_run.cfg is as follows:

[General]
use_tmpdir = true
job_type = local
#job_type = sge
#stop_all_jobs_on_failure = true

# list of files of the initial bas.h5 files
input_fofn = input.fofn
#input_fofn = preads.fofn

input_type = raw
#input_type = preads

# The length cutoff used for seed reads used for initial mapping
#length_cutoff = 1
genome_size = 5000
seed_coverage = 20

# The length cutoff used for seed reads usef for pre-assembly
length_cutoff_pr = 1

job_queue = production
sge_option_da = -pe smp 8 -q %(job_queue)s
sge_option_la = -pe smp 2 -q %(job_queue)s
sge_option_pda = -pe smp 8 -q %(job_queue)s
sge_option_pla = -pe smp 2 -q %(job_queue)s
sge_option_fc = -pe smp 24 -q %(job_queue)s
sge_option_cns = -pe smp 8 -q %(job_queue)s

pa_concurrent_jobs = 32
cns_concurrent_jobs = 32
ovlp_concurrent_jobs = 32

pa_HPCdaligner_option =   -v -B4 -t50 -h1 -e.99 -w1 -l1 -s1000
ovlp_HPCdaligner_option = -v -B4 -t50 -h1 -e.99 -l1 -s1000

#pa_DBsplit_option =   -a -x5 -s.00065536
pa_DBsplit_option =   -a -x5 -s.065536
#pa_DBsplit_option =   -a -x5 -s1
ovlp_DBsplit_option = -a -x5 -s50

falcon_sense_option = --output_multi --min_idt 0.70 --min_cov 1 --max_n_read 20000 --n_core 0
#--min_cov_aln 1 --min_len_aln 40

overlap_filtering_setting = --max_diff 10000 --max_cov 100000 --min_cov 1 --min_len 1 --bestn 1000 --n_core 0
#dazcon = 1

Then I run fc_run.py fc_run.cfg. It appeared:

[INFO]Setup logging from file "None".
[INFO]fc_run started with configuration fc_run.cfg
[INFO] No target specified, assuming "assembly" as target
[WARNING]In simple_pwatcher_bridge, pwatcher_impl=<module 'pwatcher.fs_based' from '/share/app/FALCON-integrate/pypeFLOW/pwatcher/fs_based.pyc'>
[INFO]In simple_pwatcher_bridge, pwatcher_impl=<module 'pwatcher.fs_based' from '/share/app/FALCON-integrate/pypeFLOW/pwatcher/fs_based.pyc'>
[INFO]job_type='local', job_queue='production'
[INFO]Num unsatisfied: 1
[INFO]About to submit: Node(0-rawreads/raw-fofn-abs)
[INFO]starting job Job(jobid='Pab56a8beac8601', cmd='/bin/bash run.sh', rundir='/public/home/lijing/faclon_test/0-rawreads/raw-fofn-abs', options={'job_queue': 'production', 'sge_option': '-pe smp 8 -q production', 'job_type': 'local'})
[INFO]Submitted backgroundjob=MetaJobLocal(MetaJob(job=Job(jobid='Pab56a8beac8601', cmd='/bin/bash run.sh', rundir='/public/home/lijing/faclon_test/0-rawreads/raw-fofn-abs', options={'job_queue': 'production', 'sge_option': '-pe smp 8 -q production', 'job_type': 'local'}), lang_exe='/bin/bash'))
[INFO]sleep 0.1
[INFO]sleep 0.2
[INFO]sleep 0.3
[INFO]sleep 0.4
[INFO]sleep 0.5
[INFO]sleep 0.6
[INFO]sleep 0.7
[INFO]sleep 0.8
[INFO]sleep 0.9
[INFO]sleep 1.0
[INFO]sleep 1.1
[INFO]sleep 1.2
[INFO]sleep 1.3
[INFO]sleep 1.4
[INFO]sleep 1.5
[INFO]sleep 1.6
[INFO]sleep 1.7
[INFO]sleep 1.8
[INFO]sleep 1.9
[INFO]sleep 2.0
[INFO]sleep 2.1
[INFO]sleep 2.2
[INFO]sleep 2.3
[INFO]sleep 2.4
[INFO]sleep 2.5
[INFO]sleep 2.6
[INFO]sleep 2.7
[INFO]sleep 2.8
[INFO]sleep 2.9
[INFO]sleep 3.0
[INFO]sleep 3.1
[INFO]sleep 3.2
[INFO]sleep 3.3
[INFO]sleep 3.4
[INFO]sleep 3.5
[ERROR]Task Node(0-rawreads/raw-fofn-abs) failed with exit-code=256
[INFO]recently_satisfied: set([])
[INFO]Num satisfied in this iteration: 0
[INFO]Num still unsatisfied: 1
[ERROR]Some tasks are recently_done but not satisfied: set([Node(0-rawreads/raw-fofn-abs)])
[ERROR]ready: set([])
submitted: set([])
Traceback (most recent call last):
  File "/share/app/FALCON-integrate/fc_env/bin/fc_run.py", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/share/app/FALCON-integrate/FALCON/src/py_scripts/fc_run.py", line 5, in <module>
    main(sys.argv)
  File "/share/app/FALCON-integrate/FALCON/falcon_kit/mains/run1.py", line 668, in main
    main1(argv[0], args.config, args.logger)
  File "/share/app/FALCON-integrate/FALCON/falcon_kit/mains/run1.py", line 436, in main1
    setNumThreadAllowed=PypeProcWatcherWorkflow.setNumThreadAllowed)
  File "/share/app/FALCON-integrate/FALCON/falcon_kit/mains/run1.py", line 470, in run
    wf.refreshTargets([fofn_abs_task])
  File "/share/app/FALCON-integrate/pypeFLOW/pypeflow/simple_pwatcher_bridge.py", line 206, in refreshTargets
    self._refreshTargets(updateFreq, exitOnFailure)
  File "/share/app/FALCON-integrate/pypeFLOW/pypeflow/simple_pwatcher_bridge.py", line 273, in _refreshTargets
    raise Exception(msg)
Exception: Some tasks are recently_done but not satisfied: set([Node(0-rawreads/raw-fofn-abs)])

Are there any parameters setting wrong to cause this problem?

Look in 0-rawreads/raw-fofn-abs/pwatcher.dir/stderr. My guess is that the file input.fofn does not actually exist.

I have the same problem, do you solve the problem?

What does that stderr say?

2016-11-25 21:49:43,599 - root - CRITICAL - Error in /opt/anaconda/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/do_task.py with args="{'json_fn': '/ho
me/MD3600f/puluotong/PMO/xhy/software/FALCON-integrate/FALCON-examples/run/ecoli/0-rawreads/raw-fofn-abs/task.json',\n 'timeout': 60,\n 'tmpdir': None}"

Traceback (most recent call last):
  File "/opt/anaconda/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/opt/anaconda/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/opt/anaconda/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/do_task.py", line 149, in <module>
    main()
  File "/opt/anaconda/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/do_task.py", line 141, in main
    run(**vars(parsed_args))
  File "/opt/anaconda/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/do_task.py", line 113, in run
    func = get_func(python_function_name)
  File "/opt/anaconda/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/do_task.py", line 90, in get_func
    mod = importlib.import_module(mod_name)
  File "/opt/anaconda/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named pype_tasks

what did I miss? How could I install pype_tasks？

That's strange. You are using the old pypeflow-0.1.1, but the tip of master branch of FALCON-integrate/ (also FALCON-integrate/1.8.3) definitely uses pypeflow-1.0.0 Could you re-install from 1.8.3? Somehow, you need an updated version of pypeflow, either directly in your Anaconda or indirectly via PYTHONUSERBASE.

Yes, when I re-install, everything is OK! Thank you very much! Best wishes!

I am interested in Falcon assembler, I do fc_run.py FALCON-examples/run/synth0/fc_run.cfg. I have updated version of everything. Please find below the error message: [ERROR]Some tasks are recently_done but not satisfied: set([Node(0-rawreads/raw-fofn-abs)]) [ERROR]ready: set([]) submitted: set([]) Traceback (most recent call last): File "./fc_env/bin/fc_run.py", line 6, in exec(compile(open(file).read(), file, 'exec')) File "/local/data/shilpa/FALCON-integrate/FALCON/src/py_scripts/fc_run.py", line 5, in main(sys.argv) File "/local/data/shilpa/FALCON-integrate/FALCON/falcon_kit/mains/run1.py", line 460, in main main1(argv[0], args.config, args.logger) File "/local/data/shilpa/FALCON-integrate/FALCON/falcon_kit/mains/run1.py", line 136, in main1 input_fofn_plf=input_fofn_plf, File "/local/data/shilpa/FALCON-integrate/FALCON/falcon_kit/mains/run1.py", line 170, in run wf.refreshTargets([fofn_abs_task]) File "/local/data/shilpa/FALCON-integrate/pypeFLOW/pypeflow/simple_pwatcher_bridge.py", line 226, in refreshTargets self._refreshTargets(updateFreq, exitOnFailure) File "/local/data/shilpa/FALCON-integrate/pypeFLOW/pypeflow/simple_pwatcher_bridge.py", line 292, in _refreshTargets raise Exception(msg) Exception: Some tasks are recently_done but not satisfied: set([Node(0-rawreads/raw-fofn-abs)])

How shall I fix it?

try to move (or delete) mypwatcher and restart again. if you still see it, just start from scratch. Also, check the scripts and logs in0-rawreads directory. Maybe, there is some thing in your system that makes the first few command failing there.

I re-installed everything, in stderr, I could see something like this:

aw-fofn-abs/task.json',\n 'timeout': 60,\n 'tmpdir': '/tmp'}" Traceback (most recent call last): File "/home/sgarg/new_anaconda/anaconda3/envs/py27/lib/python2.7/runpy.py", line 174, in _run_module_as_main "main", fname, loader, pkg_name) File "/home/sgarg/new_anaconda/anaconda3/envs/py27/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/local/data/shilpa/FALCON-integrate/pypeFLOW/pypeflow/do_task.py", line 190, in main() File "/local/data/shilpa/FALCON-integrate/pypeFLOW/pypeflow/do_task.py", line 182, in main run(**vars(parsed_args)) File "/local/data/shilpa/FALCON-integrate/pypeFLOW/pypeflow/do_task.py", line 136, in run run_cfg_in_tmpdir(cfg, tmpdir) File "/local/data/shilpa/FALCON-integrate/pypeFLOW/pypeflow/do_task.py", line 139, in run_cfg_in_tmpdir wait_for(fn) File "/local/data/shilpa/FALCON-integrate/pypeFLOW/pypeflow/do_task.py", line 92, in wait_for raise Exception('Timed out waiting for {!r}'.format(fn)) Exception: Timed out waiting for u'/local/data/shilpa/FALCON-integrate/input.fofn'

I tried to give TRUE path, even then it gives the same error. Thanks.

On Tue, Dec 13, 2016 at 4:58 PM, Jason Chin notifications@github.com wrote:

try to move (or delete) mypwatcher and restart again. if you still see it, just start from scratch. Also, check the scripts and logs in0-rawreads directory. Maybe, there is some thing in your system that makes the first few command failing there.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/PacificBiosciences/FALCON-integrate/issues/91#issuecomment-266777513, or mute the thread https://github.com/notifications/unsubscribe-auth/AHEq5z0JEd2Iho2kOxf0ovhXYU3tYTUKks5rHsCvgaJpZM4Km3Fi .

Seems like a configuration error. Does /local/data/shilpa/FALCON-integrate/input.fofn exist?

We install the Faclon software from git (master), we meet problems when we run test. errors as following: WARNING: autodoc: failed to import module u'pypeflow.task'; the following exception was raised: Traceback (most recent call last): File "/zfssz5/BC_PS/chenjunhui/02.genome/01.Genome/02.assembly/Pacbio_assmbly/Falcon/FALCON/lib/python2.7/site-packages/sphinx/ext/autodoc.py", line 658, in import_object import(self.modname) ImportError: No module named task

would you give us some help about this problem?

@chenjunhui, Is that an old version? We don't use sphinx anymore. If recent, you might need to update pypeFLOW also.

PacificBiosciences / FALCON-integrate

FALCON-integrate installation problem(about the final test part) #91