PacificBiosciences / FALCON-integrate

Mostly deprecated. See https://github.com/PacificBiosciences/FALCON_unzip/wiki/Binaries
https://github.com/PacificBiosciences/FALCON/wiki/Manual
31 stars 28 forks source link

Problem when old pypeflow/etc. are already installed before running virtualenv #50

Closed JohnUrban closed 8 years ago

JohnUrban commented 8 years ago

Hi,

It goes without saying, thanks for the great tools.

I have PacBio data that I can assemble with Canu, but want to try with Falcon.

I followed this procedure for installing Falcon 0.4.0 successfully:

#!/bin/bash
module load git
git clone git://github.com/PacificBiosciences/FALCON-integrate.git
cd FALCON-integrate
git checkout 0.4.0  # or whatever version you want - 0.4.0 install works, 0.5.2 latest (0.5.1,0.5.2 install has not worked)
make init
make virtualenv
make check
make -j install

The protocol above finishes without any errors -- it gets through what seems to be a test run at the end. I can then successfully assemble an ecoli dataset by making the config file compatible for SLURM:

falcon.slurm.in (batch script)

#!/bin/bash
#SBATCH -J falcon-ecoli
#SBATCH -c 4
#SBATCH --qos=biomed-sb-condo
#SBATCH --mem=10g
#SBATCH --time=4:00:00
###########################
mkdir data
cd data/
wget https://www.dropbox.com/s/tb78i5i3nrvm6rg/m140913_050931_42139_c100713652400000001823152404301535_s1_p0.1.subreads.fasta
mv m140913_050931_42139_c100713652400000001823152404301535_s1_p0.1.subreads.fasta ecoli.1.subreads.fasta 

wget https://www.dropbox.com/s/v6wwpn40gedj470/m140913_050931_42139_c100713652400000001823152404301535_s1_p0.2.subreads.fasta
mv m140913_050931_42139_c100713652400000001823152404301535_s1_p0.2.subreads.fasta ecoli.2.subreads.fasta 

wget https://www.dropbox.com/s/j61j2cvdxn4dx4g/m140913_050931_42139_c100713652400000001823152404301535_s1_p0.3.subreads.fasta
mv m140913_050931_42139_c100713652400000001823152404301535_s1_p0.3.subreads.fasta ecoli.3.subreads.fasta 
cd ..

ls data/*fasta > input.fofn

PATH=~/software/falcon/FALCON-integrate/fc_env/bin/:$PATH
fc_run.py fc_run.cfg

fc_run.cfg

[General]
input_fofn = input.fofn

input_type = raw
#input_type = preads

# The length cutoff used for seed reads used for initial mapping
length_cutoff = 12000

# The length cutoff used for seed reads usef for pre-assembly
length_cutoff_pr = 12000

job_type = SLURM
jobqueue = serial
sge_option_da = --ntasks 1 --nodes 1 --cpus-per-task 8 --mem 10g --time 00:20:00 
sge_option_la = --ntasks 1 --nodes 1 --cpus-per-task 2 --mem 10g --time 00:05:00 
sge_option_pda = --ntasks 1 --nodes 1 --cpus-per-task 8 --mem 15g --time 00:20:00 
sge_option_pla = --ntasks 1 --nodes 1 --cpus-per-task 2 --mem 10g --time 00:05:00 
sge_option_fc = --ntasks 1 --nodes 1 --cpus-per-task 16 --mem 10g --time 00:20:00 
sge_option_cns = --ntasks 1 --nodes 1 --cpus-per-task 8 --mem 10g --time 00:20:00 

pa_concurrent_jobs = 32
ovlp_concurrent_jobs = 32

pa_HPCdaligner_option =  -v -dal4 -t16 -e.70 -l1000 -s1000
ovlp_HPCdaligner_option = -v -dal4 -t32 -h60 -e.96 -l500 -s1000

pa_DBsplit_option = -x500 -s50
ovlp_DBsplit_option = -x500 -s50

falcon_sense_option = --output_multi --min_idt 0.70 --min_cov 4 --max_n_read 200 --n_core 6

overlap_filtering_setting = --max_diff 100 --max_cov 100 --min_cov 20 --bestn 10 --n_core 24

However, replacing 0.4.0 with either 0.5.0, 0.5.1, or 0.5.2 in that same install protocol results in an unsuccessful installation. e.g.:

#!/bin/bash
module load git
git clone git://github.com/PacificBiosciences/FALCON-integrate.git
cd FALCON-integrate
git checkout 0.5.1
make init
make virtualenv
make check
make -j install

gets up to what seems like a test and fails:

symlink: 'run/synth0/data/ref.fasta'
symlink: 'run/synth0/data/synth0.fasta'
cd run/synth0; fc_run.py fc_run.cfg logging.ini
falcon_kit.mains.run args: ['/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/bin/fc_run.py', 'fc_run.cfg', 'logging.ini']
[INFO] fc_run started with configuration fc_run.cfg
[WARNING] HPC.daligner option "-dal" has changed to "-B". Correcting this for you.
[WARNING] HPC.daligner option "-dal" has changed to "-B". Correcting this for you.
[INFO]  No target specified, assuming "assembly" as target 
/gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON/falcon_kit/run_support.py:299: UserWarning: Unexpected keys in input config: set(['jobqueue'])
  warnings.warn("Unexpected keys in input config: %s" %repr(unused))
[ERROR] Any exception caught in RefreshTargets() indicates an unrecoverable error. Shutting down...
Traceback (most recent call last):
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 531, in refreshTargets
    rtn = self._refreshTargets(task2thread, objs = objs, callback = callback, updateFreq = updateFreq, exitOnFailure = exitOnFailure)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 571, in _refreshTargets
    tSortedURLs = PypeGraph(rdfGraph, connectedPypeNodes).tSort( )
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 109, in __init__
    for row in self._RDFGraph.query('SELECT ?s ?o WHERE {?s pype:prereq ?o . }', initNs=dict(pype=pypeNS)):
  File "build/bdist.linux-x86_64/egg/rdflib/graph.py", line 1004, in query
    processorinst = plugin.get(processor, query.Processor)(self)
  File "build/bdist.linux-x86_64/egg/rdflib/plugin.py", line 103, in get
    return p.getClass()
  File "build/bdist.linux-x86_64/egg/rdflib/plugin.py", line 65, in getClass
    module = __import__(self.module_path, globals(), locals(), [""])
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/rdfextras-0.4-py2.7.egg/rdfextras/sparql/processor.py", line 2, in <module>
    import rdfextras.sparql.parser
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/rdfextras-0.4-py2.7.egg/rdfextras/sparql/parser.py", line 3, in <module>
    from pyparsing import (Regex, Suppress, Combine, Optional, CaselessKeyword,
ImportError: No module named pyparsing

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! Please wait for all threads / processes to terminate !
! Also, maybe use 'ps' or 'qstat' to check all threads,!
! processes and/or jobs are terminated cleanly.        !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[WARNING] #tasks=0, #alive=0
Traceback (most recent call last):
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/bin/fc_run.py", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON/src/py_scripts/fc_run.py", line 5, in <module>
    main(sys.argv)
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON/falcon_kit/mains/run.py", line 637, in main
    main1(*argv)
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON/falcon_kit/mains/run.py", line 460, in main1
    wf.refreshTargets([fofn_abs_task])
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 531, in refreshTargets
    rtn = self._refreshTargets(task2thread, objs = objs, callback = callback, updateFreq = updateFreq, exitOnFailure = exitOnFailure)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 571, in _refreshTargets
    tSortedURLs = PypeGraph(rdfGraph, connectedPypeNodes).tSort( )
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 109, in __init__
    for row in self._RDFGraph.query('SELECT ?s ?o WHERE {?s pype:prereq ?o . }', initNs=dict(pype=pypeNS)):
  File "build/bdist.linux-x86_64/egg/rdflib/graph.py", line 1004, in query
  File "build/bdist.linux-x86_64/egg/rdflib/plugin.py", line 103, in get
  File "build/bdist.linux-x86_64/egg/rdflib/plugin.py", line 65, in getClass
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/rdfextras-0.4-py2.7.egg/rdfextras/sparql/processor.py", line 2, in <module>
    import rdfextras.sparql.parser
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/rdfextras-0.4-py2.7.egg/rdfextras/sparql/parser.py", line 3, in <module>
    from pyparsing import (Regex, Suppress, Combine, Optional, CaselessKeyword,
ImportError: No module named pyparsing
make[3]: *** [run-synth0] Error 1
make[3]: Leaving directory `/gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON-examples'
make[2]: *** [test] Error 2
make[2]: Leaving directory `/gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON-examples'
make[1]: *** [test] Error 2
make[1]: Leaving directory `/gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON-make'

It is complaining that there is no pyparsing module…. the pyparsing module is certainly available in my general environment though.

When I try to assemble the ecoli data set as done above, it also gives this type of error. I found that installing pyparsing in a certain falcon dir seemed to result in bypassing this import error and throwing another error:

pip install -t ~/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/ pyparsing

Any ideas on how to successfully install the most recent version(s) would be greatly appreciated. I would be happy to answer any clarifying questions that might help you help me. Again, thanks for the great tools and support.

best,

John

p.s. I am also having trouble running falcon 0.4.0 with simulated fasta files. After searching issues and figuring out the error, it seemed like the fasta name and fasta formatting mattered. Nonetheless, I still have not been able to get falcon to run on simulated data… any advice on how to format a fasta file would be greatly appreciated.

pb-cdunn commented 8 years ago
File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg

That does not seem to be the virtualenv that you refer to elsewhere:

falcon_kit.mains.run args: ['/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/bin/fc_run.py', 'fc_run.cfg', 'logging.ini']

Did you "activate" the virtualenv? Maybe you need to re-activate. You should not be picking up any Python packages from outside the virtualenv.

I found that installing pyparsing in a certain falcon dir seemed to result in bypassing this import error ...

You should activate your virtualenv before running pip install. (And you should not use pip install -t.

p.s. I am also having trouble running falcon 0.4.0 with simulated fasta files. After searching issues and figuring out the error, it seemed like the fasta name and fasta formatting mattered.

Yes, DAZZ_DB/fasta2DB is very particular about fasta headers. It's aimed at PacBio data, but I do use it on my own simulated reads. There is a fasta2fasta in 0.5.2 that can help. Basically, you need:

>name/wellnumber/number_number
GATTACA...

Maybe the num_num part can be any string, but the wellnumber should be unique for each read. DBsplit without -a will exclude all reads of the same wellnumber except the longest one; otherwise, all reads are retained.

JohnUrban commented 8 years ago

The test at the end of the install protocol (pasted again below) fails before I can attempt to activate the virtual env later during ecoli testing. Both give the same pyparsing message.

#!/bin/bash
module load git
git clone git://github.com/PacificBiosciences/FALCON-integrate.git
cd FALCON-integrate
git checkout 0.4.0  # or whatever version you want - 0.4.0 install works, 0.5.2 latest (0.5.1,0.5.2 install has not worked)
make init
make virtualenv
make check
make -j install

It seems the above routine fails to install pyparsing due to the following message:

Requirement already satisfied (use --upgrade to upgrade): pyparsing<=1.5.7 in /gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/pyparsing-1.5.7-py2.7.egg (from rdfextras>=0.1->pypeflow==0.1.1)

In fact, a bunch of packages do not install:

Requirement already satisfied (use --upgrade to upgrade): rdflib==3.4.0 in /gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/rdflib-3.4.0-py2.7.egg (from pypeflow==0.1.1)
Requirement already satisfied (use --upgrade to upgrade): rdfextras>=0.1 in /gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/rdfextras-0.4-py2.7.egg (from pypeflow==0.1.1)
Requirement already satisfied (use --upgrade to upgrade): isodate in /gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/isodate-0.5.4-py2.7.egg (from rdflib==3.4.0->pypeflow==0.1.1)
Requirement already satisfied (use --upgrade to upgrade): html5lib in /gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/html5lib-1.0b8-py2.7.egg (from rdflib==3.4.0->pypeflow==0.1.1)
Requirement already satisfied (use --upgrade to upgrade): pyparsing<=1.5.7 in /gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/pyparsing-1.5.7-py2.7.egg (from rdfextras>=0.1->pypeflow==0.1.1)
Requirement already satisfied (use --upgrade to upgrade): six in /gpfs_home/jurban/software/localpy/lib/python2.7/site-packages (from html5lib->rdflib==3.4.0->pypeflow==0.1.1)

So - not installed: rdflib rdfextras isodate html5lib pyparsing six

If I look in FALCON-integrate/fc_env/lib/python2.7/site-packages/, I see they are indeed missing (but are there after I force install them below):

$ ls FALCON-integrate/fc_env/lib/python2.7/site-packages/
decorator-4.0.9.dist-info
decorator.py
decorator.pyc
easy-install.pth
easy_install.py
easy_install.pyc
falcon-kit.egg-link
networkx
networkx-1.10.dist-info
pip
pip-8.1.2.dist-info
pkg_resources
pypeflow.egg-link
setuptools
setuptools-21.2.2.dist-info
wheel
wheel-0.29.0.dist-info

After activating the environment, I tried to pip install all the packages above. However, "pip install package-name" is not sufficient b/c it still detects the package elsewhere. Moreover, "pip install -U package-name" works for some, but not others. Ultimately, I need to do "pip install -U --force-reinstall package-name". An example of doing only "pip install -U package-name" (without forced reinstall) is below.

cd FALCON-integrate/
. fc_env/bin/activate
pip install -U rdflib rdfextras isodate html5lib pyparsing six

/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
Collecting rdflib
  Downloading rdflib-4.2.1.tar.gz (889kB)
    100% |████████████████████████████████| 890kB 844kB/s 
Requirement already up-to-date: rdfextras in /gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/rdfextras-0.4-py2.7.egg
Requirement already up-to-date: isodate in /gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/isodate-0.5.4-py2.7.egg
Requirement already up-to-date: html5lib in /gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/html5lib-1.0b8-py2.7.egg
Collecting pyparsing
  Using cached pyparsing-2.1.4-py2.py3-none-any.whl
Requirement already up-to-date: six in /gpfs_home/jurban/software/localpy/lib/python2.7/site-packages
Collecting SPARQLWrapper (from rdflib)
  Downloading SPARQLWrapper-1.7.6.zip
Collecting keepalive>=0.5 (from SPARQLWrapper->rdflib)
  Downloading keepalive-0.5.zip
Building wheels for collected packages: rdflib, SPARQLWrapper, keepalive
  Running setup.py bdist_wheel for rdflib ... done
  Stored in directory: /users/jurban/.cache/pip/wheels/fb/93/10/4f8a3e95937d8db410a490fa235bd95e0e0d41b5f6274b20e5
  Running setup.py bdist_wheel for SPARQLWrapper ... done
  Stored in directory: /users/jurban/.cache/pip/wheels/50/fe/25/be6e98daa4f576494df2a18d5e86a182e3d7e0735d062cc984
  Running setup.py bdist_wheel for keepalive ... done
  Stored in directory: /users/jurban/.cache/pip/wheels/16/4f/c1/121ddff67b131a371b66d682feefac055fbdbb9569bfde5c51
Successfully built rdflib SPARQLWrapper keepalive
Installing collected packages: pyparsing, keepalive, SPARQLWrapper, rdflib
  Found existing installation: pyparsing 1.5.7
    Not uninstalling pyparsing at /gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/pyparsing-1.5.7-py2.7.egg, outside environment /gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env
  Found existing installation: rdflib 3.4.0
    Not uninstalling rdflib at /gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/rdflib-3.4.0-py2.7.egg, outside environment /gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env
Successfully installed SPARQLWrapper-1.7.6 keepalive-0.5 pyparsing-1.5.7 rdflib-3.4.0

As I said though - using --force-reinstall helps. Nonetheless, after installing those packages, I get different errors. For example, if I do 'make test':

$ cd FALCON-integrate
$ . fc_env/bin/activate
$ PATH=~/software/falcon/FALCON-integrate/fc_env/bin/:$PATH
$ make test
ln -sf ../../git-sym/git-sym /gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/bin/git-sym
. /gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/bin/activate; make -C ./FALCON-make/ test
make[1]: Entering directory `/gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON-make'
make -C /gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON-examples test
make[2]: Entering directory `/gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON-examples'
python -c 'import pypeflow.common; print pypeflow.common'
<module 'pypeflow.common' from '/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/common.pyc'>
python -c 'import falcon_kit; print falcon_kit.falcon'
<CDLL '/users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/ext_falcon.so', handle 139ef20 at 7fef0168cb90>
make run-synth0
make[3]: Entering directory `/gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON-examples'
git-sym update run/synth0
/bin/sh: git: command not found
Command 'git rev-parse --git-dir' returned non-zero exit status 127
Traceback (most recent call last):
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/bin/git-sym", line 549, in <module>
    global_setup()
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/bin/git-sym", line 170, in global_setup
    GIT_DIR = os.path.abspath(get_GIT_DIR())
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/posixpath.py", line 343, in abspath
    if not isabs(path):
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/posixpath.py", line 53, in isabs
    return s.startswith('/')
AttributeError: 'NoneType' object has no attribute 'startswith'
make[3]: *** [setup-synth0] Error 1
make[3]: Leaving directory `/gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON-examples'
make[2]: *** [test] Error 2
make[2]: Leaving directory `/gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON-examples'
make[1]: *** [test] Error 2
make[1]: Leaving directory `/gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON-make'
make: *** [test] Error 2

Of I try a batch script for the ecoli data described previously, I get a different error. batch script:

#!/bin/bash
#SBATCH -J falcon-ecoli
#SBATCH -c 4
#SBATCH --qos=biomed-sb-condo
#SBATCH --mem=10g
#SBATCH --time=4:00:00

###########################

. ~/software/falcon/FALCON-integrate/fc_env/bin/activate
PATH=~/software/falcon/FALCON-integrate/fc_env/bin/:$PATH
fc_run.py fc_run.cfg

batch stdout + stderr:

## SLURM PROLOG ###############################################################
##    Job ID : 12712575
##  Job Name : falcon-ecoli
##  Nodelist : node407
##      CPUs : 
##  Mem/Node : 10240 MB
## Directory : /gpfs/scratch/jurban/male-ilmn/lambda/falcon/ecoli
##   Started : Tue May 31 14:15:47 EDT 2016
###############################################################################
[INFO]fc_run started with configuration fc_run.cfg
[WARNING]HPC.daligner option "-dal" has changed to "-B". Correcting this for you.
[WARNING]HPC.daligner option "-dal" has changed to "-B". Correcting this for you.
[INFO] No target specified, assuming "assembly" as target 
/gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON/falcon_kit/run_support.py:317: UserWarning: Unexpected keys in input config: set(['jobqueue'])
  warnings.warn("Unexpected keys in input config: %s" %repr(unused))
[ERROR]Any exception caught in RefreshTargets() indicates an unrecoverable error. Shutting down...
Traceback (most recent call last):
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 531, in refreshTargets
    rtn = self._refreshTargets(task2thread, objs = objs, callback = callback, updateFreq = updateFreq, exitOnFailure = exitOnFailure)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 571, in _refreshTargets
    tSortedURLs = PypeGraph(rdfGraph, connectedPypeNodes).tSort( )
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 109, in __init__
    for row in self._RDFGraph.query('SELECT ?s ?o WHERE {?s pype:prereq ?o . }', initNs=dict(pype=pypeNS)):
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/rdflib/graph.py", line 1080, in query
    query_object, initBindings, initNs, **kwargs))
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/rdfextras/sparql/processor.py", line 49, in query
    loadContexts=loadContexts)
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/rdfextras/sparql/algebra.py", line 388, in TopEvaluate
    top.topLevelExpand(expr.constraints, query.prolog)
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/rdfextras/sparql/query.py", line 520, in topLevelExpand
    self.expand(constraints)
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/rdfextras/sparql/query.py", line 527, in expand
    if self.tripleStore.graph.store.batch_unification:
AttributeError: 'IOMemory' object has no attribute 'batch_unification'

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! Please wait for all threads / processes to terminate !
! Also, maybe use 'ps' or 'qstat' to check all threads,!
! processes and/or jobs are terminated cleanly.        !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[WARNING]#tasks=0, #alive=0
Traceback (most recent call last):
  File "/users/jurban/software/falcon/FALCON-integrate/fc_env/bin/fc_run.py", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON/src/py_scripts/fc_run.py", line 5, in <module>
    main(sys.argv)
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON/falcon_kit/mains/run0.py", line 644, in main
    main1(argv[0], args.config, args.logger)
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON/falcon_kit/mains/run0.py", line 467, in main1
    wf.refreshTargets([fofn_abs_task])
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 531, in refreshTargets
    rtn = self._refreshTargets(task2thread, objs = objs, callback = callback, updateFreq = updateFreq, exitOnFailure = exitOnFailure)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 571, in _refreshTargets
    tSortedURLs = PypeGraph(rdfGraph, connectedPypeNodes).tSort( )
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 109, in __init__
    for row in self._RDFGraph.query('SELECT ?s ?o WHERE {?s pype:prereq ?o . }', initNs=dict(pype=pypeNS)):
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/rdflib/graph.py", line 1080, in query
    query_object, initBindings, initNs, **kwargs))
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/rdfextras/sparql/processor.py", line 49, in query
    loadContexts=loadContexts)
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/rdfextras/sparql/algebra.py", line 388, in TopEvaluate
    top.topLevelExpand(expr.constraints, query.prolog)
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/rdfextras/sparql/query.py", line 520, in topLevelExpand
    self.expand(constraints)
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/rdfextras/sparql/query.py", line 527, in expand
    if self.tripleStore.graph.store.batch_unification:
AttributeError: 'IOMemory' object has no attribute 'batch_unification'

In both cases, it is still looking in "/users/jurban/software/localpy/lib/python2.7/site-packages/" as you pointed out… Perhaps I need to prepend the PYTHONPATH variable….?

Thanks for your help.

JohnUrban commented 8 years ago

As for the PYTHONPATH, I tried adding the following to the batch script…

PYTHONPATH=/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages:$PYTHONPATH

It still goes to the other location for pypeflow though…. So I tried…

PYTHONPATH=/users/jurban/software/falcon/FALCON-integrate/pypeFLOW:/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages:$PYTHONPATH

…didn't help.

pb-cdunn commented 8 years ago

Interesting. Thanks for all the detail. That definitely helps.

I think the virtualenv is keeping references to anything that already exists in the source-dir.

Goal: We want this to report that pypeflow comes from the virtualenv:

make[1]: Entering directory `/gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON-make'
make -C /gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON-examples test
make[2]: Entering directory `/gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON-examples'
python -c 'import pypeflow.common; print pypeflow.common'
<module 'pypeflow.common' from '/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/common.pyc'>
python -c 'import falcon_kit; print falcon_kit.falcon'
<CDLL '/users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/ext_falcon.so', handle 139ef20 at 7fef0168cb90>
make run-synth0

I think we have 2 main changes from 0.4.0 to 0.5.2:

  1. virtualenv code has been updated. I have no idea whether the behavior has changed slightly. That is possible.
  2. We now run pip install -e instead of python setup.py or whatever. There could be a subtle difference there too.

My theory: When you run make install, and it goes into FALCON-make, it runs pip install -e pypeFLOW. But pypeflow already exists in the virtualenv, so nothing is updated. Try to add --force-reinstall to the pip install lines in FALCON-make/makefile.

JohnUrban commented 8 years ago

I just shot myself in the foot a little bit.

I tried to do a different install protocol that I assumed was more similar to what I tried back in October 2015 that lead to falcon stuff being in that other location:

#!/bin/bash

module load git

PRE=~/software/localpy

git clone git://github.com/PacificBiosciences/FALCON-integrate.git
cd FALCON-integrate

# Fetch submodules.
git submodule update --init

# Install dependencies.

cd pypeFLOW
python setup.py install --prefix=$PRE
cd ..

cd FALCON
python setup.py install --prefix=$PRE
cd ..

cd DAZZ_DB/
make
cp DBrm DBshow DBsplit DBstats fasta2DB $PRE/bin/
cd ..

cd DALIGNER
make
cp daligner daligner_p DB2Falcon HPCdaligner LA4Falcon LAmerge LAsort  $PRE/bin
cd ..

That lead to nothing working…. here is an example batch stdout+stderr:

## SLURM PROLOG ###############################################################
##    Job ID : 12712996
##  Job Name : falcon-ecoli
##  Nodelist : node408
##      CPUs : 
##  Mem/Node : 10240 MB
## Directory : /gpfs/scratch/jurban/male-ilmn/lambda/falcon/ecoli
##   Started : Tue May 31 15:08:15 EDT 2016
###############################################################################
[INFO]fc_run started with configuration fc_run.cfg
[INFO] No target specified, assuming "assembly" as target 
[INFO]# of tasks in complete graph: 1
[INFO]tick: 1, #updatedTasks: 0, sleep_time=0.000000
[INFO]Running task from function task_make_fofn_abs_raw()
[INFO]Queued 'task:///gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py/task_make_fofn_abs_raw' ...
[INFO]tick: 2, #updatedTasks: 1, sleep_time=0.000000
[INFO]Success ('done'). Joining 'task:///gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py/task_make_fofn_abs_raw'...
[INFO]tick: 4, #updatedTasks: 1, sleep_time=0.000000
[INFO]_refreshTargets() finished with no thread running and no new job to submit
[INFO]# of tasks in complete graph: 2
[INFO]tick: 1, #updatedTasks: 0, sleep_time=0.000000
[INFO]Running task from function task_build_rdb()
[INFO]Queued 'task:///gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py/task_build_rdb' ...
[INFO]tick: 2, #updatedTasks: 1, sleep_time=0.000000
[INFO](SLURM) '/gpfs/scratch/jurban/male-ilmn/lambda/falcon/ecoli/0-rawreads/prepare_rdb.sh'
sh: sbatch: command not found
[WARNING]Call 'sbatch -J prepare_rdb.sh-task_build_rdb-task_build_rdb --ntasks 1 --nodes 1 --cpus-per-task 8 --mem 10g --time 00:20:00 /gpfs/scratch/jurban/male-ilmn/lambda/falcon/ecoli/0-rawreads/prepare_rdb.sh' returned 32512.
[ERROR]PypeTaskBase failed:
{'URL': 'task:///gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py/task_build_rdb',
 '__class__.__name__': 'PypeThreadTaskBase',
 '_status': 'TaskInitialized',
 'inputDataObjs': {'input_fofn': PypeLocalFile('file://localhost/gpfs/scratch/jurban/male-ilmn/lambda/falcon/ecoli/0-rawreads/input.fofn', '/gpfs/scratch/jurban/male-ilmn/lambda/falcon/ecoli/0-rawreads/input.fofn')},
 'mutableDataObjs': {},
 'outputDataObjs': {'rdb_build_done': PypeLocalFile('file://localhost/gpfs/scratch/jurban/male-ilmn/lambda/falcon/ecoli/0-rawreads/rdb_build_done', '/gpfs/scratch/jurban/male-ilmn/lambda/falcon/ecoli/0-rawreads/rdb_build_done'),
                    'run_jobs': PypeLocalFile('file://localhost/gpfs/scratch/jurban/male-ilmn/lambda/falcon/ecoli/0-rawreads/run_jobs.sh', '/gpfs/scratch/jurban/male-ilmn/lambda/falcon/ecoli/0-rawreads/run_jobs.sh')},
 'parameters': {'config': {'cns_concurrent_jobs': 8,
                           'falcon_sense_option': '--output_multi --min_idt 0.70 --min_cov 4 --max_n_read 200 --n_core 6',
                           'falcon_sense_skip_contained': 'False',
                           'input_fofn_fn': 'input.fofn',
                           'input_type': 'raw',
                           'install_prefix': '/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env',
                           'job_type': 'SLURM',
                           'length_cutoff': 12000,
                           'length_cutoff_pr': 12000,
                           'openending': False,
                           'overlap_filtering_setting': '--max_diff 100 --max_cov 100 --min_cov 20 --bestn 10 --n_core 24',
                           'ovlp_DBsplit_option': '-x500 -s50',
                           'ovlp_HPCdaligner_option': '-v -dal4 -t32 -h60 -e.96 -l500 -s1000',
                           'ovlp_concurrent_jobs': 32,
                           'pa_DBsplit_option': '-x500 -s50',
                           'pa_HPCdaligner_option': '-v -dal4 -t16 -e.70 -l1000 -s1000',
                           'pa_concurrent_jobs': 32,
                           'sge_option_cns': '--ntasks 1 --nodes 1 --cpus-per-task 8 --mem 10g --time 00:20:00',
                           'sge_option_da': '--ntasks 1 --nodes 1 --cpus-per-task 8 --mem 10g --time 00:20:00',
                           'sge_option_fc': '--ntasks 1 --nodes 1 --cpus-per-task 16 --mem 10g --time 00:20:00',
                           'sge_option_la': '--ntasks 1 --nodes 1 --cpus-per-task 2 --mem 10g --time 00:05:00',
                           'sge_option_pda': '--ntasks 1 --nodes 1 --cpus-per-task 8 --mem 15g --time 00:20:00',
                           'sge_option_pla': '--ntasks 1 --nodes 1 --cpus-per-task 2 --mem 10g --time 00:05:00',
                           'target': 'assembly',
                           'use_tmpdir': False},
                'work_dir': '/gpfs/scratch/jurban/male-ilmn/lambda/falcon/ecoli/0-rawreads'}}
Traceback (most recent call last):
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/task.py", line 347, in __call__
    return self.runInThisThread(*argv, **kwargv)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/task.py", line 368, in runInThisThread
    self.run(*argv, **kwargv)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/task.py", line 276, in run
    rtn = self._runTask(self, *argv, **kwargv)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/task.py", line 190, in _runTask
    return self._taskFun(self)
  File "/gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py", line 157, in task_build_rdb
    run_script(job_data, job_type = config["job_type"])
  File "/gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py", line 97, in run_script
    rc = _run_script(job_data)
  File "/gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py", line 59, in _run_script_slurm
    system(sge_cmd, check=True)
  File "/gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py", line 24, in system
    raise Exception(msg)
Exception: Call 'sbatch -J prepare_rdb.sh-task_build_rdb-task_build_rdb --ntasks 1 --nodes 1 --cpus-per-task 8 --mem 10g --time 00:20:00 /gpfs/scratch/jurban/male-ilmn/lambda/falcon/ecoli/0-rawreads/prepare_rdb.sh' returned 32512.
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/gpfs/runtime/opt/python/2.7.3/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/gpfs/runtime/opt/python/2.7.3/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/task.py", line 347, in __call__
    return self.runInThisThread(*argv, **kwargv)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/task.py", line 368, in runInThisThread
    self.run(*argv, **kwargv)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/task.py", line 276, in run
    rtn = self._runTask(self, *argv, **kwargv)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/task.py", line 190, in _runTask
    return self._taskFun(self)
  File "/gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py", line 157, in task_build_rdb
    run_script(job_data, job_type = config["job_type"])
  File "/gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py", line 97, in run_script
    rc = _run_script(job_data)
  File "/gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py", line 59, in _run_script_slurm
    system(sge_cmd, check=True)
  File "/gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py", line 24, in system
    raise Exception(msg)
Exception: Call 'sbatch -J prepare_rdb.sh-task_build_rdb-task_build_rdb --ntasks 1 --nodes 1 --cpus-per-task 8 --mem 10g --time 00:20:00 /gpfs/scratch/jurban/male-ilmn/lambda/falcon/ecoli/0-rawreads/prepare_rdb.sh' returned 32512.

[INFO]Failure ('fail'). Joining 'task:///gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py/task_build_rdb'...
[CRITICAL]Any exception caught in RefreshTargets() indicates an unrecoverable error. Shutting down...
/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py:537: UserWarning: 
            "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
            "! Please wait for all threads / processes to terminate !"
            "! Also, maybe use 'ps' or 'qstat' to check all threads,!"
            "! processes and/or jobs are terminated cleanly.        !"
            "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"

  warnings.warn(shutdown_msg)
[WARNING]#tasks=1, #alive=0
['/users/jurban/software/falcon/FALCON-integrate/fc_env/bin/fc_run.py', 'fc_run.cfg']
Traceback (most recent call last):
  File "/users/jurban/software/falcon/FALCON-integrate/fc_env/bin/fc_run.py", line 4, in <module>
    __import__('pkg_resources').run_script('falcon-kit==0.4.0', 'fc_run.py')
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 729, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1642, in run_script
    exec(code, namespace, namespace)
  File "/gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/EGG-INFO/scripts/fc_run.py", line 5, in <module>
    main(sys.argv)
  File "/gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py", line 612, in main
    main1(*argv)
  File "/gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py", line 473, in main1
    wf.refreshTargets([rdb_build_done]) 
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 548, in refreshTargets
    raise Exception('Caused by:\n' + tb)
Exception: Caused by:
Traceback (most recent call last):
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 523, in refreshTargets
    rtn = self._refreshTargets(task2thread, objs = objs, callback = callback, updateFreq = updateFreq, exitOnFailure = exitOnFailure)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 740, in _refreshTargets
    raise TaskFailureError("Counted %d failure(s) with 0 successes so far." %failedJobCount)
TaskFailureError: 'Counted 1 failure(s) with 0 successes so far.'

I tried various things to get things back the way they were when the 0.4.0 install was working… but now nothing is working. It seems even the 0.4.0 install was depending on stuff in the other location -- and that stuff was previously compatible.

I will try your new tip about --force-reinstall.

pb-cdunn commented 8 years ago

I just shot myself in the foot a little bit.

Well, maybe sort of. But you've discovered another installation environment that we should be able to accommodate. If you find changes to FALCON-make that work, I will merge them gladly.

JohnUrban commented 8 years ago

We are closing in on a solution I think.

Your --force-reinstall idea seems to have let the 'make test' finish in the installation protocol. Here is what I did specifically:

#!/bin/bash

module load git

git clone git://github.com/PacificBiosciences/FALCON-integrate.git
cd FALCON-integrate

make init

awk '{gsub(/pip\ install/, "pip install --force-reinstall"); print}'  FALCON-make/makefile > tmp-makefile
mv tmp-makefile FALCON-make/makefile

make virtualenv
make check
make -j install
make test  # to run a simple one

I do see, however, that it still does not install many of the packages it finds elsewhere:

Requirement already satisfied (use --upgrade to upgrade): rdflib==3.4.0 in /gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/rdflib-3.4.0-py2.7.egg (from pypeflow==0.1.1)
Requirement already satisfied (use --upgrade to upgrade): rdfextras>=0.1 in /gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/rdfextras-0.4-py2.7.egg (from pypeflow==0.1.1)
Requirement already satisfied (use --upgrade to upgrade): isodate in /gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/isodate-0.5.4-py2.7.egg (from rdflib==3.4.0->pypeflow==0.1.1)
Requirement already satisfied (use --upgrade to upgrade): html5lib in /gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/html5lib-1.0b8-py2.7.egg (from rdflib==3.4.0->pypeflow==0.1.1)
Requirement already satisfied (use --upgrade to upgrade): pyparsing<=1.5.7 in /gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/pyparsing-1.5.7-py2.7.egg (from rdfextras>=0.1->pypeflow==0.1.1)
Requirement already satisfied (use --upgrade to upgrade): six in /gpfs_home/jurban/software/localpy/lib/python2.7/site-packages (from html5lib->rdflib==3.4.0->pypeflow==0.1.1)

Though the make test finished fine, when I tried running the ecoli batch script (pasted again below), I got the error pasted down below.

#!/bin/bash
#SBATCH -J falcon-ecoli
#SBATCH -c 4
#SBATCH --qos=biomed-sb-condo
#SBATCH --mem=10g
#SBATCH --time=4:00:00

###########################

. ~/software/falcon/FALCON-integrate/fc_env/bin/activate
PATH=~/software/falcon/FALCON-integrate/fc_env/bin/
## also tried prepending PYTHONPATH -- got same error
## PYTHONPATH=~/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/:$PYTHONPATH

fc_run.py fc_run.cfg

batch stdout+stderr:

## SLURM PROLOG ###############################################################
##    Job ID : 12713021
##  Job Name : falcon-ecoli
##  Nodelist : node408
##      CPUs : 
##  Mem/Node : 10240 MB
## Directory : /gpfs/scratch/jurban/male-ilmn/lambda/falcon/ecoli
##   Started : Tue May 31 15:22:21 EDT 2016
###############################################################################
[INFO]fc_run started with configuration fc_run.cfg
[WARNING]HPC.daligner option "-dal" has changed to "-B". Correcting this for you.
[WARNING]HPC.daligner option "-dal" has changed to "-B". Correcting this for you.
[INFO] No target specified, assuming "assembly" as target 
/users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.5+git.1a10da1726868e2e0db05a401fdcf5f27952f170-py2.7-linux-x86_64.egg/falcon_kit/run_support.py:317: UserWarning: Unexpected keys in input config: set(['jobqueue'])
  warnings.warn("Unexpected keys in input config: %s" %repr(unused))
[INFO]# of tasks in complete graph: 1
[INFO]tick: 1, #updatedTasks: 0, sleep_time=0.000000
[INFO]tick: 2, #updatedTasks: 0, sleep_time=0.100000
[INFO]Running task from function task_make_fofn_abs_raw()
[WARNING]Missing taskObj.generated_script_fn for task. Maybe we did not need it? Skipping and continuing.
[INFO]Queued 'task:///users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.5+git.1a10da1726868e2e0db05a401fdcf5f27952f170-py2.7-linux-x86_64.egg/falcon_kit/mains/run1.py/task_make_fofn_abs_raw' ...
[INFO]Success ('done'). Joining 'task:///users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.5+git.1a10da1726868e2e0db05a401fdcf5f27952f170-py2.7-linux-x86_64.egg/falcon_kit/mains/run1.py/task_make_fofn_abs_raw'...
[INFO]_refreshTargets() finished with no thread running and no new job to submit
[INFO]# of tasks in complete graph: 2
[INFO]tick: 1, #updatedTasks: 0, sleep_time=0.000000
[INFO]tick: 2, #updatedTasks: 0, sleep_time=0.100000
[INFO]Running task from function task_build_rdb()
[INFO]script_fn:'/gpfs/scratch/jurban/male-ilmn/lambda/falcon/ecoli/0-rawreads/prepare_rdb.sh'
[INFO]jobid=Jc331e85c07f54b643d3454363a0b2e132596bf1ad3f1aec6555f7914eff891b4
[INFO]starting job Job(jobid='Jc331e85c07f54b643d3454363a0b2e132596bf1ad3f1aec6555f7914eff891b4', cmd='/bin/bash prepare_rdb.sh', rundir='/gpfs/scratch/jurban/male-ilmn/lambda/falcon/ecoli/0-rawreads', options={'sge_option': '--ntasks 1 --nodes 1 --cpus-per-task 8 --mem 10g --time 00:20:00', 'job_type': None})
[CRITICAL]Any exception caught in RefreshTargets() indicates an unrecoverable error. Shutting down...
/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py:537: UserWarning: 
            "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
            "! Please wait for all threads / processes to terminate !"
            "! Also, maybe use 'ps' or 'qstat' to check all threads,!"
            "! processes and/or jobs are terminated cleanly.        !"
            "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"

  warnings.warn(shutdown_msg)
[WARNING]#tasks=1, #alive=1
[WARNING]Now, #tasks=1, #alive=0
Traceback (most recent call last):
  File "/users/jurban/software/falcon/FALCON-integrate/fc_env/bin/fc_run.py", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON/src/py_scripts/fc_run.py", line 5, in <module>
    main(sys.argv)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.5+git.1a10da1726868e2e0db05a401fdcf5f27952f170-py2.7-linux-x86_64.egg/falcon_kit/mains/run1.py", line 505, in main
    main1(argv[0], args.config, args.logger)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.5+git.1a10da1726868e2e0db05a401fdcf5f27952f170-py2.7-linux-x86_64.egg/falcon_kit/mains/run1.py", line 344, in main1
    wf.refreshTargets([rdb_build_done])
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 548, in refreshTargets
    raise Exception('Caused by:\n' + tb)
Exception: Caused by:
Traceback (most recent call last):
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 523, in refreshTargets
    rtn = self._refreshTargets(task2thread, objs = objs, callback = callback, updateFreq = updateFreq, exitOnFailure = exitOnFailure)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 657, in _refreshTargets
    numAliveThreads = self.thread_handler.alive(task2thread.values())
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/pwatcher_bridge.py", line 172, in alive
    result = watcher.run(**watcher_args)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pwatcher/fs_based.py", line 630, in run
    return cmd_run(self.state, jobids, job_type)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pwatcher/fs_based.py", line 445, in cmd_run
    bjob = MetaJobSlurm(mjob)
TypeError: __init__() takes exactly 3 arguments (2 given)
JohnUrban commented 8 years ago

Woops. I did not mean to close the issue. Re-opened.

JohnUrban commented 8 years ago

I also tried adding "-U" to the pip install commands in the FALCON-make makefile. This also works to get through the make test. Moreover, it installs all of those packages that it usually says "Requirement already satisfied" to in FALCON-integrate/fc_env/lib/python2.7/site-packages/ … so it results in the full virtual environment I think you were going for.

#!/bin/bash

module load git

git clone git://github.com/PacificBiosciences/FALCON-integrate.git
cd FALCON-integrate

make init

awk '{gsub(/pip\ install/, "pip install -U --force-reinstall"); print}'  FALCON-make/makefile > tmp-makefile
mv tmp-makefile FALCON-make/makefile

make virtualenv
make check
make -j install
make test  # to run a simple one

However, the ecoli batch script is still broken … and it is still looking/searching in that other location….

## SLURM PROLOG ###############################################################
##    Job ID : 12713160
##  Job Name : falcon-ecoli
##  Nodelist : node408
##      CPUs : 
##  Mem/Node : 10240 MB
## Directory : /gpfs/scratch/jurban/male-ilmn/lambda/falcon/ecoli
##   Started : Tue May 31 15:39:37 EDT 2016
###############################################################################
[INFO]fc_run started with configuration fc_run.cfg
[WARNING]HPC.daligner option "-dal" has changed to "-B". Correcting this for you.
[WARNING]HPC.daligner option "-dal" has changed to "-B". Correcting this for you.
[INFO] No target specified, assuming "assembly" as target 
/users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.5+git.1a10da1726868e2e0db05a401fdcf5f27952f170-py2.7-linux-x86_64.egg/falcon_kit/run_support.py:317: UserWarning: Unexpected keys in input config: set(['jobqueue'])
  warnings.warn("Unexpected keys in input config: %s" %repr(unused))
[INFO]# of tasks in complete graph: 1
[INFO]tick: 1, #updatedTasks: 0, sleep_time=0.000000
[INFO]tick: 2, #updatedTasks: 0, sleep_time=0.100000
[INFO]Running task from function task_make_fofn_abs_raw()
[WARNING]Missing taskObj.generated_script_fn for task. Maybe we did not need it? Skipping and continuing.
[INFO]Queued 'task:///users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.5+git.1a10da1726868e2e0db05a401fdcf5f27952f170-py2.7-linux-x86_64.egg/falcon_kit/mains/run1.py/task_make_fofn_abs_raw' ...
[INFO]Success ('done'). Joining 'task:///users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.5+git.1a10da1726868e2e0db05a401fdcf5f27952f170-py2.7-linux-x86_64.egg/falcon_kit/mains/run1.py/task_make_fofn_abs_raw'...
[INFO]_refreshTargets() finished with no thread running and no new job to submit
[INFO]# of tasks in complete graph: 2
[INFO]tick: 1, #updatedTasks: 0, sleep_time=0.000000
[INFO]tick: 2, #updatedTasks: 0, sleep_time=0.100000
[INFO]Running task from function task_build_rdb()
[INFO]script_fn:'/gpfs/scratch/jurban/male-ilmn/lambda/falcon/ecoli/0-rawreads/prepare_rdb.sh'
[INFO]jobid=Jc331e85c07f54b643d3454363a0b2e132596bf1ad3f1aec6555f7914eff891b4
[INFO]starting job Job(jobid='Jc331e85c07f54b643d3454363a0b2e132596bf1ad3f1aec6555f7914eff891b4', cmd='/bin/bash prepare_rdb.sh', rundir='/gpfs/scratch/jurban/male-ilmn/lambda/falcon/ecoli/0-rawreads', options={'sge_option': '--ntasks 1 --nodes 1 --cpus-per-task 8 --mem 10g --time 00:20:00', 'job_type': None})
[CRITICAL]Any exception caught in RefreshTargets() indicates an unrecoverable error. Shutting down...
/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py:537: UserWarning: 
            "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
            "! Please wait for all threads / processes to terminate !"
            "! Also, maybe use 'ps' or 'qstat' to check all threads,!"
            "! processes and/or jobs are terminated cleanly.        !"
            "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"

  warnings.warn(shutdown_msg)
[WARNING]#tasks=1, #alive=1
[WARNING]Now, #tasks=1, #alive=0
Traceback (most recent call last):
  File "/users/jurban/software/falcon/FALCON-integrate/fc_env/bin/fc_run.py", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON/src/py_scripts/fc_run.py", line 5, in <module>
    main(sys.argv)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.5+git.1a10da1726868e2e0db05a401fdcf5f27952f170-py2.7-linux-x86_64.egg/falcon_kit/mains/run1.py", line 505, in main
    main1(argv[0], args.config, args.logger)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.5+git.1a10da1726868e2e0db05a401fdcf5f27952f170-py2.7-linux-x86_64.egg/falcon_kit/mains/run1.py", line 344, in main1
    wf.refreshTargets([rdb_build_done])
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 548, in refreshTargets
    raise Exception('Caused by:\n' + tb)
Exception: Caused by:
Traceback (most recent call last):
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 523, in refreshTargets
    rtn = self._refreshTargets(task2thread, objs = objs, callback = callback, updateFreq = updateFreq, exitOnFailure = exitOnFailure)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 657, in _refreshTargets
    numAliveThreads = self.thread_handler.alive(task2thread.values())
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/pwatcher_bridge.py", line 172, in alive
    result = watcher.run(**watcher_args)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pwatcher/fs_based.py", line 630, in run
    return cmd_run(self.state, jobids, job_type)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pwatcher/fs_based.py", line 445, in cmd_run
    bjob = MetaJobSlurm(mjob)
TypeError: __init__() takes exactly 3 arguments (2 given)
pb-cdunn commented 8 years ago

I wouldn't expect the -U to be needed. And I'm afraid it may undo our pip install -e, which is critical. We want those packages to come from their Git repos.

I'm not sure of the solution, but you don't need to run the entire workflow to test. That is providing too much info. This is all you need:

python -c 'import pypeflow.common; print pypeflow.common'

We want that to come from the virtualenv, not from the old installation.

pb-cdunn commented 8 years ago

I do see, however, that it still does not install many of the packages it finds elsewhere

That's fine. The old versions will work.

Though the make test finished fine, when I tried running the ecoli batch script (pasted again below), I got the error pasted down below.

That's troubling. Possibly it's the way you run your batch script. I think your current environment is not being used. Or maybe you're losing your environment somewhere else. Try setting job_type=local in your FALCON config (typically fc_run.cfg). Somehow, just before fc_run.py starts, try printenv|sort. I suspect that your PATH or PYTHONPATH is wrong at that point.

JohnUrban commented 8 years ago

Thanks for alerting me about "-U" -- I took it out.

I could be wrong, but it seems like the install of FALCON-integrate is unable to use its own versions of pypeflow and falcon-kit.

I was able to restore the ability to get 0.4.0 to work again on my system by restoring "~/software/localpy/lib/python2.7/site-packages/" to what it was yesterday before I tampered with it earlier when I said I shot myself in the foot. Then it stopped throwing errors about the pypeflow and falcon-kit located there. However, now I am unable to get the later versions to work -- even though before we got them working (at least for the make test) by adding --force-reinstall to the pip commands in the FALCON-make/makefile. That solution does not work any more -- this is the error message from 'make test':

Traceback (most recent call last):
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/bin/fc_run.py", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON/src/py_scripts/fc_run.py", line 1, in <module>
    from falcon_kit.mains.run1 import main
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON/falcon_kit/mains/run1.py", line 5, in <module>
    from pypeflow.pwatcher_bridge import PypeProcWatcherWorkflow, MyFakePypeThreadTaskBase
ImportError: No module named pwatcher_bridge

So I guess I am assuming for the moment that that solution with the later versions (0.5.0+) must have been working with the updated versions of pypeflow and falcon-kit in that other directory (and now 0.4.0 is working with the older versions there).

Perhaps supporting the assumption that these falcon installs are depending on the contents in "~/software/localpy/lib/python2.7/site-packages/" - it looks like pypeflow is being found there whether the virtualenv is activated or not:

$ python -c 'import pypeflow.common; print pypeflow.common'
<module 'pypeflow.common' from '/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/common.pyc'>

$ . fc_env/bin/activate
(fc_env) $ python -c 'import pypeflow.common; print pypeflow.common'
<module 'pypeflow.common' from '/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/common.pyc'>

(fc_env) $ PYTHONPATH=~/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/:$PYTHONPATH
(fc_env) $ python -c 'import pypeflow.common; print pypeflow.common'
<module 'pypeflow.common' from '/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/common.pyc'>
JohnUrban commented 8 years ago

I was able to get the install to finish the make test successfully by using the following protocol:

#!/bin/bash

module load git

git clone git://github.com/PacificBiosciences/FALCON-integrate.git
cd FALCON-integrate

make init

awk '{gsub(/pip\ install/, "pip install -U --force-reinstall"); print}'  FALCON-make/makefile > tmp-makefile
mv tmp-makefile FALCON-make/makefile

make virtualenv
make check
make -j install

python -c 'import pypeflow.common; print pypeflow.common' > install.err
python -c 'import falcon_kit; print falcon_kit.falcon' >> install.err
. fc_env/bin/activate
cd pypeFLOW
python setup.py install
cd ..
python -c 'import pypeflow.common; print pypeflow.common' >> install.err
cd FALCON
python setup.py install
cd ..
python -c 'import falcon_kit; print falcon_kit.falcon' >> install.err

make test  # to run a simple one

contents of install.err:

<module 'pypeflow.common' from '/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/common.pyc'>
<CDLL '/gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON/ext_falcon.so', handle 1eab980 at 7f1c0aeac210>
<module 'pypeflow.common' from '/users/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/common.pyc'>
<CDLL '/users/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/falcon_kit-0.5+git.11a9608751217fe13078fd409b7ca03d637a219d-py2.7-linux-x86_64.egg/ext_falcon.so', handle 1416710 at 7f6d1512ab90>

Unfortunately, I have not yet been able to get the ecoli batch script to run with job_type=SLURM….

Traceback (most recent call last):
  File "/users/jurban/software/falcon/FALCON-integrate/fc_env/bin/fc_run.py", line 4, in <module>
    __import__('pkg_resources').run_script('falcon-kit==0.5+git.11a9608751217fe13078fd409b7ca03d637a219d', 'fc_run.py')
  File "/users/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 719, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/users/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1504, in run_script
    exec(code, namespace, namespace)
  File "/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/falcon_kit-0.5+git.11a9608751217fe13078fd409b7ca03d637a219d-py2.7-linux-x86_64.egg/EGG-INFO/scripts/fc_run.py", line 5, in <module>
    main(sys.argv)
  File "/users/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/falcon_kit-0.5+git.11a9608751217fe13078fd409b7ca03d637a219d-py2.7-linux-x86_64.egg/falcon_kit/mains/run1.py", line 542, in main
    main1(argv[0], args.config, args.logger)
  File "/users/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/falcon_kit-0.5+git.11a9608751217fe13078fd409b7ca03d637a219d-py2.7-linux-x86_64.egg/falcon_kit/mains/run1.py", line 367, in main1
    wf.refreshTargets([rdb_build_done])
  File "/users/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 548, in refreshTargets
    raise Exception('Caused by:\n' + tb)
Exception: Caused by:
Traceback (most recent call last):
  File "/users/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 523, in refreshTargets
    rtn = self._refreshTargets(task2thread, objs = objs, callback = callback, updateFreq = updateFreq, exitOnFailure = exitOnFailure)
  File "/users/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 657, in _refreshTargets
    numAliveThreads = self.thread_handler.alive(task2thread.values())
  File "/users/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/pwatcher_bridge.py", line 172, in alive
    result = watcher.run(**watcher_args)
  File "/users/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pwatcher/fs_based.py", line 630, in run
    return cmd_run(self.state, jobids, job_type)
  File "/users/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pwatcher/fs_based.py", line 445, in cmd_run
    bjob = MetaJobSlurm(mjob)
TypeError: __init__() takes exactly 3 arguments (2 given)

though it seems to be running on job_type=local right now…. I will report back on whether that worked.

If job_type=local works, but not job_type=SLURM, what do you think the next step is?

JohnUrban commented 8 years ago

p.s. the notable additions to the new install protocol are the "python setup.py install" steps in the pypeFLOW and FALCON directories when the virtualenv is activated.

pb-cdunn commented 8 years ago

I've gotten lost. Fundamentally, I understand that the problem is that you have pypeflow in two locations, and for some reason it gets picked up from the wrong one. However, if you run pip install -e . in the pypeFLOW repository directory, that is the version that will be referenced from the virtualenv Python. I do not understand how you ever pick up the version from the original Python installation. Maybe something strange about virtualenvs?

If I were on your machine, I could track this down. As it is, until I manage to repro, I can only suggest that you remove the random modules which you have installed into your system Python. I do not want to go back to python setup.py, as there are significant advantages to pip install -e ..

The MetaJobSlurm command is easy to fix. I'll push that out right away, but you can change it yourself locally:

diff --git pwatcher/fs_based.py pwatcher/fs_based.py
index e4cd706..5f9c7a9 100755
--- pwatcher/fs_based.py
+++ pwatcher/fs_based.py
@@ -365,7 +365,7 @@ class MetaJobSlurm(object):
         return self.mjob.job.jobid[:15]
     def __repr__(self):
         return 'MetaJobSlurm(%s)' %repr(self.mjob)
-    def __init__(self, mjob, sge_option):
+    def __init__(self, mjob):
JohnUrban commented 8 years ago

Thanks for all your help - and apologies that I lost you. You are right -- it seems like it is basically a problem where "pip install -e" is not installing stuff if that stuff can be found elsewhere. I have no idea how it is picking up other stuff when in the virtualenv .. but it is. Annoyingly, I have it installed locally (where it is being picked up), but the system admins also have older falcon stuff installed globally for all users that I cannot erase… So I am guessing stuff outside the virtualenv may be a persistent problem…

As for job_type=local with the ecoli batch script, that is still running. It required a lot more memory … and it seems like it will take a lot longer, but it seems like it will finish.

Thanks for the info on MetaJobSlurm.

JohnUrban commented 8 years ago

Alright - as far as detecting other locations during installation goes, there are 2 fixes for you to consider.

The first is modifying FALCON-make/makefile to include this line at the top: export PYTHONPATH=""

$ head FALCON-make/makefile
export PYTHONPATH=""
FALCON_WORKSPACE?=..
FALCON_PREFIX?=../fc_env
VDIR:=${FALCON_PREFIX}
export CC=gcc
export CXX=g++

install: install-DAZZ_DB install-DALIGNER install-pypeFLOW install-FALCON
install-DAZZ_DB:
    ${MAKE} -C ${FALCON_WORKSPACE}/DAZZ_DB

The alternative -- and probably better long-term/overall solution is to modify fc_env/bin/activate such that the virtualenv never inherits PYTHONPATH from the regular environment. This is done by adding "unset PYTHONPATH" to the deactivate() function inside activate.

$ head fc_env/bin/activate
# This file must be used with "source bin/activate" *from bash*
# you cannot run it directly

deactivate () {
    unset -f pydoc >/dev/null 2>&1
    unset PYTHONPATH
    # reset old environment variables
    # ! [ -z ${VAR+_} ] returns true if VAR is declared at all
    if ! [ -z "${_OLD_VIRTUAL_PATH+_}" ] ; then
        PATH="$_OLD_VIRTUAL_PATH"

To take control of what is in the PYTHONPATH when the virtualenv is activated, you can also add PYTHONPATH="$VIRTUAL_ENV/lib/python2.7/site-packages" outside of the deactivate function after VIRTUAL_ENV is defined:

VIRTUAL_ENV="/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env"
export VIRTUAL_ENV

PYTHONPATH="$VIRTUAL_ENV/lib/python2.7/site-packages"

This way, before activating the virtual env, I see:

$ which python
/gpfs/runtime/opt/python/2.7.3/bin/python

$ python -c 'import pypeflow.common; print pypeflow.common'
<module 'pypeflow.common' from '/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/common.pyc'>

$ python -c 'import falcon_kit; print falcon_kit.falcon'
<CDLL '/users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/ext_falcon.so', handle 7d4a70 at 7fd7a7426390>

$ echo $PYTHONPATH
/users/jurban/software/macs2/lib/python2.7/site-packages:/users/jurban/software/localpy/lib/python2.7/site-packages::/users/jurban/software/pbsuite/PBSuite_15.8.24/:/users/jurban/software/ConsensusCore/build/Python/:/users/jurban/software/falcon/FALCON-integrate/bin/:/users/jurban/software/falcon/FALCON-integrate/FALCON/build/lib.linux-x86_64-2.7/falcon_kit/:/users/jurban/software/falcon/FALCON-integrate/pypeFLOW/build/lib/pypeflow/:/users/jurban/software/maligner/maligner/build/lib

…and after activating virtualenv, I see:

$ which python
/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/bin/python

$ python -c 'import pypeflow.common; print pypeflow.common'
<module 'pypeflow.common' from '/gpfs_home/jurban/software/falcon/FALCON-integrate/pypeFLOW/pypeflow/common.pyc'>

$ python -c 'import falcon_kit; print falcon_kit.falcon'
<CDLL '/gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON/ext_falcon.so', handle 1ed6950 at 7f4e7fa8bc50>

$ echo $PYTHONPATH
/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/lib/python2.7/site-packages

Both of those solutions allow the "make test" to work.

When running the batch script for ecoli (with job_type = SLURM), the first solution falls flat… I get this error:

Traceback (most recent call last):
  File "/gpfs_home/jurban/software/falcon2/FALCON-integrate/fc_env/bin/fc_run.py", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/gpfs_home/jurban/software/falcon2/FALCON-integrate/FALCON/src/py_scripts/fc_run.py", line 1, in <module>
    from falcon_kit.mains.run1 import main
  File "/gpfs_home/jurban/software/falcon2/FALCON-integrate/FALCON/falcon_kit/mains/run1.py", line 5, in <module>
    from pypeflow.pwatcher_bridge import PypeProcWatcherWorkflow, MyFakePypeThreadTaskBase
ImportError: No module named pwatcher_bridge

When using the second solution (with both lines added to activate), I do not get an error. It seems to run .. but it never launches the sh scripts in 0-rawreads: "prepare_rdb.sh prepare_rdb.sub.sh" Instead the stdout/stderr just keeps giving the [INFO]tick/sleep msg:

[INFO]fc_run started with configuration fc_run.cfg
[WARNING]HPC.daligner option "-dal" has changed to "-B". Correcting this for you.
[WARNING]HPC.daligner option "-dal" has changed to "-B". Correcting this for you.
[INFO] No target specified, assuming "assembly" as target 
/gpfs_home/jurban/software/falcon2/FALCON-integrate/FALCON/falcon_kit/run_support.py:317: UserWarning: Unexpected keys in input config: set(['jobqueue'])
  warnings.warn("Unexpected keys in input config: %s" %repr(unused))
[INFO]# of tasks in complete graph: 1
[INFO]tick: 1, #updatedTasks: 0, sleep_time=0.000000
[INFO]tick: 2, #updatedTasks: 0, sleep_time=0.100000
[INFO]Running task from function task_make_fofn_abs_raw()
[WARNING]Missing taskObj.generated_script_fn for task. Maybe we did not need it? Skipping and continuing.
[INFO]Queued 'task:///gpfs_home/jurban/software/falcon2/FALCON-integrate/FALCON/falcon_kit/mains/run1.py/task_make_fofn_abs_raw' ...
[INFO]Success ('done'). Joining 'task:///gpfs_home/jurban/software/falcon2/FALCON-integrate/FALCON/falcon_kit/mains/run1.py/task_make_fofn_abs_raw'...
[INFO]_refreshTargets() finished with no thread running and no new job to submit
[INFO]# of tasks in complete graph: 2
[INFO]tick: 1, #updatedTasks: 0, sleep_time=0.000000
[INFO]tick: 2, #updatedTasks: 0, sleep_time=0.100000
[INFO]Running task from function task_build_rdb()
[INFO]script_fn:'/gpfs/scratch/jurban/male-ilmn/lambda/falcon/ecoli/0-rawreads/prepare_rdb.sh'
[INFO]jobid=Jc331e85c07f54b643d3454363a0b2e132596bf1ad3f1aec6555f7914eff891b4
[INFO]starting job Job(jobid='Jc331e85c07f54b643d3454363a0b2e132596bf1ad3f1aec6555f7914eff891b4', cmd='/bin/bash prepare_rdb.sh', rundir='/gpfs/scratch/jurban/male-ilmn/lambda/falcon/ecoli/0-rawreads', options={'sge_option': '--ntasks 1 --nodes 1 --cpus-per-task 8 --mem 10g --time 00:20:00', 'job_type': None})
[INFO]!sbatch -J Jc331e85c07f54b --ntasks 1 --nodes 1 --cpus-per-task 8 --mem 10g --time 00:20:00 -V -D /gpfs/scratch/jurban/male-ilmn/lambda/falcon/ecoli/mypwatcher/jobs/Jc331e85c07f54b643d3454363a0b2e132596bf1ad3f1aec6555f7914eff891b4 -o stdout -e stderr -S /bin/bash /gpfs/scratch/jurban/male-ilmn/lambda/falcon/ecoli/mypwatcher/wrappers/run-Jc331e85c07f54b643d3454363a0b2e132596bf1ad3f1aec6555f7914eff891b4.bash
slurm 15.08.9
[INFO]Submitted backgroundjob=MetaJobSlurm(MetaJob(job=Job(jobid='Jc331e85c07f54b643d3454363a0b2e132596bf1ad3f1aec6555f7914eff891b4', cmd='/bin/bash prepare_rdb.sh', rundir='/gpfs/scratch/jurban/male-ilmn/lambda/falcon/ecoli/0-rawreads', options={'sge_option': '--ntasks 1 --nodes 1 --cpus-per-task 8 --mem 10g --time 00:20:00', 'job_type': None}), lang_exe='/bin/bash'))
[INFO]Queued 'task:///gpfs_home/jurban/software/falcon2/FALCON-integrate/FALCON/falcon_kit/mains/run1.py/task_build_rdb' ...
[INFO]tick: 4, #updatedTasks: 1, sleep_time=0.100000
[INFO]tick: 8, #updatedTasks: 1, sleep_time=0.500000
[INFO]tick: 16, #updatedTasks: 1, sleep_time=1.000000
[INFO]tick: 32, #updatedTasks: 1, sleep_time=1.000000
[INFO]tick: 64, #updatedTasks: 1, sleep_time=1.000000
[INFO]tick: 128, #updatedTasks: 1, sleep_time=1.000000
[INFO]tick: 256, #updatedTasks: 1, sleep_time=1.000000
[INFO]tick: 512, #updatedTasks: 1, sleep_time=1.000000
[INFO]tick: 1024, #updatedTasks: 1, sleep_time=1.000000

I tried job_type=local with the 2nd solution as well. That works -- but that was working before too.

pb-cdunn commented 8 years ago

I tried job_type=local with the 2nd solution as well. That works -- but that was working before too.

Wait a minute! I thought I understand what you were saying ... until I got to this line. If it was working with job_type=local, then the only problem is how to pass your environment into the sbatch job. That is a COMPLETELY different issue than anything else you've discussed here. It's very important to keep this separate.

Let's first discuss how to get job_type=local working reliably. Then, and only then, let's discuss what changes we need to the sbatch command-line in order to pass along the environment properly.

JohnUrban commented 8 years ago

Sounds good. Are there datasets smaller than the ecoli dataset? Ideally something that can finish in <1 hour…. It seems like job_type=local works with my installation protocol above (it just has not finished b/c I have not managed to allot enough time or memory yet) -- but I can go through the other ways I installed it to see if/when job_type=local does not work. Or could we assume it does not work when the "make test" does not work…?

pb-cdunn commented 8 years ago

This takes < 1 minute:

If you have trouble running git-sym, I can show you some other way to obtain the fasta. (Our hosting solution is evolving.)

JohnUrban commented 8 years ago

I will try to be detailed here -- but hopefully not confusing. Using the synth0 data and job_type=local for batch scripts (instead of ecoli), I will walk through the installation protocols I tried yesterday.

The config file will look something like this for all unless stated otherwise (basically that found in FALCON-integrate/FALCON-examples/run/synth0/fc_run.cfg):

[General]
use_tmpdir = true
#job_type = sge
job_type = local
#stop_all_jobs_on_failure = true

# list of files of the initial bas.h5 files
input_fofn = input.fofn
#input_fofn = preads.fofn

input_type = raw
#input_type = preads

# The length cutoff used for seed reads used for initial mapping
length_cutoff = 1
genome_size = 5000
seed_coverage = 20

# The length cutoff used for seed reads usef for pre-assembly
length_cutoff_pr = 1

jobqueue = production

sge_option_da = --ntasks 1 --nodes 1 --cpus-per-task 8 --mem 10g --time 00:20:00 
sge_option_la = --ntasks 1 --nodes 1 --cpus-per-task 2 --mem 10g --time 00:05:00 
sge_option_pda = --ntasks 1 --nodes 1 --cpus-per-task 8 --mem 15g --time 00:20:00 
sge_option_pla = --ntasks 1 --nodes 1 --cpus-per-task 2 --mem 10g --time 00:05:00 
sge_option_fc = --ntasks 1 --nodes 1 --cpus-per-task 16 --mem 10g --time 00:20:00 
sge_option_cns = --ntasks 1 --nodes 1 --cpus-per-task 8 --mem 10g --time 00:20:00 

pa_concurrent_jobs = 4
cns_concurrent_jobs = 4
ovlp_concurrent_jobs = 4

pa_HPCdaligner_option =   -v -B4 -t50 -h1 -e.99 -w1 -l1 -s1000
ovlp_HPCdaligner_option = -v -B4 -t50 -h1 -e.99 -l1 -s1000

#pa_DBsplit_option =   -a -x5 -s.00065536
pa_DBsplit_option =   -a -x5 -s.065536
#pa_DBsplit_option =   -a -x5 -s1
ovlp_DBsplit_option = -a -x5 -s50

falcon_sense_option = --output_multi --min_idt 0.70 --min_cov 1 --max_n_read 20000 --n_core 0
#--min_cov_aln 1 --min_len_aln 40

overlap_filtering_setting = --max_diff 10000 --max_cov 100000 --min_cov 1 --min_len 1 --bestn 1000 --n_core 0
#dazcon = 1

To run, the batch scripts looked like the following for all unless noted otherwise:

#!/bin/bash
#SBATCH -J falcon-synth0-1
#SBATCH -c 4
#SBATCH --qos=biomed-sb-condo
#SBATCH --mem=4g
#SBATCH --time=00:30:00

###########################
. ~/software/falcon/FALCON-integrate/fc_env/bin/activate
fc_run.py fc_run.cfg

[1] The first install protocol was:

#!/bin/bash
module load git
git clone git://github.com/PacificBiosciences/FALCON-integrate.git
cd FALCON-integrate
git checkout 0.4.0
make init
make virtualenv
make check
make -j install
make test  # to run a simple one

This protocol results in detecting and using pypeflow/falcon-kit found elsewhere on my system that I installed circa Oct 2015 -- and seems to be 0.4.0 compatible. The "make test" completed successfully, but the batch script threw an error.

batch script error:

+ touch /gpfs/scratch/jurban/male-ilmn/lambda/falcon/synth0/prevmeth01/0-rawreads/rdb_build_done.exit
Traceback (most recent call last):
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py", line 67, in _run_script_local
    system(cmd, check=True)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py", line 24, in system
    raise Exception(msg)
Exception: Call 'bash /gpfs/scratch/jurban/male-ilmn/lambda/falcon/synth0/prevmeth01/0-rawreads/prepare_rdb.sh 1> /gpfs/scratch/jurban/male-ilmn/lambda/falcon/synth0/prevmeth01/0-rawreads/prepare_rdb.sh.log 2>&1' returned 256.
[ERROR]PypeTaskBase failed:
{'__class__.__name__': 'PypeThreadTaskBase',
 '_status': 'TaskInitialized',
 'inputDataObjs': {'input_fofn': PypeLocalFile('file://localhost/gpfs/scratch/jurban/male-ilmn/lambda/falcon/synth0/prevmeth01/0-rawreads/input.fofn', '/gpfs/scratch/jurban/male-ilmn/lambda/falcon/synth0/prevmeth01/0-rawreads/input.fofn')},
 'mutableDataObjs': {},
 'outputDataObjs': {'rdb_build_done': PypeLocalFile('file://localhost/gpfs/scratch/jurban/male-ilmn/lambda/falcon/synth0/prevmeth01/0-rawreads/rdb_build_done', '/gpfs/scratch/jurban/male-ilmn/lambda/falcon/synth0/prevmeth01/0-rawreads/rdb_build_done'),
                    'run_jobs': PypeLocalFile('file://localhost/gpfs/scratch/jurban/male-ilmn/lambda/falcon/synth0/prevmeth01/0-rawreads/run_jobs.sh', '/gpfs/scratch/jurban/male-ilmn/lambda/falcon/synth0/prevmeth01/0-rawreads/run_jobs.sh')},
 'parameters': {'config': {'cns_concurrent_jobs': 4,
                           'falcon_sense_option': '--output_multi --min_idt 0.70 --min_cov 1 --max_n_read 20000 --n_core 0',
                           'falcon_sense_skip_contained': 'False',
                           'input_fofn_fn': 'input.fofn',
                           'input_type': 'raw',
                           'install_prefix': '/gpfs_home/jurban/software/falcon2/FALCON-integrate/fc_env',
                           'job_type': 'local',
                           'length_cutoff': 1,
                           'length_cutoff_pr': 1,
                           'openending': False,
                           'overlap_filtering_setting': '--max_diff 10000 --max_cov 100000 --min_cov 1 --min_len 1 --bestn 1000 --n_core 0',
                           'ovlp_DBsplit_option': '-a -x5 -s50',
                           'ovlp_HPCdaligner_option': '-v -B4 -t50 -h1 -e.99 -l1 -s1000',
                           'ovlp_concurrent_jobs': 4,
                           'pa_DBsplit_option': '-a -x5 -s.065536',
                           'pa_HPCdaligner_option': '-v -B4 -t50 -h1 -e.99 -w1 -l1 -s1000',
                           'pa_concurrent_jobs': 4,
                           'sge_option_cns': '--ntasks 1 --nodes 1 --cpus-per-task 8 --mem 10g --time 00:20:00',
                           'sge_option_da': '--ntasks 1 --nodes 1 --cpus-per-task 8 --mem 10g --time 00:20:00',
                           'sge_option_fc': '--ntasks 1 --nodes 1 --cpus-per-task 16 --mem 10g --time 00:20:00',
                           'sge_option_la': '--ntasks 1 --nodes 1 --cpus-per-task 2 --mem 10g --time 00:05:00',
                           'sge_option_pda': '--ntasks 1 --nodes 1 --cpus-per-task 8 --mem 15g --time 00:20:00',
                           'sge_option_pla': '--ntasks 1 --nodes 1 --cpus-per-task 2 --mem 10g --time 00:05:00',
                           'target': 'assembly',
                           'use_tmpdir': True},
                'work_dir': '/gpfs/scratch/jurban/male-ilmn/lambda/falcon/synth0/prevmeth01/0-rawreads'}}
Traceback (most recent call last):
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/task.py", line 330, in __call__
    return self.runInThisThread(*argv, **kwargv)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/task.py", line 363, in runInThisThread
    self.run(*argv, **kwargv)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/task.py", line 263, in run
    rtn = self._runTask(self, *argv, **kwargv)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/task.py", line 178, in _runTask
    return self._taskFun(self)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py", line 157, in task_build_rdb
    run_script(job_data, job_type = config["job_type"])
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py", line 97, in run_script
    rc = _run_script(job_data)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py", line 67, in _run_script_local
    system(cmd, check=True)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py", line 24, in system
    raise Exception(msg)
Exception: Call 'bash /gpfs/scratch/jurban/male-ilmn/lambda/falcon/synth0/prevmeth01/0-rawreads/prepare_rdb.sh 1> /gpfs/scratch/jurban/male-ilmn/lambda/falcon/synth0/prevmeth01/0-rawreads/prepare_rdb.sh.log 2>&1' returned 256.
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/gpfs/runtime/opt/python/2.7.3/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/gpfs/runtime/opt/python/2.7.3/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/task.py", line 330, in __call__
    return self.runInThisThread(*argv, **kwargv)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/task.py", line 363, in runInThisThread
    self.run(*argv, **kwargv)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/task.py", line 263, in run
    rtn = self._runTask(self, *argv, **kwargv)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/task.py", line 178, in _runTask
    return self._taskFun(self)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py", line 157, in task_build_rdb
    run_script(job_data, job_type = config["job_type"])
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py", line 97, in run_script
    rc = _run_script(job_data)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py", line 67, in _run_script_local
    system(cmd, check=True)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py", line 24, in system
    raise Exception(msg)
Exception: Call 'bash /gpfs/scratch/jurban/male-ilmn/lambda/falcon/synth0/prevmeth01/0-rawreads/prepare_rdb.sh 1> /gpfs/scratch/jurban/male-ilmn/lambda/falcon/synth0/prevmeth01/0-rawreads/prepare_rdb.sh.log 2>&1' returned 256.

[INFO]Queued 'task:///users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py/task_build_rdb' ...
[INFO]Failure ('fail'). Joining 'task:///users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py/task_build_rdb'...
[ERROR]Any exception caught in RefreshTargets() indicates an unrecoverable error. Shutting down...
Traceback (most recent call last):
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 531, in refreshTargets
    rtn = self._refreshTargets(task2thread, objs = objs, callback = callback, updateFreq = updateFreq, exitOnFailure = exitOnFailure)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 712, in _refreshTargets
    raise TaskFailureError("Counted %d failures." %failedJobCount)
TaskFailureError: 'Counted 1 failures.'
['/gpfs_home/jurban/software/falcon2/FALCON-integrate/fc_env/bin/fc_run.py', 'fc_run.cfg']

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! Please wait for all threads / processes to terminate !
! Also, maybe use 'ps' or 'qstat' to check all threads,!
! processes and/or jobs are terminated cleanly.        !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[WARNING]#tasks=2, #alive=0
Traceback (most recent call last):
  File "/gpfs_home/jurban/software/falcon2/FALCON-integrate/fc_env/bin/fc_run.py", line 4, in <module>
    __import__('pkg_resources').run_script('falcon-kit==0.4.0', 'fc_run.py')
  File "/gpfs_home/jurban/software/falcon2/FALCON-integrate/fc_env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 729, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/gpfs_home/jurban/software/falcon2/FALCON-integrate/fc_env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1642, in run_script
    exec(code, namespace, namespace)
  File "/gpfs_home/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/EGG-INFO/scripts/fc_run.py", line 5, in <module>
    main(sys.argv)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py", line 612, in main
    main1(*argv)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py", line 473, in main1
    wf.refreshTargets([rdb_build_done]) 
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 531, in refreshTargets
    rtn = self._refreshTargets(task2thread, objs = objs, callback = callback, updateFreq = updateFreq, exitOnFailure = exitOnFailure)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 712, in _refreshTargets
    raise TaskFailureError("Counted %d failures." %failedJobCount)
pypeflow.controller.TaskFailureError: 'Counted 1 failures.'

This is moderately confusing to me because yesterday this installation method (with 0.4.0) was working for the ecoli sample… and still works for the make test.

[2] I then tried that same protocol with a later version (as I did yesterday) -- for example here, I did 0.5.2:

#!/bin/bash
module load git
git clone git://github.com/PacificBiosciences/FALCON-integrate.git
cd FALCON-integrate
git checkout 0.5.2
make init
make virtualenv
make check
make -j install
make test  # to run a simple one

This protocol results in detecting and using pypeflow/falcon-kit found elsewhere on my system that I installed circa Oct 2015 -- which seems to be incompatible with 0.5.2. The "make test" threw an error and so did the batch script. Both complained that it could not find the pyparsing module. This seems to be because pip detects it elsewhere and it is not installed in the virtualenv.

[2a] make test error

Traceback (most recent call last):
  File "/gpfs_home/jurban/software/falcon3/FALCON-integrate/fc_env/bin/fc_run.py", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/gpfs_home/jurban/software/falcon3/FALCON-integrate/FALCON/src/py_scripts/fc_run.py", line 5, in <module>
    main(sys.argv)
  File "/gpfs_home/jurban/software/falcon3/FALCON-integrate/FALCON/falcon_kit/mains/run0.py", line 644, in main
    main1(argv[0], args.config, args.logger)
  File "/gpfs_home/jurban/software/falcon3/FALCON-integrate/FALCON/falcon_kit/mains/run0.py", line 467, in main1
    wf.refreshTargets([fofn_abs_task])
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 531, in refreshTargets
    rtn = self._refreshTargets(task2thread, objs = objs, callback = callback, updateFreq = updateFreq, exitOnFailure = exitOnFailure)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 571, in _refreshTargets
    tSortedURLs = PypeGraph(rdfGraph, connectedPypeNodes).tSort( )
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 109, in __init__
    for row in self._RDFGraph.query('SELECT ?s ?o WHERE {?s pype:prereq ?o . }', initNs=dict(pype=pypeNS)):
  File "build/bdist.linux-x86_64/egg/rdflib/graph.py", line 1004, in query
  File "build/bdist.linux-x86_64/egg/rdflib/plugin.py", line 103, in get
  File "build/bdist.linux-x86_64/egg/rdflib/plugin.py", line 65, in getClass
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/rdfextras-0.4-py2.7.egg/rdfextras/sparql/processor.py", line 2, in <module>
    import rdfextras.sparql.parser
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/rdfextras-0.4-py2.7.egg/rdfextras/sparql/parser.py", line 3, in <module>
    from pyparsing import (Regex, Suppress, Combine, Optional, CaselessKeyword,
ImportError: No module named pyparsing

[2b] batch-script error:

/gpfs_home/jurban/software/falcon3/FALCON-integrate/FALCON/falcon_kit/run_support.py:317: UserWarning: Unexpected keys in input config: set(['jobqueue'])
  warnings.warn("Unexpected keys in input config: %s" %repr(unused))
[ERROR]Any exception caught in RefreshTargets() indicates an unrecoverable error. Shutting down...
Traceback (most recent call last):
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 531, in refreshTargets
    rtn = self._refreshTargets(task2thread, objs = objs, callback = callback, updateFreq = updateFreq, exitOnFailure = exitOnFailure)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 571, in _refreshTargets
    tSortedURLs = PypeGraph(rdfGraph, connectedPypeNodes).tSort( )
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 109, in __init__
    for row in self._RDFGraph.query('SELECT ?s ?o WHERE {?s pype:prereq ?o . }', initNs=dict(pype=pypeNS)):
  File "build/bdist.linux-x86_64/egg/rdflib/graph.py", line 1004, in query
    processorinst = plugin.get(processor, query.Processor)(self)
  File "build/bdist.linux-x86_64/egg/rdflib/plugin.py", line 103, in get
    return p.getClass()
  File "build/bdist.linux-x86_64/egg/rdflib/plugin.py", line 65, in getClass
    module = __import__(self.module_path, globals(), locals(), [""])
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/rdfextras-0.4-py2.7.egg/rdfextras/sparql/processor.py", line 2, in <module>
    import rdfextras.sparql.parser
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/rdfextras-0.4-py2.7.egg/rdfextras/sparql/parser.py", line 3, in <module>
    from pyparsing import (Regex, Suppress, Combine, Optional, CaselessKeyword,
ImportError: No module named pyparsing

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! Please wait for all threads / processes to terminate !
! Also, maybe use 'ps' or 'qstat' to check all threads,!
! processes and/or jobs are terminated cleanly.        !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[WARNING]#tasks=0, #alive=0
Traceback (most recent call last):
  File "/gpfs_home/jurban/software/falcon3/FALCON-integrate/fc_env/bin/fc_run.py", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/gpfs_home/jurban/software/falcon3/FALCON-integrate/FALCON/src/py_scripts/fc_run.py", line 5, in <module>
    main(sys.argv)
  File "/gpfs_home/jurban/software/falcon3/FALCON-integrate/FALCON/falcon_kit/mains/run0.py", line 644, in main
    main1(argv[0], args.config, args.logger)
  File "/gpfs_home/jurban/software/falcon3/FALCON-integrate/FALCON/falcon_kit/mains/run0.py", line 467, in main1
    wf.refreshTargets([fofn_abs_task])
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 531, in refreshTargets
    rtn = self._refreshTargets(task2thread, objs = objs, callback = callback, updateFreq = updateFreq, exitOnFailure = exitOnFailure)
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 571, in _refreshTargets
    tSortedURLs = PypeGraph(rdfGraph, connectedPypeNodes).tSort( )
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 109, in __init__
    for row in self._RDFGraph.query('SELECT ?s ?o WHERE {?s pype:prereq ?o . }', initNs=dict(pype=pypeNS)):
  File "build/bdist.linux-x86_64/egg/rdflib/graph.py", line 1004, in query
  File "build/bdist.linux-x86_64/egg/rdflib/plugin.py", line 103, in get
  File "build/bdist.linux-x86_64/egg/rdflib/plugin.py", line 65, in getClass
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/rdfextras-0.4-py2.7.egg/rdfextras/sparql/processor.py", line 2, in <module>
    import rdfextras.sparql.parser
  File "/users/jurban/software/localpy/lib/python2.7/site-packages/rdfextras-0.4-py2.7.egg/rdfextras/sparql/parser.py", line 3, in <module>
    from pyparsing import (Regex, Suppress, Combine, Optional, CaselessKeyword,
ImportError: No module named pyparsing

[3] We discussed all the --forced-reinstall stuff …

#!/bin/bash
module load git
git clone git://github.com/PacificBiosciences/FALCON-integrate.git
cd FALCON-integrate

git checkout 0.4.0  # optional step

make virtualenv
make check
awk '{gsub(/pip\ install/, "pip install --force-reinstall"); print}'  FALCON-make/makefile > tmp-makefile
mv tmp-makefile FALCON-make/makefile
make -j install
make test  # to run a simple one

The "make test" works, but the batch script doesn't with and without the line: "git checkout 0.4.0".


[4] The next method is to simply add export PYTHONPATH="" to the top of FALCON-make/makefile. This can be done with and without "--force-reinstall" from above. I will show it without since emptying the PYTHONPATH variable here causes pip to not find anything elsewhere and install all the packages anyway (i.e. --force-reinstall is redundant).

#!/bin/bash
module load git
git clone git://github.com/PacificBiosciences/FALCON-integrate.git
cd FALCON-integrate

git checkout 0.4.0 ## OPTIONAL STEP

make init
make virtualenv
make check

cp FALCON-make/makefile tmp-makefile
echo "export PYTHONPATH=\"\"" | cat - tmp-makefile > FALCON-make/makefile

make -j install
make test  # to run a simple one

With and without the optional "git checkout 0.4.0" step, the "make test" finishes successfully.

Without the "git checkout 0.4.0" step, the batch script fails with:

Traceback (most recent call last):
  File "/gpfs_home/jurban/software/falcon3/FALCON-integrate/fc_env/bin/fc_run.py", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/gpfs_home/jurban/software/falcon3/FALCON-integrate/FALCON/src/py_scripts/fc_run.py", line 1, in <module>
    from falcon_kit.mains.run1 import main
  File "/gpfs_home/jurban/software/falcon3/FALCON-integrate/FALCON/falcon_kit/mains/run1.py", line 5, in <module>
    from pypeflow.pwatcher_bridge import PypeProcWatcherWorkflow, MyFakePypeThreadTaskBase
ImportError: No module named pwatcher_bridge

When including the "git check 0.4.0" step, the batch script fails with a large error message -- that I could show upon request.


[5] The ultimate goal is to avoid inheriting the PYTHONPATH variable I have in my normal environment, both when using pip to install packages in the virtualenv and later when running the program. [4] above was able to do that during the make steps in installation… This will be a better overall solution. I modified fc_env/bin/activate to avoid inheriting the PYTHONPATH variable from my normal env. I make the following 2 changes: (i) I add to the deactivate() function "unset PYTHONPATH" (ii) I add outside of the deactivate() function after VIRTUAL_ENV is defined, PYTHONPATH="$VIRTUAL_ENV/lib/python2.7/site-packages"

The second line (ii) may not be necessary, but it does not hurt. Nonetheless, when using the first line ((i) - 'unset PYTHONPATH'), the FALCON-integrate/fc_env/lib/python2.7/site-packages/easy-install.pth file defines all the paths inside the virtualenv exactly as it should -- something not seen with the other methods:

import sys; sys.__plen = len(sys.path)
./pypeflow-0.1.1-py2.7.egg
./rdfextras-0.4-py2.7.egg
./rdflib-3.4.0-py2.7.egg
./pyparsing-1.5.7-py2.7.egg
./html5lib-1.0b8-py2.7.egg
./isodate-0.5.4-py2.7.egg
./six-1.10.0-py2.7.egg
./falcon_kit-0.4.0-py2.7-linux-x86_64.egg
/gpfs_home/jurban/software/falcon3/FALCON-integrate/FALCON/.eggs/networkx-1.11-py2.7.egg
/gpfs_home/jurban/software/falcon3/FALCON-integrate/FALCON/.eggs/decorator-4.0.9-py2.7.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)

The entire activate file looks like this:

# This file must be used with "source bin/activate" *from bash*
# you cannot run it directly

deactivate () {
    unset -f pydoc >/dev/null 2>&1
    unset PYTHONPATH
    # reset old environment variables
    # ! [ -z ${VAR+_} ] returns true if VAR is declared at all
    if ! [ -z "${_OLD_VIRTUAL_PATH+_}" ] ; then
        PATH="$_OLD_VIRTUAL_PATH"
        export PATH
        unset _OLD_VIRTUAL_PATH
    fi
    if ! [ -z "${_OLD_VIRTUAL_PYTHONHOME+_}" ] ; then
        PYTHONHOME="$_OLD_VIRTUAL_PYTHONHOME"
        export PYTHONHOME
        unset _OLD_VIRTUAL_PYTHONHOME
    fi

    # This should detect bash and zsh, which have a hash command that must
    # be called to get it to forget past commands.  Without forgetting
    # past commands the $PATH changes we made may not be respected
    if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then
        hash -r 2>/dev/null
    fi

    if ! [ -z "${_OLD_VIRTUAL_PS1+_}" ] ; then
        PS1="$_OLD_VIRTUAL_PS1"
        export PS1
        unset _OLD_VIRTUAL_PS1
    fi

    unset VIRTUAL_ENV
    if [ ! "${1-}" = "nondestructive" ] ; then
    # Self destruct!
        unset -f deactivate
    fi
}

# unset irrelevant variables
deactivate nondestructive

VIRTUAL_ENV="/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env"
export VIRTUAL_ENV

PYTHONPATH="$VIRTUAL_ENV/lib/python2.7/site-packages"

_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH

# unset PYTHONHOME if set
if ! [ -z "${PYTHONHOME+_}" ] ; then
    _OLD_VIRTUAL_PYTHONHOME="$PYTHONHOME"
    unset PYTHONHOME
fi

if [ -z "${VIRTUAL_ENV_DISABLE_PROMPT-}" ] ; then
    _OLD_VIRTUAL_PS1="$PS1"
    if [ "x" != x ] ; then
        PS1="$PS1"
    else
        PS1="(`basename \"$VIRTUAL_ENV\"`) $PS1"
    fi
    export PS1
fi

# Make sure to unalias pydoc if it's already there
alias pydoc 2>/dev/null >/dev/null && unalias pydoc

pydoc () {
    python -m pydoc "$@"
}

# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands.  Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then
    hash -r 2>/dev/null
fi

I have the activate file pre-made in the dir above FALCON-integrate and follow this procedure:

#!/bin/bash

module load git
git clone git://github.com/PacificBiosciences/FALCON-integrate.git
cd FALCON-integrate

git checkout 0.4.0  # OPTIONAL STEP

make init
make virtualenv
make check

cp ../activate fc_env/bin/activate
make -j install

make test  # to run a simple one

Using this method, the "make test" works both with and without the optional "git checkout 0.4.0" step. The batch script works on both synth0 and ecoli when not including "git checkout 0.4.0". When including "git checkout 0.4.0", a big error message is seen --- I can put the last several lines below:

Traceback (most recent call last):
  File "/gpfs_home/jurban/software/falcon3/FALCON-integrate/fc_env/bin/fc_run.py", line 4, in <module>
    __import__('pkg_resources').run_script('falcon-kit==0.4.0', 'fc_run.py')
  File "/gpfs_home/jurban/software/falcon3/FALCON-integrate/fc_env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 729, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/gpfs_home/jurban/software/falcon3/FALCON-integrate/fc_env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1642, in run_script
    exec(code, namespace, namespace)
  File "/gpfs_home/jurban/software/falcon3/FALCON-integrate/fc_env/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/EGG-INFO/scripts/fc_run.py", line 5, in <module>
    main(*sys.argv)
  File "/gpfs_home/jurban/software/falcon3/FALCON-integrate/fc_env/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py", line 566, in main
    main1(*argv)
  File "/gpfs_home/jurban/software/falcon3/FALCON-integrate/fc_env/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/mains/run.py", line 428, in main1
    wf.refreshTargets([rdb_build_done]) 
  File "/gpfs_home/jurban/software/falcon3/FALCON-integrate/fc_env/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 531, in refreshTargets
    rtn = self._refreshTargets(task2thread, objs = objs, callback = callback, updateFreq = updateFreq, exitOnFailure = exitOnFailure)
  File "/gpfs_home/jurban/software/falcon3/FALCON-integrate/fc_env/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/controller.py", line 706, in _refreshTargets
    raise TaskFailureError("Counted %d failures." %failedJobCount)
pypeflow.controller.TaskFailureError: 'Counted 1 failures.'

Nonetheless -- I think a victory here is getting the batch script to work (without the optional step).

This of course is with job_type = local. The next goal is to get it to work with job_type = SLURM.

When running as job_type = slurm, it does not throw an error -- but it never seems to do anything either.

Here is a look at directories after a minute or two (when this job would be nearly done in local mode):

0-rawreads/:
input.fofn  prepare_rdb.sh  prepare_rdb.sub.sh  prepare_rdb.xsub.sh  pwatcher.dir

1-preads_ovl/:

2-asm-falcon/:

data/:
ref.fasta  synth0.fasta

mypwatcher/:
exits  heartbeats  jobs  state.py  wrappers

scripts/:

sge_log/:

Subdirs:

0-rawreads/pwatcher.dir/:

mypwatcher/exits/:

mypwatcher/heartbeats/:

mypwatcher/jobs/:
J15161be7b3fae2580125f96d49be717b00cba34554feb0e79a2338a641ceb4e4

mypwatcher/wrappers/:
run-J15161be7b3fae2580125f96d49be717b00cba34554feb0e79a2338a641ceb4e4.bash

Here is a look at the stdout+stderr feed:

[INFO]fc_run started with configuration fc_run.cfg
[INFO] No target specified, assuming "assembly" as target 
/gpfs_home/jurban/software/falcon2/FALCON-integrate/FALCON/falcon_kit/run_support.py:317: UserWarning: Unexpected keys in input config: set(['jobqueue'])
  warnings.warn("Unexpected keys in input config: %s" %repr(unused))
[INFO]# of tasks in complete graph: 1
[INFO]tick: 1, #updatedTasks: 0, sleep_time=0.000000
[INFO]tick: 2, #updatedTasks: 0, sleep_time=0.100000
[INFO]Running task from function task_make_fofn_abs_raw()
[WARNING]Missing taskObj.generated_script_fn for task. Maybe we did not need it? Skipping and continuing.
[INFO]Queued 'task:///gpfs_home/jurban/software/falcon2/FALCON-integrate/FALCON/falcon_kit/mains/run1.py/task_make_fofn_abs_raw' ...
[INFO]Success ('done'). Joining 'task:///gpfs_home/jurban/software/falcon2/FALCON-integrate/FALCON/falcon_kit/mains/run1.py/task_make_fofn_abs_raw'...
[INFO]_refreshTargets() finished with no thread running and no new job to submit
[INFO]# of tasks in complete graph: 2
[INFO]tick: 1, #updatedTasks: 0, sleep_time=0.000000
[INFO]tick: 2, #updatedTasks: 0, sleep_time=0.100000
[INFO]Running task from function task_build_rdb()
[INFO]script_fn:'/gpfs/scratch/jurban/male-ilmn/lambda/falcon/synth0/prevmeth02/0-rawreads/prepare_rdb.sh'
[INFO]jobid=J15161be7b3fae2580125f96d49be717b00cba34554feb0e79a2338a641ceb4e4
[INFO]starting job Job(jobid='J15161be7b3fae2580125f96d49be717b00cba34554feb0e79a2338a641ceb4e4', cmd='/bin/bash prepare_rdb.sh', rundir='/gpfs/scratch/jurban/male-ilmn/lambda/falcon/synth0/prevmeth02/0-rawreads', options={'sge_option': '--ntasks 1 --nodes 1 --cpus-per-task 8 --mem 10g --time 00:20:00', 'job_type': None})
[INFO]!sbatch -J J15161be7b3fae2 --ntasks 1 --nodes 1 --cpus-per-task 8 --mem 10g --time 00:20:00 -V -D /gpfs/scratch/jurban/male-ilmn/lambda/falcon/synth0/prevmeth02/mypwatcher/jobs/J15161be7b3fae2580125f96d49be717b00cba34554feb0e79a2338a641ceb4e4 -o stdout -e stderr -S /bin/bash /gpfs/scratch/jurban/male-ilmn/lambda/falcon/synth0/prevmeth02/mypwatcher/wrappers/run-J15161be7b3fae2580125f96d49be717b00cba34554feb0e79a2338a641ceb4e4.bash
slurm 15.08.9
[INFO]Submitted backgroundjob=MetaJobSlurm(MetaJob(job=Job(jobid='J15161be7b3fae2580125f96d49be717b00cba34554feb0e79a2338a641ceb4e4', cmd='/bin/bash prepare_rdb.sh', rundir='/gpfs/scratch/jurban/male-ilmn/lambda/falcon/synth0/prevmeth02/0-rawreads', options={'sge_option': '--ntasks 1 --nodes 1 --cpus-per-task 8 --mem 10g --time 00:20:00', 'job_type': None}), lang_exe='/bin/bash'))
[INFO]Queued 'task:///gpfs_home/jurban/software/falcon2/FALCON-integrate/FALCON/falcon_kit/mains/run1.py/task_build_rdb' ...
[INFO]tick: 4, #updatedTasks: 1, sleep_time=0.100000
[INFO]tick: 8, #updatedTasks: 1, sleep_time=0.500000
[INFO]tick: 16, #updatedTasks: 1, sleep_time=1.000000
[INFO]tick: 32, #updatedTasks: 1, sleep_time=1.000000
[INFO]tick: 64, #updatedTasks: 1, sleep_time=1.000000
[INFO]tick: 128, #updatedTasks: 1, sleep_time=1.000000

Conclusion/salient points:

pb-cdunn commented 8 years ago

Let's narrow this down more.

First, one of your failures named this log file: /gpfs/scratch/jurban/male-ilmn/lambda/falcon/synth0/prevmeth01/0-rawreads/prepare_rdb.sh.log. You need to get into the habit of examining the log-file for the actual failure. You are reporting long stack-traces which only say that a task failed. It's always important to learn which task and why.

The activate script is part of virtualenv, which is a popular open-source project. We can ask them to change it, but that seems unlikely. And it's unnecessary. If you need to unset PYTHONPATH before calling activate, then simply do that. We could provide that as standard advice.

The only thing we need to check is python -c 'import pypeflow; print pypeflow'. Wherever that fails, we have to fix the environment. If you run that under a slurm command, you will know whether the whole thing would work under that same slurm command.

Finally, I really need you to work on 0.5.2. It works fine. Yes, I understand that it will not work with the pypeFLOW installed in your PYTHONPATH. That is a Good Thing, and that's the problem we need to solve.

JohnUrban commented 8 years ago

I will look into log files next time.

As far as I know, I've been working with the latest code except when I specify 0.4.0. The only reason I was specifying that at all is to re-trace the steps from yesterday. Ultimately, the code that worked -- unsetting the PYTHONPATH -- for job_type=local in the batch script was not for 0.4.0 -- it was for whatever comes with the git clone. I could specify "git checkout 0.5.2".

#!/bin/bash
module load git
git clone git://github.com/PacificBiosciences/FALCON-integrate.git
cd FALCON-integrate
git checkout 0.5.2  # OPTIONAL STEP
make init
make virtualenv
make check
#cp ../activate fc_env/bin/activate ## unset PYTHONPATH one way or another
make -j install
make test  # to run a simple one
JohnUrban commented 8 years ago

Ok. I did that exact install protocol above, and:

It seems "git checkout 0.5.2" very much helped this time -- I was previously either not including "git checkout x.x.x" or just testing "git checkout 0.4.0".

I'm still stunned that this worked in slurm mode -- I mean more than just working but not doing anything (as it was before when not dong "git checkout x.x.x"). I will have to check it a few more times ….

I will also try the following installation and batch script method using "unset PYTHONPATH" outside of activate as you suggested:

#!/bin/bash
module load git
git clone git://github.com/PacificBiosciences/FALCON-integrate.git
cd FALCON-integrate
git checkout 0.5.2  # OPTIONAL STEP
make init
make virtualenv
make check
unset PYTHONPATH
make -j install
make test  # to run a simple one
#!/bin/bash
#SBATCH -J falcon-synth0-1
#SBATCH -c 4
#SBATCH --qos=biomed-sb-condo
#SBATCH --mem=4g
#SBATCH --time=00:30:00

###########################
unset PYTHONPATH
. ~/software/falcon3/FALCON-integrate/fc_env/bin/activate
fc_run.py fc_run.cfg
JohnUrban commented 8 years ago

Alright - excellent news.

The fix to all my installation woes is simply adding "unset PYTHONPATH" before doing "make -j install". There was no need to even do "unset PYTHONPATH" in the beginning of the batch script. That only needed to be done at installation. It also worked when including "unset PYTHONPATH" before activating the virtualenv in the batch script. It didn't hurt, but I didn't need it.

Final/simplest protocol that worked for the make test, local mode, and SLURM mode:

#!/bin/bash
module load git
git clone git://github.com/PacificBiosciences/FALCON-integrate.git
cd FALCON-integrate
git checkout 0.5.2  
make init
make virtualenv
make check
unset PYTHONPATH
make -j install
make test 

To run:

. ~/software/falcon3/FALCON-integrate/fc_env/bin/activate
fc_run.py fc_run.cfg

Thanks for everything. I am so happy it is working! I am still somewhat incredulous that it is working in slurm mode reproducibly… but it is! so Time to move forward onto the real stuff.

pb-cdunn commented 8 years ago

Wonderful! And that is very helpful information. We will include that advice in the docs.

One other thing: Could you try with the latest release, 0.6.2? That will use the pwatcher stuff by default, so it will need the latest pypeFLOW. Let's see...

JohnUrban commented 8 years ago

First I need to clear something up.

I do need to do "unset PYTHONPATH" both during installation (before make install) and before activating the environment -- when doing the latest protocol.

I do not need to do "unset PYTHONPATH" if it is part of the deactivate() function in activate. Since you do not want to mess around with that, then the best advice to give users is to install by:

#!/bin/bash
module load git
git clone git://github.com/PacificBiosciences/FALCON-integrate.git
cd FALCON-integrate
git checkout 0.5.2  
make init
make virtualenv
make check
unset PYTHONPATH
make -j install
make test 

and run by:

unset PYTHONPATH
. ~/path/to/FALCON-integrate/fc_env/bin/activate
fc_run.py fc_run.cfg

Here is some evidence. I do a simple check:

python -c 'import pypeflow; print pypeflow' 
which fc_run.py 
python -c 'import falcon_kit; print falcon_kit.falcon' 
which python 

#unset PYTHONPATH  ### TEST WITH AND WITHOUT THIS LINE
. ~/software/falcon/FALCON-integrate/fc_env/bin/activate

python -c 'import pypeflow; print pypeflow' 
which fc_run.py 
which python
python -c 'import falcon_kit; print falcon_kit.falcon' 

Results for 0.5.2 when "unset PYTHONPATH" is part of deactivate().

## WITHOUT UNSET LINE
<module 'pypeflow' from '/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/__init__.pyc'>
/gpfs/runtime/opt/python/2.7.3/bin/fc_run.py
<CDLL '/users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/ext_falcon.so', handle 23de670 at 23a4390>
/gpfs/runtime/opt/python/2.7.3/bin/python

<module 'pypeflow' from '/gpfs_home/jurban/software/falcon/FALCON-integrate/pypeFLOW/pypeflow/__init__.pyc'>
/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/bin/fc_run.py
/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/bin/python
<CDLL '/gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON/ext_falcon.so', handle 17918c0 at 7fc13ec5cc50>
WITH UNSET LINE
<module 'pypeflow' from '/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/__init__.pyc'>
/gpfs/runtime/opt/python/2.7.3/bin/fc_run.py
<CDLL '/users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/ext_falcon.so', handle 257e670 at 2544390>
/gpfs/runtime/opt/python/2.7.3/bin/python

<module 'pypeflow' from '/gpfs_home/jurban/software/falcon3/FALCON-integrate/pypeFLOW/pypeflow/__init__.pyc'>
/gpfs_home/jurban/software/falcon3/FALCON-integrate/fc_env/bin/fc_run.py
/gpfs_home/jurban/software/falcon3/FALCON-integrate/fc_env/bin/python
<CDLL '/gpfs_home/jurban/software/falcon3/FALCON-integrate/FALCON/ext_falcon.so', handle c988c0 at 7fec3ebaec50>

You can see with or without doing "unset PYTHONPATH" before activating the virtualenv does not matter -- both work.

However, this is different when installing by performing "unset PYTHONPATH" before "make install" -- i.e. not part of activate.

WITHOUT UNSET LINE
<module 'pypeflow' from '/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/__init__.pyc'>
/gpfs/runtime/opt/python/2.7.3/bin/fc_run.py
<CDLL '/users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/ext_falcon.so', handle d4b670 at d11390>
/gpfs/runtime/opt/python/2.7.3/bin/python

<module 'pypeflow' from '/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/__init__.pyc'>
/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/bin/fc_run.py
/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/bin/python
<CDLL '/users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/ext_falcon.so', handle 2502680 at 7f6ed2ba1d10>
WITH UNSET LINE
<module 'pypeflow' from '/users/jurban/software/localpy/lib/python2.7/site-packages/pypeflow-0.1.1-py2.7.egg/pypeflow/__init__.pyc'>
/gpfs/runtime/opt/python/2.7.3/bin/fc_run.py
<CDLL '/users/jurban/software/localpy/lib/python2.7/site-packages/falcon_kit-0.4.0-py2.7-linux-x86_64.egg/falcon_kit/ext_falcon.so', handle f4b670 at f11390>
/gpfs/runtime/opt/python/2.7.3/bin/python

<module 'pypeflow' from '/gpfs_home/jurban/software/falcon/FALCON-integrate/pypeFLOW/pypeflow/__init__.pyc'>
/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/bin/fc_run.py
/gpfs_home/jurban/software/falcon/FALCON-integrate/fc_env/bin/python
<CDLL '/gpfs_home/jurban/software/falcon/FALCON-integrate/FALCON/ext_falcon.so', handle 169d8c0 at 7f69356afc50>

In particular, without doing "unset PYTHONPATH" before activating the virtualenv, it is using pypeflow/falcon-kit from those other areas…

I find the same trends for 0.6.2.

I will report on 0.6.2 in subsequent message.

All in all -- "unset PYTHONPATH" is needed for both installation and running.

JohnUrban commented 8 years ago

Alright - here are some details on 0.6.2.

The same as above. If "unset PYTHONPATH" is in deactivate() -- then that is all one needs to do. Installing and running work. If doing "unset PYTHONPATH" right before "make install", then I also need to include it before activating the virtualenv and running.

In either case:

[INFO]tick: 4, #updatedTasks: 1, sleep_time=0.100000
[INFO]tick: 8, #updatedTasks: 1, sleep_time=0.500000
[INFO]tick: 16, #updatedTasks: 1, sleep_time=1.000000
[INFO]tick: 32, #updatedTasks: 1, sleep_time=1.000000
[INFO]tick: 64, #updatedTasks: 1, sleep_time=1.000000
[INFO]tick: 128, #updatedTasks: 1, sleep_time=1.000000
[INFO]tick: 256, #updatedTasks: 1, sleep_time=1.000000

So 0.6.2 is not working for job_type=slurm yet… 0.5.2 is though, at least with synth0.

I am currently making sure the 0.5.2 install can successfully complete in slurm mode for the ecoli assembly. So far, it keeps crashing because I have not allotted enough time for some steps. I don't know if it is not finishing in the time allotted b/c it is not enough or b/c something is wrong. . . but I will give each step an unreasonable amount of time (e.g. 12 hours) to see if that works. I will report back tomorrow.

JohnUrban commented 8 years ago

The 0.5.2 install can successfully complete in slurm mode for the ecoli assembly.

pb-cdunn commented 8 years ago

The 0.5.2 install can successfully complete in slurm mode for the ecoli assembly.

Of course! But you're much better off testing only with the small fake example.

pb-cdunn commented 8 years ago

I think I know what changed now. We updated from a very old version of virtualenv to the very latest. This may have changed:

--no-site-packages

    DEPRECATED. Retained only for backward compatibility. Not having access to global site-packages is now the default behavior.

Since that is now the default, I think we need to include this in our script:

--system-site-packages

    Give the virtual environment access to the global site-packages.

Maybe? Could you try with/and/without unset PYTHONPATH when --system-site-packages is passed to the virtualenv command?

JohnUrban commented 8 years ago

The 0.5.2 install can successfully complete in slurm mode for the ecoli assembly.

Of course! But you're much better off testing only with the small fake example.

Agreed. After all the small stuff worked, I wanted to see bigger stuff work. When the Ecoli kept failing at first I was becoming disheartened a bit. Then I realized it was failing b/c I did not allot enough time. I'm glad to see it works and am eager to try it on my even bigger data.

RE: Virtualenv stuff I need some clarification. Would you like to try this on 0.5.2, 0.6.2, or both?

Also would you like me to replace "--no-site-packages" with "--system-site-packages" in the makefile:

What it is now:
python2.7 virtualenv/virtualenv.py --no-site-packages  --always-copy ${FALCON_PREFIX}

What I can change it to:
python2.7 virtualenv/virtualenv.py --system-site-packages  --always-copy ${FALCON_PREFIX}

Also, I might be a little confused on the goal of both --no-site-packages and --system-site-packages. For --no-site-packages, I found this:

Running virtualenv with the option --no-site-packages will not include the packages that are installed globally. docs.python-guide.org/en/latest/dev/virtualenvs/

And for --system-site-packages, I found this:

If you build with virtualenv --system-site-packages ENV, your virtual environment will inherit packages from /usr/lib/python2.7/site-packages (or wherever your global site-packages directory is). This can be used if you have control over the global site-packages directory, and you want to depend on the packages there. If you want isolation from the global system, do not use this flag. https://virtualenv.pypa.io/en/stable/userguide/#the-extra-search-dir-option

Either way, it seems like you are trying to inherit packages already installed globally -- yet what I have found is that is what is messing up the installation and am only successful when I prevent that. Ultimately, it seems like you should be using an option that tells virtualenv to install everything even if it is found globally (i.e. ignore global env) so you can control/predict the exact packages/versions that all users will have… no?

There is a huge chance I misunderstood something… whether your goal for testing --system-site-packages or what those options do...

JohnUrban commented 8 years ago

Also, do you want me to try with and without unset PYTHONPATH for installation or running?

pb-cdunn commented 8 years ago

I need some clarification. Would you like to try this on 0.5.2, 0.6.2, or both?

Only 0.6.2. The sooner you upgrade, the better.

Also would you like me to replace "--no-site-packages" with "--system-site-packages" in the makefile

Oh, I didn't realize we had that. Nevermind.

Your problem is apparently not the "system" Python, but the extra stuff in your PYTHONPATH. (If that is your system python, then I don't understand why you set PYTHONPATH at all.)

From perusing pypa/virtualenv issues and documentation, I think it's behaving as expected. You need to unset your PYTHONPATH before running virtualenv.

I think everything is actually installed fine (aside from needing --force-reinstall), but since you have PYTHONPATH, that overrides the virtualenv. That is expected behavior. You and I simply misunderstood how virtualenv works. It's basically just like regular Python.

Anyway, if you don't want to spend more time on this, that's fine. We have a clear solution with unset PYTHONPATH.

By the way, another solution for you is to stop using virtualenv. You can pip install --user -e into a PYTHONUSERBASE. Then, you simply prepend your person PYTHONUSERBASE directory to PYTHONPATH, so your stuff will take precedence. I would switch to that, but then I'd get user complaints about how to see environment variables. There is no perfect solution. (Python versioning would help solve some problems, but it would create others.)

JohnUrban commented 8 years ago

Only 0.6.2. The sooner you upgrade, the better.

I did try 0.6.2. It does not work in slurm mode for me. So for now, I cannot use it.

JohnUrban commented 8 years ago

Anyway, if you don't want to spend more time on this, that's fine. We have a clear solution with unset PYTHONPATH.

I would be happy to continue trying to debug 0.6.2 for slurm mode. Happy to help in general.

pb-cdunn commented 8 years ago

This is the most important thing in this thread. We cannot test Slurm because we do not have Slurm. 0.6.2 is the current version. What you are using is not supported anymore. If you want Slurm support going forward, you may need to help us get it working.

With 0.6.2 you can use fc_run0 instead of fc_run (which is fc_run1 by default) to get the old job-dis tribution system. That will prove the problem is only the job-distribution layer, and we can concentrate on fixing that. Please open a new Issue on Slurm support.

I think we've resolved PYTHONPATH/virtualenv Issue well enough for now.

JohnUrban commented 8 years ago

I think everything is actually installed fine (aside from needing --force-reinstall), but since you have PYTHONPATH, that overrides the virtualenv.

I don't need --force-reinstall if I do unset PYTHONPATH.

JohnUrban commented 8 years ago

I will open a new 0.6.2 slurm issue and close this one. We definitely resolved the PYTHONPATH/virtualenv issue.

pb-cdunn commented 8 years ago

For future reference, here is the known issue on PYTHONPATH when --no-site-packages was provided:

It appears that the pypa folks do not intend to fix this.