Closed mdavy86 closed 9 years ago
$ cat /usr/lib/python2.6/site-packages/moa-0.11.24-py2.6.egg-info/requires.txt
Jinja2>2.0
argparse
sh
pyyaml>3
ruffus>=2.2
Yaco>=0.1.7
fist>=0.1.5
unittest2>=0.5
lockfile>=0.9
Is everything installed
$ ls -d Jinja2* argparse* sh* pyyaml* ruffus* Yaco* fist* unittest2* lockfile* mdGraph*
ls: cannot access pyyaml*: No such file or directory
argparse-1.2.2.dist-info fist lockfile ruffus sh.py unittest2-0.8.0.dist-info
argparse.py fist-0.1.5-py2.6.egg-info lockfile-0.10.2.dist-info ruffus-2.5-py2.6.egg-info sh.pyc Yaco
argparse.pyc Jinja2-2.7.3-py2.6.egg-info mdGraph-0.1.1-py2.6.egg-info sh-1.09-py2.6.egg-info unittest2 Yaco-0.1.31-py2.6.egg-info
where is pyyaml?
Seems to be mixed between /usr/lib and /usr/lib64 so that’s why your ls doesn’t work:
[shane@bbsvm site-packages]$ sudo pip install moa
Requirement already satisfied (use --upgrade to upgrade): moa in /usr/lib/python2.6/site-packages
Requirement already satisfied (use --upgrade to upgrade): Jinja2>2.0 in /usr/lib/python2.6/site-packages (from moa)
Requirement already satisfied (use --upgrade to upgrade): argparse in /usr/lib/python2.6/site-packages (from moa)
Requirement already satisfied (use --upgrade to upgrade): sh in /usr/lib/python2.6/site-packages (from moa)
Requirement already satisfied (use --upgrade to upgrade): pyyaml>3 in /usr/lib64/python2.6/site-packages (from moa)
Requirement already satisfied (use --upgrade to upgrade): ruffus>=2.2 in /usr/lib/python2.6/site-packages (from moa)
Requirement already satisfied (use --upgrade to upgrade): Yaco>=0.1.7 in /usr/lib/python2.6/site-packages (from moa)
Requirement already satisfied (use --upgrade to upgrade): fist>=0.1.5 in /usr/lib/python2.6/site-packages (from moa)
Requirement already satisfied (use --upgrade to upgrade): unittest2>=0.5 in /usr/lib/python2.6/site-packages (from moa)
Requirement already satisfied (use --upgrade to upgrade): lockfile>=0.9 in /usr/lib/python2.6/site-packages (from moa)
Requirement already satisfied (use --upgrade to upgrade): mdGraph>=0.1markdown in /usr/lib/python2.6/site-packages (from moa)
Requirement already satisfied (use --upgrade to upgrade): markupsafe in /usr/lib64/python2.6/site-packages (from Jinja2>2.0->moa)
Cleaning up...
On 28/11/2014, at 1:49 pm, Roy Storey notifications@github.com wrote:
$ cat /usr/lib/python2.6/site-packages/moa-0.11.24-py2.6.egg-info/requires.txt Jinja2>2.0 argparse sh pyyaml>3 ruffus>=2.2 Yaco>=0.1.7 fist>=0.1.5 unittest2>=0.5 lockfile>=0.9 Is everything installed
$ ls -d Jinja2* argparse* sh* pyyaml* ruffus* Yaco* fist* unittest2* lockfile* mdGraph ls: cannot access pyyaml: No such file or directory argparse-1.2.2.dist-info fist lockfile ruffus sh.py unittest2-0.8.0.dist-info argparse.py fist-0.1.5-py2.6.egg-info lockfile-0.10.2.dist-info ruffus-2.5-py2.6.egg-info sh.pyc Yaco argparse.pyc Jinja2-2.7.3-py2.6.egg-info mdGraph-0.1.1-py2.6.egg-info sh-1.09-py2.6.egg-info unittest2 Yaco-0.1.31-py2.6.egg-info where is pyyaml?
— Reply to this email directly or view it on GitHub https://github.com/biospectrabysequencing/gbs_moa/issues/14#issuecomment-64840735.
Dr Shane Sturrock shane.sturrock@biomatters.com Senior Scientist Tel: +64 (0) 9 379 5064 Level 2, 18 Shortland Street Auckland New Zealand
Fair enough, lockfile is the issue
python -v $(which moa)
import pkgutil # precompiled from /usr/lib64/python2.6/pkgutil.pyc
Traceback (most recent call last):
File "/usr/bin/moa", line 5, in <module>
from pkg_resources import load_entry_point
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2655, in <module>
working_set.require(__requires__)
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 648, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 546, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: lockfile>=0.9
importance of catching and reporting exceptions, I guess...
That’s what I was working on but we’re running the latest version (0.10.2) which is the same as I was running on Ubuntu.
I’m trying a few things - install on 6.5 instead of 6.6 and see if that’s the problem, and I’ll also try a local install of python2.7 in my own directory and see if the issue is with python2.6
Just trying to knock out the variables.
Shane
On 28/11/2014, at 1:57 pm, Roy Storey notifications@github.com wrote:
Fair enough, lockfile is the issue
python -v $(which moa) import pkgutil # precompiled from /usr/lib64/python2.6/pkgutil.pyc Traceback (most recent call last): File "/usr/bin/moa", line 5, in
from pkg_resources import load_entry_point File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2655, in working_set.require(requires) File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 648, in require needed = self.resolve(parse_requirements(requirements)) File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 546, in resolve raise DistributionNotFound(req) pkg_resources.DistributionNotFound: lockfile>=0.9 — Reply to this email directly or view it on GitHub https://github.com/biospectrabysequencing/gbs_moa/issues/14#issuecomment-64841103.
Dr Shane Sturrock shane.sturrock@biomatters.com Senior Scientist Tel: +64 (0) 9 379 5064 Level 2, 18 Shortland Street Auckland New Zealand
Then setuptools
itself - the pkg_resources
code might be the issue. 0.6c11
is ancient in comparison to 7.0
, but we have come to expect this from CentOS. If pkg_resources is solely relying on __version__
from lockfile this might be the issue and probably a big discriminator with ubuntu
python -c 'import lockfile; print lockfile.__version__'
Traceback (most recent call last):
File "<string>", line 1, in <module>
AttributeError: 'module' object has no attribute '__version__'
0.6c11 isn't even listed
OK, that is starting to look like a possibility. The supplied version is actually 0.6.10.
Shane
On 28/11/2014, at 2:23 pm, Roy Storey notifications@github.com wrote:
Then setuptools itself - the pkg_resources code might be the issue. 0.6c11 is ancient in comparison to 7.0, but we have come to expect this from CentOS. If pkg_resources is solely relying on version from lockfile this might be the issue and probably a big discriminator with ubuntu
python -c 'import lockfile; print lockfile.version' Traceback (most recent call last): File "
", line 1, in AttributeError: 'module' object has no attribute 'version' — Reply to this email directly or view it on GitHub https://github.com/biospectrabysequencing/gbs_moa/issues/14#issuecomment-64842405.
Dr Shane Sturrock shane.sturrock@biomatters.com Senior Scientist Tel: +64 (0) 9 379 5064 Level 2, 18 Shortland Street Auckland New Zealand
Working in my private virtualenv to give me some freedom:
(python)[shane@bbsvm python]$ easy_install --version setuptools 7.0
Still not working though after a clean install:
(python)[shane@bbsvm python]$ moa --help
ERROR!
Unexpected error
Warning: Encountered a run error?
Traceback (most recent call last):
File "/home/shane/python/bin/moa", line 9, in
On 28/11/2014, at 2:23 pm, Roy Storey notifications@github.com wrote:
Then setuptools itself - the pkg_resources code might be the issue. 0.6c11 is ancient in comparison to 7.0, but we have come to expect this from CentOS. If pkg_resources is solely relying on version from lockfile this might be the issue and probably a big discriminator with ubuntu
python -c 'import lockfile; print lockfile.version' Traceback (most recent call last): File "
", line 1, in AttributeError: 'module' object has no attribute 'version' — Reply to this email directly or view it on GitHub https://github.com/biospectrabysequencing/gbs_moa/issues/14#issuecomment-64842405.
Dr Shane Sturrock shane.sturrock@biomatters.com Senior Scientist Tel: +64 (0) 9 379 5064 Level 2, 18 Shortland Street Auckland New Zealand
I'm installing nothing, you're free as a :bird:
I don’t want to clash with stuff the OS has so using a virtualenv lets me do it all from source and I can do it from my non-root account.
Shane
On 28/11/2014, at 2:40 pm, Roy Storey notifications@github.com wrote:
I'm installing nothing, you're free as a
— Reply to this email directly or view it on GitHub https://github.com/biospectrabysequencing/gbs_moa/issues/14#issuecomment-64843252.
Dr Shane Sturrock shane.sturrock@biomatters.com Senior Scientist Tel: +64 (0) 9 379 5064 Level 2, 18 Shortland Street Auckland New Zealand
Same issue on CentOS 6.5 which is a clean install nothing to do with NZGL, straight install from the CentOS DVD.
Shane
On 28/11/2014, at 2:41 pm, Shane Sturrock shane@biomatters.com wrote:
I don’t want to clash with stuff the OS has so using a virtualenv lets me do it all from source and I can do it from my non-root account.
Shane
On 28/11/2014, at 2:40 pm, Roy Storey <notifications@github.com mailto:notifications@github.com> wrote:
I'm installing nothing, you're free as a
— Reply to this email directly or view it on GitHub https://github.com/biospectrabysequencing/gbs_moa/issues/14#issuecomment-64843252.
Dr Shane Sturrock shane.sturrock@biomatters.com mailto:shane.sturrock@biomatters.com Senior Scientist Tel: +64 (0) 9 379 5064 Level 2, 18 Shortland Street Auckland New Zealand
Dr Shane Sturrock shane.sturrock@biomatters.com Senior Scientist Tel: +64 (0) 9 379 5064 Level 2, 18 Shortland Street Auckland New Zealand
@afmcc time to get those Makefile
s sorted?
If someone has managed to get moa working from the current version (install clean on a CentOS machine like I did) then I would love to know what the issue is and how to solve it because I’ve got other users asking for it. Since I’ve had it working on CentOS 6.5 previously but now it doesn’t I have to assume something has changed in the dependencies which is causing this.
Shane
On 28/11/2014, at 3:04 pm, Roy Storey notifications@github.com wrote:
@afmcc https://github.com/afmcc time to get those Makefiles sorted?
— Reply to this email directly or view it on GitHub https://github.com/biospectrabysequencing/gbs_moa/issues/14#issuecomment-64844736.
Dr Shane Sturrock shane.sturrock@biomatters.com Senior Scientist Tel: +64 (0) 9 379 5064 Level 2, 18 Shortland Street Auckland New Zealand
I used Python 2.7 to get it running. Can't remember why but I think it was because of some of the dependencies not working with 2.6.
That’s what I’m thinking. It was 2.7 on Ubuntu and I’ve just installed it on my Mac under Yosemite which has python 2.7 too. Both work. I’m installing Python 2.7 in parallel and will see if that fixes it.
Shane
On 28/11/2014, at 3:21 pm, eburgueno notifications@github.com wrote:
I used Python 2.7 to get it running. Can't remember why but I think it was because of some of the dependencies not working with 2.6.
— Reply to this email directly or view it on GitHub https://github.com/biospectrabysequencing/gbs_moa/issues/14#issuecomment-64845773.
Dr Shane Sturrock shane.sturrock@biomatters.com Senior Scientist Tel: +64 (0) 9 379 5064 Level 2, 18 Shortland Street Auckland New Zealand
Here's my log:
$ /software/Python-2.7.2/bin/virtualenv moa
New python executable in moa/bin/python
Installing setuptools, pip...done.
$ source moa/bin/activate
(moa)$ pip install pyyaml
Successfully installed pyyaml
Cleaning up...
(moa)$ pip install Moa
Successfully installed Moa Jinja2 argparse sh ruffus Yaco fist unittest2 lockfile mdGraph markupsafe six
Cleaning up...
(moa)$ moa -h
usage: moa [-h] {command} ...
optional arguments:
-h, --help show this help message and exit
command:
Command legend: '.' can be executed everywhere; 'j' require a job; '*' are specified by the template
simple . Create a 'simple' adhoc job.
simple! . Create a 'simple' adhoc job.
map . create an adhoc moa 'map' job
map! . create an adhoc moa 'map' job
reduce . Create a 'reduce' adhoc job.
reduce! . Create a 'reduce' adhoc job using the bash history
set . Set one or more variables
tree . Show a directory tree and job status
version . print moa version number
archive . Archive a job, or tree with jobs for later reuse.
cp . Copy a moa job, or a tree with jobs (with -r).
mv . Move, rename or renumber a moa job.
new . Create a new job.
list . Lists all known local templates
Yep, 2.7 is the trick.
[shane@bbsvm site-packages]$ moa --help usage: moa [-h] {command} ...
optional arguments: -h, --help show this help message and exit
command: Command legend: '.' can be executed everywhere; 'j' require a job; '*' are specified by the template
simple . Create a 'simple' adhoc job.
simple! . Create a 'simple' adhoc job.
map . create an adhoc moa 'map' job
map! . create an adhoc moa 'map' job
reduce . Create a 'reduce' adhoc job.
reduce! . Create a 'reduce' adhoc job using the bash history
set . Set one or more variables
tree . Show a directory tree and job status
version . print moa version number
archive . Archive a job, or tree with jobs for later reuse.
cp . Copy a moa job, or a tree with jobs (with -r).
mv . Move, rename or renumber a moa job.
new . Create a new job.
list . Lists all known local templates
Shane
On 28/11/2014, at 3:29 pm, eburgueno notifications@github.com wrote:
Here's my log:
$ /software/Python-2.7.2/bin/virtualenv moa New python executable in moa/bin/python Installing setuptools, pip...done. $ source moa/bin/activate (moa)$ pip install pyyaml Successfully installed pyyaml Cleaning up... (moa)$ pip install Moa Successfully installed Moa Jinja2 argparse sh ruffus Yaco fist unittest2 lockfile mdGraph markupsafe six Cleaning up... (moa)$ moa -h usage: moa [-h] {command} ...
optional arguments: -h, --help show this help message and exit
command: Command legend: '.' can be executed everywhere; 'j' require a job; '*' are specified by the template
simple . Create a 'simple' adhoc job. simple! . Create a 'simple' adhoc job. map . create an adhoc moa 'map' job map! . create an adhoc moa 'map' job reduce . Create a 'reduce' adhoc job. reduce! . Create a 'reduce' adhoc job using the bash history set . Set one or more variables tree . Show a directory tree and job status version . print moa version number archive . Archive a job, or tree with jobs for later reuse. cp . Copy a moa job, or a tree with jobs (with -r). mv . Move, rename or renumber a moa job. new . Create a new job. list . Lists all known local templates
— Reply to this email directly or view it on GitHub https://github.com/biospectrabysequencing/gbs_moa/issues/14#issuecomment-64846330.
Dr Shane Sturrock shane.sturrock@biomatters.com Senior Scientist Tel: +64 (0) 9 379 5064 Level 2, 18 Shortland Street Auckland New Zealand
Looks good. Thanks @shanesturrock
Sanity checking moa
now works correctly. The following tset, just creates a moa
simple job;
## Clone bbs repo
$ git clone git@github.com:biospectrabysequencing/gbs_moa.git
$ cd gbs_moa/test
## Run the test script to make a .moa directory
$./testMoa.sh
Moa: Created a "simple" job
Moa: with title "Find location of config file"
Moa: loaded sync actor simpleRunner
find . -name config -print
./.moa/config
Moa:: Success executing "run" (<1 sec)
Now checking moa further with moa status
, and moa run
...
$ cd /tmp/$USER
$ moa status
bash: /usr/bin/moa: No such file or directory
## Moa is installed at;
$ which moa
/usr/local/bin/moa
@shanesturrock, Looks like we may also need a symbolic link for moa
to the /usr/local/bin
installation.
Works for me. I can't reproduce this.
[rstorey@bbsvm tmp]$ mkdir /tmp/$USER
[rstorey@bbsvm tmp]$ cd /tmp/$USER
[rstorey@bbsvm rstorey]$ moa status
Moa: Need a Moa job - try 'moa -h'
[rstorey@bbsvm rstorey]$ which moa
/usr/local/bin/moa
try
type moa
@mdavy86 et @kiwiroy, you would only end up with moa under the system's /usr/local/bin or /usr/bin if you're installing it as root and without using an virtual environment. I would strongly advise against this.
It is installed in a parallel installation with python2.7 compiled from source so it isn’t using our system python as moa doesn’t work with 2.6. I just needed to get it working and it is isolated in a single VM rather than system wide but yes, the plan is to create an installation within a virtualenv and have users run it from there now we’ve got it working. Users don’t have root rights anyway, or even sudo so nothing runs under admin rights.
Shane
On 2/12/2014, at 9:58 am, eburgueno notifications@github.com wrote:
@mdavy86 https://github.com/mdavy86 et @kiwiroy https://github.com/kiwiroy, you would only end up with moa under the system's /usr/local/bin or /usr/bin if you're installing it as root and without using an virtual environment. I would strongly advise against this.
— Reply to this email directly or view it on GitHub https://github.com/biospectrabysequencing/gbs_moa/issues/14#issuecomment-65133476.
Dr Shane Sturrock shane.sturrock@biomatters.com Senior Scientist Tel: +64 (0) 9 379 5064 Level 2, 18 Shortland Street Auckland New Zealand
OK, I’ve now installed moa (again) but this time it is in the /active/bbsgroup/moa python virtualenv.
You need to add:
source /active/bbsgroup/moa/bin/activate
to any scripts and then it will use the install which is in that directory. It should also (in theory) be able to run on the cluster because that directory is mounted across the entire environment. If you’ve done this you should get the following:
(moa)[root@bbsvm moa]# which moa /active/bbsgroup/moa/bin/moa (moa)[root@bbsvm moa]# moa --help usage: moa [-h] {command} ...
optional arguments: -h, --help show this help message and exit
command: Command legend: '.' can be executed everywhere; 'j' require a job; '*' are specified by the template
simple . Create a 'simple' adhoc job.
simple! . Create a 'simple' adhoc job.
map . create an adhoc moa 'map' job
map! . create an adhoc moa 'map' job
reduce . Create a 'reduce' adhoc job.
reduce! . Create a 'reduce' adhoc job using the bash history
set . Set one or more variables
tree . Show a directory tree and job status
version . print moa version number
archive . Archive a job, or tree with jobs for later reuse.
cp . Copy a moa job, or a tree with jobs (with -r).
mv . Move, rename or renumber a moa job.
new . Create a new job.
list . Lists all known local templates
Shane
On 2/12/2014, at 10:04 am, Shane Sturrock shane@biomatters.com wrote:
It is installed in a parallel installation with python2.7 compiled from source so it isn’t using our system python as moa doesn’t work with 2.6. I just needed to get it working and it is isolated in a single VM rather than system wide but yes, the plan is to create an installation within a virtualenv and have users run it from there now we’ve got it working. Users don’t have root rights anyway, or even sudo so nothing runs under admin rights.
Shane
On 2/12/2014, at 9:58 am, eburgueno <notifications@github.com mailto:notifications@github.com> wrote:
@mdavy86 https://github.com/mdavy86 et @kiwiroy https://github.com/kiwiroy, you would only end up with moa under the system's /usr/local/bin or /usr/bin if you're installing it as root and without using an virtual environment. I would strongly advise against this.
— Reply to this email directly or view it on GitHub https://github.com/biospectrabysequencing/gbs_moa/issues/14#issuecomment-65133476.
Dr Shane Sturrock shane.sturrock@biomatters.com mailto:shane.sturrock@biomatters.com Senior Scientist Tel: +64 (0) 9 379 5064 Level 2, 18 Shortland Street Auckland New Zealand
Dr Shane Sturrock shane.sturrock@biomatters.com Senior Scientist Tel: +64 (0) 9 379 5064 Level 2, 18 Shortland Street Auckland New Zealand
If for any reason you want to use Moa but without activating the environment (ie: so that your session continues to use your system's python), you can do something like this:
alias moa='PYTHONPATH=[moa's virtualenv dir]/moa/lib/python2.7/ [moa's virtualenv dir]/bin/moa'
moa -h
You would need similar aliases for the other moa bin files (moa moainit moaprompt moar moasetstatus). You can create them in bulk:
$ for mfile in [moa's virtualenv dir]/moa/bin/moa*; do alias $(basename "$mfile")="PYTHONPATH=${PYTHONPATH}${PYTHONPATH:+:}[moa's virtualenv dir]/moa/lib/python $mfile"; done
$ alias|grep moa
alias moa='PYTHONPATH=[moa's virtualenv dir]/moa/lib/python moa/bin/moa'
alias moainit='PYTHONPATH=[moa's virtualenv dir]/moa/lib/python moa/bin/moainit'
alias moaprompt='PYTHONPATH=[moa's virtualenv dir]/moa/lib/python moa/bin/moaprompt'
alias moar='PYTHONPATH=[moa's virtualenv dir]/moa/lib/python moa/bin/moar'
alias moasetstatus='PYTHONPATH=[moa's virtualenv dir]/moa/lib/python moa/bin/moasetstatus'
$ moa -h
usage: moa [-h] {command} ...
etc
It is on CentOS 6.6 using pip to install python dependencies.
Dependences on PFR systems
CentOS
What CentOS version are we running?
Python
version?
Moa
Some
Moa
dependencies;