bin/runProgressiveCactus.sh: line 32: bin/../environment: No such file or directory

[dbolser]

You mention this file in passing in the README, but I can't find any details... what should go in this file? (Environment settings presumably?)

I forgot to mention, I've just done:

git clone git://github.com/glennhickey/progressiveCactus.git
cd progressiveCactus
git submodule update --init
make

and the build seemed to complete successfully**. git log (master branch) mentions b1d0e10e92a as the latest commit.

\ The last line of make output is:

make[1]: Leaving directory `/nfs/production/panda/ensemblgenomes/development/dbolser/progressiveCactus/submodules'

[glennhickey]

the 'environment' file is created when you run make. if it's not there, it likely means there was an error building the software.

On Mon, Sep 16, 2013 at 10:37 AM, Dan Bolser notifications@github.comwrote:

You mention this file in passing in the README, but I can't find any details... what should go in this file? (Environment settings presumably?)

Cheers, Dan.

— Reply to this email directly or view it on GitHubhttps://github.com/glennhickey/progressiveCactus/issues/3 .

[joelarmstrong]

Hmm... this file should be created during compilation. It's a virtualenv environment, which is basically a sandbox containing the PATH, PYTHONPATH, python install itself, etc.

The Makefile is a little hacky right now... if you ran 'make' again after the first time, you can't just ctrl-c out of it, it has to complete fully, since I think the file is removed at the beginning and repopulated after compiling every submodule.

[dbolser]

How do I "make clean" to try again, capturing the output of make? When I try "make clean", I get this error:

make[2]: Leaving directory `/nfs/production/panda/ensemblgenomes/development/dbolser/progressiveCactus/submodules/phast/src'
make[1]: *** No rule to make target `kentToolBinariesRuleClean', needed by `clean'.  Stop.
make[1]: Leaving directory `/nfs/production/panda/ensemblgenomes/development/dbolser/progressiveCactus/submodules'
make: *** [clean] Error 2

I guess git --reset hard? But that doesn't work for submodules... OK, just cloned again...

[glennhickey]

You shouldn't need to clean or reset anything in order to reproduce any build errors: just run make again.

"make clean" (or anything else for that matter) will only work right now if make has run through at least once successfully. The reason is that some of the submodules' Makefiles which are required for make clean only get created by the first run of make...

On Mon, Sep 16, 2013 at 11:50 AM, Dan Bolser notifications@github.comwrote:

How do I "make clean" to try again, capturing the output of make? When I try "make clean", I get this error:

make[2]: Leaving directory /nfs/production/panda/ensemblgenomes/development/dbolser/progressiveCactus/submodules/phast/src' make[1]: *** No rule to make targetkentToolBinariesRuleClean', needed by clean'. Stop. make[1]: Leaving directory/nfs/production/panda/ensemblgenomes/development/dbolser/progressiveCactus/submodules' make: *\ [clean] Error 2

I guess git --reset hard? But that doesn't work for submodules...

— Reply to this email directly or view it on GitHubhttps://github.com/glennhickey/progressiveCactus/issues/3#issuecomment-24520528 .

[joelarmstrong]

That make clean issue seems to be a slight copy-paste error in the Makefile which I think I just fixed (5045e96).

But, as Glenn mentioned, it shouldn't matter much, just running make again should fix things.

[dbolser]

I wonder if this is the issue... Now that make has finished again from a clean re-install, I see the environment file. Last time, after make had run once, I immediately ran make again to look for errors, and indeed, second time round it failed. I wonder if second time round it deletes the environment file? I can try if you like.

[glennhickey]

Thanks for the feedback, Dan. The environment file gets generated each time make is run, so any failure during the make process could mess it up, even if it built successfully previously and nothing has since changed. I'm able to build repeatedly here without any problems and can't reproduce this issue at the moment.

For the environment file to have been missing entirely, something must have happened when you re-made in the very first build step of stetting up the virtual python environment (which also gets run every time make is called). This is normally pretty hassle free, but could maybe be sensitive to a local environment change or permission lock. I'm afraid that's about all I can tell you at this point without a specific error message or being able to reproduce...

On Tue, Sep 17, 2013 at 4:40 AM, Dan Bolser notifications@github.comwrote:

I wonder if this is the issue... Now that make has finished again from a clean re-install, I see the environment file. Last time, after make had run once, I immediately ran make again to look for errors, and indeed, second time round it failed. I wonder if second time round it deletes the environment file? I can try if you like.

— Reply to this email directly or view it on GitHubhttps://github.com/glennhickey/progressiveCactus/issues/3#issuecomment-24572286 .

[dbolser]

As a test I did a clean re-install. The first make succeeds, and the environment file is created. The second make fails with this error (plus a bit of context):

...
make[1]: Leaving directory `/nfs/production/panda/ensemblgenomes/development/dbolser/Test/progressiveCactus/submodules'

[dbolser@ebi-001 progressiveCactus]$ lt
total 264
-rw-rw-r--  1 dbolser nucleotide 36104 Sep 17 17:02 LICENSE
-rw-rw-r--  1 dbolser nucleotide   343 Sep 17 17:02 Makefile
-rw-rw-r--  1 dbolser nucleotide 13733 Sep 17 17:02 README.md
drwxrwxr-x  2 dbolser nucleotide    41 Sep 17 17:02 bin
drwxrwxr-x  2 dbolser nucleotide   117 Sep 17 17:02 examples
drwxrwxr-x  2 dbolser nucleotide   138 Sep 17 17:02 src
drwxrwxr-x 24 dbolser nucleotide   651 Sep 17 17:05 submodules
drwxrwxr-x  6 dbolser nucleotide   113 Sep 17 17:05 python
-rw-rw-r--  1 dbolser nucleotide  5007 Sep 17 17:36 environment

[dbolser@ebi-001 progressiveCactus]$ make
cd submodules && make all
make[1]: Entering directory `/nfs/production/panda/ensemblgenomes/development/dbolser/Test/progressiveCactus/submodules'
rm -f /nfs/production/panda/ensemblgenomes/development/dbolser/Test/progressiveCactus/submodules/../environment
python /nfs/production/panda/ensemblgenomes/development/dbolser/Test/progressiveCactus/submodules/virtualenv/virtualenv.py /nfs/production/panda/ensemblgenomes/development/dbolser/Test/progressiveCactus/python
Traceback (most recent call last):
  File "/nfs/production/panda/ensemblgenomes/development/dbolser/Test/progressiveCactus/submodules/virtualenv/virtualenv.py", line 2471, in <module>
    main()
  File "/nfs/production/panda/ensemblgenomes/development/dbolser/Test/progressiveCactus/submodules/virtualenv/virtualenv.py", line 942, in main
    never_download=options.never_download)
  File "/nfs/production/panda/ensemblgenomes/development/dbolser/Test/progressiveCactus/submodules/virtualenv/virtualenv.py", line 1043, in create_environment
    site_packages=site_packages, clear=clear))
  File "/nfs/production/panda/ensemblgenomes/development/dbolser/Test/progressiveCactus/submodules/virtualenv/virtualenv.py", line 1186, in install_python
    fix_lib64(lib_dir)
  File "/nfs/production/panda/ensemblgenomes/development/dbolser/Test/progressiveCactus/submodules/virtualenv/virtualenv.py", line 1564, in fix_lib64
    os.path.join(os.path.dirname(lib_parent), 'lib64'))
OSError: [Errno 17] File exists
make[1]: *** [virtPyRule] Error 1
make[1]: Leaving directory `/nfs/production/panda/ensemblgenomes/development/dbolser/Test/progressiveCactus/submodules'
make: *** [all] Error 2

[dbolser@ebi-001 progressiveCactus]$ lt
total 239
-rw-rw-r--  1 dbolser nucleotide 36104 Sep 17 17:02 LICENSE
-rw-rw-r--  1 dbolser nucleotide   343 Sep 17 17:02 Makefile
-rw-rw-r--  1 dbolser nucleotide 13733 Sep 17 17:02 README.md
drwxrwxr-x  2 dbolser nucleotide    41 Sep 17 17:02 bin
drwxrwxr-x  2 dbolser nucleotide   117 Sep 17 17:02 examples
drwxrwxr-x  2 dbolser nucleotide   138 Sep 17 17:02 src
drwxrwxr-x 24 dbolser nucleotide   651 Sep 17 17:05 submodules
drwxrwxr-x  6 dbolser nucleotide   113 Sep 17 17:05 python

[dbolser@ebi-001 progressiveCactus]$ git log
commit 5045e96d45b881b145d812e88cb0cfee8a3cc8c8
Author: Joel Armstrong <jcarmstr@ucsc.edu>
Date:   Mon Sep 16 09:06:06 2013 -0700

    fix naming problem in make clean rule
...

Obviously, this is a trivial issue, but just though you might like to see the error.

Cheers, Dan.

[joelarmstrong]

Thanks for the error report, I haven't seen that before. What OS are you running? Usually virtualenv just overwrites the progressiveCactus python installation without complaints. It looks like this is a special case in virtualenv, where it tries to create a symlink and isn't happy overwriting it instead.

I think a temporary fix is to remove the progressiveCactus/python directory before a make. We can do this in the makefile as well without an issue.

[dbolser]

[dbolser@ebi-001 ~]$ uname -a Linux ebi-001.ebi.ac.uk 2.6.32-358.6.2.el6.x86_64 #1 SMP Tue May 14 15:48:21 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux

[dbolser@ebi-001 ~]$ more /etc/redhat-release Red Hat Enterprise Linux Server release 6.3 (Santiago)

[glennhickey]

zapping the python directory sounds like a good workaround to me too. thanks again

On Wed, Sep 18, 2013 at 5:14 AM, Joel Armstrong notifications@github.comwrote:

Thanks for the error report, I haven't seen that before. What OS are you running? Usually virtualenv just overwrites the progressiveCactus python installation without complaints. It looks like this is a special case in virtualenv, where it tries to create a symlink and isn't happy overwriting it instead.

I think a temporary fix is to remove the progressiveCactus/python directory before a make. We can do this in the makefile as well without an issue.

— Reply to this email directly or view it on GitHubhttps://github.com/glennhickey/progressiveCactus/issues/3#issuecomment-24650964 .