Broken python environment

[cyrilcros]

With the latest stable release (msca_1508), running make fails due to some issue with the virtual environment. More generally, I have virtualenv and mkvirtualenvwrapper alredy installed locally. I see that the make process starts with:

python /mnt/share/Genomes/progressiveCactus/submodules/virtualenv/virtualenv.py . && cd /mnt/share/Genomes/progressiveCactus/submodules/networkx && setup.py install . && cd /mnt/share/Genomes/progressiveCactus/submodules/psutil/ && setup.py install . && cd /mnt/share/Genomes/progressiveCactus/submodules/biopython/ && echo y | setup.py install | cat

You could just use a requirements.txt file with the Python packages you need and use pip install -r /path/to/requirements.txt to install them all at once. They will be stored in the virtual environment anyway, and you won't need to include them yourself in progressiveCactus.

Also, take a look at autoenv (https://github.com/kennethreitz/autoenv). It overrides cd, but allows you to switch on the virtualenv you need just by entering your directory.

[cyril@synapse progressiveCactus]$ make cd submodules && make all make[1]: Entering directory '/mnt/share/Genomes/progressiveCactus/submodules' rm -f /mnt/share/Genomes/progressiveCactus/submodules/../environment python /mnt/share/Genomes/progressiveCactus/submodules/virtualenv/virtualenv.py /mnt/share/Genomes/progressiveCactus/python New python executable in /mnt/share/Genomes/progressiveCactus/python/bin/python Failed to import the site module Traceback (most recent call last): File "/mnt/share/Genomes/progressiveCactus/python/lib/python3.5/site.py", line 67, in <module> import os File "/mnt/share/Genomes/progressiveCactus/python/lib/python3.5/os.py", line 664, in <module> from _collections_abc import MutableMapping ImportError: No module named '_collections_abc' ERROR: The executable /mnt/share/Genomes/progressiveCactus/python/bin/python is not functioning ERROR: It thinks sys.prefix is '/mnt/share/Genomes/progressiveCactus/submodules' (should be '/mnt/share/Genomes/progressiveCactus/python') ERROR: virtualenv is not compatible with this system or executable Makefile:19: recipe for target 'virtPyRule' failed make[1]: *** [virtPyRule] Error 100 make[1]: Leaving directory '/mnt/share/Genomes/progressiveCactus/submodules' Makefile:10: recipe for target 'all' failed make: *** [all] Error 2

[glennhickey]

progressive cactus is only tested with python 2.7 at this point. by the looks of that log you're using 3.5.

On Fri, Oct 30, 2015 at 12:12 PM, cyrilcros notifications@github.com wrote:

With the latest stable release (msca_1508), running make outputs:

`[cyril@synapse progressiveCactus]$ make

cd submodules && make all make[1]: Entering directory '/mnt/share/Genomes/progressiveCactus/submodules' rm -f /mnt/share/Genomes/progressiveCactus/submodules/../environment python /mnt/share/Genomes/progressiveCactus/submodules/virtualenv/virtualenv.py /mnt/share/Genomes/progressiveCactus/python New python executable in /mnt/share/Genomes/progressiveCactus/python/bin/python Failed to import the site module Traceback (most recent call last): File "/mnt/share/Genomes/progressiveCactus/python/lib/python3.5/site.py", line 67, in import os File "/mnt/share/Genomes/progressiveCactus/python/lib/python3.5/os.py", line 664, in from _collections_abc import MutableMapping ImportError: No module named '_collections_abc' ERROR: The executable /mnt/share/Genomes/progressiveCactus/python/bin/python is not functioning ERROR: It thinks sys.prefix is '/mnt/share/Genomes/progressiveCactus/submodules' (should be '/mnt/share/Genomes/progressiveCactus/python') ERROR: virtualenv is not compatible with this system or executable Makefile:19: recipe for target 'virtPyRule' failed make[1]: * [virtPyRule] Error 100 make[1]: Leaving directory '/mnt/share/Genomes/progressiveCactus/submodules' Makefile:10: recipe for target 'all' failed make: * [all] Error 2`

— Reply to this email directly or view it on GitHub https://github.com/glennhickey/progressiveCactus/issues/49.

[cyrilcros]

So, I am using Archlinux, which has /usr/bin/env python, /usr/bin/python2, /usr/bin/python3 among others. I tried to switch to python2.7 but seems to have failed midway.

My workaround:

• I download the submodules
• I run find ./ -name "*.py*" -type f -exec sed -ir 's/\/usr\/bin\/env\ python/\/usr\/bin\/python2/' {} \; to make sure I always use python2 inside progressiveCactus.
• I then set :
  • the virtPy variable to use virtPy = ${virtPyDir}/bin/python2 in the Makefile
  • python2 $(PWD)/virtualenv/virtualenv.py -p python2 ${virtPyDir} in the submodules/Makefile, to create a Python2 virtual environment

Just note that you can use the virtualenv -p PYTHON.EXE flag to specify what python version you require.

[glennhickey]

Thanks! Though not an issue for us luddites who only use python 2.7, this will probably help a lot of other people. I'll try to incorporate it -- if you have a patched version, feel free to send a pull request.

On Fri, Oct 30, 2015 at 2:32 PM, cyrilcros notifications@github.com wrote:

So, I am using Archlinux, which has /usr/bin/env python, /usr/bin/python2, /usr/bin/python3 among others. I tried to switch to python2.7 but seems to have failed midway.

My workaround:

• I download the submodules
• I run find ./ -name ".py" -type f -exec sed -ir 's/\/usr\/bin\/env\ python/\/usr\/bin\/python2/' {} \; to make sure I always use python2 inside progressiveCactus.
• I then set :
  • the virtPy variable to use virtPy = ${virtPyDir}/bin/python2 in the Makefile
  • python2 $(PWD)/virtualenv/virtualenv.py -p python2 ${virtPyDir} in the submodules/Makefile, to create a Python2 virtual environment

Just note that you can use the -p PYTHON.EXE flag to specify what python version you require.

— Reply to this email directly or view it on GitHub https://github.com/glennhickey/progressiveCactus/issues/49#issuecomment-152611004 .

[cyrilcros]

Python/Java/GCC variations are an absolute horror to manage. I will test a few variations, I am sure I can skip some steps and not change every single /usr/bin/env python