installing COCO within virtualenv

[mashallAryan]

Hi, How is it possible to install COCO (python) inside a virtualenv or in an alternate/custom directory "python do.py run-python " does not seem to provide tools for this purpose.

[nikohansen]

You can use, for example, conda for this purpose. My expertise is limited, but AFAICS any virtual environment manager will do. They are made for this very purpose. After that, python do.py is called within this virtual environment.

I am not sure I understand what you mean by "alternative directory". Alternative directory for what?

https://conda.io/docs/using/envs.html https://uoa-eresearch.github.io/eresearch-cookbook/recipe/2014/11/20/conda/

[mashallAryan]

While inside the virtualenv, python do.y does not install to the virtualenv site-packages; instead tries to access the local user site-packages. Looks like --user is hardcoded inside the installation code.

By alternative directory I mean any arbitrary directory -> in legacy python installation --target is used to do so. Here is the kind of error I get. I think it can be replicated easily

$ virtualenv env
$ source env/bin/active
$ env/bin/python do.py
...
...
...EXPAND  code-experiments/build/python/setup.py.in to code-experiments/build/python/setup.py
PYTHON  setup.py install --user in code-experiments/build/python
ERROR: return value=1
NOTE: Using precompiled C file to build interface.
running install
Checking .pth file support in ~/.local/lib/python2.7/site-packages/
~/env/bin/python -E -c pass
TEST FAILED: ~/.local/lib/python2.7/site-packages/ does NOT support .pth files
error: bad install directory or PYTHONPATH..
....
...

[nikohansen]

Nice find. We are looking into it. As a quick fix you could try to replace line 183/188 (master/development) of do.py

python(join('code-experiments', 'build', 'python'), ['setup.py', 'install', '--user'])

with

python(join('code-experiments', 'build', 'python'), ['setup.py', 'install'])

and accordingly in line 241/ 246 for ./do.py install post-processing.

The disadvantage is that some users might not be able to install it this way, but I guess that is what --target could address.

Otherwise directly calling python setup.py install --target ... within the code-experiments/build/python/ folder (or code-postprocessing/ folder, respectively) after do.py failed should work as well, as do.py has done all the necessary preparations.

[mashallAryan]

@nikohansen after installation of coco inside the virtualenv I get this error:

[virtual_env]$ python -c "import cocoex"

RuntimeError: module compiled against API version 0xb but this version of numpy is 0xa
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "build/bdist.linux-x86_64/egg/cocoex/__init__.py", line 25, in <module>
  File "build/bdist.linux-x86_64/egg/cocoex/interface.py", line 7, in <module>
  File "build/bdist.linux-x86_64/egg/cocoex/interface.py", line 6, in __bootstrap__
ImportError: numpy.core.multiarray failed to import

This is caused by coco not using the virtualenv's site-packages during the installation and even not the user's site-packages. For unknown reasons it uses the system's site-packages. So incompatibility between numpy versions in these places is problematic.

[vuw-ecs-kevin]

I'm the bloke helping Mashall get this setup.

I have compiled everything from source inside a Python 2.7 virtualenv (including a bazel build of TensorFlow !)

The full list of sources used was

GPflow-0.3.4.tar.gz              numpy-1.12.0.tar.gz        
Keras-2.0.5.tar.gz               pandas-0.20.2.tar.gz       
Markdown-2.6.8.tar.gz            pbr-3.1.1.tar.gz       
Werkzeug-0.12.2.tar.gz           protobuf-3.3.0.tar.gz      
backports.weakref-1.0rc1.tar.gz  pyparsing-2.2.0.tar.gz     
bleach-2.0.0.tar.gz              python-dateutil-2.6.0.tar.gz
coco-2.1.tar.gz                  pytz-2017.2.zip          
cycler-0.10.0.tar.gz             scikit-learn-0.18.2.tar.gz
deap-1.0.2.post2.tar.gz          scipy-0.19.1.tar.gz     
dev-0.4.0.tar.gz                 setuptools-36.0.1.zip   
edward-1.3.3.tar.gz              six-1.10.0.tar.gz   
funcsigs-1.0.2.tar.gz            subprocess32-3.2.7.tar.gz
functools32-3.2.3-2.zip          tensorflow-1.2.0.tar.gz  
html5lib-0.999999999.tar.gz      webencodings-0.5.1.tar.gz
matplotlib-2.0.2.tar.gz          wheel-0.29.0.tar.gz
mock-2.0.0.tar.gz        

Note that the numpy version there matches our system's version.

I patched GPflow following the workaround here

https://github.com/tensorflow/tensorflow/issues/7570

after seeing this

>>> import GPflow
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/path/to/GPflowCoco/lib/python2.7/site-packages/GPflow-0.3.4-py2.7.egg/GPflow/__init__.py", line 18, in <module>
    from . import likelihoods, kernels, param, model, gpmc, sgpmc, priors, gpr, svgp, vgp, sgpr, tf_wraps, tf_hacks
  File "/path/to/GPflowCoco/lib/python2.7/site-packages/GPflow-0.3.4-py2.7.egg/GPflow/likelihoods.py", line 17, in <module>
    from . import densities, transforms
  File "/path/to/GPflowCoco/lib/python2.7/site-packages/GPflow-0.3.4-py2.7.egg/GPflow/transforms.py", line 18, in <module>
    from . import tf_wraps as tfw
  File "/path/to/GPflowCoco/lib/python2.7/site-packages/GPflow-0.3.4-py2.7.egg/GPflow/tf_wraps.py", line 42, in <module>
    @tf.RegisterShape("VecToTri")
AttributeError: 'module' object has no attribute 'RegisterShape'

I also removed the GCC check

diff -r GPflow-0.3.4/setup.py GPflow-0.3.k/setup.py
30,31d29
<     if gcc_version > 4:
<         compile_command += " -D_GLIBCXX_USE_CXX11_ABI=0"

following

https://github.com/GPflow/GPflow/issues/227

I have then gone and re-installed the latest numpy 1.13.0 and rebuilt coco, having removed all of the attempts to force a "user" install.

diff -r coco-2.1/do.py coco-2.k/do.py
241c241
<     python('code-postprocessing', ['setup.py', 'install', '--user'], verbose=_verbosity)
---
>     python('code-postprocessing', ['setup.py', 'install'], verbose=_verbosity)
282,283c282,283
<     # python(join('code-experiments', 'build', 'python'), ['setup.py', 'install', '--user'])
<     python(join('code-experiments', 'build', 'python'), ['setup.py', 'install', '--user'])
---
>     # python(join('code-experiments', 'build', 'python'), ['setup.py', 'install'])
>     python(join('code-experiments', 'build', 'python'), ['setup.py', 'install', ])
767c767
<            ['setup.py', 'install', '--user'], verbose=_verbosity)
---
>            ['setup.py', 'install'], verbose=_verbosity)

and then run the

python do.py run-python

I then tested the coco with the example code frm example_experiment.py

import example_experiment as ee
ee.suite_name = "bbob-biobj"
ee.SOLVER = ee.random_search  # which is default anyway
ee.observer_options['algorithm_info'] = "default of example_experiment.py"
ee.main(5, 1+9, 2, 300)  # doctest: +ELLIPSIS
COCO INFO: Results will be output to folder exdata/random_search_on_bbob-biobj_budget0002xD
Benchmarking solver '<function random_search' with budget=5*dimension on bbob-biobj suite, Fri Jun 30 14:34:14 2017
Batch usecase, make sure you run *all* 300 batches.

Jun30 14h34:14, d=2, running: f20.(1)   f40.(1)   
    done in 3.7e-03 seconds/evaluation
...
Jun30 14h34:15, d=40, running: f05.(1)   f25.(1)   f45.(1)  
    done in 3.8e-03 seconds/evaluation
  dimension seconds/evaluations
  -----------------------------
      2      3.7e-03 
      3      1.9e-03 
      5      1.4e-03 
     10      1.5e-03 
     20      2.3e-03 
     40      3.8e-03 
  -----------------------------
  bbob-biobj done (16 of 4950 problems benchmarked in batch 2 of 300)
    MAKE SURE TO RUN ALL BATCHES, Fri Jun 30 14:34:17 2017 (0h00:03 total elapsed time).

which I think means it is working ?

I have also done a

python -v -v -v -c "import cocoex" 2>&1 | tee OUT

and I attach that OUT which seems to show (I grep-ed for import) all the things that it is pulling in from both the virtualenv and system areas, in case that helps with any debugging you need to do. OUT.txt

Kevin M. Buckley

eScience Consultant School of Engineering and Computer Science Victoria University of Wellington New Zealand

[nikohansen]

which I think means it is working ?

yes

[brockho]

Looking at the code together with @nikohansen, @islandblue, and @dtusar, we realized that the python(...) method in cocoutils.py is abusing os.environ which it should not. Instead, we should use the so far unused env parameter to pass the name of the python call for the CI tests in do.py's build-python2(...) and build-python3(...).

[dtusar]

I changed the code according to the comment above. The changed are currently in the development branch.

[nikohansen]

@mashallAryan @vuw-ecs-kevin, is it possible for you check whether the changes resolved the original problem? I have troubles to get virtualenv going with my Anaconda Python installation.

[vuw-ecs-kevin]

is it possible for you check whether the changes resolved the original problem? I have troubles to get virtualenv going with my Anaconda Python installation.

We have no need to use an Anaconda Python within our institution, as our system Python provides all we need, however, I will be looking to recompile within our PVE environment with the modified Coco, yes.

As to Mashall's original question, re the forcing of a --user install by default, which then requires a user to "unpick" that, are there any plans to turn that into an option?

Clearly, we now have a set of patches that we can attempt to apply every time a change is made to the Coco codebase, but one often finds pathches fail to apply cleanly, whereas a package option for the --user install would give the user complete flexibility.

What was the commit id of the relevant point in the `development branch ?

[dtusar]

What was the commit id of the relevant point in the `development branch ?

This one: e4c6120875f82376c76d7a7df945632e99f4daf4

[nikohansen]

As to Mashall's original question, re the forcing of a --user install by default which then requires a user to "unpick" that, are there any plans to turn that into an option?

I am in favour, shouldn't be difficult to do either (just passing all --* args to build_python). There should be an informative error message though, which suggests trying --user when installation doesn't succeed (possibly due to lack of access rights).

[vuw-ecs-kevin]

OK, so

• I activated my completely-built-from-source PVE.
• I upacked the coco-2.1 tarball
• I applied our coco-2.1-rem_user_install.diff patch
• I applied a patch containing the changes from e4c61208
• I ran python do.py run-python and saw

AML     ['code-experiments/src/coco_random.c', 'code-experiments/src/coco_suite.c', 'code-experiments/src/coco_observer.c', 'code-experiments/src/coco_archive.c', 'code-experiments/src/coco_runtime_c.c'] -> code-experiments/build/python/cython/coco.c
EXPAND  code-experiments/build/python/cython/coco.c.in to code-experiments/build/python/cython/coco.c
EXPAND  code-experiments/src/coco.h to code-experiments/build/python/cython/coco.h
COPY    code-experiments/src/bbob2009_testcases.txt -> code-experiments/build/python/bbob2009_testcases.txt
COPY    code-experiments/src/bbob2009_testcases2.txt -> code-experiments/build/python/bbob2009_testcases2.txt
COPY    code-experiments/build/python/README.md -> code-experiments/build/python/README.txt
EXPAND  code-experiments/build/python/setup.py.in to code-experiments/build/python/setup.py
PYTHON  setup.py install in code-experiments/build/python
PYTHON  example_experiment.py in code-experiments/build/python

This updates the cocoex-2.1-py2.7-linux-x86_64.egg in the PVE's lib/python2.7/site-packages directory

If I now repeat the experiment, having entered my PVE's interpreter

$ python
>>> import example_experiment as ee
>>> ee.suite_name = "bbob-biobj"
>>> ee.SOLVER = ee.random_search
>>> ee.observer_options['algorithm_info'] = "default of example_experiment.py"
>>> ee.main(5, 1+9, 2, 300)
COCO INFO: Results will be output to folder exdata/random_search_on_bbob-biobj_budget0002xD
Benchmarking solver '<function random_search' with budget=5*dimension on bbob-biobj suite, Fri Jul  7 10:58:24 2017
Batch usecase, make sure you run *all* 300 batches.

Jul7 10h58:24, d=2, running: f20.(1)   f40.(1)   
    done in 3.9e-03 seconds/evaluation

Jul7 10h58:24, d=3, running: f05.(1)   f25.(1)   f45.(1)   
    done in 1.9e-03 seconds/evaluation

Jul7 10h58:24, d=5, running: f10.(1)   f30.(1)   f50.(1)   
    done in 1.4e-03 seconds/evaluation

Jul7 10h58:25, d=10, running: f15.(1)   f35.(1)   f55.(1)   
    done in 1.6e-03 seconds/evaluation

Jul7 10h58:25, d=20, running: f20.(1)   f40.(1)   
    done in 2.3e-03 seconds/evaluation

Jul7 10h58:25, d=40, running: f05.(1)   f25.(1)   f45.(1)  
    done in 3.9e-03 seconds/evaluation
  dimension seconds/evaluations
  -----------------------------
      2      3.9e-03 
      3      1.9e-03 
      5      1.4e-03 
     10      1.6e-03 
     20      2.3e-03 
     40      3.9e-03 
  -----------------------------
  bbob-biobj done (16 of 4950 problems benchmarked in batch 2 of 300)
    MAKE SURE TO RUN ALL BATCHES, Fri Jul  7 10:58:28 2017 (0h00:03 total elapsed time).
>>> quit()

which, I believe, suggests things are still working, so I then ran the simple import from issuecomment-312164990 to give the OUT file with all the import paths again (although I have attched it with a more meaningful name this time!) coco-2.1-20170707-01-OUT.txt

[vuw-ecs-kevin]

There should be an informative error message though, which suggests trying --user when installation doesn't succeed (possibly due to lack of access rights).

Why?

Python's --user is a standard python approach and, I would maintain, is what most people should be doing anyway, rather than assuming they can expect to pollute the system-level Python installation on any machine they turn up on.

And of course, if you are already in the habit of using Python's PVE facility, so as to seperate out various, and/or avoid cross pollution of, "package stacks", there's no need for a --user anyway.

You appear to be advocating that every Python package should have "commonsense" messages in it for users who don't bother to find out how to use Python before installing a Python package, which, of course, potentially involves a vast swathe of packages?

If such "commonsense" notifications are needed then, for me, they really need to be in Python's core, not written in every package.

Just my thr'pen'th though: it's your codebase.

[nikohansen]

@vuw-ecs-kevin , your insights are very welcome (I don't consider myself as a Python expert). I think we are mostly in sync. My assumption is indeed that many COCO users neither know much or even anything about Python nor are they computer scientists, and they should not need to learn anything about Python to use COCO! I am convinced we should keep this assumption around the post-processing module cocopp.

I can see that users who use cocoex are much more likely to have basic Python knowledge. Yet I always like the idea that things just work out-of-the-box. So I guess you are quite correct that I advocate the idea that users should not need to bother finding out how to use Python before to install a Python package. In this sense, it is rather a bug than a feature in Python's setup utils that the --user option is not default. When working with virtual environments it is seemingly the other way around though (polluting by default the users site could be considered rather a bug than a feature).

Given that users do not call setup.py but do.py, it seems to me acceptable to give "unconventional" (extended, non-standard) user hints.

[vuw-ecs-kevin]

your insights are very welcome (I don't consider myself as a Python expert). I think we are mostly in sync.

Oh, I wouldn't even claim to be a Python user, let alone an expert, merely someone who has had to learn a little bit about it in order to make things work for other people who come to me telling me they need to use it for their research. For me, anything that makes people learn a little about the tools they are using adds into the overall pool of what I think of as "scientific reproducibility", however, as you say, probably a bit too wide a discussion topic for a cocoex issues list.

With that in mind then, should we expect a release incorporating the fixes for the stuff Mashall found, or are we best advised to just stay with our "matches a couple of patches" version for now, given that it seems to work as expected?

[nikohansen]

It is labeled as bug and I am certainly in favour of fixing it. We just have to find someone with enough time on their hands and understanding of the problem to fix it. Seems doable.

[nikohansen]

Fixed in the development branch.