docker-stacks/optimization build fails due to a dependency failing.

[russelljjarvis]

This bug is currently being worked on at neo/quantities:

https://github.com/python-quantities/python-quantities/issues/129

docker-stacks/optimization build fails, because of an error in the quantity module. This might be a transient bug in quantities, or I might need to notify the developer.

Step 39/48 : RUN python -c "from neuronunit.models.reduced import ReducedModel"
 ---> Running in 4eff10f2a015
Traceback (most recent call last):
  File "/opt/conda/lib/python3.5/site-packages/quantities-0.11.1-py3.5.egg/quantities/quantity.py", line 231, in __array_prepare__
KeyError: <ufunc 'positive'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/lib/python3.5/site-packages/neuronunit-0.1.8.6-py3.5.egg/neuronunit/models/__init__.py", line 10, in <module>
    import neuronunit.capabilities as cap
  File "/opt/conda/lib/python3.5/site-packages/neuronunit-0.1.8.6-py3.5.egg/neuronunit/capabilities/__init__.py", line 9, in <module>
    from .spike_functions import spikes2amplitudes,spikes2widths,spikes2thresholds
  File "/opt/conda/lib/python3.5/site-packages/neuronunit-0.1.8.6-py3.5.egg/neuronunit/capabilities/spike_functions.py", line 5, in <module>
    import neo
  File "/opt/conda/lib/python3.5/site-packages/neo-0.4.0-py3.5.egg/neo/__init__.py", line 11, in <module>
    from neo.core import *
  File "/opt/conda/lib/python3.5/site-packages/neo-0.4.0-py3.5.egg/neo/core/__init__.py", line 39, in <module>
    from neo.core.recordingchannel import RecordingChannel
  File "/opt/conda/lib/python3.5/site-packages/neo-0.4.0-py3.5.egg/neo/core/recordingchannel.py", line 13, in <module>
    import quantities as pq
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
  File "/opt/conda/lib/python3.5/site-packages/quantities-0.11.1-py3.5.egg/quantities/__init__.py", line 285, in <module>
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
  File "/opt/conda/lib/python3.5/site-packages/quantities-0.11.1-py3.5.egg/quantities/units/__init__.py", line 9, in <module>
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
  File "/opt/conda/lib/python3.5/site-packages/quantities-0.11.1-py3.5.egg/quantities/units/acceleration.py", line 7, in <module>
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
  File "/opt/conda/lib/python3.5/site-packages/quantities-0.11.1-py3.5.egg/quantities/units/time.py", line 18, in <module>
  File "/opt/conda/lib/python3.5/site-packages/quantities-0.11.1-py3.5.egg/quantities/unitquantity.py", line 62, in __new__
  File "/opt/conda/lib/python3.5/site-packages/quantities-0.11.1-py3.5.egg/quantities/quantity.py", line 139, in _reference
  File "/opt/conda/lib/python3.5/site-packages/quantities-0.11.1-py3.5.egg/quantities/unitquantity.py", line 230, in __pow__
  File "/opt/conda/lib/python3.5/site-packages/quantities-0.11.1-py3.5.egg/quantities/quantity.py", line 88, in g
  File "/opt/conda/lib/python3.5/site-packages/quantities-0.11.1-py3.5.egg/quantities/quantity.py", line 321, in __pow__
  File "/opt/conda/lib/python3.5/site-packages/quantities-0.11.1-py3.5.egg/quantities/quantity.py", line 236, in __array_prepare__
ValueError: ufunc <ufunc 'positive'> not supported by quantities
                please file a bug report at https://github.com/python-quantities

[russelljjarvis]

This was blocking progress on the documentation notebook we where talking about. I managed to use the solution described at the bottom of the thread.

https://www.bountysource.com/issues/46022410-valueerror-ufunc-ufunc-positive-not-supported-by-quantities

The temporary patch is here which closely matches docker-stacks: https://github.com/russelljjarvis/ParallelPyNEURON/blob/master/neuronunit-optimization/Dockerfile#L68-L72

[rgerkin]

@russelljjarvis Can you change this so that instead of just symlinking to the checked out working copy, you install it? I.e. python setup.py install from inside the working copy?

[russelljjarvis]

Perhaps, I tried that first of course.

ie sudo /opt/conda/bin/python setup.py install but that doesn't work for me or mczerwinski in the 'developers' forum thread I cited above. I don't view this as permanent solution. I will keep you posted if a better option comes up.

mczerwinski commented on this issue 12 hours ago. 'I manage to work around it: I couldn't update the library, so I just installed it in a different place. Ugly, but working.'

I have since noticed that earlier in the thread Christian-B, recommends editing setup.py and requirements.py as below (so I will try this as recommended below):

1. In setup.py
add:
dependency_links=['http://github.com/python-quantities/python-quantities/'
'tarball/master#egg=quantities'],
and:
install_requires=[
'quantities',
.....]

Note: quantities in install_requires MUST be WITHOUT a version

In requirements.py
add the line
git+https://github.com/python-quantities/python-quantities.git

russw commented on this issue 8 hours ago. After spinning for a while on this, I've also run into this problem. I can confirm that the current "fix" is either to not upgrade numpy (or pandas), or to install the latest master from github per @Christian-B (thanks!) above:

Actually the thread has been updated even since sleeping, now it recommends trying:

pip install --upgrade git+git://github.com/python-quantities/python-quantities.git@master

For anyone who does this via a requirements.txt file, you add this exact line (no quantities at the start, since it is at the end).

-e git+git://github.com/python-quantities/python-quantities.git@master#egg=quantities

Any word on getting this updated in pypi? This is obviously the better solution. I also offer to help, but expect I wouldn't be on top of it as @apdavison, since his Neo depends on it.

[russelljjarvis]

@rgerkin

RUN sudo rm -r /opt/conda/lib/python3.5/site-packages/quantities-0.11.1-py3.5.egg
RUN /opt/conda/bin/pip install --upgrade git+git://github.com/python-quantities/python-quantities.git@master
RUN python -c "import quantities as pq"
RUN python -c "from neuronunit.models.reduced import ReducedModel"

Now works.

It would be cool if there was an easy way to temporarily wind back to older numpy/quantities/neo. Do you think there is a way?

[rgerkin]

If I simply force conda to install numpy-1.12.1, then everything seems to work. numpy 1.13 introduced a new type that quantities does not handle. I am rebuilding all the stacks now and will test them before our meeting tomorrow.

[rgerkin]

@russelljjarvis Quantities has another problem where it uses a stupid versioning package to retrieve its own version, and it doesn't work very well, causing it to report a nonsensical version number, and then other packages that rely on a certain version of quantities don't think they are getting it and error on install. So I forked the master branch of quantities to scidash and replaced the version in scidash/setup.py with a hard version number (0.11.1). That fixes one problem.

I also replaced all numpy installs in the Dockerfiles with installs of 1.12.1, which prevent quantities from having any errors. I thought that what I did in the first paragraph would be enough but for some reason it wasn't.

I prefer this solution to your implementation because it ensures that the problem is solved within sciunit and neuronunit itself, as well as in the stacks. Also, it prevents the stacks from filling up with custom installations that we lose track of. I am more motivated to resolve custom dependencies described in the setup.py files of sciunit and neuronunit themselves (in fact I have pull requests in to eventually get rid of all of them by merging my changes into future pypi releases of those dependencies).

[russelljjarvis]

Yes this makes sense.

[russelljjarvis]

I rebuilt a docker image last night. I think there were still issues with quantities installing, although ultimately they were satisfied, with an approach similar to the above.

I am leaving this issue open, as I do not believe that the solution we arrived at is the final solution.

[rgerkin]

I just changed the quantities dependency to look to the main repo instead of my fork of it, on the basis of Andrew's commments here, but this may have been premature. I'll look into it.

[rgerkin]

This was fixed with the new version of quantities on pypi.