ValueError: numpy.dtype has the wrong size, try recompiling

[smujahed]

Dear Zipline Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

• Operating System: Mac OS X El Capitan version 10.11.4 (15E65)
• Python Version: Python 2.7.11 :: Continuum Analytics, Inc.
• Python Bitness: 64-bit
• How did you install Zipline: conda
• Python packages: $ pip freeze or $ conda list
  • bcolz=1.0.0=py27_0
  • bottleneck=1.0.0=np19py27_0
  • click=6.6=py27_0
  • contextlib2=0.4.0=py27_0
  • cyordereddict=0.2.2=py27_0
  • cython=0.24=py27_0
  • decorator=4.0.9=py27_0
  • logbook=0.12.5=py27_0
  • mkl=11.3.1=0
  • networkx=1.11=py27_0
  • numexpr=2.4.6=np19py27_0
  • numpy=1.9.3=py27_1
  • openssl=1.0.2h=0
  • pandas=0.16.2=np19py27_0
  • patsy=0.4.1=py27_0
  • pip=8.1.1=py27_1
  • python=2.7.11=0
  • python-dateutil=2.5.3=py27_0
  • pytz=2016.4=py27_0
  • readline=6.2=2
  • requests=2.9.1=py27_0
  • scipy=0.17.0=np19py27_0
  • setuptools=20.7.0=py27_0
  • six=1.10.0=py27_0
  • sqlalchemy=1.0.12=py27_0
  • sqlite=3.9.2=0
  • statsmodels=0.6.1=np19py27_0
  • tk=8.5.18=0
  • toolz=0.7.4=py27_0
  • wheel=0.29.0=py27_0
  • zipline=0.8.4=np19py27_0
  • zlib=1.2.8=0

Now that you know a little about me, let me tell you about the issue I am having:

Description of Issue

• What did you expect to happen?

Tried to execute run_algo.py (fh-cap)[ smujahed @ rubix : ~/fh-capital/test-algos ] >>> run_algo.py --help

• What happened instead?

Traceback (most recent call last):

  File "/Users/smujahed/anaconda/envs/fh-cap/bin/run_algo.py", line 20, in <module>
    from zipline.utils import parse_args, run_pipeline

  File "/Users/smujahed/anaconda/envs/fh-cap/lib/python2.7/site-packages/zipline/__init__.py", line 26, in <module>

    from . algorithm import TradingAlgorithm

  File "/Users/smujahed/anaconda/envs/fh-cap/lib/python2.7/site-packages/zipline/algorithm.py", line 79, in <module>

    from zipline.pipeline.engine import (

  File "/Users/smujahed/anaconda/envs/fh-cap/lib/python2.7/site-packages/zipline/pipeline/__init__.py", line 11, in <module>

    from .loaders import USEquityPricingLoader

  File "/Users/smujahed/anaconda/envs/fh-cap/lib/python2.7/site-packages/zipline/pipeline/loaders/__init__.py", line 11, in <module>

    from .equity_pricing_loader import USEquityPricingLoader

  File "/Users/smujahed/anaconda/envs/fh-cap/lib/python2.7/site-packages/zipline/pipeline/loaders/equity_pricing_loader.py", line 19, in <module>

    from zipline.data.us_equity_pricing import (

  File "/Users/smujahed/anaconda/envs/fh-cap/lib/python2.7/site-packages/zipline/data/us_equity_pricing.py", line 23, in <module>

    from bcolz import (

  File "/Users/smujahed/anaconda/envs/fh-cap/lib/python2.7/site-packages/bcolz/__init__.py", line 64, in <module>

    from bcolz.carray_ext import (

  File "__init__.pxd", line 155, in init bcolz.carray_ext (bcolz/carray_ext.c:39345)

ValueError: numpy.dtype has the wrong size, try recompiling

Here is how you can reproduce this issue on your machine:

Reproduction Steps

1. Install conda
2. $ create -n NAME zipline
3. $ source activate NAME
4. $ run_algo.py --help

   What steps have you taken to resolve this already?

   • Googled extensively bits of the traceback. Found most link this to an incompatibility between pandas and numpy versions.
   • Installed some different combinations of pandas and numpy although I cannot remember which versions.
   • Downgraded to zipline 0.8.4, same problem.

     Anything else?

I know you guys and girls put in a lot of effort to get this working, so thank you in advance for providing a great tool.

Sincerely, smujahed

[richafrank]

Hi @smujahed ,

I'm not sure the commands that you used to install the zipline or bcolz conda packages, but seems like the bcolz package that's installed was compiled with a different version of numpy than you've got installed in the environment currently. It's not tagged with a numpy version, so I couldn't say what it's expecting. We have a python27-numpy19 tagged bcolz package available in the quantopian channel on anaconda, so you might try installing that one instead.

[smujahed]

Hi @richafrank ,

Thank you for getting back to me so quickly and my apologies for not getting back to you sooner.

I'm not sure the commands that you used to install the zipline or bcolz conda packages

The command is listed in the "Reproduction Steps" part of the issue form. Nothing was done outside of what I have listed there.

It's not tagged with a numpy version, so I couldn't say what it's expecting. We have a python27-numpy19 tagged bcolz package available in the quantopian channel on anaconda, so you might try installing that one instead.

Absolutely right! You've hit the nail on the head. For some reason when I installed zipline with conda it would automagically download bcolz from the default channel... I have no idea why, but I am going to assume that it has something to do with conda wanting to install the latest version of things.

Your solution has 100% fixed the problem that I've been having, and for the benefit of the community I'll explain what I did to fix it.

Doing conda install --override-channels -c quantopian zipline will not work because this stops conda from getting the other dependencies from the default channel... It was a stupid mistake by me thinking back at it. I got things working by doing conda install -c quantopian bcolz=0.12.1. If you omit the version number available from the quantopian channel it will bounce something back like 'Already up-to-date'. To find the version available on quantopian do conda search --override-channels -c quantopian bcolz, omitting the --override-channels will give you a list of all versions of bcolz available in the channels in your .condarc.

I hope this was helpful for anyone else having the same problem, and thanks again to @richafrank !!

All the best,

smujahed

[richafrank]

Thanks for sharing your solution @smujahed !