Closed fraka6 closed 8 years ago
I'm having the same problem, getting the same error when trying to run dual_moving_average.py.
Has this issue been fixed?
Can you try with a more recent version?
pip install -U git+https://github.com/quantopian/zipline
I did tried it at the time with the latest version and just retried after a pull and I get the same error.
Self-assigning to investigate.
I think this is the same issue as https://github.com/quantopian/zipline/issues/650.
I also just ran into this, it seems that order()
is currently broken in zipline.
OK, this actually seems unrelated to order()
. For some reason, the perf dataframe passed to analyze() does not contain a column for the record()
ed variables.
Assigning to @StewartDouglas Stewart - I believe that Thomas is right and the issue may be related to my recent refactor of the perf packet generation in the trade simulation. We'll discuss in more detail off-line.
I believe this may be the culprit commit: https://github.com/quantopian/zipline/commit/8d5bfd3c917a8bd0c80fd8237c58296eb1f84728
This one is also possibly the culprit, but I doubt it: https://github.com/quantopian/zipline/commit/16ab46b69c888a844481c08fa865a10923489c54
The reason that the perf dataframe passed to analyze()
does not contain the record()
ed variables is that the algorithm skips the first 300 days:
if context.i < 300:
return
The algorithm works correctly when we run it over 300 days or more, for example:
PYTHONPATH=. python scripts/run_algo.py -f zipline/examples/dual_moving_average.py --symbols AAPL --start 2011-1-1 --end 2012-3-12
This suggests to me that analyze()
should be modified so that it does not fail in the days < 300 case.
It's always the simple things... Thanks for figuring this out @StewartDouglas. Not sure how we can make this more obvious. The easiest for us would probably be to check in analyze whether AAPL is present and raise a helpful error message if not.
@fraka6 I have opened a PR which should address the issue you were experiencing with dual_moving_average.py, thank you for bringing it to our attention.
Regarding the other algorithms you mentioned:
examples/utils
, to make clearer the fact that it is not an algorithm.PYTHONPATH=. python zipline/examples/<algo>.py
Please do let us know if you're still have trouble with dual_ema_talib.py.
We have now removed the _analyze.py files entirely, and instead moved the analyze() methods into the algo files themselves. You can see our discussion on this and other issues here: https://github.com/quantopian/zipline/pull/697. I think the original issues raised have been addressed, so I'll close this issue. However, please do let us know of any other unexpected behavior.
Hi guys. I do confirm dual_moving_average.py works now, thanks. Concerning the others examples, I still get the same errors for those 3: dual_ema_talib.py -> ImportError: No module named talib (might have miss something in talib installation) pairtrade.py-> ValueError: You must define a handle_data function. olmar.py-> AttributeError: 'TradingAlgorithm' object has no attribute 'symbol'
talib
is an optional dependency, so it's expected that that example will fail if talib is not installed. The other two examples should always work afaik. @StewartDouglas can you check and see if those are broken for you?
ACK, looking into this.
@fraka6 I can run both pairtrade.py and olmar.py directly from the command line as follows:
PYTHONPATH=. python zipline/examples/pairtrade.py
PYTHONPATH=. python zipline/examples/olmar.py
Can you confirm whether these work for you?
The reason that pairtrade.py is not working for you is that it cannot be run using scripts/run_algo.py. I'm not sure why olmar.py is failing for you, but I'm investigating.
olmar.py works for me when I run the following:
PYTHONPATH=. python scripts/run_algo.py -f zipline/examples/olmar.py --start 2004-1-1 --end 2008-1-1 --symbols 'AMD,CERN,COST,DELL,GPS,INTC,MMM'
@fraka6 If you could let us know whether this succeeds for you, or show us a stack trace if it fails, then it would be much appreciated.
hi @StewartDouglas. Its still not good. If I set the pythonpath to zipline, it crash before (ImportError: No module named _assets -> see bellow). It seems to be related to same issue reported here: https://groups.google.com/forum/#!topic/zipline/B_cKY4HrC2Q I tried it with python anaconda, python2.7, pip install & manual git pull (latest version & pip uninstall).
Here are my notes:
which python
/home/fpieraut/anaconda/bin/python
echo $PYTHONPATH /home/fpieraut/heroku/datascience/pytrade/zipline
python scripts/run_algo.py -f zipline/examples/olmar.py --start 2004-1-1 --end 2008-1-1 --symbols 'AMD,CERN,COST,DELL,GPS,INTC,MMM'
Traceback (most recent call last):
File "scripts/run_algo.py", line 20, in
Hi @fraka6, sorry to hear it's still giving you trouble. Can you try running the following in your zipline repo (to compile the Cython files to C) and trying again:
python setup.py build_ext --inplace
Interesting. It is working, I had to install contextlib2 and bottleneck to make the example working fully. thanks. Any idea why I have to do this (python setup.py ...)? Francis
@twiecki - any idea why the Cython files would not have been generated?
TL/DR: It's expected behavior that you need to recompile Cython files if you update a development install from git. The easiest way to do this is to run python setup.py build_ext --inplace
every time you pull.
@fraka6 this is expected behavior assuming you installed from source (i.e. by doing pip install .
or pip install -e .
from inside a git checkout).
When you run a regular local pip install (i.e. pip install /path/to/zipline
), what you're saying to pip is roughly "copy all the files from my zipline repo into my site-packages
", where site-packages is a known location relative to the python executable that's running pip. Assuming you're working from a virtualenv, this will be a path like this: ~/.virtualenvs/my-zipline-virtualenv/lib/python2.7/site-packages/
. Since pip is simply taking the current state of your repo and copying it somewhere else, if you pull a newer version of zipline from git, you won't see those changes take in your virtualenv until you reinstall zipline by copying the new files.
This is reasonable behavior when you're installing a package that you don't intend to change, but if you're regularly updating a package, having to reinstall every time becomes a bit of a pain. For this reason, pip (and it's predecessors setuptools
and distutils
) provide the notion of an "editable" install. If you do pip install -e /path/to/zipline/
, you're telling pip roughly "create a symlink from my site packages to my zipline package"* The net effect of installing as editable is that every time you make changes to your zipline module (either by making changes yourself or just downloading updates), those changes become immediately visible to Python.
Having read all this, you might then reasonably ask "So if Python sees all the latest updates to my Zipline package when it's installed as editable why do I have to re-run setup.py build_ext --inplace
?". The answer is that, for performance reasons, certain files in Zipline are written in Cython, which is a Python-like language that generates C code that interoperates well with the Python. The first time you install Zipline with pip, it uses Cython to generate compiled C binaries that Python can import just like .py
files.
These are defined in our setup.py
like this:
ext_modules = LazyCythonizingList([
('zipline.assets._assets', ['zipline/assets/_assets.pyx']),
('zipline.lib.adjusted_array', ['zipline/lib/adjusted_array.pyx']),
('zipline.lib.adjustment', ['zipline/lib/adjustment.pyx']),
('zipline.lib.rank', ['zipline/lib/rank.pyx']),
(
'zipline.data.ffc.loaders._equities',
['zipline/data/ffc/loaders/_equities.pyx'],
),
(
'zipline.data.ffc.loaders._adjustments',
['zipline/data/ffc/loaders/_adjustments.pyx'],
),
])
...
which is saying "create a Python module named zipline.assets._assets
by generating a C file from zipline/assets/_assets.pyx
and then compiling it into a [Shared Object File](http://www.cprogramming.com/tutorial/shared-libraries-linux-gcc.html)
for use by Python. Do the same for zipline.lib.adjusted_array
and the rest of the listed modules. If you pull a new version of Zipline, you may have pulled updated versions of our Cython files, or we may have created entirely new Cython modules. Just pulling down new .pyx
files doesn't do anything by itself; you have to tell Python to actually compile those files into the .so
format that Python understands. The easiest way to do this is by running python setup.py build_ext --inplace
.
*For cross-platform compatibility, Python doesn't actually use symlinks, it uses "path configuration files, which for most purposes make python behave as though there were a symlink from your site-packages to Zipline."
Hello,
I try olmar.py-> AttributeError: 'TradingAlgorithm' object has no attribute 'symbol'
I try to update the newest zipline.
I use command "pip install -U git+https://github.com/quantopian/zipline"
But It shows error below:
=============================error============================================
Error [Error 2] 系統找不到指定的檔案。 while executing command git clone -q https://github.com/quantopian/zipline c:\users\leo\appdata\local\temp\pip-nbaloh-build
How to fix this problem?
Thank you very much.
Read the error message ->Cannot find command "git" Install git.
Francis
2015-10-30 0:15 GMT-04:00 easytrader notifications@github.com:
Hello, I try olmar.py-> AttributeError: 'TradingAlgorithm' object has no attribute 'symbol' I try to update the newest zipline. I use command "pip install -U git+https://github.com/quantopian/zipline" But It shows error below:
=============================error============================================
Error [Error 2] 系統找不到指定的檔案。 while executing command git clone -q https://github.com/quantopian/zipline c:\users\leo\appdata\local\temp\pip-nbaloh-build Cannot find command 'git'
How to fix this problem?
Thank you very much.
— Reply to this email directly or view it on GitHub https://github.com/quantopian/zipline/issues/663#issuecomment-152411181.
Sorry , I installed git. I use command "pip install -U git+https://github.com/quantopian/zipline" But There is another question bekow: I don't know how to fix this. I use windwos 7 x64. Thank you very much. =========================error============================================= Collecting git+https://github.com/quantopian/zipline Cloning https://github.com/quantopian/zipline to c:\users\leo\appdata\local\te mp\pip-wytg8b-build Requirement already up-to-date: pip>=7.1.0 in e:\anaconda\lib\site-packages (fro m zipline==0.8.0rc1) Requirement already up-to-date: Logbook==0.10.0 in e:\anaconda\lib\site-packages (from zipline==0.8.0rc1) Requirement already up-to-date: pytz==2015.4 in e:\anaconda\lib\site-packages (f rom zipline==0.8.0rc1) Collecting numpy==1.9.2 (from zipline==0.8.0rc1) Collecting scipy==0.15.1 (from zipline==0.8.0rc1) Using cached scipy-0.15.1.tar.gz Collecting pandas==0.16.1 (from zipline==0.8.0rc1) Using cached pandas-0.16.1-cp27-none-win32.whl Collecting patsy==0.4.0 (from zipline==0.8.0rc1) Using cached patsy-0.4.0-py2.py3-none-any.whl Requirement already up-to-date: statsmodels==0.6.1 in e:\anaconda\lib\site-packa ges (from zipline==0.8.0rc1) Collecting python-dateutil==2.4.2 (from zipline==0.8.0rc1) Using cached python_dateutil-2.4.2-py2.py3-none-any.whl Requirement already up-to-date: six==1.9.0 in e:\anaconda\lib\site-packages (fro m zipline==0.8.0rc1) Collecting requests==2.7.0 (from zipline==0.8.0rc1) Using cached requests-2.7.0-py2.py3-none-any.whl Requirement already up-to-date: Cython==0.22.1 in e:\anaconda\lib\site-packages (from zipline==0.8.0rc1) Collecting cyordereddict==0.2.2 (from zipline==0.8.0rc1) Collecting bottleneck==1.0.0 (from zipline==0.8.0rc1) Collecting contextlib2==0.4.0 (from zipline==0.8.0rc1) Requirement already up-to-date: networkx==1.9.1 in e:\anaconda\lib\site-packages (from zipline==0.8.0rc1) Collecting numexpr==2.4.3 (from zipline==0.8.0rc1) Using cached numexpr-2.4.3.tar.gz Collecting bcolz==0.10.0 (from zipline==0.8.0rc1) Collecting click==4.0.0 (from zipline==0.8.0rc1) Using cached click-4.0-py2.py3-none-any.whl Collecting toolz==0.7.4 (from zipline==0.8.0rc1) Collecting sqlalchemy==1.0.8 (from zipline==0.8.0rc1) Building wheels for collected packages: scipy, numexpr Running setup.py bdist_wheel for scipy Complete output from command e:\anaconda\python.exe -c "import setuptools;fi le='c:\users\leo\appdata\local\temp\pip-build-xppblr\scipy\setup.py';e xec(compile(open(file).read().replace('\r\n', '\n'), file, 'exec'))" bdi st_wheel -d c:\users\leo\appdata\local\temp\tmp5jhfhkpip-wheel-: lapack_opt_info: openblas_info: libraries not found in ['e:\anaconda\lib', 'C:\', 'e:\anaconda\libs'] NOT AVAILABLE
lapack_mkl_info: mkl_info: libraries mkl_lapack95,mkl_blas95,mkl_intel_c,mkl_intel_thread,mkl_core,libi omp5md,libifportmd not found in [] NOT AVAILABLE
NOT AVAILABLE
atlas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in e:\anaconda\lib libraries lapack_atlas not found in e:\anaconda\lib libraries ptf77blas,ptcblas,atlas not found in C:\ libraries lapack_atlas not found in C:\ libraries ptf77blas,ptcblas,atlas not found in e:\anaconda\libs libraries lapack_atlas not found in e:\anaconda\libs numpy.distutils.system_info.atlas_threads_info NOT AVAILABLE
atlas_info: libraries f77blas,cblas,atlas not found in e:\anaconda\lib libraries lapack_atlas not found in e:\anaconda\lib libraries f77blas,cblas,atlas not found in C:\ libraries lapack_atlas not found in C:\ libraries f77blas,cblas,atlas not found in e:\anaconda\libs libraries lapack_atlas not found in e:\anaconda\libs numpy.distutils.system_info.atlas_info NOT AVAILABLE
lapack_info: libraries lapack not found in ['e:\anaconda\lib', 'C:\', 'e:\anaconda\l ibs'] NOT AVAILABLE
lapack_src_info: NOT AVAILABLE
NOT AVAILABLE
e:\anaconda\lib\site-packages\numpy\distutils\system_info.py:576: UserWarning: Specified path C:/Program Files (x86)/Intel/Composer XE/mkl/lib/ia32 is invalid . warnings.warn('Specified path %s is invalid.' % d) e:\anaconda\lib\site-packages\numpy\distutils\system_info.py:576: UserWarning: Specified path C:/Program Files (x86)/Intel/Composer XE/compiler/lib/ia32 is in valid. warnings.warn('Specified path %s is invalid.' % d) e:\anaconda\lib\site-packages\numpy\distutils\system_info.py:576: UserWarning: Specified path C:/Program Files (x86)/Intel/Composer XE/mkl/include is invalid.
warnings.warn('Specified path %s is invalid.' % d)
e:\anaconda\lib\site-packages\numpy\distutils\system_info.py:1427: UserWarning
:
Atlas (http://math-atlas.sourceforge.net/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [atlas]) or by setting
the ATLAS environment variable.
warnings.warn(AtlasNotFoundError.doc)
e:\anaconda\lib\site-packages\numpy\distutils\system_info.py:1438: UserWarning
:
Lapack (http://www.netlib.org/lapack/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [lapack]) or by setting
the LAPACK environment variable.
warnings.warn(LapackNotFoundError.doc)
e:\anaconda\lib\site-packages\numpy\distutils\system_info.py:1441: UserWarning
:
Lapack (http://www.netlib.org/lapack/) sources not found.
Directories to search for the sources can be specified in the
numpy/distutils/site.cfg file (section [lapack_src]) or by setting
the LAPACK_SRC environment variable.
warnings.warn(LapackSrcNotFoundError.doc)
Running from scipy source directory.
Traceback (most recent call last):
File "
Failed building wheel for scipy Running setup.py bdist_wheel for numexpr Complete output from command e:\anaconda\python.exe -c "import setuptools;fi le='c:\users\leo\appdata\local\temp\pip-build-xppblr\numexpr\setup.py' ;exec(compile(open(file).read().replace('\r\n', '\n'), file, 'exec'))" b dist_wheel -d c:\users\leo\appdata\local\temp\tmpictdfipip-wheel-: Appending numexpr.tests configuration to numexpr Ignoring attempt to set 'name' (from 'numexpr' to 'numexpr.tests') running bdist_wheel running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler op tions running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler o ptions running build_src build_src building py_modules sources creating build creating build\src.win32-2.7 creating build\src.win32-2.7\numexpr building extension "numexpr.interpreter" sources build_src: building npy-pkg config files running build_py creating build\lib.win32-2.7 creating build\lib.win32-2.7\numexpr copying numexpr\cpuinfo.py -> build\lib.win32-2.7\numexpr copying numexpr\expressions.py -> build\lib.win32-2.7\numexpr copying numexpr\necompiler.py -> build\lib.win32-2.7\numexpr copying numexpr\utils.py -> build\lib.win32-2.7\numexpr copying numexpr\version.py -> build\lib.win32-2.7\numexpr copying numexprinit.py -> build\lib.win32-2.7\numexpr copying build\src.win32-2.7\numexprconfig.py -> build\lib.win32-2.7\numex pr creating build\lib.win32-2.7\numexpr\tests copying .\numexpr\tests\test_numexpr.py -> build\lib.win32-2.7\numexpr\tests copying .\numexpr\testsinit.py -> build\lib.win32-2.7\numexpr\tests running build_ext No module named msvccompiler in numpy.distutils; trying from distutils customize MSVCCompiler customize MSVCCompiler using build_ext No module named msvccompiler in numpy.distutils; trying from distutils customize MSVCCompiler Missing compiler_cxx fix for MSVCCompiler customize MSVCCompiler using build_ext building 'numexpr.interpreter' extension compiling C sources creating build\temp.win32-2.7 creating build\temp.win32-2.7\Release creating build\temp.win32-2.7\Release\numexpr creating build\temp.win32-2.7\Release\numexpr\win32 C:\Users\leo\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0 \VC\Bin\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Ie:\anaconda\lib\site-packa ges\numpy\core\include -Ie:\anaconda\include -Ie:\anaconda\PC /Tcnumexpr\win32\p thread.c /Fobuild\temp.win32-2.7\Release\numexpr\win32\pthread.obj /Zm1000 Found executable C:\Users\leo\AppData\Local\Programs\Common\Microsoft\Visual C ++ for Python\9.0\VC\Bin\cl.exe Warning: Assuming default configuration (numexpr\tests/{setup_tests,setup}.py was not found)C:\Users\leo\AppData\Local\Programs\Common\Microsoft\Visual C++ fo r Python\9.0\VC\Bin\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Ie:\anaconda\li b\site-packages\numpy\core\include -Ie:\anaconda\include -Ie:\anaconda\PC /Tpnum expr\interpreter.cpp /Fobuild\temp.win32-2.7\Release\numexpr\interpreter.obj /Zm 1000 interpreter.cpp e:\anaconda\lib\site-packages\numpy\core\include\numpy\npy_1_7_deprecated_api. h(12) : Warning Msg: Using deprecated NumPy API, disable it by #defining NPYNO DEPRECATED_API NPY_1_7_API_VERSION c:\users\leo\appdata\local\temp\pip-build-xppblr\numexpr\numexpr\str-two-way.h pp(33) : fatal error C1083: Cannot open include file: 'stdint.h': No such file o r directory error: Command "C:\Users\leo\AppData\Local\Programs\Common\Microsoft\Visual C+
for Python\9.0\VC\Bin\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Ie:\anacond a\lib\site-packages\numpy\core\include -Ie:\anaconda\include -Ie:\anaconda\PC /T pnumexpr\interpreter.cpp /Fobuild\temp.win32-2.7\Release\numexpr\interpreter.obj /Zm1000" failed with exit status 2
Failed building wheel for numexpr Failed to build scipy numexpr Installing collected packages: numpy, scipy, python-dateutil, pandas, patsy, req uests, cyordereddict, bottleneck, contextlib2, numexpr, bcolz, click, toolz, sql alchemy, zipline Found existing installation: numpy 1.8.2 DEPRECATION: Uninstalling a distutils installed project (numpy) has been dep recated and will be removed in a future version. This is due to the fact that un installing a distutils project will only partially uninstall the project. Uninstalling numpy-1.8.2: Successfully uninstalled numpy-1.8.2 Rolling back uninstall of numpy Exception: Traceback (most recent call last): File "e:\anaconda\lib\site-packages\pip\basecommand.py", line 211, in main status = self.run(options, args) File "e:\anaconda\lib\site-packages\pip\commands\install.py", line 311, in run
root=options.root_path, File "e:\anaconda\lib\site-packages\pip\req\req_set.py", line 646, in install **kwargs File "e:\anaconda\lib\site-packages\pip\req\req_install.py", line 803, in inst all self.move_wheel_files(self.source_dir, root=root) File "e:\anaconda\lib\site-packages\pip\req\req_install.py", line 998, in move _wheel_files isolated=self.isolated, File "e:\anaconda\lib\site-packages\pip\wheel.py", line 339, in move_wheel_fil es clobber(source, lib_dir, True) File "e:\anaconda\lib\site-packages\pip\wheel.py", line 317, in clobber shutil.copyfile(srcfile, destfile) File "e:\anaconda\lib\shutil.py", line 83, in copyfile with open(dst, 'wb') as fdst: IOError: [Errno 13] Permission denied: 'e:\anaconda\Lib\site-packages\numpy\
C:\Windows\system32>
Hello,
Could zipline upload the offial build (zipline>0.70) that can fix this problem. That I can use pip install to fix this problem.
Thank you very much.
@fraka6 @easytrader the zipline on PyPI is now on 0.8.3, and the docs on www.zipline.io are now being regularly updated. Hopefully that helps things going forward. Going to close this issue. If you run into new issues, feel free to reopen this or a new issue.
cannot get the simplest zipline stuff working. What a frustrating experience !!!
(quant_trader) c:\Users\lysak\Desktop>zipline
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\envs\quant_trader\Scripts\zipline-script.py", line 6, in
zipline examples do not work without ta-lib.
In fact, zipline.examples are missing when I type import zipline.examples
Better instructions are needed with zipline to install ta-lib and ALL other dependencies !!
ImportError Traceback (most recent call last)
Hi guys,
Please update http://www.zipline.io/ because dual_moving_avg.py doesn't exist, its dual_moving_average.py. This strategy work perfectly in quantopian environment but not with zipline CLI. I suspect the ichart.finance.yahoo path to be broken (see details at the end).
The only example that works or don't crash is buyapple.
dual_ema_talib.py -> ImportError: No module named talib pairtrade.py-> ValueError: You must define a handle_data function. olmar.py-> AttributeError: 'TradingAlgorithm' object has no attribute 'symbol' buyapple_analyze.py-> ValueError: You must define a handle_data function.
Thanks,
Fraka6
zipline.version Out[39]: '0.7.0'
python scripts/run_algo.py -f zipline/examples/dual_moving_average.py --symbols AAPL --start 2011-1-1 --end 2012-1-1 -o dma.pickle ... [2015-07-30 04:14:04.415502] WARNING: Loader: No benchmark data found for date range. start_date=2015-07-30 00:00:00+00:00, end_date=2015-07-30 04:14:04.295081, url=http://ichart.finance.yahoo.com/table.csv?a=6&c=2015&b=30&e=30&d=6&g=d&f=2015&s=%5EGSPC [2015-07-30 04:14:24.025207] INFO: Performance: Simulated 252 trading days out of 252. [2015-07-30 04:14:24.025510] INFO: Performance: first open: 2011-01-03 14:31:00+00:00 [2015-07-30 04:14:24.025611] INFO: Performance: last close: 2011-12-30 21:00:00+00:00 Traceback (most recent call last): ... File "scripts/run_algo.py", line 25, in
File "pandas/hashtable.pyx", line 694, in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:12231)
KeyError: 'AAPL'