quantopian / zipline

Zipline, a Pythonic Algorithmic Trading Library
https://www.zipline.io
Apache License 2.0
17.69k stars 4.73k forks source link

https://github.com/quantopian/zipline/wiki/Development-Environment #707

Closed AnthonyFJGarner closed 9 years ago

AnthonyFJGarner commented 9 years ago

I'm using Windows 8.1. I have anconda ,win-32, Python 2.7. I have zipline set up in C:\Application\github\zipline via Source Tree.

With great difficulty I set up zipline in development mode as follows: pip install -e C:\Application\github\zipline

I therefore appear to have all the dependencies.

Unfortunately while running examples I get stopped out at the following line:

File "C:\Anaconda\lib\site-packages\sqlalchemy\sql\compiler.py", line 1835, in visit_insert self.dialect.name)

CompileError: The 'sqlite' dialect with current database version settings does not support in-place multirow inserts

I am keen to help develop zipline and have a great deal of experience in system design and indeed trading. I was working with a conda download of zipline 0.7.0 until I discovered that many of the features I need to use are not included.

I can go and try and follow the instructions at https://github.com/quantopian/zipline/wiki/Development-Environment if that is what you recommend. But I have to say this is not an easy process! My partner Michael Bennett is an IT specialist based in Sydney but I would rather handle the installation on my own as part of the learning process.

I have very extensive experience in coding systems in Trading Blox and Mechanica but believe opensource is the way to go. I also intend to port systems between the development environment and Quantopian.

Any help gratefully accepted to get me over this initial frustration.

Zigguratti commented 9 years ago

I found installation a breeze and am currently using my own price and fundamental source without any issues (yet). At the risk of being naive (I am new to python, whereas you sound expert), could it be a problem with sqllite? sqllite is great but suffers certain limitations that other databases do not ... perhaps worth noting that I'm on python 3.4 and used pip install as per the tutorial (I tried installing with conda originally but had some issues). I'm also on windows 7 64 bit at the moment, although I may well move to ubuntu in due course. So far, everything is fine.

ssanderson commented 9 years ago

Zipline uses Python's built-in SQLite functionality for representing asset metadata (e.g. symbol <-> unique identifier mappings). @AnthonyFJGarner it sounds like your stdlib is linking against an older-than-expected version of SQLite. Can you post the version you're on? I get this when developing locally:

In [5]: import sqlite3

In [6]: sqlite3.version_info
Out[6]: (2, 6, 0)

Also, can you include the rest of the stack trace that you get when encountering the error? I would imagine it's bubbling out of assets.py somewhere, but it'd be informative to see the rest of the trace.

AnthonyFJGarner commented 9 years ago

Thank you gentlemen. I'm no expert I fear at anything other than the financial markets.

In [2]: runfile('C:/Application/github/zipline/zipline/examples/dual_ema_talib.py', wdir='C:/Application/github/zipline/zipline/examples') Reloaded modules: zipline.data.loader_utils, zipline.finance.performance.period, zipline.sources, zipline.data, zipline.assets.asset_writer, zipline.api, zipline.gens.utils, zipline, zipline.sources.data_source, zipline.modelling.expression, zipline.utils.algo_instance, zipline.assets.assets, zipline.utils.munge, zipline.modelling.graph, zipline.finance.controls, zipline.transforms.ta, zipline.utils.tradingcalendar, zipline.lib.rank, zipline.utils.events, zipline.history, zipline.finance.blotter, zipline.history.history, zipline.utils.factory, zipline.data.benchmarks, zipline.utils.serialization_utils, zipline.modelling.filter, zipline.finance.risk.period, zipline.transforms, zipline.errors, zipline.finance, zipline.gens.composites, zipline.assets, zipline.lib, zipline.gens, zipline.modelling.factor, zipline.modelling, zipline.finance.execution, zipline.lib.adjusted_array, zipline.transforms.batch_transform, zipline.utils.control_flow, zipline.sources.data_frame_source, zipline.modelling.engine, zipline.finance.slippage, zipline.modelling.factor.factor, zipline.finance.risk, zipline.gens.tradesimulation, zipline.algorithm, zipline.finance.risk.cumulative, zipline.utils, zipline.utils.math_utils, zipline.modelling.term, zipline.finance.risk.risk, zipline.assets.futures, zipline.utils.protocol_utils, zipline.finance.performance.position_tracker, zipline.utils.cli, zipline.finance.trading, zipline.assets._assets, zipline.finance.commission, zipline.finance.risk.report, zipline.finance.performance.position, zipline.utils.api_support, zipline.finance.performance.tracker, zipline.finance.performance, zipline.utils.lazyval, zipline.modelling.classifier, zipline.data.paths, zipline.sources.simulated, zipline.utils.data, zipline.sources.test_source, zipline.history.history_container, zipline.data.loader, zipline.protocol AAPL Traceback (most recent call last): File "", line 1, in runfile('C:/Application/github/zipline/zipline/examples/dual_ema_talib.py', wdir='C:/Application/github/zipline/zipline/examples') File "C:\Anaconda\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 685, in runfile execfile(filename, namespace) File "C:\Anaconda\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 71, in execfile exec(compile(scripttext, filename, 'exec'), glob, loc) File "C:/Application/github/zipline/zipline/examples/dual_ema_talib.py", line 122, in identifiers=['AAPL']) File "c:\application\github\zipline\zipline\algorithm.py", line 204, in **init** futures_data=kwargs.pop('futures_metadata', {}), File "c:\application\github\zipline\zipline\finance\trading.py", line 185, in write_data allow_sid_assignment=allow_sid_assignment) File "c:\application\github\zipline\zipline\finance\trading.py", line 190, in _write_data_lists .write_all(self.engine, allow_sid_assignment=allow_sid_assignment) File "c:\application\github\zipline\zipline\assets\asset_writer.py", line 196, in write_all self._write_equities(data.equities, fuzzy_char, txn) File "c:\application\github\zipline\zipline\assets\asset_writer.py", line 236, in _write_equities self.equities.insert().values([record]).execute(bind=bind) File "C:\Anaconda\lib\site-packages\sqlalchemy\sql\base.py", line 386, in execute return e._execute_clauseelement(self, multiparams, params) File "C:\Anaconda\lib\site-packages\sqlalchemy\engine\base.py", line 1003, in _execute_clauseelement inline=len(distilled_params) > 1) File "", line 1, in File "C:\Anaconda\lib\site-packages\sqlalchemy\sql\elements.py", line 494, in compile return self._compiler(dialect, bind=bind, **kw) File "C:\Anaconda\lib\site-packages\sqlalchemy\sql\elements.py", line 500, in _compiler return dialect.statement_compiler(dialect, self, **kw) File "C:\Anaconda\lib\site-packages\sqlalchemy\sql\compiler.py", line 392, in **init** Compiled.**init**(self, dialect, statement, **kwargs) File "C:\Anaconda\lib\site-packages\sqlalchemy\sql\compiler.py", line 190, in **init** self.string = self.process(self.statement, **compile_kwargs) File "C:\Anaconda\lib\site-packages\sqlalchemy\sql\compiler.py", line 213, in process return obj._compiler_dispatch(self, **kwargs) File "C:\Anaconda\lib\site-packages\sqlalchemy\sql\visitors.py", line 81, in _compiler_dispatch return meth(self, **kw) File "C:\Anaconda\lib\site-packages\sqlalchemy\sql\compiler.py", line 1835, in visit_insert self.dialect.name) CompileError: The 'sqlite' dialect with current database version settings does not support in-place multirow inserts. In [3]: import sqlite3 In [4]: sqlite3.version_info Out[4]: (2, 6, 0) In [5]:
AnthonyFJGarner commented 9 years ago

Ooops - sorry. Told you I was no expert!

AnthonyFJGarner commented 9 years ago

I am working my way through "requires.txt" and checking again that I have the correct version of everything. Requires.txt contains much duplication. Is this intentional?

I note however from issue #655 that its not that simple! To quote:

We currently do a bizarre hodge-podge of conda installs and pip installs based on grepping from our >requirements...

Zigguratti commented 9 years ago

Indeed, I'm a fundamental investor with some quant skills, hence my interest in quantopian/zipline. Our needs straddle fundamental and systematic investing, and indeed, I'm a strong believer in the convergence of the two spaces over time. There are numerous merits to using zipline rather than re-inventing the wheel ... I hope!

AnthonyFJGarner commented 9 years ago

Meticulously installed exactly the same versions of everything in "requires.txt", deleted zipline, reinstalled - perfect, no complaints, all elements necessary.

And then precisely the same error and stack as above.

Scott - any ideas?

ssanderson commented 9 years ago

@AnthonyFJGarner I think I directed you to the wrong version check before. sqlite3.version is the version of the Python library, but the error you're getting is coming from the C extension used by the python library under the hood. What do you get from running this?

In [1]: import sqlite3

In [2]: sqlite3.sqlite_version
Out[2]: '3.8.2'
Zigguratti commented 9 years ago

Looking forward to quizzing you about all this when you are up and running if you don't mind Anthony!

Scott, has Quantopian looked into using Docker for zipline setup? it might make all this simpler, and easier to maintain going forwards ... I am going to be looking into it shortly myself.

On 16 September 2015 at 18:15, Scott Sanderson notifications@github.com wrote:

@AnthonyFJGarner https://github.com/AnthonyFJGarner I think I directed you to the wrong version check before. sqlite3.version is the version of the Python library, but the error you're getting is coming from the C extension used by the python library under the hood. What do you get from running this?

In [1]: import sqlite3

In [2]: sqlite3.sqlite_version Out[2]: '3.8.2'

— Reply to this email directly or view it on GitHub https://github.com/quantopian/zipline/issues/707#issuecomment-140809075.

ssanderson commented 9 years ago

@Zigguratti we're in fact using Docker in parts of our production stack.

I've found the experience of working with docker to be a mixed bag from a usability perspective. The users who would most benefit from a plug-and-play experience for Zipline are those using OSX or Windows, where setting up and using docker is significantly more complicated because it requires a virtual machine.

Using docker on Linux, by contrast, is pretty painless, but that's the environment where it's also easiest to set up Zipline; the biggest challenge is just knowing which packages to apt-get/yum/pacman install above and beyond Python.

AnthonyFJGarner commented 9 years ago

In [1]: import sqlite3

In [2]: sqlite3.sqlite_version Out[2]: '3.6.21'

AnthonyFJGarner commented 9 years ago

pip install sqlite3==3.8.2 ?

Or ain't it that simple?

AnthonyFJGarner commented 9 years ago

Ah....no, I see that it is not.....

AnthonyFJGarner commented 9 years ago

I can of course switch to Linux and will indeed load all this on my Ubuntu desktop at some stage. But somehow, I feel it ought to work in Windows and I hate failure and partial answers. Besides which it would be handy to have Zipline on my laptop and I don't want a dual boot on my laptop and I don't want to work on my laptop with a virtual environment.

Besides which my efforts may help others.

ssanderson commented 9 years ago

@AnthonyFJGarner I expect the simplest solution will probably be to update your Python distribution, either to the newly-released 3.5, or 2.7.10 if you have a specific reason (e.g. legacy code not compatible with Py3) to stay on Python 2.

If you have a strong reason to not want to upgrade your Python installation wholesale, this might be helpful to you for updating just your SQLite dll: http://lambdathefirst.blogspot.com/2011/01/how-to-upgrade-sqlite-that-comes-with.html. (Ignore the stuff about setting up WAL journalling. The point is that the author is showing how to upgrade the sqlite library used with Python in order to access new features.)

ssanderson commented 9 years ago

Actually, given that we're currently only building Travis for Python 3.3, I'd probably stick with 2.7.10 for now. We do have developers using Zipline with 3.4 if you're feeling ambitious though.

AnthonyFJGarner commented 9 years ago

Many thanks Scott. I'm on Python 2.7.10 and will stick to if I can. I'll try the helpful link (for which many thanks).

AnthonyFJGarner commented 9 years ago

I'm going to close this issue - if that is the correct etiquette. Scott's helpful advice to load JUST the updated SQLite dll has worked in so far as the following error has now been solved:

CompileError: The 'sqlite' dialect with current database version settings does not support in-place >multirow inserts.

A whole host of further problems have been revealed and none of the alogirthms which previously worked under Zipline 7 now work for me under 8. But I suspect I will be able to work my way through these given time and patience since at least zipline 8 now seems to have the correct versions of all dependencies and of the various constituent parts of Python and anaconda.

I will just say this to the Quantopian team:

You people have done an excellent job clearly to get this far and thank you for open sourcing this project. But, a word of advice from somebody with a great deal of market experience but little IT expertise. You need to be a lot more explicit about the environment necessary for zipline. You also need to use a lot less short hand in your expressions for the non experts. Terms of art which are fully understandable to you guys will not be comprehensible to people like myself.

If you want to attract people like me with market expertise but less IT knowledge you need to compile much better and more explicit instructions on exactly what the environment should look like. This thread is a classic example. I had included on my own initiative everything in "requires.txt" in the correct version. And yet that was not enough.

As instructed I had installed anaconda with python 2.7 and yet that proved wholly inadequate. You really need to provide people with a checklist of anaconda elements and their versions. Anyone downloading anaconda today will end up with a complete mess since so many elements will be incorrect for use with zipline.The mechanism included in zipline for un-installing incorrect elements and re-installing correct elements was a disaster - I had to do it all by hand and it took days.

This is not intended as a criticism - merely a helpful suggestion.

No doubt I will face many further frustrations due to my lack of experience but in my own case I shall persevere since I have the luxury of time and the necessity of interest.

I suspect there will be many people who could add a great deal to your endeavours who will have neither and will simply move on in disgust!

Hope this helps!