search

stefan-jansen / zipline-reloaded

Zipline, a Pythonic Algorithmic Trading Library
https://zipline.ml4trading.io
Apache License 2.0
1.03k stars 199 forks source link

Ingesting data for crypto assets yields UserWarning: Ignoring values because they are out of bounds for uint32 #216

Open phelps-sg opened 10 months ago

phelps-sg commented 10 months ago

Dear Zipline Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

* Operating System: (Windows Version or `$ uname --all`) Linux pop-os 6.4.6-76060406-generic #202307241739~1692717645~22.04~5597803 SMP PREEMPT_DYNAMIC Tue A x86_64 x86_64 x86_64 GNU/Linux * Python Version: `$ python --version` Python 3.10.12 * Python Bitness: `$ python -c 'import math, sys;print(int(math.log(sys.maxsize + 1, 2) + 1))'` 64 * How did you install Zipline: (`pip`, `conda`, or `other (please explain)`) mamba * Python packages: `$ pip freeze` or `$ conda list` _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_gnu conda-forge aiofiles 0.8.0 pyhd8ed1ab_0 conda-forge aiohttp 3.8.1 py310h5764c6d_1 conda-forge aiohttp-cors 0.7.0 py_0 conda-forge aiosignal 1.3.1 pyhd8ed1ab_0 conda-forge alembic 1.12.0 pyhd8ed1ab_0 conda-forge alsa-lib 1.2.10 hd590300_0 conda-forge anaconda-client 1.12.0 pyhd8ed1ab_1 conda-forge anaconda-project 0.11.1 pyhd8ed1ab_0 conda-forge aniso8601 9.0.1 pyhd8ed1ab_0 conda-forge anyio 4.0.0 pyhd8ed1ab_0 conda-forge appdirs 1.4.4 pyh9f0ad1d_0 conda-forge argon2-cffi 23.1.0 pyhd8ed1ab_0 conda-forge argon2-cffi-bindings 21.2.0 py310h5764c6d_3 conda-forge arrow 1.2.3 pyhd8ed1ab_0 conda-forge astroid 2.15.6 py310hff52083_0 conda-forge asttokens 2.4.0 pyhd8ed1ab_0 conda-forge async-lru 2.0.4 pyhd8ed1ab_0 conda-forge async-timeout 4.0.3 pyhd8ed1ab_0 conda-forge attr 2.5.1 h166bdaf_1 conda-forge attrs 23.1.0 pyh71513ae_1 conda-forge babel 2.12.1 pyhd8ed1ab_1 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 pyhd8ed1ab_3 conda-forge backports.functools_lru_cache 1.6.5 pyhd8ed1ab_0 conda-forge bcolz-zipline 1.2.6 py310h0a54255_0 conda-forge beautifulsoup4 4.12.2 pyha770c72_0 conda-forge black 23.9.1 py310hff52083_0 conda-forge blackd 23.9.1 hd8ed1ab_0 conda-forge bleach 6.0.0 pyhd8ed1ab_0 conda-forge blessed 1.19.1 pyhe4f9e05_2 conda-forge blinker 1.6.2 pyhd8ed1ab_0 conda-forge blosc 1.21.5 h0f2a231_0 conda-forge bottleneck 1.3.7 py310h0a54255_0 conda-forge brotli 1.1.0 hd590300_0 conda-forge brotli-bin 1.1.0 hd590300_0 conda-forge brotlipy 0.7.0 py310h5764c6d_1005 conda-forge bzip2 1.0.8 h7f98852_4 conda-forge c-ares 1.19.1 hd590300_0 conda-forge c-blosc2 2.10.2 hb4ffafa_0 conda-forge ca-certificates 2023.7.22 hbcca054_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge cachelib 0.9.0 pyhd8ed1ab_0 conda-forge cachetools 5.3.1 pyhd8ed1ab_0 conda-forge cairo 1.16.0 h0c91306_1017 conda-forge certifi 2023.7.22 pyhd8ed1ab_0 conda-forge cffi 1.15.1 py310h255011f_3 conda-forge cfgv 3.3.1 pyhd8ed1ab_0 conda-forge charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge click 8.1.7 unix_pyh707e725_0 conda-forge clyent 1.2.2 py_1 conda-forge colorama 0.4.6 pyhd8ed1ab_0 conda-forge colorful 0.5.4 pyhd8ed1ab_0 conda-forge comm 0.1.4 pyhd8ed1ab_0 conda-forge conda-pack 0.7.1 pyhd8ed1ab_0 conda-forge conda-package-handling 2.2.0 pyh38be061_0 conda-forge conda-package-streaming 0.9.0 pyhd8ed1ab_0 conda-forge contourpy 1.1.0 py310hd41b1e2_0 conda-forge cryptography 41.0.3 py310h75e40e8_0 conda-forge cycler 0.11.0 pyhd8ed1ab_0 conda-forge dbus 1.13.6 h5008d03_3 conda-forge debugpy 1.8.0 py310hc6cd4ac_0 conda-forge decorator 5.1.1 pyhd8ed1ab_0 conda-forge defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge dill 0.3.7 pyhd8ed1ab_0 conda-forge distlib 0.3.7 pyhd8ed1ab_0 conda-forge dnspython 2.4.2 pyhd8ed1ab_0 conda-forge docker-pycreds 0.4.0 py_0 conda-forge dominate 2.8.0 pyhd8ed1ab_0 conda-forge email-validator 2.0.0.post2 pyhd8ed1ab_0 conda-forge email_validator 2.0.0.post2 hd8ed1ab_0 conda-forge empyrical-reloaded 0.5.9 pyhd8ed1ab_0 conda-forge entrypoints 0.4 pyhd8ed1ab_0 conda-forge et_xmlfile 1.1.0 pyhd8ed1ab_0 conda-forge exceptiongroup 1.1.3 pyhd8ed1ab_0 conda-forge exchange-calendars 4.2.8 pyhd8ed1ab_1 conda-forge executing 1.2.0 pyhd8ed1ab_0 conda-forge expat 2.5.0 hcb278e6_1 conda-forge filelock 3.12.4 pyhd8ed1ab_0 conda-forge flask 2.3.3 pyhd8ed1ab_0 conda-forge flask-bootstrap 3.3.7.1 py_0 conda-forge flask-caching 2.0.2 pyhd8ed1ab_0 conda-forge flask-jwt-extended 4.5.2 pyhd8ed1ab_0 conda-forge flask-migrate 4.0.5 pyhd8ed1ab_0 conda-forge flask-restful 0.3.10 pyhd8ed1ab_0 conda-forge flask-sqlalchemy 3.0.3 pyhd8ed1ab_0 conda-forge flask-wtf 1.1.1 pyhd8ed1ab_0 conda-forge fn 0.4.3 pyhd8ed1ab_0 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 hab24e00_0 conda-forge fontconfig 2.14.2 h14ed4e7_0 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.42.1 py310h2372a71_0 conda-forge fqdn 1.5.1 pyhd8ed1ab_0 conda-forge freetype 2.12.1 h267a509_2 conda-forge frozendict 2.3.8 py310h2372a71_0 conda-forge frozenlist 1.4.0 py310h2372a71_0 conda-forge gettext 0.21.1 h27087fc_0 conda-forge gitdb 4.0.10 pyhd8ed1ab_0 conda-forge gitpython 3.1.36 pyhd8ed1ab_0 conda-forge glib 2.78.0 hfc55251_0 conda-forge glib-tools 2.78.0 hfc55251_0 conda-forge gmp 6.2.1 h58526e2_0 conda-forge google-api-core 2.11.1 pyhd8ed1ab_0 conda-forge google-auth 2.23.0 pyh1a96a4e_0 conda-forge googleapis-common-protos 1.60.0 pyhd8ed1ab_0 conda-forge gpustat 1.1.1 pyhd8ed1ab_0 conda-forge graphite2 1.3.13 h58526e2_1001 conda-forge greenlet 2.0.2 py310hc6cd4ac_1 conda-forge grpcio 1.43.0 py310ha6b4e5e_0 conda-forge gst-plugins-base 1.22.5 h8e1006c_1 conda-forge gstreamer 1.22.5 h98fc4e7_1 conda-forge gunicorn 21.2.0 py310hff52083_0 conda-forge h11 0.14.0 pyhd8ed1ab_0 conda-forge h2 4.1.0 pyhd8ed1ab_0 conda-forge h5py 3.9.0 nompi_py310ha2ad45a_102 conda-forge harfbuzz 8.2.0 h3d44ed6_0 conda-forge hdf5 1.14.2 nompi_h4f84152_100 conda-forge hpack 4.0.0 pyh9f0ad1d_0 conda-forge html5lib 1.1 pyh9f0ad1d_0 conda-forge httpcore 0.18.0 pyhd8ed1ab_0 conda-forge hyperframe 6.0.1 pyhd8ed1ab_0 conda-forge icu 73.2 h59595ed_0 conda-forge identify 2.5.29 pyhd8ed1ab_0 conda-forge idna 3.4 pyhd8ed1ab_0 conda-forge importlib-metadata 6.8.0 pyha770c72_0 conda-forge importlib_metadata 6.8.0 hd8ed1ab_0 conda-forge importlib_resources 6.0.1 pyhd8ed1ab_0 conda-forge iniconfig 2.0.0 pyhd8ed1ab_0 conda-forge intervaltree 3.1.0 pyhd8ed1ab_1 conda-forge ipykernel 6.25.2 pyh2140261_0 conda-forge ipython 8.15.0 pyh0d859eb_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge ipywidgets 8.1.1 pyhd8ed1ab_0 conda-forge iso3166 2.1.1 pyhd8ed1ab_0 conda-forge iso4217 1.9.20220401 pyhd8ed1ab_0 conda-forge isoduration 20.11.0 pyhd8ed1ab_0 conda-forge isort 5.12.0 pyhd8ed1ab_1 conda-forge itsdangerous 2.1.2 pyhd8ed1ab_0 conda-forge jedi 0.19.0 pyhd8ed1ab_0 conda-forge jinja2 3.1.2 pyhd8ed1ab_1 conda-forge joblib 1.3.2 pyhd8ed1ab_0 conda-forge json5 0.9.14 pyhd8ed1ab_0 conda-forge jsonpointer 2.4 py310hff52083_2 conda-forge jsonschema 4.19.0 pyhd8ed1ab_1 conda-forge jsonschema-specifications 2023.7.1 pyhd8ed1ab_0 conda-forge jsonschema-with-format-nongpl 4.19.0 pyhd8ed1ab_1 conda-forge jupyter 1.0.0 py310hff52083_8 conda-forge jupyter-lsp 2.2.0 pyhd8ed1ab_0 conda-forge jupyter_client 8.3.1 pyhd8ed1ab_0 conda-forge jupyter_console 6.6.3 pyhd8ed1ab_0 conda-forge jupyter_core 5.3.1 py310hff52083_0 conda-forge jupyter_events 0.7.0 pyhd8ed1ab_2 conda-forge jupyter_server 2.7.3 pyhd8ed1ab_0 conda-forge jupyter_server_terminals 0.4.4 pyhd8ed1ab_1 conda-forge jupyterlab 4.0.6 pyhd8ed1ab_0 conda-forge jupyterlab_pygments 0.2.2 pyhd8ed1ab_0 conda-forge jupyterlab_server 2.25.0 pyhd8ed1ab_0 conda-forge jupyterlab_widgets 3.0.9 pyhd8ed1ab_0 conda-forge keyutils 1.6.1 h166bdaf_0 conda-forge kiwisolver 1.4.5 py310hd41b1e2_0 conda-forge korean_lunar_calendar 0.3.1 pyhd8ed1ab_0 conda-forge krb5 1.21.2 h659d440_0 conda-forge lame 3.100 h166bdaf_1003 conda-forge lazy-object-proxy 1.9.0 py310h1fa729e_0 conda-forge lcms2 2.15 h7f713cb_2 conda-forge ld_impl_linux-64 2.40 h41732ed_0 conda-forge lerc 4.0.0 h27087fc_0 conda-forge libabseil 20230802.0 cxx17_h59595ed_3 conda-forge libaec 1.0.6 hcb278e6_1 conda-forge libblas 3.9.0 18_linux64_openblas conda-forge libbrotlicommon 1.1.0 hd590300_0 conda-forge libbrotlidec 1.1.0 hd590300_0 conda-forge libbrotlienc 1.1.0 hd590300_0 conda-forge libcap 2.69 h0f662aa_0 conda-forge libcblas 3.9.0 18_linux64_openblas conda-forge libclang 15.0.7 default_h7634d5b_3 conda-forge libclang13 15.0.7 default_h9986a30_3 conda-forge libcups 2.3.3 h4637d8d_4 conda-forge libcurl 8.3.0 hca28451_0 conda-forge libdeflate 1.18 h0b41bf4_0 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 h516909a_1 conda-forge libevent 2.1.12 hf998b51_1 conda-forge libexpat 2.5.0 hcb278e6_1 conda-forge libffi 3.4.2 h7f98852_5 conda-forge libflac 1.4.3 h59595ed_0 conda-forge libgcc-ng 13.2.0 h807b86a_1 conda-forge libgcrypt 1.10.1 h166bdaf_0 conda-forge libgfortran-ng 13.2.0 h69a702a_1 conda-forge libgfortran5 13.2.0 ha4646dd_1 conda-forge libglib 2.78.0 hebfc3b9_0 conda-forge libgomp 13.2.0 h807b86a_1 conda-forge libgpg-error 1.47 h71f35ed_0 conda-forge libiconv 1.17 h166bdaf_0 conda-forge libjpeg-turbo 2.1.5.1 hd590300_1 conda-forge liblapack 3.9.0 18_linux64_openblas conda-forge libllvm15 15.0.7 h5cf9203_3 conda-forge libnghttp2 1.52.0 h61bc06f_0 conda-forge libnsl 2.0.0 h7f98852_0 conda-forge libogg 1.3.4 h7f98852_1 conda-forge libopenblas 0.3.24 pthreads_h413a1c8_0 conda-forge libopus 1.3.1 h7f98852_1 conda-forge libpng 1.6.39 h753d276_0 conda-forge libpq 15.4 hfc447b1_0 conda-forge libprotobuf 4.23.4 hf27288f_6 conda-forge libsndfile 1.2.2 hbc2eb40_0 conda-forge libsodium 1.0.18 h36c2ea0_1 conda-forge libsqlite 3.43.0 h2797004_0 conda-forge libssh2 1.11.0 h0841786_0 conda-forge libstdcxx-ng 13.2.0 h7e041cc_1 conda-forge libsystemd0 254 h3516f8a_0 conda-forge libta-lib 0.4.0 h166bdaf_1 conda-forge libtiff 4.6.0 h8b53f26_0 conda-forge libunwind 1.6.2 h9c3ff4c_0 conda-forge libuuid 2.38.1 h0b41bf4_0 conda-forge libuv 1.46.0 hd590300_0 conda-forge libvorbis 1.3.7 h9c3ff4c_0 conda-forge libwebp-base 1.3.2 hd590300_0 conda-forge libxcb 1.15 h0b41bf4_0 conda-forge libxkbcommon 1.5.0 h5d7e998_3 conda-forge libxml2 2.11.5 h232c23b_1 conda-forge libxslt 1.1.37 h0054252_1 conda-forge libzlib 1.2.13 hd590300_5 conda-forge lru-dict 1.2.0 py310h2372a71_0 conda-forge lxml 4.9.3 py310h9b7343a_0 conda-forge lz4-c 1.9.4 hcb278e6_0 conda-forge lzo 2.10 h516909a_1000 conda-forge mako 1.2.4 pyhd8ed1ab_0 conda-forge markupsafe 2.1.3 py310h2372a71_0 conda-forge matplotlib-base 3.7.2 py310hf38f957_0 conda-forge matplotlib-inline 0.1.6 pyhd8ed1ab_0 conda-forge mccabe 0.7.0 pyhd8ed1ab_0 conda-forge mistune 3.0.1 pyhd8ed1ab_0 conda-forge mpg123 1.31.3 hcb278e6_0 conda-forge msgpack-python 1.0.5 py310hdf3cbec_0 conda-forge multidict 6.0.4 py310h1fa729e_0 conda-forge multipledispatch 0.6.0 py_0 conda-forge multitasking 0.0.9 pyhd8ed1ab_0 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge mypy 1.5.1 py310h2372a71_0 conda-forge mypy_extensions 1.0.0 pyha770c72_0 conda-forge mysql-common 8.0.33 hf1915f5_4 conda-forge mysql-libs 8.0.33 hca2cd23_4 conda-forge nbclient 0.8.0 pyhd8ed1ab_0 conda-forge nbconvert 7.8.0 pyhd8ed1ab_0 conda-forge nbconvert-core 7.8.0 pyhd8ed1ab_0 conda-forge nbconvert-pandoc 7.8.0 pyhd8ed1ab_0 conda-forge nbformat 5.9.2 pyhd8ed1ab_0 conda-forge ncurses 6.4 hcb278e6_0 conda-forge nest-asyncio 1.5.6 pyhd8ed1ab_0 conda-forge networkx 3.1 pyhd8ed1ab_0 conda-forge nodeenv 1.8.0 pyhd8ed1ab_0 conda-forge nodejs 20.6.1 hb753e55_0 conda-forge nomkl 1.0 h5ca1d4c_0 conda-forge notebook 7.0.3 pyhd8ed1ab_0 conda-forge notebook-shim 0.2.3 pyhd8ed1ab_0 conda-forge nspr 4.35 h27087fc_0 conda-forge nss 3.92 h1d7d5a4_0 conda-forge numexpr 2.8.4 py310hd91493a_101 conda-forge numpy 1.25.2 py310ha4c1d20_0 conda-forge nvidia-ml-py 12.535.108 pyhd8ed1ab_0 conda-forge openai 0.28.0 pyhd8ed1ab_0 conda-forge opencensus 0.11.2 pyhd8ed1ab_0 conda-forge opencensus-context 0.1.3 py310hff52083_1 conda-forge openjpeg 2.5.0 h488ebb8_3 conda-forge openpyxl 3.1.2 py310h2372a71_0 conda-forge openssl 3.1.2 hd590300_0 conda-forge overrides 7.4.0 pyhd8ed1ab_0 conda-forge packaging 23.1 pyhd8ed1ab_0 conda-forge pandas 2.1.0 py310hcc13569_0 conda-forge pandas-datareader 0.10.0 pyh6c4a22f_0 conda-forge pandas-stubs 2.0.3.230814 pyhd8ed1ab_0 conda-forge pandoc 3.1.3 h32600fe_0 conda-forge pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge parso 0.8.3 pyhd8ed1ab_0 conda-forge pathspec 0.11.2 pyhd8ed1ab_0 conda-forge pathtools 0.1.2 py_1 conda-forge patsy 0.5.3 pyhd8ed1ab_0 conda-forge pcre2 10.40 hc3806b6_0 conda-forge pexpect 4.8.0 pyh1a96a4e_2 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 10.0.1 py310h29da1c1_0 conda-forge pip 23.2.1 pyhd8ed1ab_0 conda-forge pixman 0.40.0 h36c2ea0_0 conda-forge pkgutil-resolve-name 1.3.10 pyhd8ed1ab_1 conda-forge platformdirs 3.10.0 pyhd8ed1ab_0 conda-forge plotly 5.17.0 pyhd8ed1ab_0 conda-forge pluggy 1.3.0 pyhd8ed1ab_0 conda-forge ply 3.11 py_1 conda-forge pre-commit 3.4.0 pyha770c72_1 conda-forge prometheus_client 0.17.1 pyhd8ed1ab_0 conda-forge prompt-toolkit 3.0.39 pyha770c72_0 conda-forge prompt_toolkit 3.0.39 hd8ed1ab_0 conda-forge protobuf 4.23.4 py310h620c231_2 conda-forge psutil 5.9.5 py310h1fa729e_0 conda-forge pthread-stubs 0.4 h36c2ea0_1001 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pulseaudio-client 16.1 hb77b528_5 conda-forge pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge py-cpuinfo 9.0.0 pyhd8ed1ab_0 conda-forge py-spy 0.3.14 h87a5ac0_0 conda-forge pyasn1 0.4.8 py_0 conda-forge pyasn1-modules 0.2.7 py_0 conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pydantic 1.10.12 py310h2372a71_1 conda-forge pygments 2.16.1 pyhd8ed1ab_0 conda-forge pyjwt 2.8.0 pyhd8ed1ab_0 conda-forge pylint 2.17.5 pyhd8ed1ab_0 conda-forge pyluach 2.2.0 pyhd8ed1ab_0 conda-forge pyopenssl 23.2.0 pyhd8ed1ab_1 conda-forge pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge pyqt 5.15.9 py310h04931ad_4 conda-forge pyqt5-sip 12.12.2 py310hc6cd4ac_4 conda-forge pysocks 1.7.1 pyha2e5f31_6 conda-forge pytables 3.8.0 py310h374b01c_3 conda-forge pytest 7.4.2 pyhd8ed1ab_0 conda-forge pytest-mock 3.11.1 pyhd8ed1ab_0 conda-forge python 3.10.12 hd12c33a_0_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-fastjsonschema 2.18.0 pyhd8ed1ab_0 conda-forge python-interface 1.6.0 py_0 conda-forge python-json-logger 2.0.7 pyhd8ed1ab_0 conda-forge python-tzdata 2023.3 pyhd8ed1ab_0 conda-forge python_abi 3.10 3_cp310 conda-forge pytz 2023.3.post1 pyhd8ed1ab_0 conda-forge pyu2f 0.1.5 pyhd8ed1ab_0 conda-forge pyyaml 6.0.1 py310h2372a71_0 conda-forge pyzmq 25.1.1 py310h5bbb5d0_0 conda-forge qt-main 5.15.8 hc47bfe8_16 conda-forge qtconsole 5.4.4 pyhd8ed1ab_0 conda-forge qtconsole-base 5.4.4 pyha770c72_0 conda-forge qtpy 2.4.0 pyhd8ed1ab_0 conda-forge ray-core 2.6.3 py310h2ca9b2b_1 conda-forge ray-dashboard 2.6.3 py310ha1e06c9_1 conda-forge ray-default 2.6.3 py310hff52083_1 conda-forge readline 8.2 h8228510_1 conda-forge referencing 0.30.2 pyhd8ed1ab_0 conda-forge requests 2.31.0 pyhd8ed1ab_0 conda-forge requests-toolbelt 1.0.0 pyhd8ed1ab_0 conda-forge rfc3339-validator 0.1.4 pyhd8ed1ab_0 conda-forge rfc3986-validator 0.1.1 pyh9f0ad1d_0 conda-forge rpds-py 0.10.3 py310hcb5633a_0 conda-forge rsa 4.9 pyhd8ed1ab_0 conda-forge ruamel_yaml 0.15.80 py310h5764c6d_1008 conda-forge scikit-learn 1.3.0 py310hf7d194e_0 conda-forge scipy 1.11.2 py310hb13e2d6_1 conda-forge send2trash 1.8.2 pyh41d4057_0 conda-forge sentry-sdk 1.31.0 pyhd8ed1ab_0 conda-forge setproctitle 1.2.2 py310h5764c6d_2 conda-forge setuptools 68.2.2 pyhd8ed1ab_0 conda-forge sip 6.7.11 py310hc6cd4ac_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge smart_open 6.4.0 pyhd8ed1ab_0 conda-forge smmap 3.0.5 pyh44b312d_0 conda-forge snappy 1.1.10 h9fff704_0 conda-forge sniffio 1.3.0 pyhd8ed1ab_0 conda-forge sortedcontainers 2.1.0 py_0 conda-forge soupsieve 2.5 pyhd8ed1ab_1 conda-forge sqlalchemy 2.0.20 py310h2372a71_0 conda-forge stack_data 0.6.2 pyhd8ed1ab_0 conda-forge statsmodels 0.14.0 py310h278f3c1_1 conda-forge ta-lib 0.4.19 py310h278f3c1_4 conda-forge tardis-client 1.3.4 py310_0 mesonomics tardis-dev 2.0.0a13 py310_0 mesonomics tenacity 8.2.3 pyhd8ed1ab_0 conda-forge terminado 0.17.1 pyh41d4057_0 conda-forge threadpoolctl 3.2.0 pyha21a80b_0 conda-forge tinycss2 1.2.1 pyhd8ed1ab_0 conda-forge tk 8.6.12 h27826a3_0 conda-forge toml 0.10.2 pyhd8ed1ab_0 conda-forge tomli 2.0.1 pyhd8ed1ab_0 conda-forge tomlkit 0.12.1 pyha770c72_0 conda-forge toolz 0.12.0 pyhd8ed1ab_0 conda-forge tornado 6.3.3 py310h2372a71_0 conda-forge tqdm 4.66.1 pyhd8ed1ab_0 conda-forge traitlets 5.10.0 pyhd8ed1ab_0 conda-forge types-click 7.1.8 pyhd8ed1ab_0 conda-forge types-flask 1.1.6 pyhd8ed1ab_0 conda-forge types-jinja2 2.11.9 pyhd8ed1ab_0 conda-forge types-markupsafe 1.1.10 pyhd8ed1ab_0 conda-forge types-pytz 2023.3.0.1 pyhd8ed1ab_0 conda-forge types-werkzeug 1.0.9 pyhd8ed1ab_0 conda-forge typing-extensions 4.7.1 hd8ed1ab_0 conda-forge typing_extensions 4.7.1 pyha770c72_0 conda-forge typing_utils 0.1.0 pyhd8ed1ab_0 conda-forge tzdata 2023c h71feb2d_0 conda-forge ukkonen 1.0.1 py310hbf28c38_3 conda-forge unicodedata2 15.0.0 py310h5764c6d_0 conda-forge uri-template 1.3.0 pyhd8ed1ab_0 conda-forge urllib3 1.26.15 pyhd8ed1ab_0 conda-forge virtualenv 20.21.0 pyhd8ed1ab_0 conda-forge visitor 0.1.3 pyh9f0ad1d_1 conda-forge wandb 0.15.10 pyhd8ed1ab_0 conda-forge wcwidth 0.2.6 pyhd8ed1ab_0 conda-forge webcolors 1.13 pyhd8ed1ab_0 conda-forge webencodings 0.5.1 pyhd8ed1ab_2 conda-forge websocket-client 1.6.3 pyhd8ed1ab_0 conda-forge werkzeug 2.3.7 pyhd8ed1ab_0 conda-forge wheel 0.41.2 pyhd8ed1ab_0 conda-forge widgetsnbextension 4.0.9 pyhd8ed1ab_0 conda-forge wrapt 1.15.0 py310h1fa729e_0 conda-forge wtforms 3.0.1 pyhd8ed1ab_0 conda-forge xcb-util 0.4.0 hd590300_1 conda-forge xcb-util-image 0.4.0 h8ee46fc_1 conda-forge xcb-util-keysyms 0.4.0 h8ee46fc_1 conda-forge xcb-util-renderutil 0.3.9 hd590300_1 conda-forge xcb-util-wm 0.4.1 h8ee46fc_1 conda-forge xkeyboard-config 2.39 hd590300_0 conda-forge xorg-kbproto 1.0.7 h7f98852_1002 conda-forge xorg-libice 1.1.1 hd590300_0 conda-forge xorg-libsm 1.2.4 h7391055_0 conda-forge xorg-libx11 1.8.6 h8ee46fc_0 conda-forge xorg-libxau 1.0.11 hd590300_0 conda-forge xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge xorg-libxext 1.3.4 h0b41bf4_2 conda-forge xorg-libxrender 0.9.11 hd590300_0 conda-forge xorg-renderproto 0.11.1 h7f98852_1002 conda-forge xorg-xextproto 7.3.0 h0b41bf4_1003 conda-forge xorg-xf86vidmodeproto 2.3.1 h7f98852_1002 conda-forge xorg-xproto 7.0.31 h7f98852_1007 conda-forge xz 5.2.6 h166bdaf_0 conda-forge yaml 0.2.5 h7f98852_2 conda-forge yarl 1.9.2 py310h2372a71_0 conda-forge yfinance 0.2.28 pyhd8ed1ab_0 conda-forge zeromq 4.3.4 h9c3ff4c_1 conda-forge zipline-reloaded 3.0.3 py310h278f3c1_0 conda-forge zipline-tardis-bundle 0.3.1 py310he53d0f1_0 mesonomics zipp 3.16.2 pyhd8ed1ab_0 conda-forge zlib 1.2.13 hd590300_5 conda-forge zlib-ng 2.0.7 h0b41bf4_0 conda-forge zstandard 0.19.0 py310h5764c6d_0 conda-forge zstd 1.5.5 hfc55251_0 conda-forge

Now that you know a little about me, let me tell you about the issue I am having:

Description of Issue

When ingesting data for crypto pairs we often encounter large volumes which cannot be represented in uint32.

/home/sphelps/mambaforge/envs/zipline-tardis-bundle/lib/python3.10/site-packages/zipline/data/bcolz_daily_bars.py:341: UserWarning: Ignoring 422 values because they are out of bounds for uint32:                                                                                                                                                                                                                        open      high       low     close       volume
timestamp                                                                     
2022-03-30 00:00:00+00:00  0.087520  0.097370  0.084210  0.092865   9209813515
2022-03-31 00:00:00+00:00  0.092865  0.098105  0.088565  0.095280   7058370768
2022-04-01 00:00:00+00:00  0.095280  0.096950  0.088730  0.095310   7066482428
2022-04-02 00:00:00+00:00  0.095310  0.102800  0.094340  0.096920   9800596381
2022-04-03 00:00:00+00:00  0.096920  0.100835  0.094075  0.097590   6914043464
...                             ...       ...       ...       ...          ...
2023-05-27 00:00:00+00:00  0.025160  0.025505  0.025025  0.025425   5587944209
2023-05-28 00:00:00+00:00  0.025425  0.026690  0.025300  0.026265   6736129138                
2023-05-29 00:00:00+00:00  0.026265  0.027575  0.025835  0.026465  14296257656
2023-05-30 00:00:00+00:00  0.026465  0.026715  0.026115  0.026485  13632077291
2023-05-31 00:00:00+00:00  0.026485  0.026715  0.025165  0.025490  14446139237

Here is how you can reproduce this issue on your machine:

Reproduction Steps

  1. Install zipline-tardis-bundle
  2. Ingest coins with large volumes
  3. Above warning is displayed for coins with large volumes ...

What steps have you taken to resolve this already?

...

Anything else?

Can we use 64-bit int for representing volumes to support crypto? ...

Sincerely, Steve Phelps

MBounouar commented 10 months ago

Hi, By design zipline has put constraints on how the data is stored. Unfortunately, one would have to do the changes in the code. I am working on switching to parquet and relax some constraints, but I haven't had much time to finalize this.