stefan-jansen
commented
3 years ago Thanks for letting us know. However, I wasn't able replicate your experience; when testing on Windows and Ubuntu, the ZIPLINE_ROOT environment variable settings got picked up correctly when referring to an external drive. In other words, zipline bundles shows the location on the external drive and zipline ingest stores data there instead under the home directory.
It would be surprising if this had broken because the relevant code is unchanged. Please let me know if this issue persists after the usual suspect fixes like restarting shell, reinstalling zipline/virtual env etc, and I'll take another look.
EcoFin
Dear Zipline Maintainers,
Before I tell you about my issue, let me describe my environment:
Environment
3.8.8alembic 1.5.8 pyhd3eb1b0_0
anyio 2.2.0 py38haa95532_1
argon2-cffi 20.1.0 py38h2bbff1b_1
arviz 0.11.2 pyhd3eb1b0_0
async_generator 1.10 pyhd3eb1b0_0
attrs 20.3.0 pyhd3eb1b0_0
babel 2.9.0 pyhd3eb1b0_0
backcall 0.2.0 pyhd3eb1b0_0
bcolz-zipline 1.2.3 py38he6999da_0 ml4t blas 1.0 mkl
bleach 3.3.0 pyhd3eb1b0_0
blosc 1.21.0 h19a0ad4_0
bottleneck 1.3.2 py38h2a96729_1
brotlipy 0.7.0 py38h2bbff1b_1003
bzip2 1.0.8 he774522_0
ca-certificates 2021.4.13 haa95532_1
certifi 2020.12.5 py38haa95532_0
cffi 1.14.5 py38hcd4344a_0
cftime 1.4.1 py38h080aedc_0
chardet 4.0.0 py38haa95532_1003
click 7.1.2 pyhd3eb1b0_0
colorama 0.4.4 pyhd3eb1b0_0
cryptography 3.4.7 py38h71e12ea_0
curl 7.71.1 h2a8f88b_1
cycler 0.10.0 py38_0
decorator 4.4.2 pyhd3eb1b0_0
defusedxml 0.7.1 pyhd3eb1b0_0
empyrical 0.5.5 pyh9f0ad1d_0 conda-forge entrypoints 0.3 py38_0
freetype 2.10.4 hd328e21_0
greenlet 1.0.0 py38hd77b12b_2
h5py 2.10.0 py38h5e291fa_0
hdf4 4.2.13 h712560f_2
hdf5 1.10.4 h7ebc959_0
icc_rt 2019.0.0 h0cc432a_1
icu 58.2 ha925a31_3
idna 2.10 pyhd3eb1b0_0
importlib-metadata 3.10.0 py38haa95532_0
importlib_metadata 3.10.0 hd3eb1b0_0
intel-openmp 2021.2.0 haa95532_616
intervaltree 3.1.0 py_0
ipykernel 5.3.4 py38h5ca1d4c_0
ipympl 0.7.0 pyhd3eb1b0_0
ipython 7.22.0 py38hd4e2768_0
ipython_genutils 0.2.0 pyhd3eb1b0_1
ipywidgets 7.6.3 pyhd3eb1b0_1
iso3166 1.0.1 pyh9f0ad1d_0 conda-forge iso4217 1.6.20180829 py_0 conda-forge jedi 0.17.0 py38_0
jinja2 2.11.3 pyhd3eb1b0_0
joblib 1.0.1 pyhd3eb1b0_0
jpeg 9b hb83a4c4_2
json5 0.9.5 py_0
jsonschema 3.2.0 py_2
jupyter 1.0.0 py38_7
jupyter-packaging 0.7.12 pyhd3eb1b0_0
jupyter_client 6.1.12 pyhd3eb1b0_0
jupyter_console 6.4.0 pyhd3eb1b0_0
jupyter_core 4.7.1 py38haa95532_0
jupyter_server 1.4.1 py38haa95532_0
jupyterlab 3.0.14 pyhd3eb1b0_1
jupyterlab_pygments 0.1.2 py_0
jupyterlab_server 2.4.0 pyhd3eb1b0_0
jupyterlab_widgets 1.0.0 pyhd3eb1b0_1
kiwisolver 1.3.1 py38hd77b12b_0
krb5 1.18.2 hc04afaa_0
libcurl 7.71.1 h2a8f88b_1
libgpuarray 0.7.6 hfa6e2cd_0
libiconv 1.15 h1df5818_7
libnetcdf 4.7.3 h1302dcc_0
libpng 1.6.37 h2a8f88b_0
libpython 2.1 py38_0
libsodium 1.0.18 h62dcd97_0
libssh2 1.9.0 h7a1dbc1_1
libtiff 4.2.0 hd0e1b90_0
libxml2 2.9.10 hb89e7f3_3
libxslt 1.1.34 he774522_0
logbook 1.5.3 py38_0 ml4t lru-dict 1.1.6 py38h294d835_3 conda-forge lxml 4.6.3 py38h9b66d53_0
lz4-c 1.9.3 h2bbff1b_0
lzo 2.10 he774522_2
m2w64-binutils 2.25.1 5
m2w64-bzip2 1.0.6 6
m2w64-crt-git 5.0.0.4636.2595836 2
m2w64-gcc 5.3.0 6
m2w64-gcc-ada 5.3.0 6
m2w64-gcc-fortran 5.3.0 6
m2w64-gcc-libgfortran 5.3.0 6
m2w64-gcc-libs 5.3.0 7
m2w64-gcc-libs-core 5.3.0 7
m2w64-gcc-objc 5.3.0 6
m2w64-gmp 6.1.0 2
m2w64-headers-git 5.0.0.4636.c0ad18a 2
m2w64-isl 0.16.1 2
m2w64-libiconv 1.14 6
m2w64-libmangle-git 5.0.0.4509.2e5a9a2 2
m2w64-libwinpthread-git 5.0.0.4634.697f757 2
m2w64-make 4.1.2351.a80a8b8 2
m2w64-mpc 1.0.3 3
m2w64-mpfr 3.1.4 4
m2w64-pkg-config 0.29.1 2
m2w64-toolchain 5.3.0 7
m2w64-toolchain_win-64 2.4.0 0
m2w64-tools-git 5.0.0.4592.90b8472 2
m2w64-windows-default-manifest 6.4 3
m2w64-winpthreads-git 5.0.0.4634.697f757 2
m2w64-zlib 1.2.8 10
mako 1.1.4 pyhd3eb1b0_0
markupsafe 1.1.1 py38he774522_0
matplotlib 3.3.4 py38haa95532_0
matplotlib-base 3.3.4 py38h49ac443_0
mistune 0.8.4 py38he774522_1000
mkl 2021.2.0 haa95532_296
mkl-service 2.3.0 py38h2bbff1b_1
mkl_fft 1.3.0 py38h277e83a_2
mkl_random 1.2.1 py38hf11a4ad_2
mock 4.0.3 pyhd3eb1b0_0
msys2-conda-epoch 20160418 1
multipledispatch 0.6.0 py38_0
nbclassic 0.2.6 pyhd3eb1b0_0
nbclient 0.5.3 pyhd3eb1b0_0
nbconvert 6.0.7 py38_0
nbformat 5.1.3 pyhd3eb1b0_0
nest-asyncio 1.5.1 pyhd3eb1b0_0
netcdf4 1.5.3 py38h012c1a0_0
networkx 2.5.1 pyhd3eb1b0_0
norgatedata 1.0.58 pypi_0 pypi notebook 6.3.0 py38haa95532_0
numexpr 2.7.3 py38hb80d3ca_1
numpy 1.20.1 py38h34a8a5c_0
numpy-base 1.20.1 py38haf7ebc8_0
olefile 0.46 py_0
openssl 1.1.1k h2bbff1b_0
packaging 20.9 pyhd3eb1b0_0
pandas 1.2.4 py38hf11a4ad_0
pandas-datareader 0.9.0 py_0
pandoc 2.12 haa95532_0
pandocfilters 1.4.3 py38haa95532_1
parso 0.8.2 pyhd3eb1b0_0
patsy 0.5.1 py38_0
pickleshare 0.7.5 pyhd3eb1b0_1003
pillow 8.2.0 py38h4fa10fc_0
pip 21.0.1 py38haa95532_0
prometheus_client 0.10.1 pyhd3eb1b0_0
prompt-toolkit 3.0.17 pyh06a4308_0
prompt_toolkit 3.0.17 hd3eb1b0_0
pycparser 2.20 py_2
pyfolio 0.8.0 py_1 conda-forge pygments 2.8.1 pyhd3eb1b0_0
pygpu 0.7.6 py38h2a96729_0
pymc3 3.8 py_0
pyopenssl 20.0.1 pyhd3eb1b0_1
pyparsing 2.4.7 pyhd3eb1b0_0
pyqt 5.9.2 py38ha925a31_4
pyreadline 2.1 py38_1
pyrsistent 0.17.3 py38he774522_0
pysocks 1.7.1 py38_0
pytables 3.6.1 py38ha5be198_0
python 3.8.8 hdbf39b2_5
python-dateutil 2.8.1 pyhd3eb1b0_0
python-editor 1.0.4 py_0
python-interface 1.6.0 py_0 conda-forge python_abi 3.8 1_cp38 conda-forge pytz 2021.1 pyhd3eb1b0_0
pywin32 227 py38he774522_1
pywinpty 0.5.7 py38_0
pyzmq 20.0.0 py38hd77b12b_1
qt 5.9.7 vc14h73c81de_0
qtconsole 5.0.3 pyhd3eb1b0_0
qtpy 1.9.0 py_0
requests 2.25.1 pyhd3eb1b0_0
scikit-learn 0.24.1 py38hf11a4ad_0
scipy 1.6.2 py38h66253e8_1
seaborn 0.11.1 pyhd3eb1b0_0
send2trash 1.5.0 pyhd3eb1b0_1
setuptools 52.0.0 py38haa95532_0
sip 4.19.13 py38ha925a31_0
six 1.15.0 py38haa95532_0
sniffio 1.2.0 py38haa95532_1
sortedcontainers 2.3.0 pyhd3eb1b0_0
sqlalchemy 1.4.7 py38h2bbff1b_0
sqlite 3.35.4 h2bbff1b_0
statsmodels 0.12.2 py38h2bbff1b_0
ta-lib 0.4.19 py38hb572761_0 ml4t terminado 0.9.4 py38haa95532_0
testpath 0.4.4 pyhd3eb1b0_0
theano 1.0.4 py38_0
threadpoolctl 2.1.0 pyh5ca1d4c_0
tk 8.6.10 he774522_0
toolz 0.11.1 pyhd3eb1b0_0
tornado 6.1 py38h2bbff1b_0
tqdm 4.59.0 pyhd3eb1b0_1
trading-calendars 2.1.1 pyhd3deb0d_0 conda-forge traitlets 5.0.5 pyhd3eb1b0_0
urllib3 1.26.4 pyhd3eb1b0_0
vc 14.2 h21ff451_1
vs2015_runtime 14.27.29016 h5e58377_2
wcwidth 0.2.5 py_0
webencodings 0.5.1 py38_1
wheel 0.36.2 pyhd3eb1b0_0
widgetsnbextension 3.5.1 py38_0
win_inet_pton 1.1.0 py38_0
wincertstore 0.2 py38_0
winpty 0.4.3 4
xarray 0.17.0 pyhd3eb1b0_0
xz 5.2.5 h62dcd97_0
zeromq 4.3.3 ha925a31_3
zipline-norgatedata 1.2.8 pypi_0 pypi zipline-reloaded 2.0.0rc4 py38he6999da_0 ml4t zipp 3.4.1 pyhd3eb1b0_0
zlib 1.2.11 h62dcd97_4
zstd 1.4.5 ha9fde0e_0
Now that you know a little about me, let me tell you about the issue I am having:
Description of Issue
On my multiple drive system. ~/.zipline exists on < HOME>/.zipline and contains several ingested data bundles HOME is explicitly set in Win 10 environment variables to <E:\whatever> [i.e. NOT to the boot drive C:\ but to a different Anaconda working drive]
zipine-reloaded is running in fresh conda environment
In a separate conda env, zipline 1.4.0 finds < HOME>/.zipline with no problem
zipline-reloaded does not correctly resolve ~/.zipline into < HOME>/.zipline [os.getenv("ZIPLINE_ROOT") returns None]
What did you expect to happen?
CLI >zipline bundles should return a list of existing data bundles in < HOME>/.zipline
What happened instead?
CLI >zipline bundles creates a new (empty) /.zipline in C:\<user\username>
If it helps, os.environ shows: 'HOMEDRIVE': 'C:', 'HOMEPATH': '\user\username'
Here is how you can reproduce this issue on your machine:
Reproduction Steps
What steps have you taken to resolve this already?
in Jupyter Lab, set ZIPLINE_ROOT explicitly; in that case, !zipline bundles worked as expected but it still does not work from CLI
Finally, I set ZIPLINE_ROOT explicitly in Win10 user environment variables and rebooted entire system. Now it seems to be working correctly. But you shouldn't have to do that; zipline 1.4.0 doesn't need it. ...
Anything else?
There is no obvious reason for this problem since zipline 1.4.0/1.4.1 continue to work correctly It seems that, expanduser("~/.zipline") in zipline_root() in paths.py is not getting resolved correctly
The idea of an explicit config file (as used by zipline-trader) seems like a good idea actually.
Sincerely, arthur y.