search

stefan-jansen / zipline-reloaded

Zipline, a Pythonic Algorithmic Trading Library
https://zipline.ml4trading.io
Apache License 2.0
1.13k stars 210 forks source link

"'__name__' not in globals" when trying to ingest custom bundle #232

Closed heapallocation closed 1 week ago

heapallocation commented 10 months ago

Dear Zipline Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

* Operating System: **Darwin mymac.local 22.2.0 Darwin Kernel Version 22.2.0: Fri Nov 11 02:08:47 PST 2022; root:xnu-8792.61.2~4/RELEASE_X86_64 x86_64** * Python Version: **Python 3.10.13** * Python Bitness: **64** * How did you install Zipline: ``` conda create -n env_zipline python=3.10 conda install -c conda-forge zipline-reloaded=3.0.3=py310hc1335a1_0 ``` * Python packages: # # Name Version Build Channel aiohttp 3.8.2 pypi_0 pypi aiosignal 1.3.1 pypi_0 pypi alembic 1.12.1 pyhd8ed1ab_0 conda-forge alpaca-trade-api 3.0.2 pypi_0 pypi appdirs 1.4.4 pyh9f0ad1d_0 conda-forge appnope 0.1.3 pyhd8ed1ab_0 conda-forge asttokens 2.4.1 pyhd8ed1ab_0 conda-forge async-timeout 4.0.3 pypi_0 pypi attrs 23.1.0 pypi_0 pypi backports 1.0 pyhd8ed1ab_3 conda-forge backports.functools_lru_cache 1.6.5 pyhd8ed1ab_0 conda-forge bcolz-zipline 1.2.6 py310hf0b6da5_1 conda-forge beautifulsoup4 4.12.2 pyha770c72_0 conda-forge blosc 1.21.5 heccf04b_0 conda-forge bottleneck 1.3.7 py310hf0b6da5_1 conda-forge brotli 1.1.0 h0dc2134_1 conda-forge brotli-bin 1.1.0 h0dc2134_1 conda-forge brotli-python 1.1.0 py310h9e9d8ca_1 conda-forge bzip2 1.0.8 h10d778d_5 conda-forge c-ares 1.21.0 h10d778d_0 conda-forge c-blosc2 2.11.2 h354e526_0 conda-forge ca-certificates 2023.11.17 h8857fd0_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge certifi 2023.11.17 pyhd8ed1ab_0 conda-forge cffi 1.16.0 py310hdca579f_0 conda-forge charset-normalizer 2.1.1 pypi_0 pypi click 8.1.7 unix_pyh707e725_0 conda-forge comm 0.1.4 pyhd8ed1ab_0 conda-forge contourpy 1.2.0 py310ha697434_0 conda-forge cryptography 41.0.5 py310haee03e7_0 conda-forge cycler 0.12.1 pyhd8ed1ab_0 conda-forge debugpy 1.8.0 py310h9e9d8ca_1 conda-forge decorator 5.1.1 pyhd8ed1ab_0 conda-forge deprecation 2.1.0 pypi_0 pypi empyrical-reloaded 0.5.9 pyhd8ed1ab_0 conda-forge exceptiongroup 1.1.3 pyhd8ed1ab_0 conda-forge exchange-calendars 4.2.8 pyhd8ed1ab_1 conda-forge executing 2.0.1 pyhd8ed1ab_0 conda-forge fonttools 4.44.3 py310hb372a2b_0 conda-forge freetype 2.12.1 h60636b9_2 conda-forge frozendict 2.3.8 py310h6729b98_1 conda-forge frozenlist 1.4.0 pypi_0 pypi greenlet 3.0.1 py310had63691_0 conda-forge h5py 3.10.0 nompi_py310hf8115b4_100 conda-forge hdf5 1.14.2 nompi_hedada53_100 conda-forge html5lib 1.1 pyh9f0ad1d_0 conda-forge icu 73.2 hf5e326d_0 conda-forge idna 3.4 pyhd8ed1ab_0 conda-forge importlib-metadata 6.8.0 pyha770c72_0 conda-forge importlib_metadata 6.8.0 hd8ed1ab_0 conda-forge importlib_resources 6.1.1 pyhd8ed1ab_0 conda-forge intervaltree 3.1.0 pyhd8ed1ab_1 conda-forge ipykernel 6.26.0 pyh3cd1d5f_0 conda-forge ipython 8.17.2 pyh31c8845_0 conda-forge iso3166 2.1.1 pyhd8ed1ab_0 conda-forge iso4217 1.9.20220401 pyhd8ed1ab_0 conda-forge jedi 0.19.1 pyhd8ed1ab_0 conda-forge jupyter_client 8.6.0 pyhd8ed1ab_0 conda-forge jupyter_core 5.5.0 py310h2ec42d9_0 conda-forge kiwisolver 1.4.5 py310h88cfcbd_1 conda-forge korean_lunar_calendar 0.3.1 pyhd8ed1ab_0 conda-forge krb5 1.21.2 hb884880_0 conda-forge lcms2 2.15 hd6ba6f3_3 conda-forge lerc 4.0.0 hb486fe8_0 conda-forge libaec 1.1.2 he965462_1 conda-forge libblas 3.9.0 19_osx64_openblas conda-forge libbrotlicommon 1.1.0 h0dc2134_1 conda-forge libbrotlidec 1.1.0 h0dc2134_1 conda-forge libbrotlienc 1.1.0 h0dc2134_1 conda-forge libcblas 3.9.0 19_osx64_openblas conda-forge libcurl 8.4.0 h726d00d_0 conda-forge libcxx 16.0.6 hd57cbcb_0 conda-forge libdeflate 1.19 ha4e1b8e_0 conda-forge libedit 3.1.20191231 h0678c8f_2 conda-forge libev 4.33 haf1e3a3_1 conda-forge libffi 3.4.2 h0d85af4_5 conda-forge libgfortran 5.0.0 13_2_0_h97931a8_1 conda-forge libgfortran5 13.2.0 h2873a65_1 conda-forge libiconv 1.17 hac89ed1_0 conda-forge libjpeg-turbo 3.0.0 h0dc2134_1 conda-forge liblapack 3.9.0 19_osx64_openblas conda-forge libnghttp2 1.58.0 h64cf6d3_0 conda-forge libopenblas 0.3.24 openmp_h48a4ad5_0 conda-forge libpng 1.6.39 ha978bb4_0 conda-forge libsodium 1.0.18 hbcb3906_1 conda-forge libsqlite 3.44.0 h92b6c6a_0 conda-forge libssh2 1.11.0 hd019ec5_0 conda-forge libta-lib 0.4.0 h0dc2134_2 conda-forge libtiff 4.6.0 h684deea_2 conda-forge libwebp-base 1.3.2 h0dc2134_0 conda-forge libxcb 1.15 hb7f2c08_0 conda-forge libxml2 2.11.5 h3346baf_1 conda-forge libxslt 1.1.37 h20bfa82_1 conda-forge libzlib 1.2.13 h8a1eda9_5 conda-forge llvm-openmp 17.0.4 hb6ac08f_0 conda-forge lru-dict 1.2.0 py310h6729b98_1 conda-forge lxml 4.9.3 py310h479f746_1 conda-forge lz4-c 1.9.4 hf0c8a7f_0 conda-forge mako 1.3.0 pyhd8ed1ab_0 conda-forge markupsafe 2.1.3 py310h6729b98_1 conda-forge matplotlib-base 3.8.1 py310hec49e92_0 conda-forge matplotlib-inline 0.1.6 pyhd8ed1ab_0 conda-forge mplfinance 0.12.9b7 pyhd8ed1ab_0 conda-forge msgpack 1.0.3 pypi_0 pypi multidict 5.2.0 pypi_0 pypi multipledispatch 0.6.0 py_0 conda-forge multitasking 0.0.9 pyhd8ed1ab_0 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge ncurses 6.4 h93d8f39_2 conda-forge nest-asyncio 1.5.8 pyhd8ed1ab_0 conda-forge networkx 3.2.1 pyhd8ed1ab_0 conda-forge numexpr 2.8.7 py310hdba192b_4 conda-forge numpy 1.26.0 py310h0171094_0 conda-forge openjpeg 2.5.0 ha4da562_3 conda-forge openssl 3.1.4 hd75f5a5_0 conda-forge packaging 23.2 pyhd8ed1ab_0 conda-forge pandas 2.1.3 py310hdba192b_0 conda-forge pandas-datareader 0.10.0 pyh6c4a22f_0 conda-forge parso 0.8.3 pyhd8ed1ab_0 conda-forge patsy 0.5.3 pyhd8ed1ab_0 conda-forge peewee 3.17.0 py310hedf4a2c_0 conda-forge pexpect 4.8.0 pyh1a96a4e_2 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 10.1.0 py310he65384d_0 conda-forge pip 23.3.1 pyhd8ed1ab_0 conda-forge platformdirs 4.0.0 pyhd8ed1ab_0 conda-forge prompt-toolkit 3.0.41 pyha770c72_0 conda-forge prompt_toolkit 3.0.41 hd8ed1ab_0 conda-forge psutil 5.9.5 py310h6729b98_1 conda-forge pthread-stubs 0.4 hc929b4f_1001 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge py-cpuinfo 9.0.0 pyhd8ed1ab_0 conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pygments 2.17.0 pyhd8ed1ab_0 conda-forge pyluach 2.2.0 pyhd8ed1ab_0 conda-forge pyparsing 3.1.1 pyhd8ed1ab_0 conda-forge pysocks 1.7.1 pyha2e5f31_6 conda-forge pytables 3.9.1 py310ha2b9561_0 conda-forge python 3.10.13 h00d2728_0_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-interface 1.6.0 py_0 conda-forge python-tzdata 2023.3 pyhd8ed1ab_0 conda-forge python_abi 3.10 4_cp310 conda-forge pytz 2023.3.post1 pyhd8ed1ab_0 conda-forge pyyaml 6.0 pypi_0 pypi pyzmq 25.1.1 py310hd8b4af3_2 conda-forge readline 8.2 h9e318b2_1 conda-forge requests 2.31.0 pyhd8ed1ab_0 conda-forge scipy 1.11.3 py310h2db466d_1 conda-forge setuptools 68.2.2 pyhd8ed1ab_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge snappy 1.1.10 h225ccf5_0 conda-forge sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge soupsieve 2.5 pyhd8ed1ab_1 conda-forge sqlalchemy 2.0.23 py310hb372a2b_0 conda-forge stack_data 0.6.2 pyhd8ed1ab_0 conda-forge statsmodels 0.14.0 py310h91862f5_2 conda-forge ta-lib 0.4.28 py310hf0b6da5_0 conda-forge tk 8.6.13 h1abcd95_1 conda-forge toolz 0.12.0 pyhd8ed1ab_0 conda-forge tornado 6.3.3 py310h6729b98_1 conda-forge traitlets 5.13.0 pyhd8ed1ab_0 conda-forge typing-extensions 4.8.0 hd8ed1ab_0 conda-forge typing_extensions 4.8.0 pyha770c72_0 conda-forge tzdata 2023c h71feb2d_0 conda-forge unicodedata2 15.1.0 py310h6729b98_0 conda-forge urllib3 1.26.18 pypi_0 pypi wcwidth 0.2.10 pyhd8ed1ab_0 conda-forge webencodings 0.5.1 pyhd8ed1ab_2 conda-forge websocket-client 1.6.4 pypi_0 pypi websockets 10.4 pypi_0 pypi wheel 0.41.3 pyhd8ed1ab_0 conda-forge xorg-libxau 1.0.11 h0dc2134_0 conda-forge xorg-libxdmcp 1.1.3 h35c211d_0 conda-forge xz 5.2.6 h775f41a_0 conda-forge yarl 1.9.2 pypi_0 pypi yfinance 0.2.31 pyhd8ed1ab_0 conda-forge zeromq 4.3.5 h93d8f39_0 conda-forge zipline-reloaded 3.0.3 py310hc1335a1_0 conda-forge zipp 3.17.0 pyhd8ed1ab_0 conda-forge zlib-ng 2.0.7 hb7f2c08_0 conda-forge zstd 1.5.5 h829000d_0 conda-forge

Now that you know a little about me, let me tell you about the issue I am having:

Description of Issue

Here is how you can reproduce this issue on your machine:

Reproduction Steps

  1. Create two py scripts to ingest a custom bundle:

ingestion py script:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
__author__ = 'Anon'

from datetime import date, datetime, timedelta
from pathlib import Path
from os import getenv
import numpy as np
import pandas as pd
import alpaca_api as alpaca

pd.set_option('display.expand_frame_repr', False)
np.random.seed(42)

# Create a dictionary of symbols and company names
ASSETS = {
        1: ('AAPL', 'Apple Inc.'),
        2: ('TSLA', 'Tesla Inc.'),
    }

# Create a list of symbols from the asset dictionary
SYMBOLS_TEST = [value[0] for value in ASSETS.values()]

ZIPLINE_ROOT = getenv('ZIPLINE_ROOT')
if not ZIPLINE_ROOT:
    custom_data_path = Path('~/.zipline/custom_data').expanduser()
else:
    custom_data_path = Path(ZIPLINE_ROOT, 'custom_data')

def ticker_generator():
    """
    Lazily return (sid, ticker) tuple
    """
    return ((key, value[0]) for key, value in ASSETS.items())

def data_generator():

    minute_bars_results = alpaca.minute_bars(SYMBOLS_TEST)

    for symbol, asset_name in ticker_generator():
        df = minute_bars_results[minute_bars_results['symbol'] == str(symbol)]
        df.tz_localize('Pacific/Auckland').tz_convert('UTC')

        start_date = df.index[0]
        end_date = df.index[-1]

        first_traded = start_date.date()
        auto_close_date = end_date + pd.Timedelta(days=1)
        exchange = 'Alpaca'

        yield (symbol, df), asset_name, start_date, end_date, first_traded, auto_close_date, exchange

def metadata_frame():
    dtype = [
        ('symbol', 'object'),
        ('asset_name', 'object'),
        ('start_date', 'datetime64[ns]'),
        ('end_date', 'datetime64[ns]'),
        ('first_traded', 'datetime64[ns]'),
        ('auto_close_date', 'datetime64[ns]'),
        ('exchange', 'object'), ]
        # Create an empty DataFrame with the number of rows equal to the length of ASSETS
    df = pd.DataFrame(index=range(len(ASSETS)))

    # Set the data type for each column
    for col_name, col_type in dtype:
        df[col_name] = pd.Series(dtype=col_type)

    return df

def alpaca_to_bundle(interval='1m'):
    def ingest(environ,
               asset_db_writer,
               minute_bar_writer,
               daily_bar_writer,
               adjustment_writer,
               calendar,
               start_session,
               end_session,
               cache,
               show_progress,
               output_dir
               ):
        metadata = metadata_frame()

        def minute_data_generator():
            # could try without ths sid/symbol as it is in the bar data
            return (sid_df for (sid_df, *metadata.iloc[sid_df[0]]) in data_generator())

        minute_bar_writer.write(minute_data_generator(), show_progress=True)

        metadata.dropna(inplace=True)
        asset_db_writer.write(equities=metadata)
        # adjustment_writer.write(splits=pd.read_hdf(custom_data_path / 'algoseek.h5', 'splits'))
        # dividends do not work
        # adjustment_writer.write(dividends=pd.read_hdf(custom_data_path / 'algoseek.h5', 'dividends'))

    return ingest

api script to return data to script above

import pandas as pd
import getpass
from alpaca_trade_api.rest import REST, TimeFrame
from zipline.data.bundles import register

def get_bars(symbols):

    API_KEY = getpass.getpass('Enter your API key: ')
    SECRET_KEY = getpass.getpass('Enter your secret key: ')

    # Create an instance of the REST API class
    api = REST(key_id=API_KEY, secret_key=SECRET_KEY, base_url="https://paper-api.alpaca.markets")

    # time frame dates
    start_time = pd.Timestamp('2023-11-15', tz='UTC').isoformat()
    end_time = pd.Timestamp('2023-11-16', tz='UTC').isoformat()

    # Get bars for the time period
    minute_bars = api.get_bars(symbols, TimeFrame.Minute, start_time, end_time).df
    minute_bars

    return minute_bars
  1. Create an extension.py script in a .zipline folder with the following content
import sys
from pathlib import Path

sys.path.append(Path('~', '.zipline').expanduser().as_posix())
from zipline.data.bundles import register
from .zipline.ingest_alpaca_bundle import alpaca_to_bundle
from datetime import time
from pytz import timezone

register('alpaca',
         alpaca_to_bundle(),
         calendar_name='NYSE',
         )
  1. Run zipline ingest -b alpaca
  2. Get error 
    /Users/anon/opt/anaconda3/envs/env_zipline/lib/python3.10/site-packages/zipline/__main__.py:60: UserWarning: Failed to load extension: '/Users/anon/opt/anaconda3/envs/env_zipline/.zipline/extension.py'
"'__name__' not in globals"
load_extensions(

    ...

What steps have you taken to resolve this already?

I have tried to import my .py script into another python notebook successfully, but from that .zipline folder, so was wondering if it might be a context issue as the main script is being run for a different location. Initially it wasn't finding the ingest_alpaca_bundle module, but I think I resolved that by adding .zipline.ingest_alpaca_bundle. Also I wasn't sure if having another script dependency was causing issues. It's hard to tell at the moment if it is an issue with the .py script or an environmental issue

...

Anything else?

Any help would be greatly appreciated! ...

Sincerely, John.

stefan-jansen commented 1 week ago

Closing for lack of activity. Feel free to reopen if the issue persists.