JustinGuese
commented
4 years ago Still the same error for me...
Open mowoe opened 5 years ago
JustinGuese
commented
4 years ago Still the same error for me...
rcardinaux
commented
4 years ago Also encounter the same issue since I started using Zipline. I am setting manually the country code to 'US' as suggest above every time I ingest new data, but I'm pretty sure it's not the normal procedure.
Has anyone a better solution? could this be worked around by tweaking the ingest function?
rcardinaux
commented
4 years ago If it can help anyone, I solved the issue by feeding the exchanges table directly in the ingest function. It looks like that:
exchange = {'exchange': 'NYSE', 'canonical_name': 'NYSE', 'country_code': 'US'}
exchange_df = pd.DataFrame(exchange,index = [0])
asset_db_writer.write(equities=asset_metadata,exchanges=exchange_df)
chunghoony
commented
4 years ago @rcardinaux can you expand your solution a bit? where can I find the ingest function?
rcardinaux
commented
4 years ago @chunghoony : you can create your own ingest function and save it in the bundle folder of zipline. The path to that folder is the following in my environment: C:\Users\rapha\anaconda3\envs\py35\Lib\site-packages\zipline\data\bundles
The script where you define the ingest function is ultimately run whith the following command:
zipline ingest --bundle custom_data
The links below have been useful in writing my script custom_data: https://0xboz.github.io/blog/how-to-create-custom-zipline-bundles-from-binance-data-part-1/ https://0xboz.github.io/blog/how-to-create-custom-zipline-bundles-from-binance-data-part-2/
You won't be able to use my code as such, since I'm exporting data from an SQL database, but here is the content of my custom_data.py file:
import sys
sys.path.insert(0, 'C:/Users/rapha/OneDrive/Dokumente/GitHub/data/python_code')
#PROJECT'S PACAKGES
import pck_param as param
import pck_data_load as data_load
import pck_interface as interface
import pandas as pd
import numpy as np
#from pathlib import Path
from six import iteritems
from logbook import Logger
#from zipfile import ZipFile
show_progress = True
log = Logger(__name__)
#FORMAT THE DATA --> impport data from a zipfile and set the correct columns' names
def load_data_table(show_progress=show_progress):
db_info, data_source_info = param.parse_credentials()
sql_conn, sql_cursor = data_load.mysql_connect(
host=db_info['host'],
port=db_info['port'],
user=db_info['user'],
password=db_info['password'],
database=db_info['database'])
data=interface.SQL_to_python_data_loader(sql_conn=sql_conn,
sql_cursor=sql_cursor,
security_symbol_1='IBM',
security_symbol_3='AACG',
perioodicityID='D')
if show_progress:
log.info(data.info())
log.info(data.head())
return data
#CREATE THE METADATA --> The metadata table provides the mapping table for our securities list
def gen_asset_metadata(data, show_progress):
if show_progress:
log.info('Generating asset metadata.')
data = data.groupby(
by='symbol'
).agg(
{'date': [np.min, np.max]}
)
data.reset_index(inplace=True)
data['start_date'] = data.date.amin
data['end_date'] = data.date.amax
#RC: add exchange
data['exchange'] = 'NYSE'
del data['date']
data.columns = data.columns.get_level_values(0)
data['auto_close_date'] = data['end_date'].values
if show_progress:
log.info(data.info())
log.info(data.head())
return data
#STORE THE ADJUSTMENTS
def parse_splits(data, show_progress):
if show_progress:
log.info('Parsing split data.')
data['split_ratio'] = 1.0 / data.split_ratio
data.rename(
columns={
'split_ratio': 'ratio',
'date': 'effective_date',
},
inplace=True,
copy=False,
)
if show_progress:
log.info(data.info())
log.info(data.head())
return data
def parse_dividends(data, show_progress):
if show_progress:
log.info('Parsing dividend data.')
data['record_date'] = data['declared_date'] = data['pay_date'] = pd.NaT
data.rename(columns={'date': 'ex_date',
'dividends': 'amount'}, inplace=True, copy=False)
if show_progress:
log.info(data.info())
log.info(data.head())
return data
#WRITE THE DAILY BARS
def parse_pricing_and_vol(data,
sessions,
symbol_map):
for asset_id, symbol in iteritems(symbol_map):
asset_data = data.xs(
symbol,
level=1
).reindex(
sessions.tz_localize(None)
).fillna(0.0)
yield asset_id, asset_data
#--------------------
def ingest(environ,
asset_db_writer,
minute_bar_writer,
daily_bar_writer,
adjustment_writer,
calendar,
start_session,
end_session,
cache,
show_progress,
output_dir):
raw_data = load_data_table(show_progress=show_progress)
#raw_data = load_data_table(path, show_progress=show_progress)
asset_metadata = gen_asset_metadata(
raw_data[['symbol', 'date']],
show_progress
)
exchange = {'exchange': 'NYSE', 'canonical_name': 'NYSE', 'country_code': 'US'}
exchange_df = pd.DataFrame(exchange,index = [0])
asset_db_writer.write(equities=asset_metadata,exchanges=exchange_df)
#WRITE THE DAILY BARS
symbol_map = asset_metadata.symbol
sessions = calendar.sessions_in_range(start_session, end_session)
raw_data.set_index(['date', 'symbol'], inplace=True)
daily_bar_writer.write(
parse_pricing_and_vol(
raw_data,
sessions,
symbol_map
),
show_progress=show_progress
)
#STORE THE ADJUSTMENTS
raw_data.reset_index(inplace=True)
raw_data['symbol'] = raw_data['symbol'].astype('category')
raw_data['sid'] = raw_data.symbol.cat.codes
adjustment_writer.write(
splits=parse_splits(
raw_data[[
'sid',
'date',
'split_ratio',
]].loc[raw_data.split_ratio != 1],
show_progress=show_progress
),
dividends=parse_dividends(
raw_data[[
'sid',
'date',
'dividends',
]].loc[raw_data.dividends != 0],
show_progress=show_progress
)
)
evrenbingol
commented
4 years ago This worked or me. Make sure you save the database instead of closing the app(apply does not perm save it)
hosammhmahmoud
commented
1 year ago I know I am commenting so late but this fixed the issue for me. For the zipline/assets.assets.py file there is a function called _compute_asset_lifetimes which responsible for making sure the asset has exchanges and names, you can comment out this condition and everything should work fine.
with self.engine.connect() as conn:
results = conn.execute(
sa.select(
equities_cols.sid,
equities_cols.start_date,
equities_cols.end_date,
# ).where( # Comment Conditions
# (exchanges_cols.exchange == equities_cols.exchange) #& (condt)
)
).fetchall()
if results:
sids, starts, ends = zip(*results)
Dear Zipline Maintainers,
Before I tell you about my issue, let me describe my environment:
Environment
Now that you know a little about me, let me tell you about the issue I am having:
Description of Issue
ValueError: Failed to find any assets with country_code 'US' that traded between 2018-01-02 00:00:00+00:00 and 2018-01-09 00:00:00+00:00.This probably means that your asset db is old or that it has incorrect country/exchange metadata.Here is how you can reproduce this issue on your machine:
Reproduction Steps
What steps have you taken to resolve this already?
The error is caused because the country_code in the Database is somehow messed up.
~/.zipline/data/quandl/TIMESTAMP/assets-n.sqlite): 1.2. Go to the 'exchanges' Table and change country_code to 'US'. Mine was set to '???' 1.3. I used 'DB Browser for SQLite' which has an easy GUI Interface just for the record.Anything else?
I don't really know if it was necessary to open this issue, as i already resolved it, but it had driven me crazy, so if anyone else has this Problem, here is a temporary solution.
Sincerely,
mowoe