data.history returns error when some of requested assets have no data for requested period

[kooomix]

Hi,

I have ingested all data for my symbols and ran a back-test in which I use data.history to get the data for all symbols in one request. The issues is that if one of the symbols have no data for the requested period (because that symbol started to be traded after this date) it returns error instead of what I believe it should do (according to zipline documentation) - returning NAN for these specific symbols.

Help on the matter is appreciated. Thanks.

[lenak25]

Hi @kooomix ,

Please provide more details on the parameters you are using in your algorithm (mode, exchange, etc).

Thanks,

Lena

[kooomix]

Algo parameters: capital_base=CAPITAL_BASE, data_frequency='minute', initialize=initialize, handle_data=handle_data, analyze=analyze, exchange_name='poloniex', algo_namespace=NAMESPACE, quote_currency='btc', start=pd.to_datetime('2017-1-1', utc=True), end=pd.to_datetime('2018-1-1', utc=True)

Error that I get (for ingested symbol): PricingDataNotLoadedError Traceback (most recent call last)

in () 26 quote_currency='btc', 27 start=backtest_start_date, ---> 28 end=backtest_end_date 29 ) 30 ~/anaconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/utils/run_algo.py in run_algorithm(initialize, capital_base, start, end, handle_data, before_trading_start, analyze, data_frequency, data, bundle, bundle_timestamp, default_extension, extensions, strict_extensions, environ, live, exchange_name, quote_currency, algo_namespace, live_graph, analyze_live, simulate_orders, auth_aliases, stats_output, output) 599 simulate_orders=simulate_orders, 600 auth_aliases=auth_aliases, --> 601 stats_output=stats_output 602 ) ~/anaconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/utils/run_algo.py in _run(handle_data, initialize, before_trading_start, analyze, algofile, algotext, defines, data_frequency, capital_base, data, bundle, bundle_timestamp, start, end, output, print_algo, local_namespace, environ, live, exchange, algo_namespace, quote_currency, live_graph, analyze_live, simulate_orders, auth_aliases, stats_output) 361 ).run( 362 data, --> 363 overwrite_sim_params=False, 364 ) 365 ~/anaconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_algorithm.py in run(self, data, overwrite_sim_params) 399 def run(self, data=None, overwrite_sim_params=True): 400 perf = super(ExchangeTradingAlgorithmBacktest, self).run( --> 401 data, overwrite_sim_params 402 ) 403 # Rebuilding the stats to support minute data ~/anaconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_algorithm.py in run(self, data, overwrite_sim_params) 356 data.attempts = self.attempts 357 return super(ExchangeTradingAlgorithmBase, self).run( --> 358 data, overwrite_sim_params 359 ) 360 ~/anaconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/algorithm.py in run(self, data, overwrite_sim_params) 722 try: 723 perfs = [] --> 724 for perf in self.get_generator(): 725 perfs.append(perf) 726 ~/anaconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/gens/tradesimulation.py in transform(self) 222 for dt, action in self.clock: 223 if action == BAR: --> 224 for capital_change_packet in every_bar(dt): 225 yield capital_change_packet 226 elif action == SESSION_START: ~/anaconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/gens/tradesimulation.py in every_bar(dt_to_use, current_data, handle_data) 135 perf_tracker.process_commission(commission) 136 --> 137 handle_data(algo, current_data, dt_to_use) 138 139 # grab any new orders from the blotter, then clear the list. ~/anaconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/utils/events.py in handle_data(self, context, data, dt) 214 context, 215 data, --> 216 dt, 217 ) 218 ~/anaconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/utils/events.py in handle_data(self, context, data, dt) 233 """ 234 if self.rule.should_trigger(dt): --> 235 self.callback(context, data) 236 237 ~/anaconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_algorithm.py in handle_data(self, data) 377 378 def handle_data(self, data): --> 379 super(ExchangeTradingAlgorithmBacktest, self).handle_data(data) 380 381 if self.data_frequency == 'minute': ~/anaconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/algorithm.py in handle_data(self, data) 471 def handle_data(self, data): 472 if self._handle_data: --> 473 self._handle_data(self, data) 474 475 # Unlike trading controls which remain constant unless placing an in handle_data(context, data) 185 'volume', 186 bar_count=VOLUME_PERIODS*bar_steps[exchange], --> 187 frequency=freq_exchange[exchange][freq], 188 ).iloc[::bar_steps[exchange]] 189 ~/anaconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/_protocol.pyx in catalyst._protocol.check_parameters.__call__.assert_keywords_and_call() ~/anaconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/_protocol.pyx in catalyst._protocol.BarData.history() ~/anaconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_data_portal.py in get_history_window(self, assets, end_dt, bar_count, frequency, field, data_frequency, ffill) 95 field, 96 data_frequency, ---> 97 ffill)) 98 99 @abc.abstractmethod ~/anaconda3/envs/catalyst/lib/python3.6/site-packages/redo/__init__.py in retry(action, attempts, sleeptime, max_sleeptime, sleepscale, jitter, retry_exceptions, cleanup, args, kwargs) 160 logfn = log.info if n != 1 else log.debug 161 logfn(log_attempt_format, n) --> 162 return action(*args, **kwargs) 163 except retry_exceptions: 164 log.debug("retry: Caught exception: ", exc_info=True) ~/anaconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_data_portal.py in _get_history_window(self, assets, end_dt, bar_count, frequency, field, data_frequency, ffill) 69 field, 70 data_frequency, ---> 71 ffill) 72 73 def get_history_window(self, ~/anaconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_data_portal.py in get_exchange_history_window(self, exchange_name, assets, end_dt, bar_count, frequency, field, data_frequency, ffill) 323 field=field, 324 data_frequency=adj_data_frequency, --> 325 algo_end_dt=self._last_available_session, 326 ) 327 ~/anaconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_bundle.py in get_history_window_series_and_load(self, assets, end_dt, bar_count, field, data_frequency, algo_end_dt, force_auto_ingest) 896 bar_count=bar_count, 897 field=field, --> 898 data_frequency=data_frequency, 899 ) 900 return pd.DataFrame(series) ~/anaconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_bundle.py in get_history_window_series(self, assets, end_dt, bar_count, field, data_frequency, reset_reader) 1004 data_frequency=data_frequency, 1005 start_dt=asset_start_dt, -> 1006 end_dt=end_dt 1007 ) 1008 PricingDataNotLoadedError: Missing data for poloniex bch_btc in date range [2016-12-31 00:00:00+00:00 - 2017-01-01 00:00:00+00:00] Please run: `catalyst ingest-exchange -x poloniex -f daily -i bch_btc`. See catalyst documentation for details.

[lenak25]

Hi @kooomix , thanks for reporting this, we will address this.