zipline cannot ingest yahoo currency symbols

[jimbojetlag]

Having this extention:

from zipline.data.bundles import register, yahoo_equities

symbols = {
    'EURUSD=X',
}
register(
    'fx-bundle',
    yahoo_equities(symbols),
)

gives this error:

raise RemoteDataError('Unable to read URL: {0}'.format(url))
pandas_datareader._utils.RemoteDataError: Unable to read URL: http://ichart.finance.yahoo.com/table.csv

How can yahoo currencies be supported?

[richafrank]

Thanks @jimbojetlag . Currently zipline doesn't support currencies, but it would be a great feature! I'm going to close this issue in favor of the existing ones.

[jimbojetlag]

Seems like the symbol name is 'EUR=X'. But that's not all, currencies do not have dividends and splits, I had to write the below bundler to remove them.

The bundler works, but now when placing an order has no effect at all:

from zipline.api import order, symbol

stocks = ['EUR=X']

def initialize(context):
    context.has_ordered = False
    context.stocks = stocks

def handle_data(context, data):
    if not context.has_ordered:
        for stock in context.stocks:
            order(symbol(stock), 100)
        context.has_ordered = True

Too bad zipline code is so tied to equities only. Could a zipline developer make a comment on this? I'm guessing that a simple tweak is required to make it work for currencies.

Bundler code:

import os

import numpy as np
import pandas as pd
import pandas_datareader.data as web
import requests

from zipline.utils.cli import maybe_show_progress

def _cachpath(symbol, type_):
    return '-'.join((symbol.replace(os.path.sep, '_'), type_))

def forex(symbols, start=None, end=None):
    # strict this in memory so that we can reiterate over it
    symbols = tuple(symbols)

    def ingest(environ,
               asset_db_writer,
               minute_bar_writer,  # unused
               daily_bar_writer,
               adjustment_writer,
               calendar,
               cache,
               show_progress,
               output_dir,
               # pass these as defaults to make them 'nonlocal' in py2
               start=start,
               end=end):
        if start is None:
            start = calendar[0]
        if end is None:
            end = None

        metadata = pd.DataFrame(np.empty(len(symbols), dtype=[
                ('start_date', 'datetime64[ns]'),
                ('end_date', 'datetime64[ns]'),
                ('auto_close_date', 'datetime64[ns]'),
                ('symbol', 'object'),
            ]))

        def _pricing_iter():
            sid = 0
            with maybe_show_progress(
                    symbols,
                    show_progress,
                    label='Downloading forex pricing data: ') as it, \
                    requests.Session() as session:
                for symbol in it:
                    path = _cachpath(symbol, 'ohlcv')
                    try:
                        df = cache[path]
                    except KeyError:
                        df = cache[path] = web.DataReader(
                            symbol,
                            'yahoo',
                            start,
                            end,
                        ).sort_index()

                    # the start date is the date of the first trade and
                    # the end date is the date of the last trade
                    start_date = df.index[0]
                    end_date = df.index[-1]
                    # The auto_close date is the day after the last trade.
                    ac_date = end_date + pd.Timedelta(days=1)
                    metadata.iloc[sid] = start_date, end_date, ac_date, symbol

                    df.rename(
                        columns={
                            'Open': 'open',
                            'High': 'high',
                            'Low': 'low',
                            'Close': 'close',
                            'Volume': 'volume',
                        },
                        inplace=True,
                    )
                    yield sid, df
                    sid += 1

        daily_bar_writer.write(_pricing_iter(), show_progress=True)

        symbol_map = pd.Series(metadata.symbol.index, metadata.symbol)
        asset_db_writer.write(equities=metadata)

        adjustment_writer.write()

    return ingest

[jimbojetlag]

@richafrank thanks for your response. I'm not interested in trading currencies with Quantopian. I just want to use the existing code in zipline to backtest forex strategies, I don't see why it should not be possible. The only difference between equities and currencies is that currencies miss the dividend and split columns.

As I explained in the above, the bundler works fine, but the problem is that no order is placed. I'd appreciate if you could take a look at it.

[richafrank]

Oh, my mistake if I equated those incorrectly.

If your bundle is the same as yahoo, minus adjustments, then it's hard to say why orders aren't being placed. What have you done to debug it? Is handle_data actually getting called? Is order getting called? Is the order being placed, but the slippage model isn't producing any fills?

Perhaps there's a mismatch between the trading calendar and the pricing data you have? It looks like zipline run with a bundle is hardcoded to the "NYSE" calendar. If you do have a different calendar, then probably that needs to be passed to the TradingAlgorithm instance below that too.

[jimbojetlag]

Thanks for the pointers! I will look into this.