search

QuantConnect / Lean

Lean Algorithmic Trading Engine by QuantConnect (Python, C#)
https://lean.io
Apache License 2.0
8.97k stars 3.14k forks source link

[Crypto] Create Crypto Security Type #998

Closed jaredbroad closed 6 years ago

jameschch commented 6 years ago

I imagine the gdax integration will probably be blocked from completion by this task, so have started to think what this entails. The behavior of a crypto type would follow closely that of Forex. So looking at a recent build, I can see there are about 150 references to SecurityType.Forex outside of brokerage and toolbox code. We also have 14 references to QuantConnect.Securities.Forex.Forex.

Creating a crypto type would mostly entail going through each of these references and adding appropriate cases and conditions. The thing that's not clear is exactly how the crypto type should behave and how this should differ from Forex.

jaredbroad commented 6 years ago

Agree crypto could closely follow forex for the basics -- we internally brain stormed using a different sec-type because:

@StefanoRaggi can comment more but I believe the intent was separation allows us to provide more nuanced support for these features which could possibly eventually be copied into the base security type at a later date.

So -- implementation of this ticket is probably a 5-10 stage process -- lets brain storm the appropriate steps?

  1. Basic type and basic addition to enum. Global market or DAX market? The market enum was meant to distinguish between fungible assets. But I believe we only have space for 99 markets in the Symbol class. Is there a better way to do this?

  2. Adding cashbook and handling variable length names

  3. Design some way to handle name changes? We'll need a split event for crypto forks? .. .. n. Adding OrderBook and sending ticks into the orderbook to build depth model.

n. Creating slippage and fill models for crypto which are more realistic. Modelling the delayed fill.

Lots of these questions are easier answered/tested when we have data on the disk and the data-reading layers have been built. May I suggest we build the tool box data downloader and add some sample data before progressing past the # 1.

StefanoRaggi commented 6 years ago

Just some comments on a few topics:

Crypto vs Forex security type

@jaredbroad listed pretty much all the reasons for SecurityType.Crypto, I would also add this one:

@jameschch I agree there are many places to change, I remember doing something similar when we added SecurityType.Cfd

Single market vs One market per exchange

The concept of "market" is quite a "mix" at the moment:

Besides being a core component of the SecurityIdentifier class, it is also used:

Given that GDAX and Bitfinex are separate exchanges/brokerages (just as Oanda and FXCM) which quote different prices as can be seen here: https://bitcoincharts.com/markets/currency/USD.html I think it makes sense to have one market per exchange. The limitation we currently have in the SecurityIdentifier class is 999 markets, so not an issue for a long time :)

Currencies vs Trade-able pairs

Just as EUR, USD are currencies vs EURUSD, USDJPY are trade-able pairs, I think BTC, ETH etc. should be treated as currencies in the CashBook, maybe should be added here? https://github.com/QuantConnect/Lean/blob/master/Common/Currencies.cs#L233 or they could be their own separate list (CryptoCoins?)

Symbol mapping

This is a tough one, I guess exchanges can decide to do anything they like, in hard fork events. This is how GDAX planned and handled the ETH fork last year: https://blog.coinbase.com/on-the-ethereum-hard-fork-780f1577e986 I agree with @jaredbroad about getting some historical data (especially around this fork event, preferably from more exchanges), it would help us a lot in brainstorming this topic. Other related topics to discuss are new symbol additions and delistings (map files? single csv file with startDate/endDate?)

jaredbroad commented 6 years ago

Agree with all points but only clarification is that markets were "meant" to separate fungible assets. When an asset purchased on one exchange can be traded on another its the same market. With market makers like OANDA you're likely just holding a contract with OANDA and not actual currency so the units of EUR are not transferable to other brokerages.

Whether this concept continues into the future of LEAN is debatable.

On Jul 15, 2017 8:35 AM, "Stefano Raggi" notifications@github.com wrote:

Just some comments on a few topics:

Crypto vs Forex security type

@jaredbroad https://github.com/jaredbroad listed pretty much all the reasons for SecurityType.Crypto, I would also add this one:

  • in Forex we only need/support QuoteBars, in crypto we need both QuoteBars and TradeBars

@jameschch https://github.com/jameschch I agree there are many places to change, I remember doing something similar when we added SecurityType.Cfd

Single market vs One market per exchange

The concept of "market" is quite a "mix" at the moment:

  • for US equities and options, we have the single "usa" market to group them all together
  • with forex we have separate "oanda" and "fxcm" markets, in this case the market represents the exchange/brokerage

Besides being a core component of the SecurityIdentifier class, it is also used:

  • to separate multiple data sources in data file paths (oanda and fxcm have different ticks for EURUSD so they need to be stored separately)
  • to allow different market hours (in market-hours-database.json)
  • to allow different symbol properties (minimum tick size, etc.)

Given that GDAX and Bitfinex are separate exchanges/brokerages (just as Oanda and FXCM) which quote different prices as can be seen here: https://bitcoincharts.com/markets/currency/USD.html I think it makes sense to have one market per exchange. The limitation we currently have in the SecurityIdentifier class is 999 markets, so not an issue for a long time :)

Currencies vs Trade-able pairs

Just as EUR, USD are currencies vs EURUSD, USDJPY are trade-able pairs, I think BTC, ETH etc. should be treated as currencies in the CashBook, maybe should be added here? https://github.com/QuantConnect/Lean/blob/master/Common/Currencies.cs#L233 or they could be their own separate list (CryptoCoins?)

Symbol mapping

This is a tough one, I guess exchanges can decide to do anything they like, in hard fork events. This is how GDAX planned and handled the ETH fork last year: https://blog.coinbase.com/on-the-ethereum-hard-fork-780f1577e986 I agree with @jaredbroad https://github.com/jaredbroad about getting some historical data (especially around this fork event, preferably from more exchanges), it would help us a lot in brainstorming this topic. Other related topics to discuss are new symbol additions and delistings (map files? single csv file with startDate/endDate?)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/QuantConnect/Lean/issues/998#issuecomment-315531166, or mute the thread https://github.com/notifications/unsubscribe-auth/ACI6mXq9DN4gEBahJR-8LE3EfISX4dxQks5sOLISgaJpZM4OIs8j .

jameschch commented 6 years ago

It sounds like you have ideas that go way beyond my initial areas of concern (brokerage integration).

I think there's not really a functional difference between the relationships of Forex to Oanda and GDAX to Bitifinex. There's no real cross-exchange liquidity, although multi-brokerage system could do so fairly simply. The current practice of mapping an exchange to a market makes sense.

I don't know what default behavior you'd be looking at for slippage and fills. The order book is often very thin on alt-coins but I feel like the algorithm implementer can override the default in these cases.

You're right that any crypto-coin should be capable of being held in the cashbook. One big question I have is whether its best that fills on leveraged positions be reflected in the cashbook as currently with Forex.

In terms of the unstructured and dynamic symbols, this is something I have not considered. Certainly Bitfinex are now adding new alt-coins on what seems like a monthly basis. However, GDAX surprised many when they allowing ETH trading, so I'm not sure symbol dynamism will really be put through its paces for this brokerage.

I think you might be walking into a quagmire if you are thinking about hard forks in the same terms as stock splits. The Ethereum hard fork was something of an anomaly in that the minority fork did not die off almost immediately. The concern of the trading system need only be the behavior of the subscribed symbol and the manner in which the exchange choose to represent that to you. It may not always be clear cut which will be the preferred majority hard fork that will inherit the original trading symbol, however I would posit that if a hard fork were ever so contentious that it was not clear which of the hard fork chains would be the majority, this would likely spell utter disaster for both fork chains for any currency that relies on having a practical commercial use.

lucaswalter commented 6 years ago

Thanks for all the work to include crypto! I saw on the pull request that the implementation of GDAX was near competition and was in testing. Are there plans for this to be merged into master in the near future?

jaredbroad commented 6 years ago

Of course :) Just being cautious & getting to a nice stable point before merging.

jaredbroad commented 6 years ago

Merged in GDax Support.