FIX protocol support

[penberg]

HTTP/JSON is widely used by Bitcoin exchanges but traditional exchanges typically support the FIX protocol. It is a plain-text protocol so using it over the internet requires either TLS or VPN. However, there's tons of applications that already support FIX so I think it makes sense as an optional feature for Denarius.

QuickFIX/J is a widely used FIX engine for Java. We probably can wrap it in a Clojure API and use it in Denarius. It is know to be slower than proprietary FIX engines and the code hasn't been touched in a while.

Alternatively, we can use a very fast, low-latency minimal FIX engine core I have written in Java:

https://github.com/penberg/falcon

and build Clojure APIs on top of it.

[analyticbastard]

Certainly FIX should be implemented within Denarius. The only reason I started out with HTTP/JSON is that FIX is almost a world by itself and I wanted to make a proof of concept before facing the titan.

Falcon sounds great, it would be great if we had a connector using it. Denarius should be modular and connectors should be interchangeable. Also, the idea of them being able to work along each other (receiving orders from several connectors at the same time) can be discussed, but I doubt about its benefit.

An alternative regarding FIX is https://github.com/nitinpunjabi/clj-fix which comes in idiomatic Clojure, but I have not tested it yet. I suppose performance is judge here.

[penberg]

clj-fix looks really interesting! I'll take a closer look at it.

The optimizations in Falcon are (1) avoid new as much as possible, (2) use NIO APIs and ByteBuffer as much as possible, (3) implement custom formatting code (because Java APIs new a lot), and (4) cache OS timesource queries.

These are pretty much the same optimizations we ended up doing in a FIX engine written in C to get to approximately 5 μs per message overhead. In Falcon, overhead is closer to 9 μs per message but there might be room for improvement.

The most important optimization of them all is, of course, avoiding new because GC pauses are typically in 1-5 msec range.

[penberg]

Most exchanges are designed so that the core matching engine only supports a proprietary order entry protocol such as NASDAQ OUCH. These proprietary binary protocols have a very limited set of functionality and FIX connectivity is built on top of them. If Denarius ends up with similar design, I don't see a problem supporting both JSON/HTTP and FIX at the same time.

[penberg]

I asked about clj-fix performance via email from @nitinpunjabi and got permission to include his reply here:

Thanks for reaching out. I'm afraid your worries are justified--I would not recommend clj-fix for low-latency applications. The underlying networking is pretty good as it leverages Netty, however the bottleneck I believe is the FIX encoding and decoding. When I created clj-fix for work, the trade-off was made to make messages very human-friendly and to easily create and support multiple FIX implementations. This came at the cost of speed.

In production, we processed around 3000 messages per second on a commodity Linux server box. More than good enough for us, but I doubt it meets your requirements.

3000 messages/second translates to 330 μs/message on average but latency spikes are even worse because of GC pauses.

For reference, BATS claimed 191 μs order entry latency on average in 2011 (I'm sure they improved since then):

http://batstrading.com/resources/features/bats_exchange_Latency.pdf

And that number includes 10 Gigabit network overhead, accepting an order, processing it in the matching engine, and acknowledging or filling the order.

[analyticbastard]

@penberg Nice work! We have a baseline now.

Regarding performance, I think that settles it. I'd go for wrapping Falcon into Denarius. I was thinking of creating namespaces for each connector. Since the cross function (core) executes all functions attached to the key :on-matching of the order's metadata, it is trivial for each connector to have its own callback executed upon order matching.

What do you mean by NASDAQ OUCH and Fix supported on top of proprietary protocols? I was imagining a flat approach, a connector using a proprietary protocol and another supporting FIX, but reading your post sugguest me FIX recevies orders and translates it to OUCH, creating a second network layer.

[penberg]

From latency point of view, it's very difficult to distribute an order book across multiple machines. So it makes sense to keep the core matching engine as lean and mean as possible and distribute by asset. The problem with a flat approach is that FIX connections have session data which makes it harder to scale.

I think a typical architecture is where you have at least two per-asset share-nothing matching engine nodes (for fault tolerance) and N connector nodes (JSON/HTTP and FIX) that can be scaled up independently of assets and matching engines.

[analyticbastard]

Basic design question: How to include these connector nodes? Would they be an independent project, with their own main function in lein? Should they be included within the Denarius main distribution insted? If so, how would we run them?

Apart from that, it would be a good idea for them to check the trading desk margin/balances prior to sending orders to the matching engine.

[penberg]

I think they should be part of Denarius main distribution, sure. I was only suggesting that they should talk to each other via sockets so that they can be physically distributed on different machines (if necessary).

Sorry, I'm a Clojure newbie so I don't know if there's a nice, idiomatic way to package such an application.

:+1: for moving buying power checks to the connector nodes.