NT8 broker execution issues with OnOrderUpate and OnExecutionUpdate - Rithmic and Interactive Brokers

MicroTrendsTom commented 3 years ago

Feedback received with thanks and Reported by Sidlercom https://github.com/MicroTrendsLtd/NinjaTrader8/discussions/41#discussioncomment-290550

The crux is that NT8 is not 100% compatible or broker neutral for rithmic and Interactive brokers from within a strategy. OnOrderUpdate and OnExecutionUpdate and OnPoisitonUpdate may or may not be called in sequence or if at all. Broker centric workarounds are required to make NT8 it work explicitly in the derived strategy classes instead of being handled in the plumbing below internally within NT8.

This is a debug trace supplied by Sidlercom

partiall fill

13.05.2021 19:44:39:189 > SidiSystemBase > OnExecutionUpdate > Long > ↑LE.MKT#3.ST > execution='123673938|1193980675|1193980675' instrument='MNQ 06-21' account='' exchange=Globex price=13018 quantity=3 marketPosition=Long orderId='1193980675' time='2021-05-13 19:44:39' sod=False statementDate='2021-05-13' 13.05.2021 19:44:39:265 > SidiSystemBase > OnOrderUpdate > Long > ↑LE.MKT#3.ST | state=Filled | limitPrice=0 | stopPrice=0 | qty=10 | filled=10 |avgPrice= 13017.8 | time=13.05.2021 19:44:39 | Id=1193980675 | error=NoError

it doesn't go on here because Rithmic no longer calls the OnExecutionUpdate()

Initial Solution draft Pattern be implemented within the ATSQuadroStrategyBase that can handle these scenarios so the strategy is broker neutral and works the same for any connection etc Order Routing etc

It could be it detects Interactive Brokers and Rithmic and handles workflow processing or a generic solution will one with the option for a parameter on/off to engage a monitor to check back on the state of the execution after an interval and then move the workflow forwards - if CEP event driven model fails to move it forwards.

Risk mitigation with NT8 we are best to avoid multithreading and timers - due to deadlocks etc A reliable pattern is to use events to drive the workflow forwards and use the tick/market flow to drive a virtual monitoring/timer interval/timeout to check on the order/execution state.. This kind of pattern is used within the system already and so it should be easy to implement without risk of breaking the system.

In summary if events dry up or are missed due to connectivity the market data will call a method to call back into the workflow to check the executions are filled and then move forwards.

https://github.com/MicroTrendsLtd/NinjaTrader8/blob/main/ATSQuadroStrategyBase/NinjaTrader%208/bin/Custom/Strategies/AlgoSystemBase.cs

For partial fills Checks OnOrderUpdate also to fire OnExecutionUpdate

protected override void OnMarketData(MarketDataEventArgs marketDataUpdate)
is used to check back in after an interval //call back in to workflow - Monitor for Workflow events or for Error State Assessment of workflow changes and when to set the call back to workflow pattern.

jmscraig commented 3 years ago

HI Tom,

Saw this issue posted. Just wanted to check if you have seen the significant work NT_Jim put in to address at least some of these issues. Reply if not and I will do find some links to those conversations and sample templates.

James

jmscraig commented 3 years ago

I did not quickly find links to those conversations but did locate a key sample file I was looking to post.

In the attached .cs file by NT team member NT_Jim ...

.. from within OnMarketUPdate() calls a function "private void InlineExecutionUpdate(Order order, double AvgFillPrice, int quantity, MarketPosition marketPosition, string orderId)" to act as a pseudo OnExectutionUpdate() event manager and attempts to address at least some of the same concerns you list above.

Note the attached file is NOT an NT8 export file. I just zipped the *.cs file in place so that GitHub would let me attach it to this message.

SampleUnmanagedRithmicInteractiveBrokersNonNTOnOrderUpdate2020.zip

`

MicroTrendsTom commented 3 years ago

HI Tom,

Saw this issue posted. Just wanted to check if you have seen the significant work NT_Jim put in to address at least some of these issues. Reply if not and I will do find some links to those conversations and sample templates.

James

i hadnt even looked to be honest but i will now you have posted the example excellent

MicroTrendsTom commented 3 years ago

I did not quickly find links to those conversations but did locate a key sample file I was looking to post.

In the attached .cs file by NT team member NT_Jim ...

.. from within OnMarketUPdate() calls a function "private void InlineExecutionUpdate(Order order, double AvgFillPrice, int quantity, MarketPosition marketPosition, string orderId)" to act as a pseudo OnExectutionUpdate() event manager and attempts to address at least some of the same concerns you list above.

Note the attached file is NOT an NT8 export file. I just zipped the *.cs file in place so that GitHub would let me attach it to this message.

SampleUnmanagedRithmicInteractiveBrokersNonNTOnOrderUpdate2020.zip

`

AWESOME thank you - sounds like i was thinking along the same lines a psuedo onExec - excellent i will have to try find time i hope in the next few weeks to get into this - bogged down with asp.net core and blazor SignalR atm to buikld a cloud based correlation gui and api for a product - ETA 2 weeks

MicroTrendsTom commented 3 years ago

Ok I had a brief look at the code example some caveats are 1) OnOrder doesnt always fire - connection or other issues 2) Position tally cannot safely be done and left within in scope OrderFills due to caveats external and internal

Part A: So we might look to quiz the connection for rithmic or IB or via parameter turn on a different workflow that would push through a call to onExec or other method to finalize the order enrty cycle.

OnOrder to drive it and of possibly push a simulated call to OnExec etc

Possible also update the executions collection to push through the onExec event - but care needs to be taken not to upset executions and trade calculations - if it worked

Part B: then also a pseudo onMArketData timeout monitor can look for the workflow and assess its complete and nudge it forwards as plan b pattern

MicroTrendsTom commented 3 years ago

Sorry but still far off from being able to address this myself due to other obligations kidnapping my time 24/7 almost coding for months on end etc

MicroTrendsTom commented 2 years ago

Unable to find time to investigate meanwhile waiting for NT to fix this in the platform as this is clearly just an omission their end and no doubt with no fiscal reasons to fix it as a priority.

Please note there are other projects as above that have fixed this issue by adding code to the engine - for all other broker users this wont be required etc

Time permitting will wheel back to this if this projects becomes active as a dependency on other projects etc

MicroTrendsTom commented 2 years ago

One way to look at this is: This type of engine is only really good for backtesting and limited in live trading for attended and non fault tolerant commercial swing - but excellent for rapid algo design and testing.

So then we would use an engine that does not suffer from any brokerage limiations as its totally generic bound to the API layers internally within NT8 itself such as the type we use here: https://algofuturestrader.zendesk.com/hc/en-us/articles/360055568611-NinjaTrader-7-AFT7-day-trading-futures-system-overview-of-components

jmscraig commented 2 years ago

So much in the market has shifted .. and a huge amount of liquidity has shifted to crypto that rather than lights-out auto-trading I am presently more interested in first in automation assisted trading, then leading toward human assisted auto-trading and finally some lights-out algo execution again.

These images demonstrate the depth of complexity to be managed and in general what I am working to build with an eye toward crypto and classic futures for deployment. https://algofuturestrader.zendesk.com/hc/en-us/articles/360055568611-NinjaTrader-7-AFT7-day-trading-futures-system-overview-of-components

If for no other reason than to respect the NinjaTrader Terms of Service for me NT8 needs to be an optional companion to the core platform.

For the interface stack I have been investigating

1) non-NT8 WPF, WinUI 3 (including for the server side) and a few other Windows centric things.

2) A few high speed bi-directional C# <> Javascript webhook interface scheme caught my eye:

2.1) One depends upon cref chromium, and another broad Javascript. cref might be a little old but the author claims the integration is very light.. and can be free from asyc dependencies etc. This would open up the possibility to leverage nice fast and responsive existing React and Vue UIs.

2.2) I saw another bi-directional C# <> Javascript webhook interface for integrating the slower-and-less-sexy-than-React Microsoft Blazor web UI ... another variant using Microsoft's not yet mature replacement for cref.

3). Regarding WinUI 3.. Third Party dependencies are a path that raises concerns for me but the three (WinUI3, Maui, Xamarin) free forever libraries on this page have caught my eye.. they do simplify the work to build a UI.. https://www.devexpress.com/Products/Try/
A repository with the DevEx WinUI3 demo code https://github.com/DevExpress/winui-demos I attached the installer for the more reliable pre-compiled version of the WinUI3 component demo. This demo shows the two code pages (UI screen, C# code behind the screen) for each demo screen.. You also get access to the precompiled demo once you sign up for the free WinUI3 component library. I don't know how long term I would use the WinUI3 package, I am thinking if entry is low enough cost and the NET6 libraries it depends on run and Windows clients as well as cloud data center servers might give a lot of UI fast and let me work on the other topics faster.

FeatureDemo (Package).zip

4) What about WPF? NT8 WPF is hugely complex to build in given the restrictions the NT added.

If building a NT8 collaborating but independent window panels: 1) Github has popular WPF build kits that might simplify a path forward with WPF

Your thoughts? James

MicroTrendsTom commented 2 years ago

im a long fan of devexpress and have an active annual sub raw WPF in Visutal studio works great NT8 WPF yes i try to avoid it as much as possible.

the only caveat is for things like property girds if you need them the one in nt8 is not good to work with. devexpress and others might be better or use a winforms host etc...

so this convo we can put into a discussion on uis as this is about rithmic and so on... point being im not going to resolve it as the best layer for a UI is not in a strategy as NT8 strategy is not the best thing for hybrid algo trading in NT8 - this will demonstrate exactly how soon as i can drop onto it- http://algofuturestrader.com

we can discuss more at #52

MicroTrendsLtd / NinjaTrader8

NT8 broker execution issues with OnOrderUpate and OnExecutionUpdate - Rithmic and Interactive Brokers #45