fast market entry and exit order rejects

[MicroTrendsTom]

Ideally we will need to add a trailing Stop order for entry for fast market - in order to this a pricing engine/price for long or short needs to be passed in and updated in real-time on tick/bar close then the - call it a TSAR - Trailing Stop and Reverse -that would mean a lighter weight workflow for that mode.

In the absence of TSAR The system in safe mode will que signals long/short/short/long as they come in -and will when ready to take a new trade -when the prior is safely dealt with it will take the last signal in last in first out and cancel the others.

the system has an unsafe mode - this will short cut some of the trade workflow for testing the context of entry and exits etc.- stoploss placement etc so it can be unlocked and is ready for a new trade etc.

The unsafe mode needs to have some kind of feature to detect the incorrect fill or different intended position and close or add one for example or simply change the exit orders placed to match that fill etc... - this might mean to lock the system for a time period and then to only allow trades every 1 to 3 seconds or so... which would miss some trade whipsaw entries... this adds in many caveats - a fast market is typically filtered from a shared data feed such as CQG or TT - and it can even fall behind reality real time - so assessment of tick date stamp versus real time now - and the delta can be used to filter out trading lag/delayed pricing etc. NT8 suffers from what is called Render delay.... so caution for fast market trading and trying to scalp small bars in fast markets...

entry with a stop before the reversal is a better system...

Currently unsafe mode is a compromise of speed and safety and makes no attempt to correct course if the system has gone into a spin of orders and positions due to fast market reversals etc...

The way it is dealt with is - allow a human copilot to deal with any caveats thrown up via a user Interface etc.

Of course there are other items to be considered. @jmscraig i started this in response to your ideas/focus on the topic please do feel free to add etc.

[MicroTrendsTom]

have optimised the code for backtest - bypass the q. and conditional logic for realtime to bypsss the q - unless a problem or wait condition arises when a realtime signal cant be executed as the engine is busy waiting for order statues updates or postions etc - the signal will be added to the Q. - or by default will push through more rapidly this should speed things up

in addition onmarketUpdate rearranged the conditional flow and tests to make it more optimal

[jmscraig]

Hi Tom,

A few thoughts on the Fast Market Topics.

Update: text below was not intended to be bold, just have not figured out how to turn the bold off.

First just wanted to share that in really thin quick functionality test I confirmed with the latest updates that I now see trades occur via both Backtest and Optimization modes in Strategy Analyzer.

---

On to the Fast Market topics.

Environment Context driving my observations and conclusions in the comments below: Latest rev of NT8 Lifetime license, a focus that includes trading NQ, ES, etc. futures from very short period tick charts from a low latency VPS in Chicago with NinjaTrader as the primary broker for these fast-moving futures trades. I believe broker-side Entry execution at NinjaTrader brokerage might be slower than some other Brokers. On the upside tactical execution at NinjaTrader brokerage might have more features to work with and fewer process flow constraints than some of other brokers.

Now a few thoughts on the topics you raised

“The unsafe mode needs to have some kind of feature to detect the incorrect fill or different intended position and close or add one for example or simply change the exit orders placed to match that fill etc... - this might mean to lock the system for a time period and then to only allow trades every 1 to 3 seconds or so... which would miss some trade whipsaw entries... “

I have been chasing the fast markets topics for some time. I am still on a journey. The comments below share where I am now.

This reply is already long so for now I will skip the history and reasoning behind the comments and just cut to the chase.

Given the long list of trade execution operational latency delays upstream and outside of my control, where I do have control I am designing and building against the following principles or biases. 1) Strive for ‘Ultra-Fast” “zero-wait state biased” design and operations leveraging learnings and concepts from proven high capacity and high-speed Exchange management and OrderBook Management projects (one example being (https://github.com/mzheravin/exchange-core). 2) Structure Risk Management & Order Sizing to expect that all required Pre-Checks, Order-Pre-Validations will be completed prior to receiving ticks inducing operational execution (E.g. Entries, Order Management, Exits, etc.) or after execution takes place. (E.g. responding to Overfills, missed limit Entries or Exits and rejected orders).

A. Pre-processed and Out of Band Processing for Risk Management: Enabling 1 & 2 above requires robust ‘always-on’, continually processed, pro-active, pre-processed Risk Management measures like Max Loss for Period, Consecutive Losses, Data Lag, Connectivity, General Market Readiness for Long/Short entries, Level of Risk/Sizing Allowed for the next few moments, # of Entries in and Allowed Concurrently, etc. Pragmatically these metrics and statuses are held in simple shared Class Level Bools or Doubles so when the signal hits execution is immediate without trade execution dependency on function calls or complex logic flows.

The robust work-flow engine you built would be a good toolkit to manage this constant preparatory work. Possibly in a way that is not really that far from what is happening now, maybe just more of the workflow occurring ‘at the end of the previous tick or happens every X ticks’ and less workflow required prior to entry execution just after the entry signal ticks come in.

B) Given that the market will not wait for Ninja to deliver to us all the confirmations we want before we need to initiate the next execution step embrace the expectation and norm that we will receive high fills, low fills, overfills and accommodate this reality:

1) Adjust default Order Size down so that High Fills and Overfills fit your Risk profile perfectly rather than disrupt it.

2) Build Tick–by–Tick prioritized rapid response workflow identify and respond to what fills have actually-transpired or not as they happen. For example, when receiving High Fills or an Overfill if the Market micro-direction is moving in the opposite direction seek the most profitable near-term exit (not just a panic costly Market order exit). However, if the Market micro-direction aligns to your new unplanned but not unanticipated position then adopt the entry with risk appropriate Profit Target and Stop Loss orders.

The robust work-flow engine you built already includes a lot of the logic required to identify and respond to what has actually transpired.

For me, for Fast Markets rather than thin it out I envision the fully suite of what you have built and more as being very important for managing execution in Fast Markets. I recommend positioning as much of that Risk Management and Readiness processing prior to receiving the execution bar of data, and (I have not covered this yet) as pushing as much possible of the execution wait-state and workload Out-of-Band in parallels threads leaving core the strategy thread ready for fast immediate execution for Entry & Exit Assessment and Execution as soon as new data comes in.

There is more. I am hoping these few ideas provide a general sense my conclusions to date.

Your thoughts?

James

[MicroTrendsTom]

wonderful feedback really very nice to read from someone with tech skills and in depth understanding of this topic. (I got too much code to write for 2 late projects so I sadly I cannot write what I want to, but I hope to make a good reply.)

I like this a lot from your reply "designing and building against the following principles or biases." and your thoughts I agree in what i have read and i think that it can be implemented or even spread out against the modes and if needed lateral approaches of de-coupling.

This engine premise and aim was definitely not a fast market scalping type of engine more of a family saloon - but with a sports mode with caveats and limits. Its aim is to not sink in a fast market - rather than to try attempt to trade it etc.)

Other designs in short - I do have different engines for each trade scenario and this one was really just a generic example and experiment to see if it appealed to anyone, aimed at the general use for most retail trading applications, that were not really low latency or for fast scalp reversal pseudo HFT- given that retail is not low latency and DMA is etc. - so we have to be aware and avoid data lag and limitations of shared datafeed and order routing etc

I believe there is room for different modes in the system and to evolve it also and this is the current process now with collaboration this is possible - but it will be a compromise and there will be a limit compared to DMA and solution directly with a Fix engine etc. My vision for the usage of the system when I trade is 1 to 3 trades per session... yours might be 10 to 50 and they go through all market states... so there is an attempt to cater for other usages and market speeds/ signals speeds - low or high frequency.

So, there are 3 modes

1 Default mode - Sync/ mode The rather verbose but reduced instruction set workflow from its Ancestor code base - we wouldn’t want to try to take a trade every second - I doubt it would be a profitable system anyway- but if we had a long short long occur in very quick succession we want it to overcome and continue safely etc. - if the system state is busy the signal gets placed on the q for later execution - Async.

2 Realtime Signal Q mode - Sync + Async mode - this is aimed more at fast market handling - Signal LIFO Execution - this places signals into a q in Realtime - and if the q has count of 1 it executes if possible -if greater >1 then we wait for on Market Update calls to the Q process method to deal with it. - So, if the q has 10 signals in it by the time execution context returns to process it would de-Q them all and execute the most recent/last signal if appropriate etc.

3 unsafe mode -this is synchronous and will attempt to take all trades with no special order workflow therefore not checking up on system balances etc. - Copilot required...

NOTE: Later This copilot could be a monitor... in a different thread and process.... looking at the account and instrument with a set of base rules to manage to. 100% agreed that a fast execution system cannot have time due to the nature of trading to try to correct and get involved with the flow of trading -to add one take one off - it could simply accept the difference of planned outcome and adapt to it -for example the expected position of -5 ends up as a fill of -6 - ok so then the exit orders need to balance -6 not -5 etc.

but also, we might take time to close or add one etc.

NT8 and Parallel threading etc.
My experience may be different, but it resulted in a dumbing of code and patterns as a compromise to get it to work at all. The problem with NT8 is that we cannot safely or reliably use multithreading solutions . When launching operations in parallel they simply stop responding and NT8 doesn't return etc... we are talking Plinq and parallelism tasks/actions queues -all of that amazing stuff we cannot use. So we can’t have too much fun in that respect as coders and we cant take advantage of it as algo traders we need reliability via compromise - or we approach it differently and decouple the logical units see below... But to keep it all housed in 1 process the “strategy”, we have to mimic that parallel/ async model as much as possible.
N.B. Even NinjaTrader took down support and code methods and best practices for multithreading from its guide…in about 2018 when I was working in that area and reporting to them etc. I found that even dispatcher timers were not ideal so I mimic timers via OnMarketData. recently: I also found that using a “signal q” in back testing caused it to stop responding yet in real time its fine... but that’s not a problem – we don’t want to use that for backtest 😊

So what I have tried to do is to use the On Marketdata as the driver to emulate as much as possible that Nirvana you describe at the end as I failed to make it reliable prior and spent a good 18months fighting it. and so we could not roll out artwork and code we have to make it clunky and dream of the other common Parallel patterns available now instead and hope we can use them in NT9.

Decoupled Logical units HOWEVER - we can add them as side by side processes that monitor and interact and are in loosely coupled… so we have a pattern such as a signal producer, a signal consumer/executor and a trade manager that monitors for fills and acts accordingly for exits and so forth. I have done this in a different project – but you cannot backtest…. this method doesn’t even run a NT strategy etc. rather they hang off a data series for signals and the execution/ monitoring ,management are 100% parallel and hang off the account object subscribed to events as Addons etc.

Conclusion Admittedly it is a bit bleeding edge atm

so in short test evolve feedback etc and add in modes and patterns/workflow to suit for different styles even look to decoupling parallel processes if need be to monitor for issues or use a human copilot etc

[MicroTrendsTom]

Note: Your patterns for parallelism might work where my past has failed.

[jmscraig]

"wonderful feedback really very nice to read from someone with tech skills and in depth understanding of this topic."

I could not agree more from my side.

Today made a few small changes to AlgoSystemBase and the ATS sample strat to try and address the constant popups error for attempted change orders while order status is "Cancel Pending" Testing now for this and to try and find deadlocks by running only one strategy each on two machines with all tracing and logging turned on. With only one strategy running per machine should be easier to use the logging to help debug.

One these topics in your posts above

Very good thoughts and I have much to reply on but would like to start with very near-term pragmatic questions related to what I am designing and coding on this week.

---

A quick FYI just to clarify my design and development interest. Though I know the discussion I present sounds a lot like HFT and hard core scalping I agree that is not possible with the NT8 platform. As well chasing 500-3000 trades a day is not my present interest.

While my work leverages much of the same concepts as scalping for entries I am much more interested in 'Fast, Early Entries into sustained moves and Fast Exits and in-between those two 'well managed chaos cycles' robust trade and risk management to enable stronger risk management.

So yes I am working to get entries, exists and the trade management engine moving as fast as I can (I target 200-700 nanoseconds execution time for actual entry and exit decisions ) and there may be a lot of management of chaos at the open of the trade but that trade may run for a number of minutes and I am attempting with some portion of the entry to catch 40-120 points a go of larger moves.

So while I am chasing the some of the same speed capabilities of someone interested in entries on each tick I am not currently pursing a collection of trades that average only 1-3 seconds in length.

The idea that I might need to wait the really, really, really long time of 1-3 seconds to have some of these chaotic moves fully settled out does not bother me.

Later I expect to look at trading 401Ks through IB or other brokers and also expect to look at systems for trading a collection of managed accounts and for those topics the features and goals of the AlgoSystemBase as you first shared it are of strong interest.

---

So on to the near-term pragmatic conversation ... I would like to first raise conversation on the topics in this quote

"The unsafe mode needs to have some kind of feature to detect the incorrect fill or different intended position and close or add one for example or simply change the exit orders placed to match that fill etc... - this might mean to lock the system for a time period and then to only allow trades every 1 to 3 seconds or so..."

First let's assume over-watch disaster prevention monitoring is already running today (as it is for my work today on two separate machines in separate geographic locations monitoring each account with code that, unlike a NT8 strategy, can not be externally randomly disabled).

Now the Questions: From all your experience to date ....

Question 1) In regards to the ideas you share that quote just above. So given that real-time out-of-band over-watch is in place, using either 'Unsafe mode" the real-time queuing or both, and an eye toward the AlgoSystemBase design what are the minimum features you think are required for to enable reasonable risk-managed 'Unsafe' and queued high speed trading? (This might already be in the AlgoSystemBase?)

---

On to Question 2

"this might mean to lock the system for a time period and then to only allow trades every 1 to 3 seconds or so..."

Question 2) Would you mind describing more of the vision you have here? What actions, tests, validations do you see taking place during this 1-3 seconds between trades and which bits actually drive the time delays?

Thanks!

[MicroTrendsTom]

ok fantastic a well received FYI: understood we are on the same page -we both want timely entry's and exits and are not scalping fast reversals etc. The system needs to be rock solid during chaotic periods and fast market and if a series of fast reversals do occur it can manage and be fault tolerant. this might mean not all entries are attempted or taken or are late and should be ignored etc.

For Q1: Unsafe mode to safeguard against erroneous positions and order balances... is the main item and how to correct and or go with it with the exit orders etc

The RT Q needs testing.... :-) but should really work the same as the default mode with less bumps...atleast that was the plan As of time of writing it was migrated from a NT7 engine long ago

For Q2: so realty this is about waiting for the OrderStates and Position quantity to be updated and make sure all loose ends are accounted for -so we can say with certainty all orders are cancelled or filled or working -and the position size is what we expect and we are able to allow trade management or new signals to take place.. .so some or all of the tests represented by the TradeWorkflow Enum. as we have not implemented all of them as yet.

[jmscraig]

"understood we are on the same page -we both want timely entry's and exits and are not scalping fast reversals etc." I want to make sure I am not misrepresenting myself Lol...
trying to improve timely entries I evolved to Fast Entries.
trying to get Fast Entries right I finally capitulate to the concept that I just needed to learn and execute the best scalping entries I possibly could. There is just no getting around it.. a really good fast early Entry looks so much like a scalp ... "if it looks like a Duck, walks like a Duck, sounds a Duck ... its a Duck"

Really powerful concepts here.. I identifying market-micro reversals to your entry early enough and developing capability to exit fast, seeking all exits to be profitable... very powerful to reduce drawdown

AND.. your long term trade signal still say enter in the original direction so IF the market-micro reversal going in support of the trade your back again.

I am exploring adoption of this approach because it will allowed me to consider real big swing Algos that come with risk of really high drawdown. I can dramatically reduce the drawdown profile because I am always out fast during downturns while the trade is still pointing to stay for the long haul.

By the way for market times outside of the opening hour and closing 35 minutes so far it looks like AlgoSystemBase might be able to do the job WITHOUT Unsafe enabled.

============================================= "AlgoSystemBase might be able to do the job WITHOUT Unsafe enabled."

I envision a layered approach. A small, tight "Exit Readiness Assessment and Execution" region followed by a "Fast Entry Readiness Assessment and Execution" (aka "Fast Exits-Entries")right at the top of key event handlers (E.g. OBU(), OEU(), etc.). This "Fast Exits-Entries" region can act screaming fast tick by tick (especially for Exit, Exit Chase, etc.) for brief milliseconds-seconds of chaos then letting the workflow engine handle the rest.

In a loose way, allowing the Workflow Engine to at least sometimes operate sort of out of band because Fast Exit-Entry region governs when it passes control on executes "return;" withholding / denying further execution for that specific tick level OBU() event, Especially during Escape, Exit Chase or Entry Chase when every tick counts.

Things like Bracket Entries, ongoing trade management (updates of Profit Targets, Stop Loss, graduation of an open entry that has achieved X ticks of favorable movement to the next higher level Exit Management Plan, etc.), completion of FlattenAll, CancelAllOrders(), ClosePosition(), ongoing Risk Management Assessment, Trade Sizing, assessment of overall market state and trend etc. all go to the workflow engine.

Tongue-in-check one might say AlgoSystemBase workflow engine used to operating on 1 minute chart might not know they have ever been denied access to any OBU() events because they have ongoing workload to carry.

---

I also posted related thoughts in the discussion area for this issue.. https://github.com/MicroTrendsLtd/NinjaTrader8/issues/4#issuecomment-737595210

Your thoughts?

[jmscraig]

Good answers to Q1 and Q2 thanks

[MicroTrendsTom]

"understood we are on the same page -we both want timely entry's and exits and are not scalping fast reversals etc." I want to make sure I am not misrepresenting myself Lol... trying to improve timely entries I evolved to Fast Entries. trying to get Fast Entries right I finally capitulate to the concept that I just needed to learn and execute the best scalping entries I possibly could. There is just no getting around it.. a really good fast early Entry looks so much like a scalp ... "if it looks like a Duck, walks like a Duck, sounds a Duck ... its a Duck"

Your thoughts?

Scalping is really defined to me as taking small profits and a higher trade frequency etc slippage and liquidity are key etc as well perhaps as a cyclical trending market which is not ischemic as the ideal conditions. 2 to 4 to 8 lots for example depending on market and time of day etc perhaps 10 to 40 trade operations per day - with inverted risk reward - high win ratio etc. 75 to 90% wins

For fast micro reversals i have a different engine altogether and approach that would probably be the fastest type possible... for NT/desktop trading And this could be inserted as a mode here and would use a subset of the trade workflow,. create a price engine to calculate the long and short prices - When the desired conditions exist to place the entry at the long or short price via stop entry is placed... and it can be OCO the price will calculated on each or tick and the order moved in as the price changes... the market price hits the stop and you are in - and perhaps even within 1 second or more depending on biz logic you next opposite order is in the market waiting... and this would be behind the stoplosses etc - or they would move up to be filled before it SAR so your existing position continues and is not closed unless by logic - it will close at fill of target or stop loss and the entry order maybe filled then also

so we can be clever with order types and pricing etc and that is so much the fastest way to trade etc you can even have your exits above ready - liquidity is the key to avoid slippage etc a method like that on FDAX is good for 4 to 8 lots for example and provides 15 good days of cyclical trend per month and 5 terrible on average and the hours to do it are the first 2 to 3 hours of Eruex or NYSE Etc

NQ is candidate also

So we build with either safety in mind at the outset and to compromise on speed with generic functions. or we build super fast lightweight solutions with specific methods imprinted - that avoid high latency actions etc by way of order techniques etc which are inherently safe fit by design without extra overhead etc...
so this engine is more generic and cumbersome in comparison etc but would fit most applications in day trading etc

[jmscraig]

Scalping is really defined to me as taking small profits and a higher trade frequency etc slippage and liquidity are key etc as well perhaps as a cyclical trending market which is not ischemic as the ideal conditions. 2 to 4 to 8 lots for example depending on market and time of day etc

For fast micro reversals i have a different engine altogether and approach that would probably be the fastest type possible... And this could be inserted as a mode here and would use a subset of the trade workflow,.

Yes.

"And this could be inserted as a mode here and would use a subset of the trade workflow" Yes, I think component layering is the way to go. I get the feeling that asking one trade engine to cover all contexts would lead to an overly complex, hard to maintain set of compromises that finds it hard to do any one or several jobs extremely well.

"and it can be OCO"

For fast trading do we want OCO? OCO adds up to 200ms per leg on entry, I suspect OCO delays ChangeOrder execution and I was eyeing OCO delays after a CancelOrder request was sent or initiated by the exchange facing server as a contributor to Rejects due to CancelPending (there is no way for us in advance to see Broker exchange facing side delays as the legs of OCO are managed through a cancel cycle).

I have been migrating toward on-the-VPS simulated OCA for lower tier (trades with a shorter expected life) bracketed by two layers of outer Overwatch monitor to prevent disaster.

What do you feel the right role is for OCO on the shorter lived trades?

[MicroTrendsTom]

"What do you feel the right role is for OCO on the shorter lived trades?" For some trades OCO is the solution for Entry so Entry OCO long and short with stop same time. For example a pullback - will it reverse or be a continuation - OCO is the tool to use for it as fast response already in the market well almost... etc

im trying to spin up the engine i was referring to... its been gathering dust for years as its pilot passed away (RIP Paul) so the project was canned after some 2 years - i haven't met anyone yet who understands it yet and can pull the sword from the stone. ha the othe team member uses it daily for academic R&D - i dont have time to use it... it came from a new jersey uni dept of financial systems and was part of a project - i was the one to code it for them.

i think there is plenty of scope to add modes to the engine and remove limitations on 4 in and out - even if its a collection of in and outs in chunks of 1 to 4

[jmscraig]

Sounds similar in goals what I have been iterating through with a focus on gaining good entries, some short lived to shore up the risk profile but always with a goal to keep pushing a portion contracts into longer and longer looking Exit Management Plans to catch long runs.

To me good fast entries and fast exits are to most important bit and this part I have well enough sorted "to keep pushing a portion contracts into longer and longer looking Exit Management Plans to catch long runs."

If you feel that evolving that code would be good component fit with the overall system I can have a look at it. If not, that is totally OK as well.

[MicroTrendsTom]

"If you feel that evolving that code would be good component fit with the overall system I can have a look at it. If not, that is totally OK as well.: yes for sure as long as its backwards compatible with any systems derived from it now. - so the idea to abstract out layers and have the clunky 4 pot cylinder engine was one idea where it shared some of the base with some othertypes.. .or we squeeze it all in and use property overriding and browsable false so any derived systems that didn't want some extra features to appear are ok etc

[jmscraig]

That is a great interface for manual trading or semi manual trading through NT. Some people would grump that it is too complex but once you have lived in the reality of tracking the hard right edge moment by moment that complexity is what you need to stand on solid ground.

I could see that as an interface for controlling or influencing strategies as they run faster than a human could ever respond.

[jmscraig]

"... so the idea to abstract out layers and have the clunky 4 pot cylinder engine was one idea where it shared some of the base with some othertypes.. .or we squeeze it all in and use property overriding and browsable false so any derived systems that didn't want some extra features to appear are ok etc"

This is exactly the path I have been thinking, some, perhaps the default implementation does not include it. And has no hard dependencies on it, or any other peer "drop-in" systems that may join.

[jmscraig]

In the case of the Fast trading layer, NT8 design is set up nicely to pretty easily use OBU() / OMDU(0) / OOU() as a Factory Controller pattern to decide which component gets first or only access to events as they come in which appears to make it seem difficult to build a stack of layered components and retain independence.

Events Off the fast dataset needed by the Fast focused component get those events first and then handoff to the next layer or not, depending on case.

So following the Factory pattern, control of which component gets the incoming event first or timed workflow initiation or at all might be a case structure at the top of or called in the top of the key event handlers as events come it or timed workflow initiation signals are fired.

Layered workflow components do their bit by case and return control to the Factory Controller workflow case structure who decides if any and which Component would gain access to the event data or workflow action signal next.

This top level workflow case structure has to stay very thin and be free of constraints that may not be shared by all components over the long term so it has to stay very thin and be a layer above the current workflow engine. Maybe in some ways like ProcessWorkflow() is today.