jmscraig commented 3 years ago

Greetings! Happy Monday. I have 20+ of hours test execution completed against the latest code posted here using my development environment overnight processing with live market date on a Sim account on a VPS during high volume daytime hours.

Completed thousands of successful trade executions but surfaced three issues.

1) Deadlocks 2) LIST<> read write index related conflicts, 3) ChangeOrder submission rejects because the order is in “Cancel Pending” status

3) ChangeOrder submission rejects because the order is in “Cancel Pending” status

Resolving this one seems pretty simple .. I will see how far checking Order status to == Active or Working just prior to submission.

Just thinking out loud... If that does not work and we find that the Popup error messages have a connection with NT8 / strategy stability then at what point do we consider a transition to setting error handling to "IgnoreAllErrors"

Your thoughts? Code fixes to paste in? Better Ideas? James

MicroTrendsTom commented 3 years ago

in fact there are things in the distance in 2021 i cant speak of here but some very exciting stuff you might like to get involved with so at some point we will catch up and talk of the universe and possibilities its 4am i got a glitch and my brain cant see it hahaha so i cant release the last commit

MicroTrendsTom commented 3 years ago

ok delete the email all good

jmscraig commented 3 years ago

Cool I just left an Enhancement post. I have built this and use.. it is really part of my secret sauce / proprietary jewels and I wanted to share the concepts with you but will probably delete the post soon because I am not yet sure I want to leave it public

https://github.com/MicroTrendsLtd/NinjaTrader8/issues/20#issue-757368693

jmscraig commented 3 years ago

Partial fills .. in fact i have a customer who wants to use this but i dont have time to code for him.. so it could be an option for you if you are interested in that type of thing?

Could be .. if that is still a hot question in a few weeks and would double as value for adding non-existent capability. I have not coded for others but have consulting business license.

I have a pretty good partial fill solution that could be migrated to this stack. In my mind the pre-step before that coding is learning what you had in mind or figuring out the desired approach here to Limit entries, how to respond to Limit entries left at the alter by the market moving away, how long they stay open, is there any action response (Entry Chase, Entry Split, etc.) handling more uniqueness in PT and SL orders

Any existing plans on these?

jmscraig commented 3 years ago

got the email ... that can be deleted ..

I have a commercial use license for Zoom so that is a tool as well.

MicroTrendsTom commented 3 years ago

cool do you like AZURE? messaging and the such like? SignalR? push notifications?

im hatching a vision hahah im going to stop coding soon as i will write garbage been here since 10am hahah

MicroTrendsTom commented 3 years ago

ideas on limits will come back to yep

jmscraig commented 3 years ago

On Limits .. cool I am asking to help better design and prioritize my near term work and want to build core, compatible or supporting .. rather than conflicting.

MicroTrendsTom commented 3 years ago

a layer of abstraction above the parent for orders types and methods.. and patterns a layer above that for trade entry filters, and exits off the shelf etc a gui for interaction and so on

jmscraig commented 3 years ago

Simulate OCA: Caveat Constrained within the limits of Broker specific exchange facing servers and rules...

I am hoping the Simulated OCA helps in solve or reduce likely of the some problems this thread is focused on because it

reduces the number of times Broker side rules are violated
reduces entry time
reduces exit time, as one leg is not held up waiting for confirmation/risk management activities to make sure the other leg is managed as well.

---- added one

I am expecting that optional, use case specific use of Simulated OCA allows this stack to sometimes run a faster with a little less heavy breathing trying to keep it all in sync because it does not need to risk manage the broker side OCO and fewer ordering timing delays.

jmscraig commented 3 years ago

a layer of abstraction above the parent for orders types and methods.. and patterns a layer above that for trade entry filters, and exits off the shelf etc a gui for interaction and so on

Cool! Can't wait to see it. Or some of it as it starts coming visible.

I figured you had a stack of really good options already running you would want to start by cutting and pasting from.

jmscraig commented 3 years ago

been here since 10am

Oh ya, for sure.

Go. Sleep. Sleep in! Its the weekend

jmscraig commented 3 years ago

I put another six hours in on this work today. Signing off to sleep so doing a quick handoff.

During Market Hours froze again while using/testing both strategy files in the latest commit.

I put together a collection of things we can do (code) to avoid deadlocks.. we first need to identify where/how/why it is happening.

Debugging through DEBUG log files VS was not that helpful to try and find deadlocks >>> I suggest we extend DEBUG Logs to more directly identify when setting locks is the next step in the code (edit a few existing DEBUG message strings, copy-paste a few new "if(tracing) Print(.." blocks to just prior to the lock calls so it is more clear when a specific lock as contributed to a deadlock

After even more reading if we get any hint that a lock on Account.Orders may not have prevented a deadlock switch to locking on Account.All. Reasoning: Interesting NT help forum posts identify that NT does not tell us what objects they are locking on under the hood so we get deadlocks by locking on the wrong objects. The solution on our side is to lock on Account.All if Account.Orders does not pan out.

jmscraig commented 3 years ago

Here is a sample bit of code that we can use to prevent deadlocks when a lock is not free, allow us the ability to add response logic and also help us with better Debugging because we will know which locks often busy, etc.

Did not paste in cleanly but worth a look.

        try
        {

    //isRealtime
    if (!IsHistoricalTradeOrPlayBack && !orderCancelInspectEachOrDoBatchCancel) 
            {
                // lock (Account.Orders)

                // use of try-Monitor.TryEnter-finally (similar to source code inside "Lock") solves a number of needs
                //   1) Enchances DEBUG tracing with the new information on if a lock was free or blocked and what our code did in response
                //   2) Prevent deadlocks because it allows the code to bypass a locked object rather than "hang" waiting for access
                //   3) Allows us to respond to the block in a variety of ways by coding logic into the 'finally' block

                bool accountLockAcquired = false;

                try
                {
                    Monitor.TryEnter(Account.Orders, ref accountLockAcquired);

                    if (accountLockAcquired)
                    {
                        if (tracing)
                            Print("Lock Acquired Account.CancelAllOrders() ");

                        Account.CancelAllOrders(this.Instrument);
                    }
                }
                finally
                {
                    if (accountLockAcquired)
                    {
                        Monitor.Exit(Account.Orders);
                    }
                    else 
                    {
                        if (tracing)
                                Print("Lock BLOCKED Account.CancelAllOrders() ");

                        /// Tom, here I am thinking that sometimes logic gets added for a retry loop
                        /// Or in the case of not getting a lock on Account.Orders maybe in finally we ensure execution is 
                        ///    attempted against Account.Cancel(OrdersActive); ... or vis-versa and then if that fails do the retry loops.  
                        ///  The current code has three pragmatically redundant methods to close all open orders.   
                        ///  It seems like if one of the two objects we are trying to Lock on is blocked, as long 
                        ///      as we are able to fully execute two of the three MAYBE we wont need any retry cycles. 
                    }   
                }           
            } // End of try - finally               

          }

jmscraig commented 3 years ago

Night.. Enjoy your Saturday.

jmscraig commented 3 years ago

Just an FYI.. I am brewing up a good sized Pull Request for a further refactorying of CancelAllOrders().

Extends your recent Pull Requests but adds in

Real-time functionality to just cancel Stop orders on OCO TP SL
Wait Free Locking - dramatically reduces hang or deadlock risk due to locks
Immediate Auto-cascade to Fallback exit methods if defined default route is locked

MicroTrendsTom commented 3 years ago

Hi

ok looks some pretty awesome lock ideas indeed. of course one way to fix that is we cancel the order refs directly and keep them all in a structure -a list etc so none are lost... .then there is no need for arbitrary sweeping statements to close all in the account

so with this in mind the cancel event should have a mode to directly cancel those order instances in realtime

Note: I have to fix the problem i introduced with the close by limit technique and so possibly edit the workflow case statement then i got to whirl off to a project and will be flat out all week in an attempt to complete it.

So there may indeed be some code merging and problems to address as i am also likely editing the same code parts etc on any pulls etc

Testing and Realtime Q mode In my testing and beta testers no one gets any locks but that is because its not being heavily soak tested etc ie real world usages are no where near what your testing does Your testing in fact should use the Realtime Q mode probably.. .so it evens out the bumps caused by a spikey profile of activity

MicroTrendsTom commented 3 years ago

Debugging through DEBUG log files VS was not that helpful to try and find deadlocks >>> I suggest we extend DEBUG Logs to more directly identify when setting locks is the next step in the code (edit a few existing DEBUG message strings, copy-paste a few new "if(tracing) Print(.." blocks to just prior to the lock calls so it is more clear when a specific lock as contributed to a deadlock

Running VS2019 attached to NT8 instanace and the code open in the debugger will allow you to see where the deadlock was? So when if its stopped you can then click pause on the VS debugger and it will show where the code execution entered not to return?

jmscraig commented 3 years ago

Morning,

"So there may indeed be some code merging and problems to address as i am also likely editing the same code parts etcon any pulls etc"

Just go. Be very productive. I will Merge in with what ever is the latest. or tap you back if there is a major merge to work through

I have fully restructured CancelAllorders() with a number of models to test through. Is a jump toward providing longer term desired functionality. I expect it has more options and code than we would want long term. To me is a test platform where we can really firm up what we like best and also flush solutions for the remaining unanswered questions.

I am re-writing a bit this morning and am thinking about posting it as a separate temp dev branch so you can look it over, test it and we can close on design without disruption to the main branch until we feel it is pretty firmed up.

Dev branch - https://github.com/jmscraig/NinjaTrader8/blob/CancelAllOrders/ATSQuadroStrategyBase/NinjaTrader%208/bin/Custom/Strategies/AlgoSystemBase.cs

And then submit a Pull Request to merge with Main.

Your Thoughts?

MicroTrendsTom commented 3 years ago

And then submit a Pull Request to merge with Main. Yep sounds a good idea or write a new CancelAllOrders - call it OrdersCancelAll() etc and we can use it based on a public property OrderCancelMode etc and test it in parallel and merge it in so the rollback is easy if needed or the roll forwards etc

One idea im having is to call TriggerCustomEvent - from a Dispatcher Timer spun up when there is a problem with straight forwards cancellation instead of using OnMarketUpdate

jmscraig commented 3 years ago

"Your testing in fact should use the Realtime Q mode probably."

Not sure I understand the full capabilities of the RT Q but have not yet felt a need for it.

The soak testing is identifying new entries most often 1min or more apart and almost never less than 40 ticks or more apart. I have not yet recognized any time when the system has struggled to keep up with new entry signals.

jmscraig commented 3 years ago

call it OrdersCancelAll() etc and we can use it based on a public property OrderCancelMode etc

Great idea.. will do

MicroTrendsTom commented 3 years ago

"Your testing in fact should use the Realtime Q mode probably."

Not sure I understand the full capabilities of the RT Q but have not yet felt a need for it.

The soak testing is identifying new entries most often 1min or more apart and almost never less than 40 ticks or more apart. I have not yet recognized any time when the system has struggled to keep up with new entry signals.

i very surprised to hear it locks up under that context -the testing im doing we can have 5 signals in 1 second sometimes when the market moves etc... so its for these peaks the test is relevant for.

Really realtime q - is for fast market usage where you can get a flood of signals etc

jmscraig commented 3 years ago

i very surprised to hear it locks up under that context -the testing im doing we can have 5 signals in 1 second sometimes when the market moves etc... so its for these peaks the test is relevant for.

Really realtime q - is for fast market usage where you can get a flood of signals etc

I think in reality I am already throttling per bar signals in logic above before I would send an entry to the workflow.

Next time I dig into Reversal Entry management or Exit Chase on full reversal I will look at the RT-Q ...

MicroTrendsTom commented 3 years ago

Did you see this one -a sample strategy added ? ATSSamplePriceReversalTest.cs this could in theory reverse every 1 second. on a 1 second bar etc but does depend a bit on the bar price action etc

if we can establish the engine can do that which it could i will retest then it would seem NT8 is locking up on fast bar types mixed in etc...

trade management cancels new signals

Ok it failed spectacularly so my regression testing at some point was not done... So i will test can code against a reversal each 1 second interval as prior and see what part of the workflow is the culprit perhaps long/short rejection etc with a fill just prior that goes into the error reject workflow and so on etc

jmscraig commented 3 years ago

Running VS2019 attached to NT8 instanace and the code open in the debugger does not look right .. think I need to re-install VS or reset to default configs.

MicroTrendsTom commented 3 years ago

Running VS2019 attached to NT8 instanace and the code open in the debugger does not look right .. think I need to re-install VS or reset to default configs.

make sure you compile the code in NT8 in debug mode from the open source Right click the NT8 code editor window > Debug Mode = Checked -after compile then attach from VS2019 to NinjaTrader instance process

MicroTrendsTom commented 3 years ago

ok my best theory at this point is:

For some reason this part is now subject to coming into play alot -rejections.... and i can think of the cure to stop before it happens but i want to address the lock up symptom

Waiting:> StrategyTradeWorkFlowStates=GoLong GoLong:> ProcessWorkFlow(GoLong) GoLong:> ProcessWorkFlow(GoLongValidationRejected) GoLong:> StrategyTradeWorkFlowStates=GoLongValidationRejected GoLongValidationRejected:> ProcessWorkFlow(Error) GoLongValidationRejected:> StrategyTradeWorkFlowStates=Error

then soon after it goes bang....deadlocks maybe prevention is cure in this case...also

ok so history shows i used to use the SignalQ to process all signals This was taken off for the reason to fix backtesting as the SignalQ did not work with the backtesting mode - EnQ never returned.

Then the process of the Q used to block attempts to go long when there was already ad long or block shorts when there was a short etc...

So when this was taken out and the SignalQ not used this feature was removed...bypassedd.. This means we get rejections which arise to Errors.. these seem to come before a deadlock....scenario

so first element is to remove that initial cause

There were some basic issues here Dont allow to try long when its long this is not a compounding Trade Engine OnMarketUpdate would not reset its timeForNextProcessing using timerItnerval (in the prior engine in NT7 this was done with a windows timer so this was an issue We are not using a dispatcher as this has proved to be unreliable in some scenarios but might be used with some trivial cycles and implemented with triggerCustomEvent....) WFTimerInterval set to min of 3 seconds not 100ms that will allow a lot more time for items events to complete before we need to give it a nudge....

For Cancel dont use account.CancelAllOrders or any account object use the methods from base class Strategy etc such as Cancel one must assume any suitable locking if any is needed goes on in this -from an unmanaged strategy etc OCO order just cancel the stoploss portion and use the reference variable or iterate through the list and cancel from strategybase

Summary for not it looks like a more stable build as it was prior to the start of backtesting fixes etc a 1 second chart reversal test of 1000 trades went ok Which moved some rules around now added back - simple stuff Also providing better breathing space for events to callback such as OnOrderUpdate, OnExec and OnMarket In OnMarket has used TriggerCustomEvent so that pointers have a chance to line up as some reference errors were occuring CancelOrders Public propertys to control its mode OnMarketUpdate the same Long/short workflow rejections prevented Long/short checking - now a long/short qty will be an overfill etc

jmscraig commented 3 years ago

Hi Tom,

just an FYI. I went to test the updates I made to CancelAllOrders() and saw the same Workflow looping I posted on last thursday or friday.

I will check out the commits you submitted three hours ago.

MicroTrendsTom commented 3 years ago

ok the checked in code ran on a s second bar reversal all night and no lock ups so i'm happy with that its working as it used to. Prior to the backtest edits which were as far as i know the start of the issues etc as detailed in changes here: https://github.com/MicroTrendsLtd/NinjaTrader8/issues/6#issuecomment-739991382

The system was soak tested on 1 sec bars, small range ,renko etc prior so that is looking good. Back to how it was. Some mods to workflow were made which indeed could result in fast market loops and other behaviors

Extra Locks - overtime i will look at the locks put in and consider to remove any redundant, to keep it lean and mean -however that is really optimization and if it aint broke i wont fix etc - there is a lot of other items to come now.

So the thing to do now would be to test what is there and try bust it etc
1 second bars using the reversal algo - this allows trail in and that's what we want. fast Renko, range, ticks. then onwards to the layers..

This end that soak testing passed... on NQ all night ( EST day) only stopped when my server connected to same datafeed and did not lock or crash!!! and that is a test also.... connect/disconnect etc

jmscraig commented 3 years ago

I have four test charts running. Two with the code exactly how you released it earlier today. Two with some of the modifications I wanted to start testing.

The PriceReversal Sample is a good tool to see the queue in action. Without it the market action is faster and one of two charts because disabled with 3 contracts still active. With it for sure a slow down is visible and has not yet locked up or let free any rouge positions.

MicroTrendsTom commented 3 years ago

I have four test charts running. Two with the code exactly how you released it earlier today. Two with some of the modifications I wanted to start testing.

The PriceReversal Sample is a good tool to see the queue in action. Without it the market action is faster and one of two charts because disabled with 3 contracts still active. With it for sure a slow down is visible and has not yet locked up or let free any rouge positions.

Please do add your method so we can see - and we can add a mode etc I just remember now what i need to do is fix the close with profit targets... hmmm - ok shouldnt be hard after some chocoffee

jmscraig commented 3 years ago

1 sec bars

It is interesting to see how time based bars, even as small as a one second bar, can go a long way to leveling whipsaw and spikes that surface in a fast tick charts. Especially during the market open.

jmscraig commented 3 years ago

So I am running the PriceReversalExample on a 2 tick MNQ chart which sounds fast but the market has been really, really slow.

The Queue is on. There is one rogue contract contract and no live orders. I was also running the SampleMA on MES so the output of both are intermingled but reported against two different instruments. I was not running he debugger.

Below is the config of the PriceReversal.. Do you want the log files or limit follow-up to 1 second charts and beyond?

jmscraig commented 3 years ago

Maybe this the limit close issue you mentioned ..

Looks like market position was 3 short.

Even though position was 3 short, submitted close order quantity of 4 ... leaving one open long position uncovered.

MicroTrendsTom commented 3 years ago

Maybe this the limit close issue you mentioned ..

Looks like market position was 3 short.

Even though position was 3 short, submitted close order quantity of 4 ... leaving one open long position uncovered.

Limit close is not in effect as of yet - Limit Exit Caveats So limit exit is better provided the balance was correct in the first place etc - always a pro and con to all Some traders dont understand the " target filled -" when its a close in loss

Extras Base on a close order scenario - the system will send an exit order base don the quantity reported at the time it prepares the submit, so that is where you can get differences and overfills underfills - ie it sends a 4 but its only a 3 by that time it was prepared - that is where post exit comes in to assess it was flat or not - if that misses then post trade entry fill is a double check - not yet implemented

Strategy position versus account position. Strategy can be out of sync with underlying account etc

MicroTrendsTom commented 3 years ago

based on the thread title and lack of rejections etc we can close this - and if it comes back re-open for any other matter start a new issue for each one

System closing is another thread will make new for each etc

jmscraig commented 3 years ago

ie it sends a 4 but its only a 3 by that time it was prepared - that is where post exit comes in to assess it was flat or not - if that misses then post trade entry fill is a double check - not yet implemented

I know the sample strategies are intended to be very simple.. you don't like the check Filled=X or Position.Quantity=X before sending PT and SL orders?

This is a new issue we can post to a new thread Please see https://github.com/MicroTrendsLtd/NinjaTrader8/issues/23#issue-759053499

jmscraig commented 3 years ago

Cool .. all goes into backlog .. only the highest priority work gets attention first.

MicroTrendsTom commented 3 years ago

ie it sends a 4 but its only a 3 by that time it was prepared - that is where post exit comes in to assess it was flat or not - if that misses then post trade entry fill is a double check - not yet implemented

I know the sample strategies are intended to be very simple.. you don't like the check Filled=X or Position.Quantity=X before sending PT and SL orders?

We will discussed in a new issue workflow

Pre Trade Check Post Check but also = adaptive stop loss/exits to the position which is underfill or overfill

jmscraig commented 3 years ago

So far zero issues on the two SMACrossover test clients.

jmscraig commented 3 years ago

We will discussed in a new issue workflow

Pre Trade Check Post Check but also = adaptive stop loss/exits to the position which is underfill or overfill

Cool.

jmscraig commented 3 years ago

.. all goes into backlog

I agree with a theme in your posts above. Fundamental System stability is paramount.

Features and advancement needs to be constrained by keeping the system solid and stable

MicroTrendsLtd / NinjaTrader8

Third of Three Issues - Rejected - order Cancel Pending status #6

1) Deadlocks 2) LIST<> read write index related conflicts, 3) ChangeOrder submission rejects because the order is in “Cancel Pending” status