Test Report 2020-12-08 - #2

[jmscraig]

90 mins into this test .. PriceReversal 1 second and 2 secnod running great. 100% default configs. MACross MNQ 18tick Hung and frooze the chart in 15 min (no logging or tracing enabled). MACross MNQ 8tick is running great(no logging or tracing enabled).

---

3 hours in, now both MACross clients have Hung / Frozen. (no logging or tracing enabled).

Switching logging to the MACross client.

[MicroTrendsTom]

What settings for the MACross client? i guess its latest v etc? NOTE: beware of OCO ORder type signal and the ISPositinCloseLimit etc

[MicroTrendsTom]

Switching logging to the MACross client.

VS Debubbing with output window on will show the full gamut of methods and events up to the lock not seen in trace plus trace interweaved Also then if a lock has occurred you can click pause and see where it was etc - for this reason i test 1 strategy per server for that

[jmscraig]

Hi Tom, just logging off to go to bed.

What is the config

Given the rapid recent changes I am typing to stay with the exact defaults you are submitting the strategies with... at least for the first rounds of testing.

I always reduce the values of Fast and Slow so the Workflow engine gets a better workout.

This is config is currently running with an intent for strong tracing to help debug any issues that surface. Any recommended changes?

Night!

[MicroTrendsTom]

Note: Trade management will produce those kind of errors or stop placement too near market in a fast bar type perhaps Defaults i have made an attempt to write the defaults out in the constructor and the State Default also as im not sure how the reset to factory defaults works, So i run with what is the constructor - however after adding the new functions im testing with concDict now also :-)

[jmscraig]

This morning Test Reportout:

MACross crashed at 9:06am with one Sell Stop order hung, un-cancelable in the Sim account.

The last few popups all said something like "Unable to verify ErrorFlattenAllConfirmed"

Next round will enable full tracing, it was not on.

Log shows repeated rounds of "Unable to verify ErrorFlattenAllConfirmed". Two different message strings (e.g. likely sent from at least two diff locations )

183 instances of the same error message starting when the market go faster at 8:31am.

I have seen this messaging as a precursor to hang consistantly in all MACross/AlgoSystemBase version combinations.

Ok, so...

• is this a weakness in timings or workflow that should be fixed? or
• is this bugs in the Sim Account/Sim system?
• is this from timing delays from Sim system processing errors?

[jmscraig]

... more on ... is this from timing delays from Sim system processing errors?

Re-writing this line "is this from timing delays from (or related to) Sim system processing errors?"

Over the whole night:

• At 6:58am there were to rejected orders via "Cancel Pending" status. These did not produce or were not near any visible errors from the system.

• At 8:33 am there were four rejections due to Sell stop or sell stop limit orders (most likely ChangeOrder() ) can't be placed above the market.' These are likely to system lag or data lag at the open. (will add a 1 tick BarArray) to for order execution to reduce the risk of the "Above the Market' error.

• The first of many ErrorFlattenAllConfirmed errors started at 8:31. Could be related to the Sim processing errors, but at that time of day system slow down might alone be the breaking point.

[jmscraig]

Next test is on the latest commit

[MicroTrendsTom]

Ok So the Error workflow has started and cannot close or cancel or confirm it has been closed/flattened So it keeps trying then when its resetting state - it does another attempt of a different nature on all "Orders" Ok this would be great to replicate so i can see why it tries and tries againg when that never happens on any machine i have or strategy what is diferrent etc

ak ok thats from the control center log - we need to see the trace log to see the issue so it recovered perhaps its not clear from the date toime

What DataSeries are you using? I can get the settings from the above

[jmscraig]

I think this time (though it happened afterhours as well) I think it is most likely related to the heavy processing load at the open.

18tick.. now running a 1 second which is more like 24-40tick

[MicroTrendsTom]

The part in question is

look from case StrategyTradeWorkFlowState.Error: to case StrategyTradeWorkFlowState.ErrorFlattenAllPending:

what part can it not deal with... if (Position.MarketPosition == MarketPosition.Flat)

that ends in a friendly Flatten to keep it all alive etc as a Position.Close or Account.Flatten(new[] { Instrument }); would halt it etc

[jmscraig]

we need to see the trace log to see the issue looks like Default properties did not have trace turned on so enabling workflow trace

traceorder did not add any value this round so disabling that for this test.

[jmscraig]

that ends in a friendly Flatten to keep it all alive etc as a Position.Close or Account.Flatten(new[] { Instrument }); would halt it etc

Will watch

[MicroTrendsTom]

BTW here is a new one: CancelOrdersIfExists(OrdersActive);

[jmscraig]

Is that in the commit from 8 hours back? That is what I am running

Cool

[MicroTrendsTom]

Is that in the commit from a few hours back? a few before

[MicroTrendsTom]

in the error flow we will get reset messages and popups if (connectionStatusOrder != ConnectionStatus.Connected).....

or a stuck position....

[jmscraig]

What about bumping up the retry alarm, Timeout, Reset numbers a little?

[MicroTrendsTom]

Error workflows are usually kicked off by: OverFiledOrderDetect OrderRejection StrategyTradeWorkFlowState. GoLongValidationRejected StrategyTradeWorkFlowState. GoShortValidationRejected

definetly good idea to try increase the intervals and timeout/alarms etc mayeb the default is too fast

[jmscraig]

test ConDict ?

[MicroTrendsTom]

What about bumping up the retry alarm, Timeout, Reset numbers a little? sometimes this collaboration is like playing a Jazz improv - when you understanding is immediate and others would take years haha

[jmscraig]

Cool!

[MicroTrendsTom]

test ConDict ? thats new today a few hours back and is only implemented sparsely with the mode on IsOrdeActiveConcDict

[jmscraig]

Test ConDict or not yet?

[MicroTrendsTom]

Cool! serious we walk a lonely path ha

[MicroTrendsTom]

Test ConDict or not yet?

yeah i did do a short test and it worked ok on the small sample its ready to test out

[jmscraig]

Will do

[MicroTrendsTom]

awesome

[jmscraig]

One client testing ConDict and one not.

FYI. On launch of both hundreds of this log entry.

I think (maybe not) when launching this one.

[jmscraig]

But both clients were running fine very soon

[jmscraig]

Using those settings above I launched it on a 12 tick of the MNQ to push it to see where the pressure points are..

not long and the first on error file popped up.. both clients still look happy

[MicroTrendsTom]

One client testing ConDict and one not.

FYI. On launch of both hundreds of this log entry.

I think (maybe not) when launching this one.

ok please raise new bug for Print() method so you get notified of fixes etc did you manage to get an output window of it etc? or in trace?

[MicroTrendsTom]

https://github.com/MicroTrendsLtd/NinjaTrader8/issues/33#issue-760587951

[jmscraig]

Client testing MACross and ConDict on 1 second .. is still running fine

[jmscraig]

Follow up or Close this issue however you want.. Starting a new one with a new topic

[MicroTrendsTom]

ok ty