Unknown Race Condition/State Machine Deadlock in States Motion

[ZLLentz]

Current Behavior

TMO's AL1K4 and LI1K4 ended up in a bad state deadlock where they could not move.

There are two function blocks I'd like to point out here.

First: https://github.com/pcdshub/lcls-twincat-motion/blob/master/lcls-twincat-motion/Library/POUs/Motion/States/FB_StatesInputHandler.TcPOU Where:

• variable nSTATE = IDLE
• output bExecMove = TRUE
• input bMoveBusy = FALSE

Second: https://github.com/pcdshub/lcls-twincat-motion/blob/master/lcls-twincat-motion/Library/POUs/Motion/FB_MotionRequest.TcPOU Where:

• variable nState = WAIT_EXEC
• input bExecute = TRUE (set from previous bExecMove output)
• output bDone = FALSE (indirectly used as the bMoveBusy input above)

The problems are:

• The motion state machine was waiting for an execute rising edge, and the input was giving out a constant execute TRUE that would never be able to rise.
• In this state, there is no escape hatch for the input handler to drop bExecMove back down to FALSE. As far as it's aware, we're still waiting on the previous move to end.
• In this state, there is no condition for the move state machine to say "I'm waiting for an exec, I currently have an exec, let's move on". As far as it is aware, the execute input is stale and it needs to wait for a new input.
• I don't know how we got into this state: it needs more investigation.

This is recoverable with an online write to bExecMove := FALSE

Expected Behavior

States should always move when asked, if possible and safe.

Context / environment

lcls-twincat-motion 4.0.1 lcls-twincat-common-components 3.0.1

Steps to Reproduce (for bugs)

Unknown

Suggested Solution

1. Understand how we get into this state deadlock and prevent it
2. Add an escape hatch to escape the deadlock- perhaps we can adjust some things when the FB errors are cleared?

[ZLLentz]

Some further thinking: the path to this bad state must begin when a move is requested, since at startup none of the execute booleans are TRUE.

[ZLLentz]

Update: this is caused by the following sequence:

1. We get an error
2. We ask for another move during an error without clearing the error first

And then the states function block becomes permanently stuck

The solution should involve one or more of:

1. Reset more variables on epics user reset
2. Reject moves when in an error state