BUG: fix "invalid goal position" fast fault initialization bug

ZLLentz commented 9 months ago

Description

There is a bad state that can happen at PLC startup when we initialize the states FBs. This is done using the first cycle positions which are not guaranteed to be initialized at that point. The impact of this bad state is essentially a fast fault until the first move or reset command: "Invalid goal position in state move"

The solution here is to allow the states FB to re-initialize on the fly when the read state changes. This is implemented by caching the previous cycle's read state, and re-doing the "current goal" initialization if we are idle and the state has updated.

Motivation and Context

https://jira.slac.stanford.edu/browse/ECS-4486

How Has This Been Tested?

This PR adds a test case that was failing prior to the PR and passing after it. I need to get permission to test this from one of the following POCs/projects: Josh - kfe-motion Tong - tmo motion Jyoti - rix motion (tested: working!)

Where Has This Been Documented?

In the jira ticket and in this PR text

Pre-merge checklist

[x] Code works interactively
[x] Test suite passes locally
[x] Code contains descriptive comments
[x] Libraries are set to Always Newest version (Library, *)
[x] Committed with pre-commit or ran pre-commit run --all-files

jyotiphy commented 9 months ago

@ZLLentz you can use rix-motion to test this functionality.

ZLLentz commented 9 months ago

@jyotiphy I'll give this a go on rix today or tomorrow after you've merged/deployed your setpoints update, thank you!

ZLLentz commented 9 months ago

let's pencil it for tomorrow, rather than try to squeeze it in at the end of the day

ZLLentz commented 9 months ago

This worked in RIX today, though I may have found a new, unrelated (old) race condition. Here's the test I ran:

plc-kfe-rix-motion:

restart 1 (no patch, restart in run mode): more than 10 instances of "Invalid goal position" (also: BPTM timeout, separate, deeper issue)
restart 2 (apply patch via activate config): all "Invalid goal position" faults are gone, still a few timeouts/bp mismatch (expected)
restart 3 (restart in run mode): all "Invalid goal position" faults are still gone, but there was a safe beam params load issue (resolved by using pmpsdb reload plc-kfe-rix-motion)

(pcds-5.8.1)zlentz@rix-console:~$ pmpsdb reload plc-kfe-rix-motion
INFO: Last file reloaded at: Wed Dec 31 16:00:00 1969
INFO: Refreshing...
INFO: This file reloaded at: Fri Feb  9 14:33:06 2024

I think the pmps db file loading failure here was unrelated since I didn't touch that code at all and since it only happened one of the two times.

ZLLentz commented 9 months ago

I'm convinced this works based on the unit test + the interactive usage. If @ghalym has a chance to review then I'll feel good about merging and making the fix available.

pcdshub / lcls-twincat-motion