AL1K4 Pico Motor Readback/Setpoint Desync

ZLLentz commented 3 years ago

Bug Report

When we move axis 1, axis 2's readback moves
When we move axis 2, axis 3's readback moves
When we move axis 3, axis 4's readback moves
When we move axis 4, axis 1's readback moves

This is perplexing because:

This worked correctly last month
Each axis is connected to the controller by exactly 1 cable (and I'm not even sure they are encoded).

Expected Behavior

The axis setpoint/readback mapping should be consistent

Context

Makes TMO reference laser alignment confusing Reported by @aegger13

Steps To Reproduce

Move any of the al1k4 pico motors

Your Environment

Standard LCLS EPICS environment IOC config file is:

RELEASE=/reg/g/pcds/epics/ioc/common/picoMotion/R3.0.0
ENGINEER=Zachary Lentz (zlentz)

IP=mcp-tmo-al1k4
IOCNAME=ioc-mcp-tmo-al1k4
PREFIX=AL1K4:L2SI:MCP
IOC_PREFIX=AL1K4:L2SI:MCP:IOC

MOTOR(NAME=01,DESC="TIP")
MOTOR(NAME=02,DESC="TILT")
MOTOR(NAME=03,DESC="X")
MOTOR(NAME=04,DESC="Y")

Modules used are:

IOCADMIN_MODULE_VERSION = R3.1.15-1.0.1
AUTOSAVE_MODULE_VERSION = R5.6.2-2.2.0
ASYN_MODULE_VERSION = R4.31-0.1.0
MOTOR_MODULE_VERSION = R6.10.1-1.0.1

ZLLentz commented 3 years ago

I observed via telnet that the controller sends the response to the nth command ago, rather than my most recent command.

Theory: the controller falls behind, and if we are out of sync by an even number of steps, then the axes are shifted. We poll for two values (position and is_done), then receive the value from a previous query. IOC basically stops working if you are out of sync by an odd number of steps since positions all be come 1 or 0 counts (is done, is not done)

To fix live:

disable the IOC
ssh psdev, telnet to the controller on port 23
*RST to reset
wait for controller to reboot
start the IOC

klauer commented 3 years ago

This is sadly a very common issue with motor record drivers and other devices that have a too simple communication protocol. A hacky way to fix this is to find a way to get the device to respond with a unique string, so you can correlate request/response. Looks like there's no reasonable way to avoid this according to the command set for the 8742.

ZLLentz commented 3 years ago

You can probably mitigate this by polling very slowly... I agree that the device protocol for the 8742 where it responds with just a number (4 instead of something like POS1=4) makes it rough to know when a problem has occurred.

You could potentially do a periodic check of *IDN? and do... something? when that doesn't respond with the correct string? (not sure what)

pcdshub / Bug-Reports-and-Requests

AL1K4 Pico Motor Readback/Setpoint Desync #57