search

fermi-ad / controls

Central repo for reporting bugs, making feature requests, managing RFCs, and requesting seminar topics.
https://www-bd.fnal.gov/controls/
2 stars 0 forks source link

Settings to CAMAC ramp cards "jump" and do not match desired setting #24

Open beauremus opened 9 months ago

beauremus commented 9 months ago

Describe the bug

I'm transposing an email thread here so it doesn't get lost.

This originated with DVM.

Devices like D:SXSINO and D:SXCOSO get set to 0.0 A on a parameter page, then change to a small non-zero value for some mysterious reason.

Yesterday afternoon I would send 0.0 A and within 10 seconds, I'd get 0.011 for D:SXSINO and 0.003 for D:SXCOSO. I repeated this maybe five times with the same results every time.

This morning I sent the changes and nothing happened this time - I sent 0.0 A and after several minutes, the settings were still 0.0 A.

473ppage

Mark Austin responded:

I talked to James Smedinghoff and he looked into the data log of when D:SXCOSO had changed. He said on the 23rd at 7:35ish (I don’t remember exactly, but around that time), the setting was set to Zero. I believe that was you right before you emailed us with the problem. Then at 8:15, the sequencer set the device to it’s default value of 1. When you plug in a value of 1 into that devices scaling on D80, a value of 1 is .011. So the sequencer is making changes to the devices outside the parameter page.

Also, D:SXCOSO is an array device, so on the parameter page, there will only be index zero of that array. The sequencer is set up to set a 32 bit value and sets all the devices in the array. However, the parameter page is only a 16 bit setting and you are only setting one device in the array. On a side note, the device is currently set to .021 which is a raw 2. However, the log doesn’t show the sequencer or the parameter page ever changing it to 2. Which is more confusing, but James said it is possible for the FE to change the value outside of the sequencer or the parameter page and it wouldn’t get logged but he wasn’t able to say if that was occurring.

DVM summarizing:

One thing that has not been mentioned in this e-mail chain is we have a similar(?) problem for the D:QD, D:QF, and D:QSS supplies in that same crate ($18). These are all 468 ramps cards, and the problem comes and goes. Sometimes months pass between having problems. Here is a summary of what has already been done to troubleshoot that problem:

Log entries:

Tue 2021-02-16 14:14:21 - First mention of the setting for D:QF making a large jump when being knobbed. 468 card was replaced.

Tue 2021-02-16 15:23:52 - Card replacement didn't fix the D:QF problem. Further tests show the same problem exists with D:QD and D:QSS. Crate power supply checked.

Thu 2021-02-18 12:37:07 - Tried to recreate the settings problem with D:QD, D:QF, and D:QSS. Didn't happen this time.

Fri 2021-02-19 11:52:31 - Replaced the crate controller in $18.

Tue 2021-04-13 10:23:20 - 473 card for resonant extraction added to crate $18.

Fri 2021-11-12 15:48:57 - Problems with D:QF and D:QSS ramps noted.

Thu 2022-03-24 11:56:40 - Problems with settings for D:QD, D:QF. It was noticed there were a couple of anomolies in ramping the supplies during studies.

Mon 2022-05-02 17:35:29 - Mention of a step in the QF ramp the studiers could not explain.

Tue 2022-05-24 18:35:14 - While trying to calibrate tune mults, it was noticed QD / QF both had erratic behavior.

Fri 2022-06-10 12:11:39 - Brian Hendricks helps diagnosing the problem of references jumping for D:QD, D:QF, and D:QSS with an ACL script he wrote. Could not get D:QD to misbehave that day, but D:QF and D:QSS did. There is a note that during the setting process, bits were being dropped. After this study, Controls took another look at the crate.

Wed 2022-06-15 09:22:50 - Crate $18 power supply replaced.

Wed 2022-12-14 22:03:48 - Ops notice a problem when investigating trouble with the D:QDRF setting. When trying to restore the initial setting from a parameter page (82A), the setting jumps to 4.9A. This problem was repeatable.

Thu 2023-01-12 18:53:57 - Same problem as seen on Wed 2022-12-14 22:03:48

Have we done a test on the 473 card like Brian did on the 468 cards (Fri 2022-06-10 12:11:39) to see if this problem is similar? As far as I could see, there is no explaination anywhere in the logbook as to why the bits were being dropped to the 468 cards.

Additional context:

To Reproduce

It doesn't happen consistently 😞

Affected system

rneswold commented 8 months ago

... James said it is possible for the FE to change the value outside of the sequencer or the parameter page and it wouldn’t get logged but he wasn’t able to say if that was occurring.

Yes, technically it's possible because it's only the FE that sends CNAFs to the ramp card. However, the ramp drivers aren't background processes that are always running. They receive SETS32 requests and update the ramp cards based on that -- they don't do it on their own.

rneswold commented 8 months ago

Kent Triplett did a lot of work with Chris Olsen to debug the BOOSTER CAMAC link. The old copper lines were getting out of tune and affecting the shape of the signal so incorrect crates were responding to commands.

MUONFE and SWYD front-ends are using old, copper links and may need the same maintenance. The MI is fiber optic, so it should be in better shape.