search

HERA-Team / hera_corr_f

HERA F-Engine on SNAP
1 stars 5 forks source link

WR Stability Problem #47

Closed AaronParsons closed 3 years ago

AaronParsons commented 3 years ago

Many nodes are routinely losing sync over a day of observing. Suspicion is that at least one port of the switch has something that is misbehaving attached to it, causing others to lose their lock.

AaronParsons commented 3 years ago

This was tracked to a misconfigured termination in the GPS antenna input by @david-macmahon and fixed.

david-macmahon commented 3 years ago

Just for completeness, I changed the termination of the antenna input of the GPS clock and rebooted the switch and the problem was resolved. But then I changed the termination back to the original setting and rebooted the switch to test whether the change in termination was really the culprit. To my surprise, the problem remained resolved. Try as I might, I was not able to recreate the problematic state. I think perhaps the real fix was that I installed the daemongps process on paper3. I wonder whether maybe the GPS clock doesn't output the reference signals until/unless requested to do so (e.g. by the daemongps process). The daemongps process had ben installed/running on the now defunct paper1 server. Fortunately, we had a power cycle test of the container with the recent chiller work a few days ago and the white rabbit Grand Master switch successfully locked to the GPS signals when power was restored, so I think we are out of the woods on this issue.