epics crashes when using three slots simultaneously to operate detectors

[dpdutcher]

Describe the bug

In Princeton testing of the SO detector modules, operating three slots simultaneously causes epics to crash on one of the boards, and that board needs to be hammered to become operational again.

Here, "operating" means taking either IV curves or bias-step data using sodetlib functions. This problem has not occurred when operating two slots simultaneously, but regularly occurs when operating three. The crash is not immediate, sometimes happening after one hour, two hours, or 12 hours. It is not always the same slot that crashes. The other two slots remain operational while the third one has crashed.

Additional details

• Have briefly discussed with Jack and Shawn, mentioned on March 15 SO Readout and Detectors call.
• The smurf server CPU and RAM are not overly taxed during these events, according to Grafana metrics.
• If after one of these crashes I attach to the pysmurf-ipython session and try to run commands, I am met with either the same "cannot connect to smurf_server_s3..." errors as in the traceback below, or RuntimeError: epics failed to respond.

Machine is smurf-srv15. Relevant log files for such a crash that happened today are:

• Core dump: /data/cores/core_1647369167_python3_3917_4508_0_0
• Docker logs directory: /data/logs/16473/1647374577
  • Could not connect to epics for the slot that crashed, so no rogue variables dumped.

Beginning of traceback from ocs-pysmurf docker:

2022-03-15T18:32:49+0000 Unexpected problem with CA circuit to server "localhost:5064" was "Connection reset by peer" - disconnecting
CA.Client.Exception...............................................
    Warning: "Virtual circuit disconnect"
    Context: "localhost:5064"
    Source File: ../cac.cpp line 1223
    Current Time: Tue Mar 15 2022 18:32:49.911947848
..................................................................
2022-03-15T18:33:49+0000 ufm_biasstep_sodetlib.py: cannot connect to smurf_server_s3:AMCc:FpgaTopLevel:AppTop:AppCore:RtmCryoDet:SpiCryo:write
2022-03-15T18:34:49+0000 ufm_biasstep_sodetlib.py: cannot connect to smurf_server_s3:AMCc:FpgaTopLevel:AppTop:AppCore:RtmCryoDet:SpiCryo:write
2022-03-15T18:34:54+0000 ufm_biasstep_sodetlib.py: cannot connect to smurf_server_s3:AMCc:FpgaTopLevel:AppTop:AppCore:RtmCryoDet:SpiCryo:read

[msilvafe]

@dpdutcher what was the solution for this?

[dpdutcher]

The interim solution was to run S._caput('smurf_server_s2:AMCc:SmurfProcessor:SOStream:BuilderEncode', 1) for each slot, and the longer term solution was Jack set that to be the default in an update to the smurf-streamer. v0.4.0 certainly has the fix.