ipm_ctrl resetting periodically (Josh's issue #21)

[janineA]

(Josh is tracking issues at \eol-epicac.eol.ucar.edu\raftechs\RAF_Instruments\iPM_PowerMonitor\Testing\iPM Bug Tracking.xlsx)

3/26/2024 Update on this issue from today - testing on gigajoules/dsmspare and GV/dsm303: (Note: Got gigajoules working. Just had to restart DSM process on gigajoules. Also, missing commas are a non-issue/display artifact.)

For TF02, commented out iPM start in XML and started ipm_ctrl manually. It ran for the entire flight without having to be restarted.

Started ipm_ctrl from nidas on both platforms:

• Ran on GV for an hour.

• Only saw one segfault restart so not going to worry about it at that frequency.

• Set up printing process id every 10 seconds ("while true; do psg ipm; sleep 10; done")

• Observed ipm periodically resetting

• Reset at 2 minutes 30 sec, 4 min 10 sec, 9 min 40 sec (test cut short by end of day)

• Ran on gigajoules for 90 minutes.

• Saw zero segfaults

• Set up printing process id every 10 seconds ("while true; do psg ipm; sleep 10; done")

• Saw zero resets

[janineA]

• Ran ipm_ctrl on gigaojoules on 3/26 for 90 min and on 3/27 for hours and it never reset. Logs are in /var/log/ads
• Ran ipm on 3/26 on GV. Not sure where ipm log files is. Saw 3 restarts in an hour. Take a look at ads file. (data_dump, data_stats).

To Do:

• Run on GV after ACES and see if restarts.
• Check process id of ipm and add to notes

On GV iPM is connected directly to DSM. On gigaojoules goes through... (I forgot what Josh said, but it should stabilize the connection).

[janineA]

From Chris, March 28th after ACES TF03: I pulled tf03 and ran it through nimbus. In the usual spot. I also pulled the iPM log files and they are in Raw_Data/ACES/ipmLogs

[janineA]

ACES notes: 3/26 - Uncommented NAIiPM in XML and got ipm_ctrl running successfully when launched from nidas. ipm_ctrl restarts (Josh’s issue #21) on GV/DSM303 persist but could not replicate on gigajoules/dsmspare today during 90 minute test.

3/27 - Commented out NAIiPM in XML per Josh request and started ipm_ctrl on dsm303 manually. Had to have the tech power cycle iPM (unplug and plug back in on back of ads rack) to get it to work. As of pre-flight, iPM is running well (pid 977)

3/28 - Created second XML, default_ipm.xml that has the iPM uncommented. We agreed to use that for the ferry flights. Manual start on TF04, and no iPM at all for TF05 and RF01.

Ensure running on dsm303 to restart ipm_ctrl??