Issue #20: ipm_ctrl not opening serial port correctly

(Josh is tracking issues at \eol-epicac.eol.ucar.edu\raftechs\RAF_Instruments\iPM_PowerMonitor\Testing\iPM Bug Tracking.xlsx. Issue number here correspond to the lines in his Excel spreadsheet)

From 11/1/2023 email:

Playing around a little more with ipm_ctrl, it looks like there is an issue opening the serial port after a fresh DSM boot and I believe that it is related to the options specified when ipm_ctrl opens the serial port. I am not able to get data acquisition with ipm_ctrl running properly immediately after a fresh DSM reboot without a workaround. I think this is related to serial flow control not getting set right.

I understand working on iPM is lower priority right now than CAESAR, but I'm hoping we can address this particular issue relatively soon because it prevents a normal data acquisition startup.

Thanks, Josh

Reproduce the problem with the following steps. (Note: I have the DSM setup to automatically configure the COM1 port register at boot) 1) reboot DSM 2) stop_dsm 3) ipm_ctrl -v -i -a 2 -c VER?

After executing the python test scripts, then the ipm_ctrl will work: 1) reboot DSM 2) stop_dsm 3) python3 /home/ads/test/ipm_testing/app_scripts/naii_ipm/data_logger_ipm.py (and ctrl-C to stop) 4) ipm_ctrl -v -i -a 2 -c VER?

It looks like the python scripts are changing the default state options of the serial port (thereby alleviating the responsibility of ipm_ctrl to set the options)

The flow control option (ixon) is enabled after reboot and disabled after running Python script.
A line termination option (icrnl) is also different.

BEFORE PYTHON: ads@dsmspare:~/aircraft_ipm$ stty -F /dev/ttyS0 -a speed 115200 baud; rows 0; columns 0; line = 0; intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = ; eol2 = ; swtch = ; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W; lnext = ^V; discard = ^O; min = 1; time = 1; -parenb -parodd -cmspar cs8 hupcl -cstopb cread clocal -crtscts -ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl ixon -ixoff -iuclc -ixany -imaxbel -iutf8 opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0 isig icanon iexten echo echoe echok -echonl -noflsh -xcase -tostop -echoprt echoctl echoke -flusho -extproc

AFTER PYTHON ads@dsmspare:~/test/ipm_testing/app_scripts/naii_ipm$ stty -F /dev/ttyS0 -a speed 115200 baud; rows 0; columns 0; line = 0; intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = ; eol2 = ; swtch = ; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W; lnext = ^V; discard = ^O; min = 0; time = 0; -parenb -parodd -cmspar cs8 hupcl -cstopb cread clocal -crtscts -ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr -icrnl -ixon -ixoff -iuclc -ixany -imaxbel -iutf8 -opost -olcuc -ocrnl -onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0 -isig -icanon -iexten -echo -echoe -echok -echonl -noflsh -xcase -tostop -echoprt -echoctl -echoke -flusho -extproc

Email from Josh Feb 15th: I have attached the function used to open the serial port I use. Part of a large application that can be found on Directory: Gigajoules -> /home/ads/Josh/ipm_testing/ Top Script: Gigajoules -> /home/ads/Josh/ipm_testing/app_scripts/naii_ipm/data_logger_ipm.py

device_interface_serial.py.txt

TF01 3/22/2024 Logs from acserver (repeats every minute)

Mar 22 14:41:01 acserver systemd[1]: Started Session 131 of user catalog. Mar 22 14:41:01 acserver systemd[1]: Started Session 130 of user ads. Mar 22 14:41:01 acserver systemd[1]: session-131.scope: Succeeded. Mar 22 14:41:01 acserver systemd[1]: session-130.scope: Succeeded. Mar 22 14:41:05 dsm303 kernel: [ 1151.468394] ipm_ctrl[1119]: segfault at 4244c89 ip 04244c89 sp bf907f3c error 4 Mar 22 14:41:20 dsm303 kernel: [ 1166.764291] ipm_ctrl[1121]: segfault at 4244c89 ip 04244c89 sp bfa8f5cc error 4 Mar 22 14:41:45 dsm303 kernel: [ 1191.644407] ipm_ctrl[1123]: segfault at 4244c89 ip 04244c89 sp bfaad41c error 4

Logs from dsm303 (repeats every 10 seconds)

Mar 22 14:43:30 dsm303 dsm[617]: NOTICE|closing: sock::30230, #timeouts=64 Mar 22 14:43:30 dsm303 dsm[617]: INFO|closing: sock::30230 Mar 22 14:43:30 dsm303 dsm[617]: INFO|7 active sensors Mar 22 14:43:40 dsm303 dsm[617]: NOTICE|opening: sock::30230 Mar 22 14:43:40 dsm303 dsm[617]: INFO|UDPiPMSensor: forking command ipm_ctrl -p /dev/ttyS0 -m 1 -r 1 -n 1 -0 2,7,30223 Mar 22 14:43:40 dsm303 dsm[617]: INFO|8 active sensors

Josh has been seeing this error in the lab (where dsm process keeps failing to start ipm_ctrl). Josh set up an iPM in the lab so I can test.

The above may simply be due to ipm_ctrl quitting immediately because of a missing serial configuration. On gigajoules, after a dsm reboot, Josh’s python script has to be run in order for iPM to be able to connect to the serial port and run successfully. The thought is that the python code is setting a flag that the C code needs that persists until the next reboot.

Python serial configuration https://pyserial.readthedocs.io/en/latest/pyserial_api.html serial.Serial(port = dev_name, baudrate = self._DEFAULT_BAUDRATE, bytesize = serial.EIGHTBITS, parity = serial.PARITY_NONE, timeout = self._rd_timeout, # default = 2.0 seconds xonxoff = False, # Enable software flow control write_timeout = self._wr_timeout, # default = 1.0 inter_byte_timeout = None, # inter-character timeout; none is default exclusive = True) # Exclusive access mode

C++ serial configuration (cfsetspeed(&port_settings, get_baud()) port_settings.c_cflag &= ~CRTSCTS; // turn off hardware flow control port_settings.c_cflag |= CS8; // 8n1 (8bit,no parity,1 stopbit) port_settings.c_cc[VTIME] = 1; // 0.1 second timeout

Differences Read timeout is 2 in Python and 1 in C; Change C to 20 (so 2 seconds) Python disables software flow control and C disables hardware flow control; Add port_settings.c_iflag &= ~(IXON | IXOFF | IXANY); // turn off s/w flow ctrl Python sets a write timeout, … ?? inter-character timeout, … ?? and exclusive access mode. From googling, looks like cpp doesn’t support this

3/24/2024 Installed Josh's Python code as a backup workaround in case can't get serial configurated sorted by tomorrow's flight

Updated and upgraded python3 on dsm303, installed pyserial, copied over Josh’s python script, debugged (had to move a block of code and change ttyUSB0 to ttyS0), and got it running.
Reboot dsm303, ran ipm_testing/app_scripts/naii_ipm/data_logger_ipm.py, then ran ipm_ctrl and it worked.
NAIiPM sensor is currently commented out on GV so nidas won’t try to start ipm_ctrl. Will need to login and manually run data_logger_ipm, then start ipm_ctrl before tomorrow's flight.

Debugging termios port config: Termios settings on GV (end of day Sunday; would like to confirm at next opportunity )	dsm303 right after boot	After running Josh’s Python code*	After running ipm_ctrl
c_iflag	00000500	00000000	00000000
c_oflag	00000005	00000000	00000000
c_cflag	00000cbd	00001cb2	00001cb2
c_lflag	00008a3b	00000000	00000000

*Josh's script generated errors so not sure if it is working correctly - iPM in weird state?; ipm_ctrl wouldn’t run consistently, though it did earlier in the day.

Termios settings on gigajoules	dsmspare right after boot	After running ipm_ctrl	After running Josh’s Python code*	After running ipm_ctrl
c_iflag	00000500	00000100	00000000	00000000
c_oflag	00000005	00000005	00000000	00000000
c_cflag	00000cbd	00001cb2	00001cb2	00001cb2
c_lflag	00008a3b	00008a3b	00000000	00000000

*Josh's script generated errors so not sure if it is working correctly - iPM in weird state?; ipm_ctrl wouldn’t run consistently, though it did earlier in the day.

So for c_cflag, which is an unsigned int, 32 bits: 00000cbd = 00000000000000000000110010111101 - after boot 00001cb2 = 00000000000000000001110010110010 - after python code

Of these 32 bits, the leftmost one is hardware flow control (CRTSCTS)
11 in spots 5 and 6 from the right is 8 bit (CS8)
00 in spots 9 and 10 from the right is no parity (PARENB | PARODD)
0 in spot 7 from the right is 1 stopbit (1 would be two stopbits) (CSTOPB)
1 in spots 2 and 13 from the right sets the speed to 115200 That is all the cflag stuff I have been changing, which doesn’t turn off bits 1,3 and 4 from the right like the python code does, so still need to figure out what those values are..

Set CREAD to set bit 8 Set HUPCL and CLOCAL Zero out iflag, oflag and lflag

See /usr/include/bits/termios.h to decode (/usr/include/i386-linux-gnu/bits/termios.h on dsmspare)

3/25/2024 Committed changes that fix serial issue: https://github.com/NCAR/aircraft_ipm/commit/01a12f5e5b663f46badf91f09046045892839d67

Was able to upload changes to iPM code to GV/dsm303 preflight and start ipm_ctrl manually. Left NAIiPM commented out in XML since data acq was already started. If all goes well this flight, can uncomment post-flight and let nidas start ipm_ctrl next flight.

NCAR / aircraft_ipm

Issue #20: ipm_ctrl not opening serial port correctly #2