Determine why TCs are not being properly displayed on GUI.

[harrisonCassar]

Symptom

When viewing the DAQ GUI, we expect to see a nice visualization of current TC data, but we receive a blank canvas, despite the TC box being plugged into the Ground Systems RPi:

This is a pretty critical issue, due to the requirement for live TC data support on GUI for Cold Flow (10/29/2022) and Hotfire (11/5/2022) operations.

Work To Be Done

Investigate why this is occurring, and what part of the system/dataflow is failing (if any at all).

[harrisonCassar]

We note the full data flow (and therefore the possible failure points) as follows:

• Breakout boards processing signal from physical TC
• Teensy polling breakout boards for signal via SPI
• Teensy transmitting aggregated TC data over Serial to RPi
• RPi reading from specific Serial port, processing data, and outputting to local IP address/port
• Grafana Live reads from IP address/port, and displays data on configurated dashboard

[harrisonCassar]

Due to this same dashboard setup working last year, we can likely eliminate a Grafana config issue, leaving us with the following:

• Breakout boards processing signal from physical TC
• Teensy polling breakout boards for signal via SPI
• Teensy transmitting aggregated TC data over Serial to RPi
• RPi reading from specific Serial port, processing data, and outputting to local IP address/port
• ~Grafana Live reads from IP address/port, and displays data on configurated dashboard~

[harrisonCassar]

To rule out an issue with the actual TC signal reading, we re-uploaded updated TC code (added in #18) to the TC Teensy, and observed on serial monitor our expected comma-separated output:

Therefore, we leave ourselves to only be an issue with the RPi and/or its tc.py code:

• ~Breakout boards processing signal from physical TC~
• ~Teensy polling breakout boards for signal via SPI~
• ~Teensy transmitting aggregated TC data over Serial to RPi~
• RPi reading from specific Serial port, processing data, and outputting to local IP address/port
• ~Grafana Live reads from IP address/port, and displays data on configurated dashboard~

[harrisonCassar]

We first check to see if the TC log file is being continuously appended to, finding that it is not (has static values):

We attempt to restart the TC service (systemctl restart tc.service), but the same result occurs.

Taking a look at the RPi tc.py code, we expect to be able to log a barely-modified version of what was received over a specific serial port (defined to be /dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.3:1.0 exactly, in the source code).

We confirm that this is the correct port we expect when plugging the TC into the top-left-most USB port by confirming it's presence in the file system after being plugged in as follows:

We confirm that Serial data is, in fact, being streamed in when the TC Teensy is plugged into the RPi as follows, finding that the output stream is continuously growing:

Therefore, the script must be having some issue with reading from the port, likely running into some exception and silently-failing the running Service. We attempt to view this Exception failure message by running the script directly via command-line, receiving:

As seen here, we have a permission issue when the script attempts to open the log file!

We modify the permissions, and try again:

With the command-line script now running, we attempt to check if the log file is now being appended to, and we find that it is:

[harrisonCassar]

We note, however, that the GUI is still not operational, perhaps that the Grafana dashboard is attempting to read from a stream that is improperly labeled?

We note that the current dashboard is attempting to read from stream ares/lox_fill_tc, and the source code is supposedly outputting to that same label...

[harrisonCassar]

Seems to be that this was the case! We're now reading TCs perfectly fine on our GUI, even running a functional test with the Propulsion team.

I've thrown together a nice TC testing dashboard (with several different means of visualizing the TC data):

[harrisonCassar]

There's a few small QoL changes that should be made as a part of this issue to the tc.py Python code handling TC serial data (as a part of the tc.service). This will be completed in a PR linked to this issue.