search

uniclogs / yamcs

Yamcs-based mission control software for UniClOGS
GNU Affero General Public License v3.0
1 stars 3 forks source link

Fix: TM Ingest Error-Handling #53

Closed jiangha4 closed 1 year ago

jiangha4 commented 1 year ago

Describe the bug A bad UDP Packet crashes Yamcs server

To Reproduce Send packet with payload "hi" on the UDP port.

Expected behavior Server handles packet instead of crashing

dmitri-mcguckin commented 1 year ago

This seems to be attributed to the Sink being raw UDP, though in a normal case, the UDP sink should just toss the stream if it doesn't fit into a packet.

Though in my recreation, the sink just freezes? @aSmig can you corroborate this? Is there any other details you can provide at this time?

Will need to enable debug logging on the UDP sink and investigate further.

aSmig commented 1 year ago

Yeah, that sounds right, Dmitri. My recollection is that Yamcs does not crash. Yamcs continues running normally-ish but processing of future good beacons silently stops. The UDP socket receive queue stops getting flushed. As additional packets come into UDP:10015, we can watch the queue size grow by running netstat -uln | awk '/:10015/{print$2}'. This should never be above 0 for more than a fraction of a second during normal operation. To create the bad state, echo hi | nc -u -c localhost 10015 from the same host Yamcs is running on. The issue has been observed at least from Mar 19th, 2022 through Feb 08, 2023.

dmitri-mcguckin commented 1 year ago

Found the problem.

We set the field timestampOffset to a non-zero value which is passed along to the packet pre-processor, which attempts to scrub the raw UDP packet bytes for a timestamp based on byte position.

Ofcourse, running echo "hi" | nc -u localhost 10015 is going to produce a packet that's smaller than the byte offset required for timestamp-scrubbing.

Thus the following is thrown and the sink dies (without closing gracefully and reporting that it is no longer accepting data):

SEVERE: Uncaught exception 'java.lang.ArrayIndexOutOfBoundsException: Index 26 out of bounds for length 3' in thread Thread[Thread-1,5,main]: [org.yamcs.utils.ByteArrayUtils.decodeInt(ByteArrayUtils.java:237), org.oresat.uniclogs.tctm.Packet.getSequenceNumber(Packet.java:31), org.oresat.uniclogs.tctm.BeaconPacket.<init>(BeaconPacket.java:12), org.oresat.uniclogs.tctm.BeaconPacketPreprocessor.process(BeaconPacketPreprocessor.java:24), org.yamcs.tctm.UdpTmDataLink.getNextPacket(UdpTmDataLink.java:104), org.yamcs.tctm.UdpTmDataLink.run(UdpTmDataLink.java:68), java.base/java.lang.Thread.run(Thread.java:833)]
May 01, 2023 9:12:39 PM org.yamcs.LogCrashHandler handleCrash

This is a problem with org.oresat.uniclogs.tctm.BeaconPacketPreprocessor, testing out some changes now.