opendcs / opendcs

Open Data Collection System for Hydro/Meteorologic Data
Apache License 2.0
31 stars 25 forks source link

Combine split Iridium messages before passing to DECODES #726

Open DanielTOsborne opened 1 week ago

DanielTOsborne commented 1 week ago

Is your feature request related to a problem? Please describe. Iridium allows sending larger message sizes than GOES, but messages containing > 330 bytes are automatically split during transmission. These messages must be recombined at the receiving end to reform the original message. The current implementation in LRGS means that each split message will be received and operated on independently, with no prior context. This can cause decoding errors.

Describe the solution you'd like I'd like to see the LRGS detect and combine messages automatically (possibly as an option), so the message saved in the archive is the reformed full message. The goal is to have the full, combined message available for DECODES in one piece.

Describe alternatives you've considered Currently, the way to reduce transmission size is to either use pseudobinary, another format, or switch to more frequent (15-minute, instead of hourly) transmissions.

Additional context Here's an example of a large message, and how it appears when received by the LRGS:

ID=xxxxxxxxxxxxxxx,TIME=24283001211,STAT=00,MO=24556,MT=00000,CDR=27C16F06,LAT=38.5,LON=-121.4,RAD=9 IE:020154 1,20,0,432::Batt 7 #15 12.82 :AT 7 #15 99.2 :RH 7 #15 16 :Stage 7 #15 0.00 :Dewpoint 7 #15 81.49 :YB 12.84 :YN Iridium Test :YD 000705 :SN 2001238 :YDT 241009000745 This is a message purely to add length for testing purposes. The goal of this extra length is to better understand how iridium handles multipart transmissions.At this length,

ID=xxxxxxxxxxxxxxx,TIME=24283001228,STAT=00,MO=24557,MT=00000,CDR=27C174C8,LAT=38.6,LON=-121.5,RAD=8 IE:020070 1,20,329:in addition to the legitimate sensor information, multiple messages should be sent to the EMSS gateway.

Here's an example what I might expect to see after being combined:

ID=xxxxxxxxxxxxxxx,TIME=24283001211,STAT=00,MO=24556,MT=00000,CDR=27C16F06,LAT=38.5,LON=-121.4,RAD=9 IE:020154 1,20,0:Batt 7 #15 12.82 :AT 7 #15 99.2 :RH 7 #15 16 :Stage 7 #15 0.00 :Dewpoint 7 #15 81.49 :YB 12.84 :YN Iridium Test :YD 000705 :SN 2001238 :YDT 241009000745 This is a message purely to add length for testing purposes. The goal of this extra length is to better understand how iridium handles multipart transmissions.At this length, in addition to the legitimate sensor information, multiple messages should be sent to the EMSS gateway.

Depending on if it's a single, first in multi-part, or subsequent message of a multi-part, there's a few different starting markers.

If the message is a single self-contained message, it starts with 0 (after the IE:xxxxx part). For the first message in a multi-part message, it starts with 1,20,0,432: (seems the colon is included here). For subsequent messages in a multi-part message, it starts with 1,20,329:.

I have not identified what the first two fields are (the 1 & 20), but the third field is the offset into the message, and in the case of the first message, the last one is the length of the entire combined message. In the examples above, the total message length is 432, and the second message starts at offset 329.

DanielTOsborne commented 1 week ago

Ok, additional information, from the Sutron docs (section 16.12.1): https://cdn.hach.com/1XMCM0ZF/at/9kvcqv4x6b53fxgtq8j3pwt4/sl3_O_M.pdf

The 1,20 in the example above are the packet-type, and message-id, respectively. So 1 means it's a self-timed extended message (vs 0 is just a self-contained self-timed message). The message-id I'm assuming can be used as a key for piecing together the split messages.