Add support for message precedence/priority

[xylo04]

C Matthew Curtin on the mailing list says:

It looks like Pat has no awareness of message precedence. Not only do messages with higher precedence posted while connected not move ahead, but even when they're all already posted at the time that connection is made, even routine (R) messages go ahead of even Flash (Z).

It sounds like we just need to define a way to flag this on messages in the composer, then respect those priorities when sending.

[xylo04]

This article describes how precedence is indicated on messages: https://www.winlink.org/content/how_use_message_precedence_precedence. Basically, it's just a special string in the beginning of the subject. This is also described in detail in Winlink FAQ Q690.

• //WL2K Z/ - Flash (for urgent message use only)
• //WL2K O/ - Immediate (for urgent message use only)
• //WL2K P/ - Priority
• //WL2K R/ - Routine (Default)

As a bare minimum to support the feature, we don't need to make any changes to the composer, we just need to make the sending queue detect and respect these subject prefixes.

As an enhancement, we could add this as an option in the composer. However, not even Winlink Express goes that far. Users who need message precedence already know the prefixes.

[xylo04]

Looks like this is handled down in the base project https://github.com/la5nta/wl2k-go right here. We're already sorting the messages, currently by size, smallest first. By refining that sorting logic, we can support this without too much difficulty.

[THE-cmcurtin]

This looks really nice. A little more background can be found in ACP 121, with some extra detail in ACP 123.

Critical bits: Precedence and general time objectives for delivery, which means the communication facility of the origination to the communication facility of the destination.

• Flash (Z) - As fast as humanly possible with time objective of not less than 10 minutes
• Immediate (O) - 30 minutes to 1 hour
• Priority (P) - 1 to 6 hours
• Routine (R) - 3 hours to the start of business on the next working day

In practice this also means that Flash traffic can interrupt transmissions from lower-precedence traffic. If I'm transmitting and a Flash message comes in for me to transmit I will stop my transmission and start the Flash transmission unless it takes more time to stop the transmission that's going.

We have some discussion of how this is being used in the Weather Alert Project 21 (WAP21):

• Adding Z https://groups.io/g/QTC/message/495
• Introducing O to amateur operators, discussion of precedence in amateur procedure https://groups.io/g/QTC/message/495

Ideal handling should also allow for a client to confirm listed traffic for the station and then to include the traffic to pull in the traffic list and to handle by precedence. For example:

• KD8TTE has QTC to send: 1 O, 1 P, 1 R
• KD8TTE has incoming: 1 P, 1 O

When KD8TTE attaches to the RMS, the order of handling should be (I'm not certain about the correctness of send vs receive on the same precedence but for the sake of this discussion, I assume sending should happen before receiving and I'm ignoring all other variables like message size): 1 Send O 2 Receive O 3 Send P 4 Receive P 5 Send R

Further, let's say that as 1 (Send O) above is completing, KD8TTE posts a Z message, in which case before Receive O begins, a new step (Send Z) should complete.

Info beyond the scope of Pat (and wl2k-go) but related follows. I'll raise these issues with the WL2K development team. Your work is a big step forward for wl2k-go for emergency communications use.

I suspect that even Winlink Express doesn't present a UI for control over the precedence because it's not well understood, and probably isn't much used. This did come up during exercise BLACK SWAN 20 while we were moving messages over long-haul circuits with Winlink via SHARES between IPAWS R&D and AUXCOMM stations in Ohio, who in turn took the message moved by IPAWS to an amateur relay net for transmission to the target area. (An AAR is available for the part of the exercise that dealt with amateur radio at https://www.blackswancomex.org/2020/arrl-ohio-aarip.)

I further note that in the Winlink Templates, there are two specific to SHARES, and both refer to precedence.

• SHARES SPOTREP-2 has precedence hardwired to R (at the top, just before the date-time group), but when the user submits the form, the Subject is hardwired to use precedence P -- this is a bug for sure, and
• SHARES Message Form2 does not format as a 16-line and does not refer to precedence except in what it calls paragraph 1, where options are "routine," "exercise," and "actual event," and the Winlink subject token to specify precedence is not used.

[xylo04]

Thanks for the detailed additional information about the ideal send/receive cycles, that will definitely take some more thought. PR https://github.com/la5nta/wl2k-go/pull/64 will help within a single send cycle, but doesn't address some of the wider issues you bring up.

FWIW, what you're proposing seems to be a pretty big paradigm shift in these exchanges. We have previously thought about sessions as being relatively infrequent and fast, with the total number of messages to be exchanged being relatively low. With that model in mind, we treat a session as an immutable batch, and only account for one list of messages to send and one list to receive. If something changes during the session (like a new message being posted), a new session needs to be started after the current one finishes.

Your description of precedence challenges us to think of a session as being long-lived enough to accommodate mid-stream changes like new messages, to go through multiple send/receive phases, and to be a near-continuous process. We can certainly evolve toward that, but it's a big change from the way we think about sessions now.

EDIT: corrected below by @martinhpedersen

[martinhpedersen]

If something changes during the session (like a new message being posted), a new session needs to be started after the current one finishes.

Actually, that is not true 🙂 Pat will check for new messages in the outbox for each "turn-over" of the FBB protocol. Meaning, if a message is posted while we're sending/receiving a batch (chunk of max 5 msgs per batch), then Pat will include that message in the next batch. However, changes to a batch (list of proposals) in-flight is not possible as per protocol design.

So with the changes in PR la5nta/wl2k-go#64, if a user (while in-session) posts a new message with higher priority than the rest of the messages in the outbox, it will take precedence and be transmitted as the first message in the next batch before the session is terminated.

EDIT: This behavior is by design, and implemented by passing an OutboundHandler the fbb package's session. The callback is triggered on each protocol turn-over. Pat's OutboundHandler reads the entire outbox on every call, to account for mid-session changes.

[THE-cmcurtin]

Pat will check for new messages in the outbox for each "turn-over" of the FBB protocol. Meaning, if a message is posted while we're sending/receiving a batch (chunk of max 5 msgs per batch), then Pat will include that message in the next batch. However, changes to a batch (list of proposals) in-flight is not possible as per protocol design.

This makes sense, and suggests that it could even be possible for the client to have a runtime option to allow for different batch sizes. The station could then be optimized for efficiency (larger batches) or flexibility (smaller batches). I think a batch size of 1 would take care of the movement of Immediate traffic and even Flash (with the exception of the "override" [interrupt] part of the procedure.

Perhaps the option could be set by config, able to override with a command-line option, maybe a UI element that can adjust batch size at turn-over.

The only condition that would be left unaddressed by a change like that is what to do if a Flash message comes in mid-stream. Obviously if it's incoming, unless there's a Flash-override-protocol-aware exchange that I don't know about there are a lot of things to address. Let's just call that a Hard Problem for now.

Override (halt ongoing exchange to handle the Flash message) probably gets into issues in protocols that I'm only now learning to exist. Assuming that's the case it's probably worth calling a Hard Problem and addressing in the right context.

Flash is handled in voice procedure is pretty straightforward but it requires that the operators all understand what's happening and what to do. We train operators to let up on PTT so they can hear what's happening on the circuit whenever possible. The operator of a station with Flash traffic, upon hearing a break in the transmission will voice FLASH FLASH FLASH and then call the intended receiving station direction -- even in a directed net. Net control stays out of it unless it's to facilitate the movement of the message. To illustrate why this isn't silly, Flash precedence is for handling something like a tornado sighting. Waiting 10 or 5 or even 2 minutes could mean life-saving information doesn't make it to the destination on-time. A "watch" (in U.S. National Weather Service terminology) where conditions are favorable and a tornado is even likely would be handled as Immediate, and the batch size of 1 addresses that case completely.