Unable to split messages across frames when using QueueManager/MACManager

[berkowski]

I seem unable to split messages across frames for when using the MicroModem driver with a QueueManager/MACManager. This specific case uses a MAC slot w/ rate 1, which is a 192B packet split into 3 64B frames. The encoded message is 71 bytes and it gets refused because it is too big.

Here is the snippet where we see the message get pushed into the queue:

Example [2016-May-27 20:00:34.603099] {goby::acomms::queue::push::1}: D: dslcore.acomms.protobuf.DslTextMessage (16386): attempting to push message (destination: 2)
Example [2016-May-27 20:00:34.603194] {goby::acomms::queue::push::1}: D: pushed to send stack (queue size 2/100)
Example [2016-May-27 20:00:34.603290] {goby::acomms::queue::push::1}: D2: Message: [[DslTextMessage]] header {
Example [2016-May-27 20:00:34.603364] {goby::acomms::queue::push::1}:   src: 1
Example [2016-May-27 20:00:34.603433] {goby::acomms::queue::push::1}:   dst: 2
Example [2016-May-27 20:00:34.603501] {goby::acomms::queue::push::1}:   timestamp: 1464379234601972
Example [2016-May-27 20:00:34.603568] {goby::acomms::queue::push::1}:   msg: 2
Example [2016-May-27 20:00:34.603635] {goby::acomms::queue::push::1}: }
Example [2016-May-27 20:00:34.603701] {goby::acomms::queue::push::1}: message: "HELLO FROM USB0  THIS IS A MUCH LONGER MESSAGE THAT WONT FIT"
Example [2016-May-27 20:00:34.603768] {goby::acomms::queue::push::1}: 
Example [2016-May-27 20:00:34.603849] {goby::acomms::queue::push::1}: D2: Meta: [[QueuedMessageMeta]] non_repeated_size: 71

And here is the portion where goby tries to send it:

Example [2016-May-27 20:01:00.054759] {goby::acomms::amac::1}: D: Cycle order: [ d1/-1@1 d2/-1@1  ]
Example [2016-May-27 20:01:00.055118] {goby::acomms::amac::1}: D: Starting slot: src: 1 dest: -1 time: 1464379260000000 rate: 1 type: DATA slot_seconds: 15 slot_index: 0
Example [2016-May-27 20:01:00.055415] {goby::acomms::modemdriver::out::1}: D: Beginning to initiate transmission.
Example [2016-May-27 20:01:00.055685] {goby::acomms::modemdriver::out::1}: D:   this is a DATA transmission
Example [2016-May-27 20:01:00.056026] {goby::acomms::queue::priority::1}: D2: Finding next sender: [[ModemTransmission]] src: 1
Example [2016-May-27 20:01:00.056265] {goby::acomms::queue::priority::1}: dest: -1
Example [2016-May-27 20:01:00.056474] {goby::acomms::queue::priority::1}: time: 1464379260000000
Example [2016-May-27 20:01:00.056677] {goby::acomms::queue::priority::1}: rate: 1
Example [2016-May-27 20:01:00.056876] {goby::acomms::queue::priority::1}: type: DATA
Example [2016-May-27 20:01:00.057074] {goby::acomms::queue::priority::1}: max_num_frames: 3
Example [2016-May-27 20:01:00.057271] {goby::acomms::queue::priority::1}: max_frame_bytes: 64
Example [2016-May-27 20:01:00.057468] {goby::acomms::queue::priority::1}: frame: ""
Example [2016-May-27 20:01:00.057665] {goby::acomms::queue::priority::1}: slot_seconds: 15
Example [2016-May-27 20:01:00.057863] {goby::acomms::queue::priority::1}: slot_index: 0
Example [2016-May-27 20:01:00.058088] {goby::acomms::queue::priority::1}: D: Starting priority contest
Example [2016-May-27 20:01:00.058294] {goby::acomms::queue::priority::1}:       Requesting 3 frame(s), have 0/64B
Example [2016-May-27 20:01:00.058562] {goby::acomms::queue::priority::1}: D:    dslcore.acomms.protobuf.DslTextMessage next message is too large {71}
Example [2016-May-27 20:01:00.058790] {goby::acomms::queue::priority::1}: D:    all other queues have no messages
Example [2016-May-27 20:01:00.059380] {goby::acomms::queue::priority::1}: D: ending priority contest
Example [2016-May-27 20:01:00.059844] {goby::acomms::queue::out::1}: D: No data found. sending empty message to modem driver.

The rejection notice is from goby/src/acomms/queue/queue.cpp:

    // wrong size
    else if(request_msg.has_max_frame_bytes() &&
            (next_msg.non_repeated_size() > (request_msg.max_frame_bytes() - data.size())))
    {
        glog.is(DEBUG1) && glog << group(parent_->glog_priority_group()) << "\t" << name() << " next message is too large {" << next_msg.non_repeated_size() << "}" << std::endl;
        return false;
    }

I would expect messages to get rejected if they're too big to fit in a transmission packet, but you should be able to span frames without issues. Or am I misunderstanding something here?

[tsaubergine]

This is working as expected. The Micro-Modem treats the frames as the smallest atomic units, so the Goby MMDriver currently does as well. More concretely, you could get frames 1&2 but not 3 within a packet. All packetization must be handled at the application layer.

It wouldn't be hard to implement a configuration option for MMDriver to treat multiframe messages (say N frames of B bytes) as single frame messages (of size N*B) that are rejected if any of the frames are bad.

I wouldn't want this to be the only option, though. Especially with high rates it's not uncommon to get <N frames through (but not all of them bad).

[berkowski]

Ah, ok. That makes sense though it seems to make things a bit awkward in practice.

The MACManager entries know about rates, and obviously the modem drivers do as well, but the QueueManager entries do not. How am I at the application level supposed to handle splitting a message up into frame-size chunks for pushing with QueueManager::push_message if I do not know what rates the MACManager entries have been configured for?

[tsaubergine]

There's no one solution here, since you could be working with mixed rates (e.g. rate 1 and 3 alternating, where the former has 64 bytes frames and the latter has 256 byte frames). What I do is to target (i.e. at configuration time) the lowest MTU that I'll be using (in the above example, 64 bytes), and then queue will concatenate smaller packets to fill the 256 byte frame.

If you want to introspect on the frame sizes at runtime, you can attach a callback slot to the ModemDriverBase::signal_data_request signal:

http://gobysoft.org/doc/2.1/classgoby_1_1acomms_1_1ModemDriverBase.html#a84f659cdd5d56f3b432547b3813b2ba8

At this point ModemTransmission will include max_num_frames() and max_frame_bytes(), where

max_num_frames() == 3 and max_frame_bytes() == 64 for rate 1 and max_num_frames() == 2 and max_frame_bytes() == 256 for rate 3

If you attach your application level callback slot before the QueueManager slot (e.g. using goby::acomms::bind), I would assume that you could push some messages during that callback that will then get added to the Queue priority contest and sent out.

Let me know if that doesn't make sense.

[tsaubergine]

And if you have ideas on how to better handle this, let me know. Also keep in mind that if you want to have complete control over the data buffering and packetization, you may be best off not using Queue at all (and attaching your applications signals directly to those in ModemDriverBase).

[berkowski]

I ended up going with the later there and wrote my own simple queue class that can interact with both the MMDriver and the MACManager. So far it's survived some testing on the dev box. I don't have a better way of handling it that wouldn't require some extensive changes to the existing QueueManager, and I don't really think that's the way to go for a whole host of reasons.

I thought about something similar to your suggestion of splitting based on the smallest frame size, but I didn't want to give up on ensuring that at some level a protobuf object doesn't get split across discrete modem packets. I'd lose that control by frame-chunking a protobuf message at the queueing stage with how QueueManager currently works.

Let me also take the time to thank you again for the continued help, explanations, and fixes!