send() with acknowledgement

[daverolli]

Could possibly just wrap at the Matlab layer for most things? The main pain point is when setting gains, which seems to occasionally miss, even when there doesn’t seem to be packet loss normally.

[iamtesch]

:+1:

[ennerf]

Should really be supported by the API though.

@iamtesch how did you handle back-pressure, multiple sends, ordering, and all the other corner cases? Would it make sense to just open a TCP port that sends guaranteed messages?

[iamtesch]

I think you're overthinking this.

I would not add a TCP port, because then you're doing all the session set up potentially on a per-message setup.

I don't need to handle back-pressure, as I'm not working with ring buffers.

I'm not sure what the mutliple sends and ordering have to do with a single packet send-with-acknowledgement unless you're just checking the very first received packet.

Basically, I send a request, and I block that thread after the send while I wait for received packets that match the sent ones until the timeout value is exceeded, returning 'true' as soon as I get the responses or "false" if I timeout.

[iamtesch]

Overall, I think it should really be in the API as well, but if that isn't likely to happen soon, then it probably makes sense for @daverolli or @ennerf (or me) to just write a MATLAB wrapper so we at least have this functionality. Kind of like multiselect, I think this is something that would be helpful sooner in any form then waiting for an implementation that handles everything perfectly, or a 2.0 release of MATLAB, for example

[rLinks234]

I think those corner cases are not important for sub KHz frequencies. Even if there was a crufty non-optimized second code path for the sendCommandWithAck method, that would be entirely fine. It isn't a common use case to continuously send commands and receive ack. The call itself is going to block until the timeout/ack has been received, so making it optimal (speed wise) isn't important.

The only time I ever use it to begin with is what dave said - with gains.

[iamtesch]

As some background -- this came up in discussion here the other day because of an observed problem when sending gains in the office while setting up a demo, and I noted I had problems with this when giving training to Gaitech as well. There is at least a workaround (just send gains a bunch, or send gains and then request and then compare), so it's not a huge blocker, but I just wanted to bump the discussion to elevate the relative priority a bit.

[ennerf]

I don't really like to implement it that way, but I guess it'd be one way to do it.

It's a bit annoying on the MATLAB API because all modules in a group share a single socket (for commands as well as feedback), and the encode/tx/rx/decode is split across 4 different threads. The send method itself is also fundamentally non-blocking, and MATLAB always needs a timeout-fallback to all Java calls so that it can't hang the entire program forever.

If we are ok with the send call being blocking (e.g. send('guaranteed', true)), I think there is a decently simple hack to get it in. We just have to be clear with users that it shouldn't be used within the main control loop (a call may take >100ms on a larger group).

[iamtesch]

I would be shocked if send('guaranteed', true) was not blocking -- how else would you get feedback to know if the command went through or not? Also, fwiw, I wouldn't expect this to retry, but I would expect it to give you a true/false on whether or not it succeeded.

This way, instead of needing to write a loop to get info back from the modules to verify, you could just do something like the following:

got_gains = false;
while ~got_gains
   got_gains = send('guaranteed', true);
end

or

if ~send('guaranteed', true)
  disp('Could not send gains to module!');
  return; % Or something else here if you can continue...
end

[ennerf]

In many cases with guaranteed delivery (e.g. TCP) sends are non-blocking and go into an output queue that is guaranteed to be delivered at some point. The logic for handing retries, backing-off, session handling, keep-alive, etc. can be pretty nasty though.

(Technically the current group.send() method also only adds messages to an output queue and doesn't wait for anything to actually be sent on the network)

I like your suggestion that the call blocks, and returns whether all modules have returned ACKs (acknowledgements) to a single sent packet within a specifiable timeout. That prevents infinite hangs, and provides users with enough control to implement guaranteed delivery on the user level.

The common case would always return false as nothing has received ACKs.

[iamtesch]

Yep -- this isn't guaranteed delivery. This is just acknowledgement of response. There are no retries, backing off, session handling, or keep-alive -- that would get crazy hard pretty quick!

Done this way, it would also match the C-family APIs.

[ennerf]

Yep, for some reason I thought that the C API was doing guaranteed delivery.... Even though re-reading your comments above it's pretty clear that it's not. I should be able to add that soon.

[daverolli]

Sorry, just catching up on this. A blocking call, send w/ ack, is exactly what I was thinking. Something where if I need it to absolutely make I put it in a while statement like Matt had above.

ackReceived = false;
while ~ackReceived 
   ackReceived  = send('gains', gains, 'ack', true);
end

[ennerf]

Added in latest snapshot