REPLY with flexible destination

[deRemo]

-) Optionally change the destination of a REPLY. This is useful for DAG topologies with no-reply-back. It will be the last node of the chain to reply to the client -) Optionally exclude a final REPLY for fire-and-forget use-case, even in the case of FORWARD messages. In the latter case, it is needed a mechanism to join a forked stream without relying on the node that called FORWARD (however, we probably want to reply to the client nonetheless).

A first step may be converting REPLY to a "special" FORWARD case.

[tomcucinotta]

For enhanced clarity, we might:

• rename the current FORWARD into a CALL/RPC command, that (synchronously) waits for a REPLY back from the called node;
• merge the MULTI_FORWARD command with the same CALL/RPC command (unless I'm mistaken, one is a special case of the other); with MULTI_FORWARD, we still wait for REPLY(ies);
• use the FORWARD name for a new "fire-and-forget" command that forwards everything that remains in the message to the next hop, then forgets about the request;
• introduce a new REPLY_TO command, or just a variant of the current REPLY command, handling the reply to the client.

How to reply to a client:

1. we might re-use the (new fire-and-forget) FORWARD, specifying the client IP address; this would only work if the client has a reachable IP, so it might not represent the general case of using a DAG/service-chain from an outside network going through a firewall/NAT;
2. we might have the client connecting to the end-point supposed to provide the final reply, and receive the final REPLY through this 2nd socket.

In both cases 1) and 2) above, we have (again) the burden of how to couple the individual request sent by a given client with a given request ID, with the final reply referring to it (the client computes the response times based on that), i.e., using a per-request UUID.

At the moment, requests IDs are "reassigned" by each node executing a current FORWARD/MULTI_FORWARD, but they could not even be used earlier, because each client was reusing anyway the same request ID sequences. On a related note, probably requests that are forwarded in a fire-and-forget way don't strictly need a reassignment of their ID, and they might bring forward the same ID as sent by the client (which might be what the client expects to see anyway in a final REPLY to couple it with the initial request message).