Using gNMI SubscribeRequest/SubscribeResponse within Tunnel sent as Data after new tunnel session is established

[ravh-ciena]

Hi Carl,

We are implementing the tunnel client/server and we are able to get the client/server session establishment and we have started with writing the C++ wrappers for the tunnel client/server. Now, I have few questions about using the gNMI subscribe request/response as Tunnel Data in the RPC - (rpc Tunnel(stream Data) returns (stream Data)). I am looking at reusing the gNMI SubscribeRequest (https://github.com/openconfig/gnmi/blob/master/proto/gnmi/gnmi.proto#L211) and SubscribeResponse (https://github.com/openconfig/gnmi/blob/master/proto/gnmi/gnmi.proto#L235) as raw 'bytes' - streamed between target and collector via the tunnel and so the below questions.

Questions/Issue 1: In Tunnel.proto Data exchanged is only as bytes – defined here

1. Dial-out using tunnel.proto would require gNMI subscribe request RPC and subscribe response sent/exchanged as Data/bytes via tunnel.proto.

• With our approach, this means, gNMI SubscribeRequest RPC and SubscribeResponse RPC needs to be converted to raw bytes before and after exchange, serialize and de-serialize.

• Question is how to convert, serialize and de-serialize without losing the reference to the structure defined in gNMI.proto but used as raw bytes in tunnel.proto ? should importing gNMI.proto within tunnel.proto could be a solution to have a single stack that could do both tunnel as well as gNMI - with respective wrappers being autogenerated from the .proto ? or any other suggestions ?

• Are there API’s in grpc c++ stack (or C stack) which does such conversion of gNMI SubscribeRequest/SubscribeResponse to raw-bytes that could be used as 'bytes' in the message Data used in the rpc Tunnel(stream Data) returns (stream Data) ?

• Are there any examples from the unit-test client for tunnel.proto where this is attempted ? (I couldn’t find one in the repo for this use-case)

2. Following that, and relevant to some of the latest conversation we have had related to gNMI encoding/format, if gNMI request and gNMI response is all going to be just raw-bytes in tunnel.proto based on the above approach, would it make any difference in selection of formats/encoding types in the gNMI used for dial-out (in other words, for dial-out, it looks like it really doesn’t matter about the different format types/encoding – as its expected the entire request/response on the wire is eventually raw bytes only. In which case, what is the relevance of different format types defined in gNMI – like json_ietf or even scaler proto ?

• What is puzzling to me with this is how to reconstruct a tree structure format of a gNMI subscribe response that will be received as just raw-bytes within the tunnel.proto (when gNMI subscribe is used within the tunnel). Any reference/comments ?

Issue 2: On a related note – will there be C++ or C or python (basically any other language wrapper support for tunnel client/server) based on tunnel.proto besides golang ?

[gcsl]

gRPC handles serialization and deserialization directly and you should not need to do so within the application. The tunnel is replacing the raw TCP transport and embedding it within another gRPC stream. There should be no need to modify any of the semantics within gNMI. This was the primary motivation of using a generalized tunnel approach. We've shown in examples that this tunnel is general enough to carry both gRPC and SSH but should be possible for any TCP based protocol given willingness to write the glue.

The tunnel.proto was designed such that other languages could be supported as either the tunnel client or tunnel server. We published a complete reference implementation in Go with several examples to demonstrate the end to end use. We welcome community contributions for reference implementations in other languages.

[ravh-ciena]

Thanks, Carl. I agree with the view and motivation beyond making the ‘data’ exchanged inside the tunnel to be general enough to carry data of any TCP based protocol. However, that super valid when grpctunnel is used ‘only’ as a tunnel allowing raw bytes exchanged in a new session and it poses a significant limitation for any applications that will be built on top of tunnel-server and tunnel-client – and by limitations here, I mean those applications not having any common interface to interpret request/response structure besides having to parse everything as raw bytes – and further not having a reference/common definition – those applications would either require another .proto defined on top of grpctunnel where the respective structures/format of request/response is aligned.

Specifically, in the case of using the grpctunnel for dial-out use-cases where on one-side of the tunnel, the application is a gNMI-client and on the other side of the tunnel, the application is gnmi server, and to picture it, using the grpc-tunnel as a tunnel for registerOp server as well as Tunnel service for actually streaming a ‘specific’ type of request/response as data within the tunnel - as below, would pose a limitation of not having a reference to define a vendor agnostic Request/Response – and leaves for using off-band understanding of what those request/response types could be and those definitions would be required to be established by some other means – other than tunnel.proto itself. In the examples of tunnel, all such information is essentially read from respective config files and the strings used as bytes/data to be sent/received over the tunnel. Attached snap-shot to visualize the illustration of using tunnel.proto for end-to-end dial-out use-cases.

To summarize the limitation I see with the grpctunnel to be used for dial-out with only bytes as data, below would be the points:

1. Current tunnel.proto allows only exchange of bytes as data between tunnel server and tunnel client as well as end to end.
2. requires a parsing layer to parse raw bytes to match gNMI subscribe Request on the Node where gNMI server is running.
3. requires a parsing layer to parse raw bytes to match gNMI subcribe Response on the dial-out collector-side of the tunnel-server ,Or, requires a parsing layer on the other vendor gnmi-client in the dial-out collector to parse raw bytes to gNMI Subscribe Response.
4. raw bytes are unit8 slices – provides no identification of the beginning or completion of an entire request over the continuous stream capable of having bidirectional request/response exchanged in any order by applications, inside the tunnel.
5. W.r.to gnmi.Subscribe request, entire subscribe request gets chunked into many unit8 bytes slices – that can arrive at deferred-time over the tcp stream – causes for broken request/response messages, to have a parser-glue-layer-waiting to make a full Request/Response. Further without a standard structure nor a delimiter, parseing-glue-layer are left for complete ambiguity.
6. Even when bytes are used as a raw string, reading from a config file, to interpret per string of UTF-8 length – it still requires an out-of-band understanding of what each individual string means between dial-out collector side of applications and the gNMI server on the nodes. No consistent/standard interpretation would be possible for any consistent vendor agnostic implementation.

To illustrate this further, a below proposal to change the tunnel.proto might help visualize the limitations and one possible proposal to overcome it:

1. Make tunnel.proto import gnmi.proto

import "github.com/openconfig/gnmi/proto/gnmi/gnmi.proto"; //make tunnel.proto to import gnmi.proto

2. introduce a new message ‘Type’ that can take a union/oneof below types of data that can be exchanged inside the tunnel session. Bytes is also one Type of data – that can be used for applications who want grpc tunnel to just do tunnel/TCP replacement and provide bidirectional copy of raw bytes, keeping the original motivation intact.

`

message Type {
     oneof {

        bytes data = 1         // raw bytes exchange

        SubscribeRequest gNMISubRequest = 2  //introduce where SubscribeRequest is a message defined in gnmi.proto 

         SubscribeResponse gNMISubResponse = 3   //introduce where SubscribeResponse is a message defined in gnmi.proto

       … 

       … // any other types of messages like capability request/response could as well be included to provide the easier
           //separation of types of request/response used in dial-out collector – those that exists in gnmi.proto as well as any
           //new/future message type additions between two end points using tunnel where raw bytes won’t provide implicit
           //identification of a request/response type.
      }
   }

`

3. Modify existing ‘Data’ message to include new defined message Type

message Data { int32 tag = 1; Type type = 2; bool close = 3; }

If a bidi.copy of gNMISubscribeRequest and gNMISubscribeResponse – is provided and available to be used at (tunnel-client and tunnel-server) – then even in case of 4 separate binaries as mentioned above is used in end-to-end scenarios of application use-cases (or two binaries in case the tunnel-client and tunnel-server are embedded within the applications – either as a package/library for respective languages) – it would significantly eliminate a lot of clutter glue code which different vendors will end up hand-writing – where each of it will not necessarily be compatible with one another as there is no common interface embedded as reference at the parsing-glue-layer to use while converting bytes to application specific request/response. Also, it will eliminate the significant performance degradation/huge-latency that would otherwise occur – as the request/response would need to go thru multiple glue-layers and conversions over every message for an end-to-end use-cases. And the possibility that each applications as well as tunnel-client/tunnel-server itself could be language independent – on top of glue-layer itself, huge latency getting introduced due to independent choices of the separate language based glue-layer – could be minimized by embedding standard request/response types into tunnel.proto itself.

Pls let me know your thoughts/comments ?

I am experimenting with above changes and would be happy to push the change for the proto change as well as the reference implementation in Go for using it.

[gcsl]

I think we are talking past one another.

We have no intention of having the tunnel.proto have any reference to the gnmi.proto. This is an intentional design decision. The examples we have published have demonstrated that the tunnel can be used to carry anything over it without bespoke marshaling and unmarshaling of specific proto messages because that interchange is handled directly by the gRPC library. When dealing with the tunnel, we only ever deal in raw bytes and never introspect within that data because we are using the tunnel bytes purely as transport. This modularity offers immense flexibility. It is true that one "could" attempt to introspect that data and unmarshal the raw bytes into a given proto, but one shouldn't. What we have done is to simply embed the TCP stream of the gNMI session as bytes within the tunnel and one shouldn't expect to interact with those bytes any more than one would read the raw TCP buffer of any gRPC stream.

This general design was put forth specifically to avoid needing to write any code tailored to a given RPC. I realize the context of this question is to handle the gNMI.Subscribe RPC but we are looking forward to the complexity of handling all of the various interfaces to a given device, gNMI including Get, Set and Subscribe, gNOI for all the operational RPCs, gRIBI, gNSI, gNPSI, SSH, P4 and others. The dial-out paradigm is applicable to them all. The most decoupled way to support the tunnel without any modification to existing server code is to deploy it as a separate binary as a local port forwarder where the raw TCP is embedded directly into the tunnel.proto bytes. This can be done with the existing Go implementation if integrated into the build for a given device.

A tighter coupling can be done by adding tunnel support directly a client or server, but for a binaries not already written in Go, the existing published example demonstrating this cannot be used directly and a port of the same concept would need to be made.

[ravh-ciena]

Thanks Carl. I was able to get the end to end dial-out working with decoupled tunnel-client and gnmi-server - without a need for the earlier mentioned changes or any other changes to tunnel.proto. Issue can be closed.