Including network management protocols

[idomingu]

When it comes to identifying model-driven telemetry (MDT), the details regarding the network management protocols enabled by a network platform must be known.

Firstly, the capabilities of the protocol server implemented by the network platform determine the options for streaming telemetry data. For instance, a platform that implements the NETCONF protocol may not support the XPath capability, thus telemetry collectors could only subscribe by specifying a subtree filter. To this end, the Platform Manifest model shall be extended to include such information in a similar fashion as done in YANG Catalog. A snapshot is shown below:

+--rw protocols
     |  +--rw protocol* [name]
     |     +--rw name                identityref
     |     +--rw protocol-version*   string
     |     +--rw capabilities*       string  

Secondly, as part of describing an MDT subscription, the encoding format that is chosen for the streamed data should also be accounted. Network protocols such as gNMI allow for streaming telemetry data using JSON or Protobuf encoding formats. Furthermore, data encoded with JSON could even follow different structures, namely, IETF and OpenConfig. This information is required by data engineers to properly deserialise the ingested data for further processing. In this sense, the Collection Manifest could be extended to describe this kind of information – for example, by means of YANG leaf node.

[JeanQuilbeufHuawei]

Hi Nacho, for the first point, which is specifying the capabilities of the telemetry on the given device, I think that it is out of scope of this draft and maybe more for something like the metadata for collection draft. The goal of the platform manifest is to enable a posteriori interpretation of the collected data, not what configuration methods are supported.

First point, still, currently we identify each collected node via XPath (+ datastore at the collection root). The XPath is used as the key in our data manifest YANG model. My current conjecture is that supported subtrees and supported XPath for configuring collection have the same expressivity (basic path + filtering on keys only), and that XPath can represent any subtree. If that’s not the case, we might need to revisit the YANG model to include another way than XPath to represent collected nodes. Please let us know if you see any issues here?

For the last point, about the data encoding. I had the same idea as you that encoding is important to interpret the data. But actually YANG is supposed to resolve that issue by assigning a well-defined type to each leaf. So by having:

• the platform manifest with the module names and revisions as well as device and OS versions,
• the path collected and the datastore, one should be able to retrieve the expected schema for the streamed values. It’s up to the collector to specify the encoding used for streaming from the device and the encoding used for storing in the db. If that is true we need to clarify in the draft that encoding needs to be managed by the collector/DB system. What do you think?

[idomingu]

Hi Jean,

Thanks for your comments. I will go through each point:

• First, the specification of telemetry capabilities on the device. I agree with you, this information is out of scope of this draft. I like your suggestion on defining this kind of information to a draft like metadata for collection draft. I believe it is important to know the protocol capabilities for a given device in advance (i.e., at implementation-time) so that MDT-based mechanisms can be selected and configured properly.

• Second, identifying each collected node via XPath. Here I think XPath filtering cannot fit all use cases. Subtree filtering should also be taken into account as it provides features that XPath cannot. For example, in the case of subscribing to interface counters from the ietf-interfaces module, say we only want to retrieve in-pkts and out-pkts counters. With XPath we would have to either configure a separate subscription for each counter, or a single subscription to the whole statistics container, which would be inefficient for this use case. By using subtree filtering, this could be achieved with a single subscription where these two counters are selected. Additionally, subtree filters also allow matching list members by other attributes that may not necessarily be the key of the list (https://datatracker.ietf.org/doc/html/rfc6241#section-6). For these reasons, I suggest to update the data manifest and to follow the YANG Push approach for defining subscription filters:

     +--rw subscriptions
        +--rw subscription* [id]
           |  ...
           +--rw (target)
           |  +--:(stream)
           |  |   ...
           |  +--:(yp:datastore)
           |     +--rw yp:datastore                     identityref
           |     +--rw (yp:selection-filter)?
           |        +--:(yp:by-reference)
           |        |  +--rw yp:selection-filter-ref
           |        |          selection-filter-ref
           |        +--:(yp:within-subscription)
           |           +--rw (yp:filter-spec)?
           |              +--:(yp:datastore-subtree-filter)
           |              |  +--rw yp:datastore-subtree-filter?
           |              |          <anydata> {sn:subtree}?
           |              +--:(yp:datastore-xpath-filter)
           |                 +--rw yp:datastore-xpath-filter?
           |                         yang:xpath1.0 {sn:xpath}?
           | ...

• Third, data encoding. If the YANG module does not capture the encoding format that is used for each subscription, this must be clarified in some way in the draft. Then the question is, how does the consumer of the DB know which format was used to encode the YANG data? Can the consumer interact with the collector to discover this particular information? I think this approach would likely hamper the exploitation of data (for example, in a data lake). Since this draft proposes the platform manifest and data collection manifests to be the sources of truth to interpret the YANG data, I suggest to include the encoding format for each subscription in the data collection manifest.

[JeanQuilbeufHuawei]

We now have the same information as in the YANG push module. @idomingu I believe this closes the issue.

[idomingu]

Indeed. Closing this issue.