Proposal: Adopting Structured Paths

[robshakir]

gNMI Proposal: Adopting Structured Paths

Contributors: {hines,csl,robjs,aashaikh}@google.com\ Date: May 2017 Status: New Proposal

Summary

This contribution proposes to modify the Path message of gNMI to contain structured elements rather than simply consisting of simple string elements.

Problem Statement/Motivation

The motivation for this proposal is to simplify the parsing of path elements which contain keys. Particularly, the current format:

• Requires an implementation to walk through each string element character-by-character to determine whether it contains key elements, and extract them. In addition, careful consideration must be made as to whether there are escaped characters such that keys are extracted successfully, and the name of key elements are correctly determined.
• Loses type information for a schema-unaware entity parsing a path. For example, without the schema, an implementation that parses a path such a /interfaces/interface[name=eth0]/subinterfaces/subinterface[index=32] that does not have access to the schema, cannot know the type of 32. Hence, implementations must either assume all keys are strings, or have access to determine the type of the corresponding element.

This proposal aims to simplify the complexity of parsing paths within gNMI, and avoid type loss for schema unaware clients. Additionally, it seeks to provide a path format which can be converted to a number of human-readable formats, such that client systems can flexibly structure their path format in the most suitable format for their users.

Proposal

We proposed that the Path message in gNMI is modified, such that the element field is deprecated. It would be replaced with a repeated message (tenatatively named PathElement), which corresponds to structured data for each element of the path. Each PathElement has a name, corresponding to the node's name in the tree, and an optional map<string,TypedValue> field which is used to express keys.

A straw-man proposal for this change is below:

message Path {
  repeated string element = 1 [deprecated=true];
  string origin = 2;
  repeated PathElement elem = 3;
}

message PathElement {
  string name = 1;
  map<string, TypedValue> key = 2;
}

Examples

Taking the examples from the gNMI Path Conventions document, we can show their alternate encoding.

Example 1: /interfaces/interface[name=Ethernet1/2/3]/state/counters:

v0.3.1 encoding:

<
  element: "interfaces"
  element: "interface[name=Ethernet1/2/3]"
  element: "state"
  element: "counters"
>

Proposed encoding:

<
  elem: <
    name: "interfaces"
  >
  elem: <
    name: "interface"
    key: <
      name: "name"
      value: <
        string_val: "Ethernet1/2/3"
      >
    >
  >
  elem: <
    name: "state"
  >
  elem: <
    name: "counters"
  >
>

Example 2: /network-instances/network-instance/tables/table[protocol=BGP][address-family=IPV4]

v0.3.1 encoding:

<
  element: "network-instances"
  element: "network-instance"
  element: "tables"
  element: "table[protocol=BGP][address-family=IPV4]"
>

Proposed Encoding:

<
  elem: <
    name: "network-instances"
  >
  elem: <
    name: "network-instance"
  >
  elem: <
    name: "tables"
  >
  elem: <
    name: "table"
    key: <
      key: "protocol"
      value: <
        string_val: "BGP"
      >
    >
    key: <
      key: "address-family"
      value: <
        string_val: "IPV4"
      >
    >
  >
>

Example 3: /interfaces/interface[name=eth0]/subinterfaces/subinterface[index=42]/ipv4/neighbors/neighbor[ip=192.0.2.1]

v0.3.1 encoding:

<
  element: "interfaces"
  element: "interface[name=eth0]"
  element: "subinterfaces"
  element: "subinterface[index=42]"
  element: "ipv4"
  element: "neighbors"
  element: "neighbor[ip=192.0.2.1]"
>

Proposed Encoding:

<
  elem: <
    name: "interfaces"
  >
  elem: <
    name: "interface"
    key: <
      key: "name"
      value: <
        string_val: "eth0"
      >
    >
  >
  elem: <
    name: "subinterfaces"
  >
  elem: <
    name: "subinterface"
    key: <
      key: "index"
      value: <
        uint_val: 42
      >
    >
  >
  elem: <
    name: "ipv4"
  >
  elem: <
    name: "neighbors"
  >
  elem: <
    name: "neighbor"
    key: <
      key: "ip"
      value: <
        string_val: "192.0.2.1"
      >
    >
  >
>

Open Questions

• We prefer the use of map over a repeated message for keys, due to the terseness and advantage of enforcing unique keys within language bindings. We are interested in other implementor's feedback as to their preferred approach.
• The TypedValue message is re-used within the proposal. In theory this allows keys to include invalid types of data (e.g., proto.Any, or ascii_val) fields to be created. This is similar to ScalarArray - whereby invalid data can be included, though specification that this is not allowable.

[robshakir]

@tsuna, @jsterne, @kuvempu, @nileshsimaria - we would appreciate your feedback on this proposal. Thanks!

[aaronbee]

This sounds reasonable for when the network-device has to create a path, but what about when there is a client that doesn't have access to the schema has to create the path.

Specifically, I am thinking about a gnmi command line app (for example) that takes in a path as a string eg. "/interfaces/interface[name=eth0]/status/counter" and has to construct a valid gnmi request. The client app could assume a value is always a string and the target device will use its better knowledge of the schema to reinterpret the string as the real type. But, then you won't be able to actually trust the types that are used in a gnmi path.

[jsterne]

How do wildcards work in the structured paths ? Like this ?

<
  elem: <
    name: "interfaces"
  >
  elem: <
    name: "interface"
    key: <
      name: "name"
      value: <
        string_val: "*"
      >
    >
  >
  elem: <
    name: "state"
  >
  elem: <
    name: "counters"
  >
>

What if the key type is a uint ? I assume we would not use string_val (i.e. a type that is explicitly different than the schema type for the key). Would we have to use one of json_val or json_ietf_val for that one key ?

[gcsl]

On Mon, May 29, 2017 at 10:09 AM, jsterne notifications@github.com wrote:

How do wildcards work in the structured paths ? Like this ? < elem: < name: "interfaces"

elem: < name: "interface" key: < name: "name" value: < string_val: "*"

elem: < name: "state"

elem: < name: "counters"

What if the key type is a uint ? I assume we would not use string_val (i.e. a type that is explicitly different than the schema type for the key). Would we have to use one of json_val or json_ietf_val for that one key ?

We had thought that since the container was named, to wildcard a key within it the key could just be left off in the query. As in, only specify the keys one would wish to filter on. Based on your question, I now realize we didn't get this thought expressed in the proposal.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/openconfig/reference/issues/54#issuecomment-304702327, or mute the thread https://github.com/notifications/unsubscribe-auth/ARfILzDR6Llh8xwHRtubrmOAnDcXtLBNks5r-vvFgaJpZM4Nldq_ .

[gcsl]

On Wed, May 24, 2017 at 2:30 PM, Aaron Beitch notifications@github.com wrote:

This sounds reasonable for when the network-device has to create a path, but what about when there is a client that doesn't have access to the schema has to create the path.

We had also discussed making the map<string, string> which gets around this issue, and as you point out, would make schema agnostic handling of keys easier. I am leaning toward not using TypedValue in the map.

Specifically, I am thinking about a gnmi command line app (for example) https://github.com/aristanetworks/goarista/blob/master/cmd/gnmi/main.go that takes in a path as a string eg. "/interfaces/interface[name= eth0]/status/counter" and has to construct a valid gnmi request. The client app could assume a value is always a string and the target device will use its better knowledge of the schema to reinterpret the string as the real type. But, then you won't be able to actually trust the types that are used in a gnmi path.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/openconfig/reference/issues/54#issuecomment-303857445, or mute the thread https://github.com/notifications/unsubscribe-auth/ARfIL5U9t7Us6sDh-rbN_6MEG1UtvZIWks5r9KFzgaJpZM4Nldq_ .

[robshakir]

When we discussed map<string,string> then I think that we ended up erring on the side of using TypedValue based on the idea that we would lose type information, which could not be recovered if all keys were cast to string. I had thought that it was possible to express keys in YANG as non-string values, however, looking at this further, I'm not sure that this is actually the case.

XPATH 1.0 seems to state that any attribute that is used for an XML node is a string - https://www.w3.org/TR/1999/REC-xpath-19991116/#attribute-nodes - if this is the case, then erring towards map<string,string> is probably better, as it solves the issue that @aaronbee mentions - and the server is already implied to know how to handle this in the XPATH case.

[robshakir]

map<string,string> would also help with cases that we have wildcards. As @gcsl says, the wildcard key could be expressed as * in the key value, or by omitting the key (per the XPATH approach).

Wildcards for path elements would be encoded as string_val in the existing TypedValue proposal - e.g., we would have string_val: "..." for a multi-level match.

[kuvempu]

map<string,string> works. Would prefer omitting the key if it is just * - prefer tightening the spec. Also, is wildcarding of a numeric key being discussed for completeness, or, are there real use cases. I can see ranges and sets for expressing numerics. Generally, ranges would cover most of the wildcard cases as well.

[robshakir]

At the moment, we're not proposing to have specific support for anything other than the wildcard key case. For extensibility, we could define this map to be map<string,KeySpec> where we specify:

message KeySpec {
  string value = 1;
}

At this point. This would give us room to add new query types in the future if we identify use cases.

@gcsl @marcushines -- any thoughts?

[gcsl]

On Sat, Jun 3, 2017 at 11:02 AM, Rob Shakir notifications@github.com wrote:

At the moment, we're not proposing to have specific support for anything other than the wildcard key case. For extensibility, we could define this map to be map<string,KeySpec> where we specify:

message KeySpec { string value = 1; }

I suggest keeping map<string,string> in the update case and considering your proposal on the SubscribeRequest case. Clearly we would never expect an update to use anything other than the explicit string. On the client side, I can see the desire to parameterize the request, but I am reluctant to open this door too wide without more consideration as we could quickly end up with arbitrary regex expressions which could have substantial performance impacts (e.g. matching prefixes in RIB). I would like to have concrete problems to solve before proposing generic solutions.

At this point. This would give us room to add new query types in the future if we identify use cases.

@gcsl https://github.com/gcsl @marcushines https://github.com/marcushines -- any thoughts?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openconfig/reference/issues/54#issuecomment-305991485, or mute the thread https://github.com/notifications/unsubscribe-auth/ARfIL1CO8mQgJ-COBE8W9RvkjLf5ngpqks5sAZ_HgaJpZM4Nldq_ .

[robshakir]

I got the same feedback from the folks at Juniper - so I agree. If there were future requirements for this then there is always extensibility through adding a key query leaf that can be populated rather than the map.

At this point, seems like we have rough consensus. I'll plan to start working on updating the proto this week. @aashaikh and I are working on keeping the spec up-to-date.

[mberroug]

I would like to add my point of view on the definition of paths, I think that defining the path using the xpath standard as a simple string is less expensive and covers all situations. You will not have to reinvent the wheel!

In fact, if the gRPC client or the gRPC server already uses xpath to manage its openconfig model data, they will have to do a coding / decoding between the xpath and the gnmi-path-convention, which is not practical.

I also add that the YANG standard also uses the xpath format.

I think that to remain "open" in the definition of paths, it is better to use directly xpath definition

Regards,

[robshakir]

Respectfully, we'd disagree with this, there are a few motivations:

• gNMI does not solely carry YANG modelled data, per the specification any data that can be represented in a tree format can be carried by the protocol. As such, "YANG does this" should be only one of the considerations. For example, protobuf modelled data does not require any understanding of XPATH, and can be carried via gNMI.
  • XPATH is a complex mechanism to specify paths - it comes with the cost of string parsing to extract keys and predicates, per the previous specification of paths within the gNMI spec (prior to this standard). We should not load this complexity onto the path specification, and should rather look at how best to implement such logic (whether server-side filtering is actually more complex than simple client-side filtering).
  • Translating gNMI paths of this format into XPATH seems trivial to me, can you provide any datapoints that this is impractical? It should be noted that to use some other XPATH libraries, then even YANG XPATH expressions need to be translated (e.g., those that require a @ character before predicates).

Given the consensus that we've reached on this issue - we will merge it into gNMI 0.4.0.

[jsterne]

Just a clarification for keys that are non-strings (e.g. integers, like subinterface index 234). How is the 234 encoded ? As a string of 3 ASCII chars ?

elem: < name: "subinterface" key: < key: "index" value: "234"

[robshakir]

Yes, as the UTF8 string-representation of the value. So "234" in this case.

I have made this clarification in the path specification document.

r.

On Wed, Jun 21, 2017 at 4:21 PM jsterne notifications@github.com wrote:

Just a clarification for keys that are non-strings (e.g. integers, like subinterface index 234). How is the 234 encoded ? As a string of 3 ASCII chars ?

elem: < name: "subinterface" key: < key: "index" value: "234"

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/openconfig/reference/issues/54#issuecomment-310231969, or mute the thread https://github.com/notifications/unsubscribe-auth/AEnN1b30ZgK2exj9b2hyXp9-vO9ZjOGPks5sGaV9gaJpZM4Nldq_ .

[mberroug]

It is true that in protobuf we do not have this notion of tree.

So I understand that the goal is not to cover all the syntax of xpath (we cannotdo for example with the gNMI syntax : bookstore/ book[price>35]/ title)

On the other hand it would be welcome that gNMI offers a translation library between what is supported in xpath syntax and gNMI-path-conventions

Respectfully,

[robshakir]

This would be relatively trivial to implement (it is certainly easier with the structured path format here). The question is really what language should such a library be published in? We have some code that does something similar in Go, which I'd expect to update as we adopt 0.4.0 internally to Google.

[robshakir]

This was adopted and merged.