Extension of Subscription message

[mirovr]

I would like to ask for an extension in Subscription message allowing to define a "filter criteria". In case the collector subscribes to large tables, it would be beneficial to define a filter criteria in order to limit number of notifications sent by the target. For instance, in case of subscribing to forwarding table, it is sometimes required to subscribe only to routes form certain subnets iso of all forwarding entries. Similarly, only routes originated by a particular protocol (e.g. isis) could be filtered out. Also other tables, (e.g. subscriber-hosts) which are typically large and experience relatively high number of changes would benefit possibility to filter outgoing notifiactions. In Nokia we have currently the server implementation which allows to filter subscription to a forwarding table based on the prefix, but this is based on using proprietary "keyword" in the path. In order to make it more generic, applicable to more use cases, it would be better to have ability to define generic filter in Subscription message.

[jacossio]

I would second @mirovr 's request. In our case, we are trying to subscribe to large acl tables. This generic filter approach should let us pass either a list or a range of acl entries in the table to which we would like to subscribe, limiting the update messages to send only relevant information.

[robshakir]

Hi,

Apologies for the delay responding here.

This is something that we've discussed previously, and left as an open discussion based on the different use cases that might emerge. Today, we solely filter based on list keys, and with exact matches. This is relatively simple, and generally performant for an underlying infrastructure.

I'm interested in what the proposed types of filtering would be. It seems like the use cases here are all based on key leaves, and are generally:

• IP address within a subnet.
• Subscriber filtering -- it's not clear what this would be. Do you have further information as to the type of filter that would be used here?
• Integer ranges -- presumably this is always in the key?

Are there cases where filtering is on a non-key leaf? It'd be good to understand if this needs to be something wider than the key.

Thanks, r.

[gcsl]

It would be good to solicit possible use cases with as much detail as possible. We have considered filtering options previously and have left it out of the current spec to avoid providing too much flexibility that would be difficult to implement efficiently. For example, we specifically decided not to include arbitrary regular expressions on paths to avoid the potential of a poorly constructed requests from burdening a target with excessive computation. On the other end of the spectrum, I think we also want to avoid an unbounded set of explicit filters, each of which might or might not be implemented on a given platform.

Carl

On Mon, Oct 1, 2018 at 5:12 PM Rob Shakir notifications@github.com wrote:

Hi,

Apologies for the delay responding here.

This is something that we've discussed previously, and left as an open discussion based on the different use cases that might emerge. Today, we solely filter based on list keys, and with exact matches. This is relatively simple, and generally performant for an underlying infrastructure.

I'm interested in what the proposed types of filtering would be. It seems like the use cases here are all based on key leaves, and are generally:

• IP address within a subnet.
• Subscriber filtering -- it's not clear what this would be. Do you have further information as to the type of filter that would be used here?
• Integer ranges -- presumably this is always in the key?

Are there cases where filtering is on a non-key leaf? It'd be good to understand if this needs to be something wider than the key.

Thanks, r.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/openconfig/gnmi/issues/37#issuecomment-426104421, or mute the thread https://github.com/notifications/unsubscribe-auth/ARfIL3lIa1YaxrdQwXnzc6ioNo2Yx8GKks5ugq9UgaJpZM4WDxuU .

[mirovr]

Rob, Carl,

thanks for looking to this request.

As you have mentioned, there are two aspects to this request: i) filtering criteria key-leaf vs. non-key-leaf ii) matching methiod - range, exact-match, match-on-list, match-string and etc...

Let me describe couple of use examples which address various combinations.

Consider table of ipv4-routes in FIB (simplified example):

/state/routing-context[name={}]/route-fib/ipv4/route[ip-prefix={}] | +-- last-updated <date&time> +-- route-protocol +-- next-hop

in the first example, I want to subscribe to FIB entries with following subscription path:

/state/routing-context[name=base]/route-fib/ipv4/route[ip-prefix=*]

but I would like to receive only information on the routes which have prefix 10.10.0.0/16 or longer (this means /17,/18..../32). This is example of filtering on the key, with the range of ip-addresses

As a consequence I need the filter criteria expressing this:

/state/routing-context[name=base]/route-fib/ipv4/route[ip-prefix=range{10.10.0.0 ... 10.10.255.255}] or /state/routing-context[name=base]/route-fib/ipv4/route[ip-prefix=10.10.0.0/16#or-longer]

in the second example, I want to subscribe to all ISIS routes. This would be an example of filtering on non-key-Leaf (route-protocol).

The subscription path would be the same: /state/routing-context[name=base]/route-fib/ipv4/route[ip-prefix=*]

but the filter criteria would be something like this: /state/routing-context[name=base]/route-fib/ipv4/route[ip-prefix=*]/route-protocol[protocol-name=isis] // this is probably not right syntax as it looks as the subscription path to "route-protocol" leaf.

The example of @jacossio is very much like the first example (filtering on key-leaf):

/state/ip-filter[filter-name={}]	+-- entry

                                 + hit-count <number>

the subscription would be

/state/ip-filter[filter-name=my-filter]/entry[entry-id=*]

and the filter criteria: /state/ip-filter[filter-name=my-filter]/entry[entry-id=range{10...100}] //in case of range /state/ip-filter[filter-name=my-filter]/entry[entry-id=List{10;20;50;100}] // in case of list

The description above is probably not very good from syntax perspective. I am not sure what is the best solution to express the filtering criteria. I just wanted to describe some examples from usage perspective.

I hope this helps.

Miro

[hellt]

I wonder if this can be approached with gNMI extensions?

[ashu-ciena]

@robshakir @gcsl Was any progress made on this request? It seems more and more network element vendors need such type of filtering. If regular expression could introduce computation issues for the server/target, then is there any alternative to this? Please note - the gnmi spec has already reference to this:

2.4.2 Interpretation of Paths Used in RPCs When a client specifies a path within an RPC message which indicates a read, or retrieval of data, the path MUST be interpreted such that it refers to the node directly corresponding with the path and all its children. The path refers to the direct node and all descendent branches which originate from the node, recursively down to each leaf element. If specific nodes are expected to be excluded then an RPC MAY provide means to filter nodes, such as regular-expression based filtering, lists of excluded paths, or metadata-based filtering (based on annotations of the data schema being manipulated, should such annotations be available and understood by both client and target).

I was wondering if we could define a standard convention of providing regular expression/range/list in filter-value.

[dplore]

@ashu-ciena I think we need specific operational use cases from you and others to inform what the filter structure should be. A general use case Regex is not favored for scaling / efficiency concerns. (see https://github.com/openconfig/gnmi/issues/37#issuecomment-426758074)

A more structured approach is here: https://github.com/openconfig/gnmi/issues/37#issuecomment-428105230.

I have a specific operational use case to filter a subscription to /network-instances/network-instance/afts to only match a known set of ip prefixes. The style in https://github.com/openconfig/gnmi/issues/37#issuecomment-428105230 would meet this I think.

For example: /network-instances/network-instance/afts/ipv4-unicast/ipv4-entry[prefix="10.0.0.0/8#or-longer"]

So +1 to this style approach. Thoughts?

[robshakir]

It's also possible to implement this through a gNMI extension -- which would likely be the way that we'd need to introduce this anyway unless we decided it was applicable everywhere. If there's an extension already that we could review, that would be great.

[karimra]

We have been working on an extension to add filtering to gNMI. We realized that the paths will have to be duplicated in the extension to add the filtering conditions to it. Would it be possible to add an extensions field to other messages like PathElem ? That way the condition can be appended to each Path at the PathElem level allowing for filtering on any leaf and parent leaf.

[ashu-ciena]

@karimra I was also thinking on the same lines. @dplore @robshakir ,

Operational Use Cases:

• Support range (..) based filter-values to optimize the subscription query and response. E.g. We are only interested to fetch information on operationally up interfaces and those are found to lie between in3 and in8 both inclusive. So, the subscription query could become /openconfig-interfaces:interfaces/interface[name=range("in3".."in8")]/state/counters

• Support a list of filter-values to optimize the subscription query and response. E.g. We are only interested to fetch information on operationally up interfaces and those are not sequential but random. So, the subscription query could become /openconfig-interfaces:interfaces/interface[name=list(2,3,6,7,13)]/state/counters

The following is my proposal as a possible solution through gNMI extension.

message PathElem { string name = 1; // The name of the element in the path. map<string, string> key = 2; // Map of key (attribute) name to value. repeated gnmi_ext.Extension extension = 3; // Add extensions to PathElem } message Extension { oneof ext { RegisteredExtension registered_ext = 1; // A registered extension. // Well known extensions. MasterArbitration master_arbitration = 2; // Master arbitration extension. History history = 3; // History extension. Commit commit = 4; // Commit confirmed extension. Depth depth = 5; // Depth extension. PathElemFilterHint hint = 6; // Path element filter hint extension. } }

message PathElemFilterHint { string key = 1; oneof hint { Range range = 2; List list = 3; Regex regex = 4; } }

message Range { string start = 1; // start filter value string end = 2; // end filter value }

message List { repeated string value = 1; }

message Regex { string regex = 1; RegexEngine engine = 2; }

enum RegexEngine { python = 0; javascript = 1; php = 2; perl = 3; cplusplus = 4; java = 5; last = 999; }

The extension can be repeated for those filters within the path element, whose key-value is not a wildcard or a specific value but a pattern of Range or List type.

[ashu-ciena]

@robshakir / @gcsl / @dplore @karimra has opened a pull request on this topic with Where extension. Any discussion happened prior to this proposal in any other discussion thread? link ?

[dplore]

@robshakir / @gcsl / @dplore @karimra has opened a pull request on this topic with Where extension. Any discussion happened prior to this proposal in any other discussion thread? link ?

Here's one recent topic that is similar: https://github.com/openconfig/gnmi/pull/169

[ashu-ciena]

@robshakir / @gcsl / @dplore @karimra has opened a pull request on this topic with Where extension. Any discussion happened prior to this proposal in any other discussion thread? link ?

Here's one recent topic that is similar: #169

Thanks for sharing. I hope for a similar comprehensive discussion to happen on filtering proposals so that we cover all use cases and still keep things simple. Some part of proposal can be rejected for not to make target computation complex.