Proper filtering of state data requests in backend plugins

[wkz]

I am in the process of writing my first backend plugin for clixon (great project, btw!). Existing examples cover configuration data handling pretty thoroughly, but I'm having some trouble understanding how to know whether a ca_statedata callback is applicable to a particular plugin.

From the documentation, it is clear that...

The plugin should ensure that xpath is matched (using namspace context nsc)

...but from what I can tell, all examples always generate their state data, and rely on the clixon core to filter out the superfluous data.

Are there any examples of how a plugin should perform this filtering?

It seems to me like the procedure should be something like:

1. Using the xpath and nsc arguments, find the set of matching nodes in the model (from clicon_dbspec_yang())
2. Filter the set with a static xpath to retrieve the subset that my plugin is responsible for
3. Retrieve the actual state for all paths in the resulting set
4. Convert state data to XML and insert it in xtop

Step 3 is obviously specific to each plugin, and 4 is well described in existing examples - but I have not found any examples of steps 1 and 2. Or maybe I've just completely misunderstood the proper way of doing this :smile:

Any hints on how to proceed would be much appreciated!

[olofhagsand]

Good observations. The state API is somewhat problematic for several reasons, and is the reason that the docs is unclear. (1) The first reason is that xpath filtering sometimes cannot be done transitively. In particular if an xpath is applied twice such as first in a callback and later in the generic get code. For example, an xpath such as: x[y='a']/z where "y" is not a key. If applied twice on a structure such as <x><z>42</z><y>a</y><x> will return NULL since the first xpath will remove the branch. One example is https://github.com/clicon/clixon/issues/414. A planned solution in 6.2 is to refactor the configuration tree where instead copying and then removing parts of the tree with xpath, instead make zerocopy and "mark" parts of the tree that matches . In this way all parts of the tree will remain present for xpaths that references other parts, even if they are filtered. (2) However, while this is a solution for config trees, state data is different. The current API is functional and relies on that callbacks builds the state data from scratch each time. To be able to reference any part of the state tree with an xpath (or other constraint) the whole tree needs to be accessible. The solution to this is probably to keep a state tree that is modified by callbacks, instead of always re-creating it from scratch, and then following the marking scheme in the same way as for config data. This has been proposed by @dcornejo and some prototype code exists but we have talked about it a long time. Until then, these limitations exists and is solved for now in the state callbacks by mostly returning the whole tree independently of the xpath. This is what most applications do. However, in some cases known in advance that there are no xpath cross-dependencies, then one may actually filter the returned xml from the provided xpath, but when to do that is ad-.hoc.

[olofhagsand]

Best examples are in existing system state functions, such as in https://github.com/clicon/clixon/blob/585823a6099783809f03a7c40652d9c6ea3acffd/apps/backend/backend_get.c#L202 and https://github.com/clicon/clixon/blob/585823a6099783809f03a7c40652d9c6ea3acffd/lib/src/clixon_netconf_monitoring.c#L246

[wkz]

Thanks for the detailed response. I'm new to all this, so I'll try to rephrase what you said in my own words, and hopefully you can let me know if I've understood:

Fundamentally, it's very hard to know, based only on the requested XPath, whether a particular leaf in the model needs to be included in a response or not. Presumably, this has to do with the fact that references may exist in other parts of the model, that either point directly at the leaf in question or any node above it.

Thus, state collection and inclusion can only be skipped in cases where:

1. No such references can ever exist
2. All such references are known by the plugin and can be matched using the requested XPath

If that is correct, let's say that I am implementing support for /ietf-system:system-state in my plugin, let's also assume that no external references exist into that branch of the model. What would be the proper procedure to use xpath and nsc to check if something below /ietf-system:system-state is requested or not?

[olofhagsand]

yes that is correct. I need to come back with such an example. it is quite straightforward but may be some cornercases with the namespace prefixes, especially if you dont use canonical namespaces (= the prefixes as defined by the YANG prefix statements).