Embedded Python API for callbacks

[olofhagsand]

Clixon provides a C-API for callbacks. A recurring topic in clixon is a foreign function interface enabling a developer to use another programming language when interacting with other tools. The reason is to increase usability, development speed, flexibility, etc. So far, this has been up to the application developer, but there are multiple users having activities in this area. It could be beneficial to maintain an optional ffi in Clixon, or maybe as an add-on to the core clixon functionality. A quick slack investigation seem to be in favor of an embedded python C-API, although other alternatives (or even multiple) are not ruled out. Other alternatives could be golang, php, etc, but python seems the best option at this time.

[akaevgen]

Hi Olof,

Has there been any progress on this request? 🙂

[rcmcdonald91]

This was something I originally proposed with Olof. It continues to be a side-project of mine. The idea here is to provide an interface that many different interpreters can plug into (Python, PHP, Lua, etc.). If you want to start a public discussion (design objectives, wishlists, etc), I say lets do it!

[olofhagsand]

FYI, There is currently a python project which is someaht different in scope here: https://github.com/clicon/clixon-pyapi That is a client written in python using netconf to communicate with the backend. What is proposed in this FR is an embedded plugin API. I would surely welcome such a discussion please go ahead. Maybe on https://matrix.to/#/#clixonforum:matrix.org

[rcmcdonald91]

That's cool. Any reason not to instead build this as CPython module that wraps the Clixon client API? I know the client API last I checked wasn't very mature, but you could kill two birds with one stone :smile: , then any language that provides a C interface could share the same client library and build a high level interface on top

[dima1308]

Hi @olofhagsand maybe I'm missing something, but it is not clear to me why not use 3rd party Netconf library to communicate with CLIXON? I did it in the past using https://github.com/ncclient/ncclient

[olofhagsand]

Yes agree, a 3rd party client-side like ncclient is a prefered choice for a netconf client lib. In the controller work we started out with a a python wrapper of the plugin API but instead made a more loosely coupled architecture where python code just uses netconf. I guess we could have used a 3rd party for the client (@krihal ?) but then the emphasis of pyapi is in the network service script code where a class structure is added. The reason why we did not pursue a plugin wrapper was it was that for this application (the controller) it seemed an easier way forward to use a regular client.n An embedded python API wrapper serves a different purpose and would be useful in other scenarions. The

[dima1308]

Nclient is a de-facto standard library for client side implementation in Python. Developmant of another solution will take a significant effort. From my point of view, if I can reuse existing solution that well maintained, I will do it.

[cminyard]

Any word on this? I am just starting with clixon, and this is an important piece we need for our application.

I have also done this on a couple of different libraries with a number of different languages.

Generally, calling from another language into C is not that hard. Calling the other way is generally harder. I've done this a number of different ways:

• Hand-write the interface. This is tedious if you have a large number of APIs.
• Use swig for calling into C, and hand-write the callbacks out of C. I did this with the openipmi library into perl and python by creating a printf-like API for doing callbacks. This was complicated and error-prone. I also did this with the gensio library for python, but I hand-coded the callbacks. That was better, I think.
• Use swig for calling both directions. You have to have a C++ API for swig to generate callbacks, since it does that through an OO interface. I did this in the gensio library for python and go. I have to say that it wasn't a pleasant experience. For python it adds some odd constraints about how a callback class is created and initialized. It's easy to forget. For go, well, go is just a mess. Not really any fault of swig, it's more on go for this.

I have also done a rust interface for gensio. rust is relatively easy; the hardest part is handling rust errors and such at the api boundaries.

The big question for this is: What parts of the API are needed for this interface? The whole API is big and probably not necessary. For what I need you would need very little. You could pass the XML object across and call into the C code like you would in C, but that seems sub-optimal. It seems better to convert it to a string, pass it over, and convert that to a native XML representation on the target language. You would need to add the flags into the XML somehow, like a clixon-flag attribute. Then outside of that, what do you need? Logging is important, of course. Event notification is important, but for that you need threads or some sort of mechanisms for knowing when data is ready on inputs. I assume clixon is thread safe on these interfaces. Is there anything else?

For the callbacks into the target language, that is well-bounded by the api structure. All of the api structure probably isn't necessary (maybe not error message rewriting, for instance), but most of it is probably relevant.

I've been working on allowing multiple plugins to be instantiated from a module and to bind plugins to a specific top-level namespace so you don't pass the entire XML tree to each plugin (https://github.com/clicon/clixon/pull/525). That should be helpful for this.

The other way to do this would be to run it in an external program and use XML for communication, much like the frontends talk to the backend. I'm not sure if that would be better or not.

[olofhagsand]

Nclient is a de-facto standard library for client side implementation in Python. Developmant of another solution will take a significant effort. From my point of view, if I can reuse existing solution that well maintained, I will do it.

There is now a summer project for adding ncclient to pyapi

[olofhagsand]

@cminyard

Any word on this? I am just starting with clixon, and this is an important piece we need for our application.

Not really. We talked about it a lot and it is (very) important for usability. I know people have done it, or partly, but not up-streamed. I agree with your analysis. The pyapi https://github.com/clicon/clixon-pyapi addresses a different usecase, but uses the external program and XML interface approach. For a plugin I think it needs a tighter coupling. As you say, the datatypes are essential, primarily XML(cxobj) and YANG. Marshaling them is an interesting idea, havent thought about that. I dont think performance is really an issue, could be a nice way to solve it. Personally, I have no plans doing it, trying to focus on the core functionality.

[cminyard]

On Tue, Jun 04, 2024 at 03:00:52AM -0700, Olof Hagsand wrote:

@cminyard

Any word on this? I am just starting with clixon, and this is an important piece we need for our application.

Not really. We talked about it a lot and it is (very) important for usability.

I agree. The people I'm working with asked: "Are we really going to do all this in C?"

I know people have done it, or partly, but not up-streamed.

Ok. That's too bad that people didn't upstream their work.

I agree with your analysis. The pyapi https://github.com/clicon/clixon-pyapi addresses a different usecase, but uses the external program and XML interface approach.

Yeah, I saw that and got excited, but then was let down :).

For a plugin I think it needs a tighter coupling. As you say, the datatypes are essential, primarily XML(cxobj) and YANG. Marshaling them is an interesting idea, havent thought about that. I dont think performance is really an issue, could be a nice way to solve it.

I'm not terribly worried about performance (but if you don't worry enough about performance then you end up with issues). I'm more concerned about having a foreign interface instead of using the more standard tools available in the language.

Personally, I have no plans doing it, trying to focus on the core functionality.

Ok, I completely understand. I'm looking at doing the work.

My current plan is to not do this as something built in to clixon. With the changes I proposed for being able to add plugins dynamically, or something like that, I think an external backend could handle this.

My thought is to have python packages in a directory that get loaded and then register themselves, which would then translate to registration of plugins into the main clixon library.

[cminyard]

I realized a problem with this. The backend API doesn't have any sort of backend-specific data, like a (void *) to hold information about the backend in question. This is pretty customary, but it would require reworking the APIs to add in.

The problem is that if you have a general purpose mechanism to take callbacks and map them to specific python object, you need a way to hold the specific python object and when the callback comes in, use it for the call into python.

You could handle a lot of this by passing it in to the function to add a plugin, and putting it into the transaction data. But that doesn't help with the callbacks that don't have transaction data.

The other option would be to create a backend plugin that had it's own plugin API that other plugins would then use. It would keep track of all the sub-plugins, the registration by namespace, and the specific backend data. It would require no changes to the base clixon code. Having two separate plugin infrastructures, however, seems less than optimal.

I guess I'm looking for guidance here.

[olofhagsand]

I am not sure I understand.

To start from another perspective, there are some variants of callbacks, as described here: https://clixon-docs.readthedocs.io/en/latest/plugins.html#clixon-plugins

1. The generic ones registered in clixon_plugin_init, see https://clixon-docs.readthedocs.io/en/latest/plugins.html#clixon-plugins
2. The backend transaction callbacks for validate/commit, etc: https://clixon-docs.readthedocs.io/en/latest/backend.html#backend-plugins
3. The CLI callbacks for all clispec commands: https://clixon-docs.readthedocs.io/en/latest/cli.html#cli-callbacks
4. The "registered" callbacks for eg RPCs: https://clixon-docs.readthedocs.io/en/latest/plugins.html#registered-callbacks

They are slightly different in characteristics (parameters/datatypes). Maybe one should start with a limited set to narrow down the solution space? My guess is that the transactions and cli callbacks are most interesting? As an example, https://github.com/clicon/clixon/pull/522 targets the latter.

[cminyard]

Yeah, I was pretty sure I wasn't doing a good job of explaining this. Let me give a concrete example.

Let's say we have a backend plugin that supports python plugins, and let's assume it uses the per-namespace plugin I proposed, where you have a function like:

clixon_add_plugin(clixon_handle h, const char *name, const char *ns, clixon_plugin_api *api, clixon_plugin_t **cpp)

and let's just worry about commit. You have some processing in this plugin that does something like:

int python_commit(clixon_handle h, transaction_data td) {
    // How do I know which particular python plugin this belongs to?  I need the handler
    // passed into python_register_plugin, but there is no way to get it from there to here.
}

struct clixon_plugin_api python_api {
   .ca_commit = python_commit
}

// Called by the python code to register their interest in a backend.  It passes in a
// handler object who's commit function is called when a commit comes in.
python_register_plugin(clixon_handle h, char *name, char *namespace, python_object *handler) {
    // What do I do with the handler here?
    clixon_add_plugin(h, name, namespace, &python_api, NULL);
}

int python_plugin_init(clixon_handle h) {
   for (each file in /var/lib/exec/python_plugins) {
      load the file
    }
    return NULL;
}

In most callback infrastructure I have used (X-Windows, glib, all through the Linux kernel, a myriad of other places) you have one of three ways of doing this.

One is you pass in a void to the register function and the callback functions are all passed the void when done for that instance. That way you can allocate a data structure that holds data, like the plugin callback, and maybe ways to keep a list of the objects you have registered. Consider glib's timer add function:

guint
g_timeout_add_full (
  gint priority,
  guint interval,
  GSourceFunc function,
  gpointer data,
  GDestroyNotify notify
)

That gpointer thing is a void *, and when you get a timeout callback it's passed to you so you can cast that to your own data structure to see what the timeout is all about. This is the most common way I have seen in C in userland.

The second is to make the data structure you pass in (like the API data structure) part of a bigger data structure. For instance, the api data structure would be inside another data structure. You would pass the api data structure back in the callbacks, and you can use offsetof() in C to find a pointer to the actual structure the api is part of. The Linux kernel uses this extensively.

The third is in OO languages, where you register in an object that implements some interface and then the callbacks come to methods in that interface. That way you naturally have the object you care about in the callbacks.

If the plugins are the way they are now, then it doesn't really matter. Every plugin gets all the data, so there's no need to differentiate them, you just call all of them with all the data. But with a python interface that's going to be pretty inefficient if you have more than a few backends registered. I would guess that most backends would only care about a portion of the XML tree, and marshaling all that data (or doing all the python-to-C calls to extract object info) is going to be inefficient.

So if the per-namespace backend is added to clixon, it's really going to need a change to the APIs to add a void *.

I could also do all this work in my own backend, scanning the XML tree, generating the differences, etc. That's how I started, and it's easier for me to handle, but it seemed unnatural to have two plugin infrastructures, one with just a namespace added (and a void *, but I didn't think realize that until now). But maybe that's the best.

[krihal]

Years ago we implemented something similar for a project named Grideye. The plugin API in the client part of that project is similar to Clixons plugin API. The Python plugins defined a list of name, callbacks etc and the client used that to populate the API struct. The code is simple and available here:

Plugin loading: https://github.com/cloudmon360/grideye_agent/blob/master/grideye_agent.c#L392 Executing plugins: https://github.com/cloudmon360/grideye_agent/blob/master/grideye_agent.c#L263 Example of plugin: https://github.com/cloudmon360/grideye_agent/blob/master/plugins/grideye_iperf.py#L97

Do you think an approach like that would be suitable here?

[cminyard]

On Wed, Jun 05, 2024 at 03:19:50AM -0700, Kristofer Hallin wrote:

Years ago we implemented something similar for a project named Grideye. The plugin API in the client part of that project is similar to Clixons plugin API. The Python plugins defined a list of name, callbacks etc and the client used that to populate the API struct. The code is simple and available here:

Plugin loading: https://github.com/cloudmon360/grideye_agent/blob/master/grideye_agent.c#L392 Executing plugins: https://github.com/cloudmon360/grideye_agent/blob/master/grideye_agent.c#L263 Example of plugin: https://github.com/cloudmon360/grideye_agent/blob/master/plugins/grideye_iperf.py#L97

Do you think an approach like that would be suitable here?

Yes, that's sort of what I was thinking. That's something where I can at least see how things were done. Thanks.

[krihal]

@cminyard have you started implementing this? I've started looking at implementing something similar as I did för Grideye to Clixon but can't guarantee any quick progress (or any progress at all). If you have started working on something it might be an idea to cooperate. Whatever I do end up here before eventually being merged upstream: https://github.com/krihal/clixon/

[cminyard]

On Fri, Jun 07, 2024 at 10:48:46AM -0700, Kristofer Hallin wrote:

@cminyard have you started implementing this? I've started looking at implementing something similar as I did för Grideye to Clixon but can't guarantee any quick progress (or any progress at all). If you have started working on something it might be an idea to cooperate. Whatever I do end up here before eventually being merged upstream: https://github.com/krihal/clixon/

I have started working on this as an external backend. I should have something in a few days as a prototype.

[cminyard]

@krihal, I have a prototype ready that's mostly functional. It's at https://github.com/MontaVista-OpenSourceTechnology/clixon-backend-helper

I have not yet implemented the external program part of the helper. I'm trying to figure out how that's going to work, or if it's even reasonable to do.

[krihal]

Nice progress. I don't really understand the benefit of implementing an external backend instead of extending the existing mechanism for plugins, are there any reasons for that? Also first time I've ever seen Meson.

[cminyard]

Nice progress. I don't really understand the benefit of implementing an external backend instead of extending the existing mechanism for plugins, are there any reasons for that?

A few reasons: I originally started that way, but I wasn't really able to re-use the existing API; I needed a new API. One of the things that was important to me for the Python interface was to be able to only provide the XML tree that was required for that plugin, and to do that would require an entire new API out of clixon. I had originally started working on a built-in version in https://github.com/clicon/clixon/pull/525 but it wasn't working out like I liked.

Second, it seems to me that this may not be the only way to provide this API for all uses, and it may be best not to have an "official" interface for this.

Third, @olofhagsand seemed to approve this method in an email.

These changes could be pulled into clixon at some point, I think. And if Olof prefers it that way, I'm happy to do it that way.

Also first time I've ever seen Meson.

Yeah, I'm just learning meson. Someone is working on converting a project I maintain over to meson and I'm a qemu subsystem maintainer and qemu switched over to it, and it really seems much better than autotools. And I'm an autotools master. I had used cmake for a while, but it turned out to be a huge pain; more complex than autotools in the end. meson is becoming the defacto standard for building; lots of large projects are switching over to it because of its speed and simplicity.