Add run-time channels to ZBus

[NelsonFerragut]

Is your feature request related to a problem? Please describe. Currently, in Zephyr OS, the zbus mechanism requires channels to be defined at compilation time. While this design choice offers simplicity and efficiency, it limits the flexibility of applications, particularly in scenarios where dynamic channel creation or modification is necessary during runtime.

We're developing an application with Linkable Loadable EXTensions (LLEXT). We would like these llext modules to communicate with each other using ZBus. Ideally the modules would create ZBus channels and register themselves as observers at run-time. This minimizes coupling and promotes independence between modules.

Describe the solution you'd like We propose the implementation of a feature in Zephyr OS's zbus mechanism that allows for the dynamic definition of channels at runtime. This enhancement would provide developers with greater flexibility and enable more dynamic communication patterns between components or processes within embedded applications.

The ZBUS_CHANNEL_DEFINE() macro does not allow for run-time definition of ZBus channels. We would like a function that can be used to create a ZBus channel at run-time. Something similar to the function shown below.

    /* ===== EXAMPLE API ===== */
    /**
     * @brief Add a channel to the channels list.
     *
     * @param name Channel name.
     * @param msg_init The message initialization.
     * @param msg_size The channel's message size.
     * @param user_data User data available to extend zbus features.
     * @param validator Message validator.
     * Every message is valid when this field is empty.
     * @param obs The observers list. The sequence indicates the priority of
     * the observer. The first the highest priority.
     * @param timeout Waiting period to add a channel,
     *                or one of the special values K_NO_WAIT and K_FOREVER.
     * @param[out] chan Pointer to reference to set to newly added channel.
     *
     * @retval 0 New channel was added to the channels list.
     * @retval -EALREADY The channel is already present in the channels list.
     * @retval -ENOMEM Returned without waiting.
     * @retval -EAGAIN Waiting period timed out.
     * @retval -EINVAL Some parameter is invalid.
     */
    int zbus_add_chan(const char *const name,
                      void *const msg_init,
                      const size_t msg_size,
                      void *const user_data,
                      bool (*const validator)(const void *msg, size_t msg_size),
                      const struct zbus_observer *obs,
                      k_timeout_t timeout,
                      const struct zbus_channel **chan);

As shown above, the caller could accept the new channel through a pointer and return an error code. Alternately, the function could return the new channel and let any error be inferred through a NULL return value.

Describe alternatives you've considered

Alternative 1:

Create a pool of zbus channels available for modules to dynamically repurpose.

PROS:

• Can be implemented with no updates to Zephyr's ZBus.

CONS:

• Inefficient memory use since there will probably be "just in case" unused zbus channels.
• All channels must have the same message size in order to have a predictable behavior. The method described in samples/subsys/zbus/dyn_channel could be used to have dynamic message sizes.

Alternative 2:

Implement a custom publish/subscribe messaging system.

PROS:

• Meets our system requirements.

CONS:

• More issues/problems expected due to less community support.

[github-actions[bot]]

Hi @NelsonFerragut! We appreciate you submitting your first issue for our open-source project. 🌟

Even though I'm a bot, I can assure you that the whole community is genuinely grateful for your time and effort. 🤖💙

[rodrigopex]

@NelsonFerragut,If you were not using zbus, what should you use to make llext modules communicate? Just to be inspired.

[gbmhunter]

I was just looking for this functionality myself! Dynamic creation of zbus channels would be powerful. Didn't expect there not to be a dynamic way of creating them before reading the docs.

[NelsonFerragut]

@NelsonFerragut,If you were not using zbus, what should you use to make llext modules communicate? Just to be inspired.

The value of topics and subscribers is very high for our application. So, we haven't given much thought to Alternative 3. In response to your question, I gave it some more thought. I think it is not a valid alternative. All the other data passing methods available in Zephyr are one-to-one message channels. We need many-to-many message channels. I will remove Alternative 3 from the feature request. I've moved it here for reference...

Alternative 3:

Use an existing Zephyr solution (Data Passing) for sending data between threads.

PROS:

• Can be implemented with no updates to Zephyr's ZBus.

CONS:

• Cannot use the channel feature with this solution.

[NelsonFerragut]

For alternative 2, we would probably modify the existing zbus implementation to customize it for ourselves. This, too, is very undesirable. We would prefer that the Zephyr community continue to maintain it and test it.

[rodrigopex]

@NelsonFerragut Taking an initial view of the zbus code. I could not find a good reason that prevents the dynamic channel feature addition despite the increasing complexity of the code base. You should use that in conjunction with runtime observers.

For alternative 2, we would probably modify the existing zbus implementation to customize it for ourselves. This, too, is very undesirable. We would prefer that the Zephyr community continue to maintain it and test it.

I won't be able to work on that for the next month. Do you have an initial implementation? If so, could you open a PR with the change?

[NelsonFerragut]

@rodrigopex I, too, think this could be incorporated into the current code base. Maybe with a KConfig option to enable/disable it. We are working on something, but so far it was not focused on updating the Zephyr code base. I will look at it from that perspective and see if there is anything we can do.

[rodrigopex]

@NelsonFerragut, I'm thinking even more about the extensions (LLEXT) and zbus usage... I imagine the external module already would have its channel defined. It may not be necessary to have that dynamically. Don't you think? The only essential dynamic allocated element necessary should be the observer, which already is. Why should you need dynamic channels?

[NelsonFerragut]

@rodrigopex, Sorry for the delayed response...

I agree with you. An external module can pre-define its zbus channel(s). However, doing so would require the other external modules that it communicates with to know where the channel is defined. In order to get the channel, they must directly call each other. I'm not sure how feasible that is since an external module may or may not be present.

I'd like to achieve two simultaneous goals:

1. Provide a messaging infrastructure that does not constrain the architecture.
2. Minimize coupling between external modules.

The current plan is to use an abstraction layer in the software.

The abstraction layer allows the external modules to determine what ZBus channels are required without forcing extra dependencies between the external modules. The external modules will request ZBus channels by name (and maybe message size); the abstraction layer returns a reference to the channel – creating it, if necessary. Dynamically created channels are required so that the abstraction layer can create a requested channel on the first request.

It is true that the abstraction layer can define all the channels statically. However, that would effectively move much of the software architecture definition into the abstraction layer. I want to avoid that. I want the abstraction layer to be stable software that supports varying architectures defined through the presence and implementation of external modules.

[rodrigopex]

@NelsonFerragut, I see your point.

I still don't think the dynamically created channel is necessary. At some point, the external modules must know how to interact with other external modules. You must imply some standardization to make communication possible, right? Maybe your Abstraction Layer should know which external module is working and make that information available to other external modules. However, creating a new channel from scratch dynamically seems unnecessary flexibility, which you should not enjoy based on the communications restrictions of the modules.

I imagine the external modules have well-defined APIs/Channels, right? How would you add a new channel that does not know the existing module's APIs?

I would love to talk to you (in a meeting) to understand better the scenario. If you think that is appropriate, send a message on Discord, and we can schedule a meeting.

The change you are asking for will require a lot of effort and time. We must avoid increasing zbus' complexity.

[pillo79]

Found this after our talks at ZDS - I'm sure this would be pretty useful to us as well. However, we also need to access other iterable section stuff, like mutexes and such. I think we need to cover this from the llext side and figure out a way to "merge in" kernel objects that are defined in the extension itself. With that done, I figure ZBus could then use that infrastructure to extend its lookup tables to any extension.