Add kconfig templates to avoid code repetition

[nordic-krch]

Many modules (ideally all) has own, unique logging level configuration in kconfig. From user perspective this configuration should look the same for each module (choice vs raw int, configuration message, help, etc.). Ideally there should be some way of defining template or macro. It could be generic or log specific if we don't see other use case.

It could be something like:

macro LOG_CONFIG(MODULE, module_str)
choice
prompt "<module_str> logging level"

config <MODULE>_LOG_LEVEL_OFF
bool "Off"
config <MODULE>_LOG_LEVEL_ERR
bool "Error"
config <MODULE>_LOG_LEVEL_WRN
bool "Warning"
config <MODULE>_LOG_LEVEL_INF
bool "Info"
config <MODULE>_LOG_LEVEL_DBG
bool "Debug"
endchoice

config <MODULE>_LOG_LEVEL
int
default 0 if <MODULE>_LOG_LEVEL_OFF
default 1 if <MODULE>_LOG_LEVEL_ERR
default 2 if <MODULE>_LOG_LEVEL_WRN
default 3 if <MODULE>_LOG_LEVEL_INF
default 4 if <MODULE>_LOG_LEVEL_DBG

endmacro

Usage:

source "subsys/logging/Kconfig.log_macro"
LOG_CONFIG(BLUETOOTH_LL, Bluetooth link layer)

Initially it was proposed in the following review: https://github.com/zephyrproject-rtos/zephyr/pull/8816#discussion_r208098277

[nordic-krch]

@nashif @carlescufi @ulfalizer what do you think about this? Maybe similar feature already exists in KConfig?

[SebastianBoe]

It does not exist. The closest thing I am aware of is Microchip's 'source' Kconfig extension:

http://ww1.microchip.com/downloads/en/DeviceDoc/MPLAB%20Harmony%20Configurator%20Developers%20Guide.pdf#page=13

But it does not allow arbitrary macro definitions.

It is a shame that Kconfig uses '#' as a comment, it would have been convenient to use CPP for preprocessing ...

[SebastianBoe]

Perhaps when codegen is merged it could be used.

It would be beneficial to keep the number of preprocessing languages in the project to a minimum at least ...

[ulfalizer]

I'm always a bit wary of generated code, because it can make things harder to understand.

Another option would be to generate the checked-in files themselves, with a comment at the top like

# Note: Do not hand-edit. Generated by foo.py.

If something like this was added to Kconfig itself, I'd want to base it on the existing Kconfig preprocessor (which is a recent addition, and is supported by Kconfiglib): https://github.com/torvalds/linux/blob/master/Documentation/kbuild/kconfig-macro-language.txt

It should be possible to support something like this by extending the preprocessor to support expansion in more places (it's deliberately limited to just expressions and strings now). You could then do something like this to generate symbols:

Kconfig.template:

config $(name-prefix)_LOG_LEVEL_OFF
    bool "Off"

...

Using the template:

name-prefix = BLUETOOTH_LL
source "Kconfig.template"

...

name-prefix = OTHER_SYSTEM
source "Kconfig.template" 

(Could set other parameters too.)

[ulfalizer]

@nordic-krch The other contexts where you're using <MODULE> are already supported by the preprocessor by the way (just with the $() syntax instead). It's not supported when giving a symbol a name though (though it wouldn't be hard to add, as it's a pure preprocessing pass, similar to the C preprocessor).

[ulfalizer]

Hmm... just realized that $(module)_LOG_LEVEL_OFF isn't supported within an expression though (just $(module)), so would take some more changes.

The preprocessor was mostly added to be able to run shell commands to add depends on and defaults. You can do stuff like depends on $(success,some-test-script.sh) (where success is defined as a macro that uses the $(shell,...) function, and expands to either y or n).

[nordic-krch]

Looking at https://github.com/torvalds/linux/blob/master/Documentation/kbuild/kconfig-macro-language.txt

It seems that variables could be used for that:

The variable reference can take parameters, in the following form:

$(name,arg1,arg2,arg3)

You can consider the parameterized reference as a function. (more precisely, "user-defined function" in contrast to "built-in function" listed below).

Though, i cannot find how this 'user-defined functions' are defined. I guess that this is not yet supported by kconfiglib?

[ulfalizer]

@nordic-krch It's supported. You define functions as regular variables, with $(1), $(2), etc., in the locations where the parameter values should be substituted.

For example, the following would give var the value xxx foo bar baz xxx:

fn = foo $(1) baz
var = xxx $(fn,bar) xxx

Side note: I used to dislike Make syntax, but I kinda started respecting it while implementing the Kconfig preprocessor. The reason it's "weird" is that it's deliberately designed to not clash with shell syntax, which would be bad in a language used to build shell commands. It all kinda starts to make sense when you think of it like that. People did it better in the 70s. ;)

[ulfalizer]

Ah, and now I get what you mean by using functions instead. Yeah, that might work as a workaround for the can't-have-$()-within-a-token problem.

[nashif]

Many modules (ideally all) has own, unique logging level configuration in kconfig.

the question is, do we need this for every module? What does qualify as a module? My concern is that we are defining too many modules and while this seem very flexible, it just introduces too many Kconfigs and then makes using the logger difficult. In many of the PRs converting to the new logger I see new modules being added based on the file name and sometimes the function being logged, this is a bit excessive, we probably need to maintain list of modules and just use those, any additional modules will need to be application specific after that and not part of zephyr.

[nordic-krch]

@nashif quantification is another story but even if we specify entity/module to something bigger, multi-file there will be easily more than 50 of those, i guess. It's vital to have them presented in the uniform way in Kconfig. Let say that we want to add another log level, with template it will be a change in one kconfig instead of 50.

@ulfalizer I played a bit with those variables. As you wrote, this preprocessor scope is now limited. Can you then remove this limitation? What would be the way to create multiline variables which would instantiate whole config entry?

[nordic-krch]

fyi @Olivier-ProGlove

[ulfalizer]

@nordic-krch Are you sure all those subsystems need a separate log level btw? I wonder if it's too fine-grained.

Don't think there's a ton of different log level settings in the Linux kernel, for example.

[ulfalizer]

@nordic-krch No way to instantiate whole symbols at the moment. The scope was deliberately limited.

What could be done would be to allow expansion in symbol names though (config $(name)). You could then include the same file multiple times to define different symbols.

[nordic-krch]

Increased granularity is driven by embedded systems limitations: ram, flash and throughput of the transport. Soon (already in PR) user will be able to dynamically reconfigure logger using shell so fine-granularity will be very much welcomed.

config $(name) should be enough.

[ulfalizer]

@nordic-krch If the log levels are dynamic, then what do the Kconfig settings represent? Just the default log levels?

Was thinking they might be there to save space by compiling out certain log levels, but if it's dynamic, then I guess that can't be it.

[nordic-krch]

@ulfalizer There are 2 levels of logs filtering: static/compile time and dynamic. Kconfig configuration decides which are compiled in. When runtime filtering is enabled then additionally those which are compiled in can be disabled/enabled on runtime. Static configuration is to reduce space and slightly improve performance (if log is compiled in but disabled logger still need to evaluate a flag on runtime).

Runtime configuration is optional since it only make sense if you have UI to control it (e.g. shell).

[ulfalizer]

@nordic-krch Alright, makes more sense then, as long as it's not overly fine-grained.

I'll see if I can change things to allow $() anywhere within a (symbol) token while I'm at it. That should make things general enough for a long time.

Could do things like

config LOG_$(module)
    ...

then. Just hope people don't get too crazy with it. :)

[ulfalizer]

Got it implemented now. Could make a PR tomorrow.

Still can't put macros in help texts, but pretty sure that'd be overkilling it. Can customize the prompt text at least.

The template would look like this:

choice
    prompt "Max compiled-in log level for $(module-str)"

config $(module)_LOG_LEVEL_OFF
    bool "Off"

config $(module)_LOG_LEVEL_ERR
    bool "Error"

config $(module)_LOG_LEVEL_WRN
    bool "Warning"

config $(module)_LOG_LEVEL_INF
    bool "Info"

config $(module)_LOG_LEVEL_DBG
    bool "Debug"

endchoice

config $(module)_LOG_LEVEL
    int
    default 0 if $(module)_LOG_LEVEL_OFF
    default 1 if $(module)_LOG_LEVEL_ERR
    default 2 if $(module)_LOG_LEVEL_WRN
    default 3 if $(module)_LOG_LEVEL_INF
    default 4 if $(module)_LOG_LEVEL_DBG

(Think it's better to go for a less cramped style.)

You'd use it like this:

module = FOO
module-str = foo
source "Kconfig.logging_template"

...

module = BAR
module-str = bar
source "Kconfig.logging_template"

[Olivier-ProGlove]

Thanks @nordic-krch I will refresh all my pull-requests with this macro once we the v1.13 release is done. My understanding is the new logging subsystem has moved to v1.14, correct?

[carlescufi]

@Olivier-ProGlove the logging subsystem itself has been merged and was part of 1.13. Using the subsystem from other subsystems is part of 1.14

[carlescufi]

@Olivier-ProGlove @nordic-krch #9797 is now merged

[Olivier-ProGlove]

Thanks @carlescufi I noticed it, I will start to rebase my logging subsystem pull-requests.