clarify configuration

[bmcdonnell-ionx]

I don't understand how to use the Mbed OS configuration system. I think the reference documentation regarding it is unclear. If I'm missing something(s) obvious, please point me to it. Otherwise, please clarify the documentation to answer questions such as those I pose below. (Also, I'd appreciate some quick answers here, if possible.)

For instance, the page linked above provides an example of how to configure the stack size for the main thread, by putting this in mbed_app.json:

{
    "config": {
        "main-stack-size": {
            "value": 2000
        }
    }
}

But how does this work? What does it do? I couldn't find "main-stack-size" anywhere in the mbed-os or mbed-cli code trees. (I used grep -R.)

On further searching (grep -Ri "main[^ ]stack[^ ]size" .), I find the conditional #define MBED_CONF_APP_MAIN_STACK_SIZE in mbed_boot.c, so I infer that lower-case-hyphenated labels in .json files get somehow translated to MBED_CONF_ALL_CAPS #defines - but how? Where is the code that does this so I can learn more when the documentation falls short, or I want to improve something? How/where do the custom #defines I configure get made visible to my code? (e.g. In makefiles, or is a new header file generated? What about when I use mbed export?) [EDIT: I see that it generates mbed_config.h. Please consider mentioning this in the documentation.]

Where can I find the list of configurable items? Are there more than what are listed on the pages linked below?

How does the info on the Platform, Drivers, RTOS, and Connectivity pages translate to what I should put in a .json file? And how do the .json files get named?

For instance, the Platform page begins with this:

Configuration parameters
------------------------

Name: platform.default-serial-baud-rate
    Description: Default baud rate for a Serial or RawSerial instance (if not specified in the constructor)
    Defined by: library:platform
    Macro name: MBED_CONF_PLATFORM_DEFAULT_SERIAL_BAUD_RATE
    Value: 9600 (set by library:platform)

What do I put in a .json file to #define MBED_CONF_PLATFORM_DEFAULT_SERIAL_BAUD_RATE to the desired value? How do I name the .json file? What is the meaning of the . separator in the Name? What is the meaning of the : separator in Defined by?

[AnotherButler]

@bmcdonnell-ionx Thank you for raising this issue. We have more config content in our tools section: https://os.mbed.com/docs/v5.7/tools/configuring-tools.html

I think the content here can answer most of your questions. If not, @geky could you please take a look?

Also, would you find it useful if we moved all of our configuration content to the same location, instead of linking them together?

[0Grit]

@bmcdonnell-ionx I've had similar problems in the past interpreting what a configuration parameter is versus a macro.

275 and mbedOS: #5225 were mildly helpful in clearing this up.

It doesn't feel intuitive yet though. I'm not sure if it is the documentation or the actual config system implementation.

[bmcdonnell-ionx]

@AnotherButler Thanks for the link. The answers to most of my questions are still unclear to me, though. And that was a long read to try to glean the (IMO) important bits I was looking for.

I feel like the first document I linked to isn't really telling me how to use the info it's providing, unless what I want to do happens to coincide with an example. And the second seems like how to arrange a bunch of magic words.

I think for these configuration system documents, the authors need to consider it from the Mbed OS users' perspective, instead of from an Mbed OS developer's perspective. I think it should begin within the first section or subsection with something like, "These tools [which?] read the .json config files, and translate label : value pairs (or whatever) in such-and-such syntax and format, into macros via _____ method. (Or does it do something beyond generate macros?) All generated macros are output to mbed_config.h. [Right?]" Then show a simple example to demonstrate the syntax and translation. And then tell me where I can find a list or lists of all of the available config options/params. And tell me how I can find them in the code (i.e. where the macros are used). And then get into the details and all the flexibility.

There are some parts near the bottom of the Configuration System page which speak to some of this, but still leave me with questions. (That is, in the last two sections, "Configuration data precedence" and "Using configuration data in the code".) For instance, I think this bit should be in the intro of the first section, or in the first subsection:

"When compiling, the configuration system will automatically generate macro definitions for the configuration parameters and all the macros defined in libraries and the application in their macros keys. These definitions will be written in a file named mbed_config.h, located in the build directory." But how does target.serial_console_speed of 9600 translate to #define MBED_SERIAL_UART_SPEED 9600? Note the different words, "console" vs. "UART". And which prefixes are applied when?

[bmcdonnell-ionx]

@loverdeg-ep

Thanks for your input. Glad to know I'm not the only one confused.

what a configuration parameter is versus a macro.

The configuration system generates macros, which it puts in mbed_config.h. I'm not sure yet if that's all it does, though.

It doesn't feel intuitive yet though. I'm not sure if it is the documentation or the actual config system implementation.

Documentation, IMO so far.

[geky]

Hi @bmcdonnell-ionx, thanks for the feedback. I think you're right we probably need at least one more document targeting users (@AnotherButler a tutorial maybe?).

Lets see if I can't help clarify with what I know:

The library developer defines a configuration option

A library can provide a special file name "mbed_lib.json", the details are in the config docs, but basically this has each config option name, documentation, and defaults values:

   "name": "events",
   "config": {
       "use-lowpower-timer-ticker": {
          "help": "Enable use of low power timer and ticker classes. May reduce the accuracy of the event queue.",
          "value": false
       }
   }

Each config option is converted to a C macro by prepending the prefix MBED_CONF_<LIBRARY>_ (where is the library name), and then converting all characters to uppercase and any -s to _. So for example, the above config option is equivalent to the following:

   #define MBED_CONF_EVENTS_USE_LOWPOWER_TIMER_TICKER false

Alternatively, the developer can provide a "macro_name" for a config option to change the name of the emitted define:

   "name": "events",
   "config": {
       "use-lowpower-timer-ticker": {
          "macro_name": "EVENTS_LP_TICKER",
          "help": "Enable use of low power timer and ticker classes. May reduce the accuracy of the event queue.",
          "value": false
       }
   }

Becomes:

   #define EVENTS_LP_TICKER false

If the user compiles through the online compiler or with the mbed tools (mbed compile blablabla), these defines are emitted automatically by the toolchain.

If the user exports to another development environment, the mbed tools will create an "mbed_config.h" file with each configuration option defined. This means that if the user changes a configuration option, the program will need to be exported to get the updated values.

Either way, the user should be able to use the config option as though it had been #defined in their own code before every file in their codebase.

The application developer modifies a config option

Like a library, and application can provide an "mbed_app.json", which has the same rules as an "mbed_lib.json", but strangely does not have a name.

However, not only can we provide our own config options, we can override config options from other libraries. Now stay with me, because the syntax gets a bit complicated. An override is provided in the form of a "target_overrides" and it looks like this:

   "target_overrides": {
       "*": {
            "events.use-lowpower-timer-ticker": true
       }
   }

This override will change the "use-lowpower-timer-ticker" config option in the "events" library that we defined above to the value true. All overrides are resolved before compiling the code base, so all the user will see is this definition:

   #define MBED_CONF_EVENTS_USE_LOWPOWER_TIMER_TICKER true

What about that asterisk? The "target_overrides" field actually has one more trick up its sleeve. An application can actually specify different overrides for different targets by specifying the target's name in the target override:

   "target_overrides": {
       "*": {
            "events.use-lowpower-timer-ticker": true
       },
       "K64F": {
            "events.use-lowpower-timer-ticker": false
       },
   }

This would evaluate to true on all targets except the K64F, where it would evaluate to false. This becomes very helpful when assigning pins on different targets.

Examples

Here's some real world examples we have been using:

• mbed_lib.json: https://github.com/ARMmbed/mbed-os/blob/master/features/filesystem/littlefs/mbed_lib.json
• mbed_app.json: https://github.com/ARMmbed/mbed-os-example-client/blob/master/mbed_app.json

---

Does that help clarify things?

@theotherjimmy, @bridadan, feel free to add anything I've missed. This may be a good start to a tutorial in the docs.

[geky]

Ah, so in the case of main-stack-size, I believe it's currently not provided in an "mbed_lib.json", but is just expected to be created in "mbed_app.json", which isn't suggested. @bulislaw?

[bmcdonnell-ionx]

@geky

This may be a good start to a tutorial in the docs.

I think so!

Some questions remain for me.

A library can provide a special file name "mbed_lib.json"

1. Do you mean "library" here in the generic sense of modular reusable code? As opposed to a "library" pulled in by mbed-cli due to a repo reference in a mystuff.lib file.

Each config option is converted to a C macro by prepending...

2. Where is the tool(s)/code that does this?

If the user compiles through the online compiler or with the mbed tools (mbed compile blablabla), these defines are emitted automatically by the toolchain.

3. What does "emitted automatically by the toolchain" mean? Does it mean that the toolchain passes the #defines as command line arguments to the compiler?

An override is provided in the form of a "target_overrides"

4. Are there other overrides besides "target_overrides"? If so, what are their names, where can I find them, and how do I know when to use which? If not, why "target" overrides for things that are not target-specific?

5. [Refer to the "main-stack-size" example in Configuration.] Why are the syntax and labels ("config", not "target-overrides") here different from "target-overrides"? How do I know which to use?

6. If I want to override some values from the Platform, Drivers, RTOS, or Connectivity pages, what is the opening label? "config", "target-overrides", or something else? How could I figure that out for myself?

7. I looked at a few mbed-os/features/**/module.json files. They seem different from what you've described here. What are they for, how do they work, etc.? Please link to documentation, if there is any.

8. Same for mbed-os/targets/targets.json.

9. Should I ask about mbed-os/tools/*.json and mbed-os/tools/**/*.json?

10. Is there some tool that parses the "help" values and emits them somewhere else? Or are they just there for human reference when reading the .json files themselves?

[geky]

Let's see what I can answer!

1. A library can provide a special file name "mbed_lib.json"

   Do you mean "library" here in the generic sense of modular reusable code? As opposed to a "library" pulled in by mbed-cli due to a repo reference in a mystuff.lib file.

   Yes, to your first sentence. The configuration options and .lib files are independent features.

2. Each config option is converted to a C macro by prepending...

   Where is the tool(s)/code that does this?

   Oh uh, here I think, I'm not that familiar with how it's implemented: https://github.com/ARMmbed/mbed-os/blob/master/tools/config/__init__.py#L74-L75

3. What does "emitted automatically by the toolchain" mean? Does it mean that the toolchain passes the #defines as command line arguments to the compiler?

   Yes. How this happens is toolchain specific. The important part is that you do not have to explicitly include a special file.

4. Are there other overrides besides "target_overrides"? If so, what are their names, where can I find them, and how do I know when to use which? If not, why "target" overrides for things that are not target-specific?

   None that I know of. I'm not sure, but I think the name comes from the fact that the overrides are per target, with the target "*" being special. I think there is motivation to provide just an "overrides" field that does just this.

5. [Refer to the "main-stack-size" example in Configuration.] Why are the syntax and labels ("config", not "target-overrides") here different from "target-overrides"? How do I know which to use?

   You can have both "config" and "target_overrides" fields in a config file (either mbed_lib.json or mbed_app.json).

   "config" = creating a field "target_overrides" = replacing an existing field

   If you provide a "target_overrides" for a config option that is not declared in a "config" field, the mbed tools will error.

   So the rule of thumb: If you're creating a new config option, add it to the "config" field, if you're replacing an existing config option, add it to the "target_overrides" field.

6. If I want to override some values from the Platform, Drivers, RTOS, or Connectivity pages, what is the opening label? "config", "target-overrides", or something else? How could I figure that out for myself?

   If you're overriding a value, the label should be "target_overrides". For example here is an mbed_app.json that overrides the platform.stdio-baud-rate option (from here):

   {
      "target_overrides": {
          "*": {
              "platform.stdio-baud-rate": 115200
          }
      }
   }

   All options provided in the docs are intended to be used with "target_overrides". The "config" section is only useful if you want to add your own config options.

7. I looked at a few mbed-os/features/**/module.json files. They seem different from what you've described here. What are they for, how do they work, etc.? Please link to documentation, if there is any.

   The module.json files are deprecated files from mbed OS v3. This is the documentation. But no tools in mbed still process the module.json file, and they are replaced by mbed_lib.json files. They're still lingering in our repository because for some of the nested libraries there wasn't a reason to remove them.

8. Same for mbed-os/targets/targets.json.

   target.json is different and to be honest I know a lot less about that file. It provides the description for individual targets and includes things like the target name, the target's inheritance, and what device_has features there are. More info can be found here: https://os.mbed.com/docs/v5.7/tools/adding-and-configuring-targets.html

9. Should I ask about mbed-os/tools/*.json and mbed-os/tools/*/.json?

   Any json files in the tools are just used by the python scripts that compose the tools and shouldn't contain any configuration options in themself.

10. Is there some tool that parses the "help" values and emits them somewhere else? Or are they just there for human reference when reading the .json files themselves?

    Not yet, though the intention of the field is to be both human readable and parsable for future tools.

[bmcdonnell-ionx]

@geky, thank you for quickly answering my immediate questions. I hope this also helps you in improving the documentation. 😄

About where to put the documentation, I can see how some wound up in Configuration under Reference, as it has to do with writing your code; and some wound up in Config Tools under Tools. Consider putting it all in one place, though - and then maybe link to it from the other place for discoverability. If not, I recommend at least linking to each from the other. Likewise, if your info here turns into a 3rd document, link to each from the other.

5. Refer to the "main-stack-size" example in Configuration.] Why are the syntax and labels ("config", not "target-overrides") here different from "target-overrides"? How do I know which to use?

You can have both "config" and "target_overrides" fields in a config file (either mbed_lib.json or mbed_app.json).

"config" = creating a field "target_overrides" = replacing an existing field

Ah. I was confused because "main-stack-size" is also overriding a default value, but now I understand the difference that in that case the default is in the code, not another .json config file.

6. If I want to override some values from the Platform, Drivers, RTOS, or Connectivity pages, what is the opening label? "config", "target-overrides", or something else? How could I figure that out for myself?

If you're overriding a value, the label should be "target_overrides".

I recommend adding that on those pages.

[0Grit]

@geky Personally, I found it confusing that mbed_lib.json is optional.

A library can provide a special file name "mbed_lib.json",

It made it hard for me to determine what a library actually was; Seeing that some libraries had the file and others didn't. Same goes for mbed_app.json.

Could a case be made to make mbed-cli require these files? Even if they just tracked a version number or held the name of the library. Slippery slope but you could even use it to have mbed-cli enforce some basic standards for library creation.

[bridadan]

@geky Excellent clarification!

Regarding this question:

10. Is there some tool that parses the "help" values and emits them somewhere else? Or are they just there for human reference when reading the .json files themselves?

There is actually an option in Mbed CLI to do this. The command is mbed compile --config -v:

$ mbed compile --config -v
[mbed] Working path "C:\Users\bridan01\onedrive_arm\Documents\dev\mbed-os-example-blinky" (program)
[mbed] Exec "c:\python27\python.exe C:\Users\bridan01\onedrive_arm\Documents\dev\mbed-os-example-blinky\mbed-os\tools\get_config.py -t GCC_ARM -m K64F --source . -v" in C:\Users\bridan01\onedrive_arm\Documents\dev\mbed-os-example-blinky
Scan: .
Scan: FEATURE_STORAGE
Scan: FEATURE_LWIP
Configuration parameters
------------------------
Name: configuration-store.storage_disable
    Description: Configuration parameter to disable flash storage if present. Default = 0, implying that by default flash storage is used if present.
    Defined by: library:configuration-store
    No value set
Name: drivers.uart-serial-rxbuf-size
    Description: Default RX buffer size for a UARTSerial instance (unit Bytes))
    Defined by: library:drivers
    Macro name: MBED_CONF_DRIVERS_UART_SERIAL_RXBUF_SIZE
    Value: 256 (set by library:drivers)
Name: drivers.uart-serial-txbuf-size
    Description: Default TX buffer size for a UARTSerial instance (unit Bytes))
    Defined by: library:drivers
    Macro name: MBED_CONF_DRIVERS_UART_SERIAL_TXBUF_SIZE
    Value: 256 (set by library:drivers)
Name: events.present
    Defined by: library:events
    Macro name: MBED_CONF_EVENTS_PRESENT
    Value: 1 (set by library:events)
Name: events.shared-dispatch-from-application
    Description: No thread created for shared event queue - application will call dispatch from another thread (eg dispatch_forever at end of main)
    Defined by: library:events
    No value set

<removed for brevity>

Macros
------
Defined with "macros": ['UNITY_INCLUDE_CONFIG_H']
Generated from configuration parameters: ['MBED_CONF_LWIP_ENABLE_PPP_TRACE=0', 'MBED_CONF_FILESYSTEM_PRESENT=1', 'MBED_CONF_LWIP_TCP_ENABLED=1', 'MBED_CONF_LWIP_IPV4_ENABLED=1', 'MBED_CONF_LWIP_DEFAULT_THREAD_STACKSIZE=512', 'MBED_CONF_LWIP_UDP_SOCKET_MAX=4', 'MBED_CONF_EVENTS_PRESENT=1', 'MBED_CONF_LWIP_TCPIP_THREAD_STACKSIZE=1200', 'NSAPI_PPP_IPV4_AVAILABLE=1', 'MBED_CONF_DRIVERS_UART_SERIAL_TXBUF_SIZE=256', 'MBED_LFS_PROG_SIZE=64', 'MBED_CONF_LWIP_PPP_THREAD_STACKSIZE=768', 'NSAPI_PPP_IPV6_AVAILABLE=0', 'MBED_CONF_PLATFORM_STDIO_FLUSH_AT_EXIT=1', 'MBED_CONF_DRIVERS_UART_SERIAL_RXBUF_SIZE=256', 'MBED_CONF_NSAPI_PRESENT=1', 'MBED_CONF_LWIP_ADDR_TIMEOUT=5', 'MBED_CONF_LWIP_IP_VER_PREF=4', 'MBED_CONF_PPP_CELL_IFACE_AT_PARSER_TIMEOUT=8000', 'MBED_LFS_BLOCK_SIZE=512', 'MBED_CONF_PPP_CELL_IFACE_AT_PARSER_BUFFER_SIZE=256', 'MBED_CONF_PLATFORM_FORCE_NON_COPYABLE_ERROR=0', 'MBED_CONF_PLATFORM_STDIO_BAUD_RATE=9600', 'MBED_CONF_LWIP_IPV6_ENABLED=0', 'MBED_CONF_EVENTS_SHARED_HIGHPRIO_STACKSIZE=1024', 'MBED_CONF_LWIP_TCP_SERVER_MAX=4', 'MBED_CONF_PPP_CELL_IFACE_BAUD_RATE=115200', 'MBED_LFS_READ_SIZE=64', 'MBED_CONF_PLATFORM_DEFAULT_SERIAL_BAUD_RATE=9600', 'MBED_CONF_LWIP_TCP_SOCKET_MAX=4', 'MBED_CONF_RTOS_PRESENT=1', 'MBED_CONF_EVENTS_SHARED_DISPATCH_FROM_APPLICATION=0', 'MBED_CONF_EVENTS_SHARED_EVENTSIZE=256', 'MBED_CONF_LWIP_DEBUG_ENABLED=0', 'CFSTORE_STORAGE_DISABLE=0', 'MBED_CONF_EVENTS_SHARED_STACKSIZE=1024', 'NSAPI_PPP_AVAILABLE=0', 'MBED_CONF_PPP_CELL_IFACE_APN_LOOKUP=0', 'MBED_LFS_LOOKAHEAD=512', 'MBED_CONF_EVENTS_USE_LOWPOWER_TIMER_TICKER=0', 'MBED_CONF_LWIP_USE_MBED_TRACE=0', 'MBED_CONF_PLATFORM_STDIO_CONVERT_NEWLINES=0', 'MBED_CONF_LWIP_ADDR_TIMEOUT_MODE=1', 'MBED_CONF_LWIP_ETHERNET_ENABLED=1', 'MBED_LFS_ENABLE_INFO=0', 'MBED_CONF_LWIP_SOCKET_MAX=4', 'MBED_CONF_EVENTS_SHARED_HIGHPRIO_EVENTSIZE=256']

The -v option enables the help text. Removing it will provide a less verbose summary of the config options.

Regarding @loverdeg-ep's question:

Could a case be made to make mbed-cli require these files?

This currently isn't enforced I believe to be backward compatible with Mbed 2 libraries (libraries in the sense of the .lib files used to pull in repos with reusable code).

[theotherjimmy]

@bmcdonnell-ionx @geky Some corrections:

Each config option is converted to a C macro by prepending..

Where is the tool(s)/code that does this?

~That is here: https://github.com/armmbed/mbed-os/tree/master/tools/config/__init__.py~ I misread your link as targets (facepalm)

What does "emitted automatically by the toolchain" mean? Does it mean that the toolchain passes the #defines as command line arguments to the compiler?

Yes. How this happens is toolchain specific. The important part is that you do not have to explicitly include a special file.

Nit: This is toolchain independent. The configuration macros are emitted into a header file named mbed_config.h, and included before any other code when compiling. The important part is still true :D

Any json files in the tools are just used by the python scripts that compose the tools and shouldn't contain any configuration options in themself.

Well, build profiles are configuration, and have documentation. I can't seem to find an os.mbed.com link for that one though.

[theotherjimmy]

Could a case be made to make mbed-cli require these files?

The answer is [unfortunately] no. For 2 reasons: 1) We don't have any way to determine where we should have an mbed_lib.json but don't. 2) Many libraries depend on the current behavior.

[theotherjimmy]

It made it hard for me to determine what a library actually was; Seeing that some libraries had the file and others didn't. Same goes for mbed_app.json.

~@bmcdonnell-ionx~ (wrong person, caffeine not working yet) @loverdeg-ep I think your confusion comes from the fact that we don't define library. We define sub-repos across hg and git, with .lib files, and we define a method for creating configuration that may later be overwritten.

[0Grit]

@theotherjimmy I suppose what I should have meant was have mbed-cli generate mbed_lib.json by default.

Though this would require separate mbed new library and mbed new application commands.

[theotherjimmy]

@loverdeg-ep I suppose that would aid in discovering the configuration system. It might be worth implementing. Certainly we could create an empty mbed_app.json on mbed new

[theotherjimmy]

@geky I'm going to start an overhaul of the current documentatoin using snippets that you wrote, and trying to correct the bits that @bmcdonnell-ionx and @loverdeg-ep found confusing. Looks like most, if not all, of the examples are incorrect.

[theotherjimmy]

see https://github.com/ARMmbed/Handbook/pull/463. Sorry that took so long, I think it was in great need for an overhaul.