.murdock: Build apps for different environments.

[miri64]

Description

Both https://github.com/RIOT-OS/RIOT/pull/14665 and https://github.com/RIOT-OS/RIOT/pull/14668 showed me once more, that we really should build some tests in their different compile time configurations.

While it is of course really easy to just duplicate the tests for that, I have an idea floating around in my head for some time now, that I don't really have time to execute right now: A descriptive file to define different build environments for Murdock. This optional file could be written in YAML and be structured somewhat like (example would be for the tests/lwip* applications):

- name: ipv4
  env:
    LWIP_IPV4: 1
- name: ipv6
  env:
    LWIP_IPV6: 1
- name: dualstack
  env:
    LWIP_IPV4: 1
    LWIP_IPV6: 1

and be fetched in a similar manner as boards and toolchains are. The name can then be used for the results file to identify the environment (which is why they should be unique).

I think we can also remove a lot of existing code duplication with that approach (see e.g. https://github.com/RIOT-OS/RIOT/issues/6596 by adding USEMODULE to that env list).

If this would be still possible or make sense with Kconfig, I don't know.

[leandrolanzieri]

The CI would indeed take advantage of #14654 when changing the configurations using Kconfig. I can imagine we can provide multiple .config files and compile once per file. With #14654 only the affected object files would be re-compiled.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you want me to ignore this issue, please mark it with the "State: don't stale" label. Thank you for your contributions.

[leandrolanzieri]

Still of interest

[chrysn]

Do I understand this issue right in that it is about exercising the freedoms the application definition (the main Makefile, which describes the modules the written code depends on to be present, like "there must be the sockets module") leaves to further configuration (of which the build system usually picks a default one)?

If so, it sounds a lot like that should be way easier to do using the more high-level Kconfig than the Makefiles (as at Kconfig level it is known AFAICT whether a module was selected by the application or just as a consequence of needing "some" socket and, say, dual-stack lwIP was picked but others would work too).

[maribu]

Yes, also, but not limited to. E.g. an concrete example would be the test of GNRC's SOCK API: The test does check for the AUX API the expected behavior depending on whether some auxiliary field is provided or not. E.g. not clearing the flags of a requested field when it cannot be provided by the stack, but clearing and providing the correct values when the corresponding pseudo modules are present.

For complete test resulte, each unique combination of compile time configurations relevant to the test should be checked.

Once such a feature is available, it is likely that a single test application per SOCK API could be provided instead, with the virtual network device that allows injecting messages into the network stack to test being selected as a compile time option. Indeed, this would make it easier to make sure all network stacks adhere to the API.