Parametrize some FreeRTOS config fields (configIDLE_SHOULD_YIELD and configUSE_PREEMPTION) (IDFGH-6716)

[carlessole]

Module or chip used: ESP32-WROVER (With ESP_PROG)
IDF version: v4.3
Build System: CMake/idf.py (usingEclipse Plugin)
Compiler version: esp-2020r3-8.4.0
Operating System: Windows
(Windows only) environment type: -.
Using an IDE?: Yes. Eclipse + ESP-IDF plugin
Power Supply: USB

Hello, I have seen that almost all the freeRTOS config fields (defined in FreeRTOSConfig.h) are hardcoded and can not be modified (not by sdkconfig and as far as I know, this file is recommended to not modify it, right?).

I see that in master branch is the same: https://github.com/espressif/esp-idf/blob/master/components/freertos/esp_additions/include/freertos/FreeRTOSConfig.h

Is there any idea to support/let modify some config fields? I would like to be able to modify the:

• configIDLE_SHOULD_YIELD: https://github.com/espressif/esp-idf/blob/a470ae224d2479cbb1019a0b69490a1249e6a562/components/freertos/esp_additions/include/freertos/FreeRTOSConfig.h#L145
• configUSE_PREEMPTION: https://github.com/espressif/esp-idf/blob/a470ae224d2479cbb1019a0b69490a1249e6a562/components/freertos/esp_additions/include/freertos/FreeRTOSConfig.h#L64

As for example: https://github.com/espressif/esp-idf/blob/a470ae224d2479cbb1019a0b69490a1249e6a562/components/freertos/esp_additions/include/freertos/FreeRTOSConfig.h#L68

Thanks

[Dazza0]

@carlessole

There are some IDF system level considerations that force most (if not all) FreeRTOS configurations to be hardcoded. We've already made menuconfigurable what we can.

• configIDLE_SHOULD_YIELD will be tricky. I'll need to check how it interacts with configUSE_TICKLESS_IDLE as it may prevent the IDLE task from putting the system into light sleep.
• configUSE_PREEMPTION 0 might break a lot of higher level components if they are written with the assumption that preemption is always enabled. It's rare to disable preemption in FreeRTOS as it basically makes the scheduler no longer Real Time.

[carlessole]

Thanks for the answer @Dazza0 .

I understand the configUSE_PREEMPTION dependency on many higher level code.

Regarding the configIDLE_SHOULD_YIELD for my needs it would be great to have that possibility. Without it what is happening is that once an IDLE priority TASK will not be executed after one FreeRTOS tick once sched_yield has been called on it.

I have tried (in order to avoid changing the configIDLE_SHOULD_YIELD) to use an IDLE hook an perform an sched_yield from there, but as it is normal, the behaviour of my code was quite undefined.

https://docs.espressif.com/projects/esp-idf/en/release-v3.0/api-reference/system/hooks.html#_CPPv431esp_register_freertos_idle_hook22esp_freertos_idle_cb_t

I'm assuming that the warnings are already preventing to use 'sched_yields' and similars in this hooks.

Carles

[carlessole]

This issue is related with: https://github.com/espressif/esp-idf/issues/7256

I don't know if the fix that you have introduced in version 4.4

has already solving it (configIDLE_SHOULD_YIELD is still not configurable):

https://github.com/espressif/esp-idf/blob/8153bfe4125e6a608abccf1561fd10285016c90a/components/freertos/include/esp_additions/freertos/FreeRTOSConfig.h#L219

Thanks

[carlessole]

Any news on that? Thanks

[o-marshmallow]

Hi @carlessole ,

There is a way to force the macros that you want when compiling ESP-IDF without modifying ESP-IDF. Let's take the example hello-world in esp-idf/example/get-started/hello_world/.

Edit its CMakeLists.txt to add the following line:

add_compile_options(-include ${CMAKE_CURRENT_SOURCE_DIR}/main/override.h)

Right after include, before project(hello_world)

This will force the inclusion of the file main/override.h inside all the source files compiled in IDF. This file will be included at first, so before any other includes the source file may contain.

Thus, we need to create that file, main/override.h, with the following content inside:

#pragma once

// Force the inclusion of FreeRTOS config to make sure we can redefine any macro right after
#include "freertos/FreeRTOSConfig.h"

// New value for configIDLE_SHOULD_YIELD
#undef configIDLE_SHOULD_YIELD
#define configIDLE_SHOULD_YIELD                         1

Now you can compile IDF, your new macro will be taken into account for all sources, including FreeRTOS'

[Dazza0]

@carlessole I'm still hesitant to expose configUSE_PREEMPTION and configIDLE_SHOULD_YIELD as Kconfig options due to the reasons mentioned above (i.e., lack of preemption and idle yielding could cause other higher level components to break). Could you elaborate further as to your use case for wanting to set configUSE_PREEMPTION = 0 and configIDLE_SHOULD_YIELD = 1. Maybe there's another way to achieve the same thing.

I'm also open to allowing users to override FreeRTOS config (but not exposing them as Kconfig options). One method is described above by @o-marshmallow. But, adding something like user provided #include "FreeRTOSConfigOverrides.h" into FreeRTOSConfig.h so that users can undef/define is also an option.

[carlessole]

@o-marshmallow Thanks!! It is really a nice way to override some symbols in ESP-IDF without modifying its source code :). But anyway, I need to be sure that if 'configIDLE_SHOULD_YIELD' is '1', all the architecture is readyy to support that.

[carlessole]

@Dazza0 I understand it.

I will try to expose my reasons to wanting that.

Main reason is that we are migrating many existing projects from another microcontroller to ESP32. Then all the code is already done and it is expecting a certain scheduler behavior. They are big projects that involves many and many files.

Then, regarding 'configUSE_PREEMPTION': All the existing code supposes that if no 'delay(n)' or 'sched_yield()' is called, no one can take the control of a running task. Then many data structures shared between tasks are not well protected with 'mutexes' or 'semaphores' as 'configUSE_PREEMPTION = 0' was a precondition.

That has been solved protecting them with 'mutexes' and 'semaphores', but anyway keeping the same behavior as the old scheduler (which one has been taken as reference to develop ALL the existing code) should be very nice for us. As I said I understand that many ESP-IDF components have been created with 'configUSE_PREEMPTION = 1' in mind, so, not big deal about not being able to change this option ('configUSE_PREEMPTION')

Regarding 'configIDLE_SHOULD_YIELD': The existing projects have many tasks running (about 20). None of them perform a 'delay(n)', they just perform and 'sched_yield()', that means that once a tasks gives away the uP control, the task is immediately ready again to enter (it is placed in the scheduler queue). To achieve that all the tasks have the same priority (the IDLE task priority). The refresh time of each task has been calculated with oscilloscope and in the old platform it was OK as there were no tasks that keeps running too much time. But when 'configIDLE_SHOULD_YIELD = 0' (like it is in ESP-IDF), then each time all our tasks have performed its job and they have been placed in the scheduler queue to enter again ASAP, it takes at least a freeRTOS tick (fastest configuration is 1ms) to be running again as IDLE tasks are keeping the execution until a freeRTOS tick . See this issue for more details. For example, we have many tasks that are reading the UART input with polling it and they have the requirement to be able to answer with an ACK very very fast (less than 3 ms). With current ''configIDLE_SHOULD_YIELD = 0' we see that the responsiveness of our code is much more slower and in some cases we are not matching the needs of the sytem where our devices are installed.

So it is very important that IDLE tasks can yield once they have finished their job and not waiting until a freeRTOS tick happens (see here).

[carlessole]

@Dazza0 Any feedback on that?

Thanks!