WHY I cannot have `delay()` or `yield()`?

[MacDada]

server.on("/test", HTTP_GET, [](AsyncWebServerRequest *request) {
    // 1s OK, 3s crash -> when no wdtFeed
    auto seconds = 20;

    auto end = millis() + seconds * 1000;

    // this loop, when empty, causes a reboot after about 3 seconds
    // but imagine this is not an empty loop, but a real stuff that is just taking a bit of time…
    while (end > millis()) {
        //yield(); // panic, crash, restart
        //ESP.wdtFeed(); // no crashes even for long times, but everything else is being blocked
    }

    request->send(200, "text/plain", "It is OK :-)");
});

Hey, I'm trying to understand what's going on with the AsyncWebServer and delay() conflict. As you can see from the code:

1. while (end > millis()) {} – empty not infinite loop causes a restart – I found out it is a Watchdog not being fed (it would be fed after every pass of the main loop, but ESPAsyncWerbServer runs "outside" of the main loop and actually blocks the main Arduino loop).

2. while (end > millis()) { yield(); } – OK, so let's try to give a bit of time to the main Arduino's tasks that should be running in the background => nope, one cannot use yield() in the web controller. It causes a crash due to a panic.

3. Same as above when using delay() – I discovered this issue while trying to do web controlled servo, with delay()s, ofc. Actually delay does yield inside, hence the problem.

4. while (end > millis()) { ESP.wdtFeed(); } – OK, let's become a cheater and feed the dog within our delaying loop. It works! My device actually is waiting 20 seconds with no problem, and then giving the response back to the browser. Working like a charm. Almost… In the main Arduino loop I send information to the Serial every second – it is blocked until my web controller finishes.

Of course I read important things to remember, it is said what I should not use.

It is NOT said what should I use instead (I already know workarounds, like scheduling action for the main loop to actually do stuff).

It is not said WHY is that a problem.

---

So, after me wasting like 2 days trying to figure out what's going on [aka "learning the hard way"], I would like to ask:

1. WHY does yield() in the web controller crash Arduino?
2. Isn't it like hmm, dangerous (?) – the fact that even background Arduino stuff is being stopped while the web is being handled? Won't the WiFi connection drop? Or any other issue?
3. I believe everything like that should be explained in the docs – am I missing something? I can make PR with docs update after I know it all [mind that I've started learning C++ like a month ago, if I were able to understand everything from analysing the source code, I would probably not be asking here…]

[Pablo2048]

If we are talking about ESP8266 I've described the situation here https://github.com/me-no-dev/ESPAsyncWebServer/issues/1187#issuecomment-1196566925 (I've seen that You already read it). So I try to explain more with my current level of knowledge: I've already described the two cooperative tasks. The "user"(arduino) task can be yielded and the delay() is modified that it internally uses yield to allow the "system" task to run. The "system" task is different - in this context the WiFi driver and lwip stack are executed. The ESPAsyncTCP, which is used fro ESPAsyncWebServer, uses the lwip callbacks so they are called directly from the "system" task.

[MacDada]

I've already described the two cooperative tasks.

What do you mean by "tasks"? Those aren't like C threads, right? It is just "an abstract concept" of two areas of code trying to be run at (almost) the same time by constantly switching execution from one to another, right?

I imagine it like this:

• we have an "outer" function that makes the infinite loop – in the loop it makes internal basic arduino stuff [lets call it doArduinoInternals(), and then calls my loop() –> so we're switching between arduino's and user's code in the outer loop
• if yield() is called, it calls extra doArduinoInternals()
• if delay is called, they might loop doArduinoInternals() until required time has passed, and then come back to user's code

So, that would not be like "real" task switching, but as an abstract concept it would be.

The ESPAsyncTCP, which is used fro ESPAsyncWebServer, uses the lwip callbacks so they are called directly from the "system" task.

So, from what I imagine, the problem is that the outer loop called my loop(), it finished or I called delay/yield, so doArduinoInternals() is called; from iside of arduino_internals() my web controller is called => so we are deep into shit and cannot go back to "outer layer", which would be able to call my loop() again or other stuff in doArduinoInternals() – right? ;-)

// pseudocode of what i imagine is happening

user_main_loop():
  // our arduino user code
  // we might call yield() or delay()

user_web_controller(AsyncWebServerRequest request):
  // here we handle the request
  // we CANNOT call yield/delay because we actually might be INSIDE of yield/delay!

handle_esp_async_web_server():
  if we_have_http_request:
    user_web_controller(request)

arduino_internals():
   handle_wifi_etc()
   handle_esp_async_web_server()

yield():
  arduino_internals()

delay(time):
  while !has_time_passed(time):
    arduino_internals()

the_real_outer_main():
  while true:
    arduino_internals()
    user_main_loop()

the_real_outer_main()

EDIT: one more thing: Watchdog.feed() would be called from arduino_internals() here ofc ;)

[Pablo2048]

Your imagination is wrong IMHO - there is nothing like handle_esp_async_web_server - there are just callbacks from tcp_pcb (lwip) and timers - take a look inside ESPAsyncTCP. If you really need so deep knowledge just take a look into internal functions here https://github.com/esp8266/Arduino/blob/master/cores/esp8266/core_esp8266_main.cpp

[softhack007]

Most important is to understand the concept of "tasks" (aka thread aka process): https://www.freertos.org/taskandcr.html

Async Webserver seems to run in a separate task. As this task is not part of "arduino execution context", you cannot use yield(). The best solution might be to use vTaskDelayUntil() so that other tasks can still execute while async Webserver waits for 20 seconds - see example provided here https://www.freertos.org/vtaskdelayuntil.html .

[Pablo2048]

We were talking about ESP8266, so no FreeRTOS here...

[gibo77]

I think the AsyncWebserver already used the interrupt. Only one interrupt can be used in ESP. You need to get out of AsyncWebserver before using delay() since it uses an interrupt hardware. 

Sent from Yahoo Mail for iPhone

On Sunday, September 25, 2022, 10:23 AM, Pablo2048 @.***> wrote:

We were talking about ESP8266, so no FreeRTOS here...

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.***>

[stale[bot]]

[STALE_SET] This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[MacDada]

bump

i believe this should be better explained in the docs

[gkostov]

I understand the reasoning behind this limitation but still long running response handlers are something quite natural so there needs to be a way to get those to work. I tried to quickly get around this in a separate task like here (sure, creating and terminating a task on each call is far from great but it's perfect for the POC below)

void handler(AsyncWebServerRequest *request) {
    xTaskCreate(
        [](void *arg){
            AsyncWebServerRequest *request = (AsyncWebServerRequest *)arg;
            // something runs here that takes a while and calls delay() and yield() a number of times

            request->send(200, "text/plain", "some result for the client to read");
            vTaskDelete(NULL);
        },
        "handler",
        5000,
        request,
        1,
        NULL
    );
}

And this runs just fine until on some call the task needs to run for over 3 seconds and then the web server closes the connection and the client sees ERR_EMPTY_RESPONSE.

In relation to this problem the docs say

The server is smart enough to know when to close the connection and free resources

but since obviously it can't know everything, I wonder if there is a way to tell it to wait a bit more for me to prepare the response before terminating the connection? I tried with request->beginResponse( and request-> beginResponseStream( in hope that this may give it a hint that I'm on it and just need a little more time, but that didn't do it either. Does anyone know where in the code is the "smart" logic that the server uses to close the connection?

---

Edit:

After more digging I came to this

void handler(AsyncWebServerRequest *request) {
    xTaskCreate(
        [](void *arg){
            AsyncWebServerRequest *request = (AsyncWebServerRequest *)arg;
            request->client()->setRxTimeout(50 /*a number of seconds that you expect to be enough for the job to get done*/);
            // something runs here that takes a while and calls delay() and yield() a number of times

            request->send(200, "text/plain", "some result for the client to read");
            vTaskDelete(NULL);
        },
        "handler",
        5000,
        request,
        1,
        NULL
    );
}

All good now, I have what I need - a long-running response handler that can take its time to do its job before responding :)

[MacDada]

@gkostov I do a workaround (kinda asynchronically):

1. web controller only sets a (global) var for something "to be done" and returns an immediate response
2. the global loop "does" it
3. another web request can check if it is already done

[stale[bot]]

[STALE_CLR] This issue has been removed from the stale queue. Please ensure activity to keep it openin the future.