beginChunkedResponse stops asking for rest of file randomly

[davepl]

I'm using chunked responses to fulfill some larger files. Rarely, it works, but usually it bails out at some random point. There's no indication of why, and in Chrome it just says "stalled". It's like I'm returning data to the handler that's not making it to the client.

Would appreciate any suggestions on what to look at!

` // The default method of fulfilling a file doesn't work on large files because it tries to hold // the entire thing in RAM, and it chokes. So, for files that are too large to serve from RAM, // I use this function. It registers a file-specific handler and then does chunk based IO.

void ServeLargeStaticFile(const char pszName[], const char pszType[])
{
   _server.on(pszName, HTTP_GET, [pszName, pszType](AsyncWebServerRequest *request)
    {
        Serial.printf("GET for: %s\n", pszName);
        File SPIFFSfile = SPIFFS.open(pszName, FILE_READ);
        if (SPIFFSfile)
        {
            Serial.printf("[HTTP]\tOpening [%d]\r\n", SPIFFSfile);
        } 
        else 
        {
            Serial.printf("[HTTP]\tSPIFFS File DOESN'T exists [%d] <<< ERROR !!!\r\n", SPIFFSfile);
        }
        AsyncWebServerResponse *response = 
            request->beginChunkedResponse(pszType, [SPIFFSfile](uint8_t *buffer, size_t maxLen, size_t index) -> size_t 
            {
                auto localHandle = SPIFFSfile;
                Serial.printf("[HTTP]  [%6d]    INDEX [%6d]    BUFFER_MAX_LENGTH [%6d]\r\n", index, localHandle.size(), maxLen);
                size_t len = localHandle.read(buffer, maxLen);
                Serial.printf(">> Succcessful read of %d\n", len);
                if (len == 0)
                {
                    Serial.printf("Closing [%d]\n", SPIFFSfile);
                    localHandle.close();
                }
                return len;
            }
        );
        request->send(response);
    });
}    `

Example log:

GET for: /static/js/main.js [HTTP] Opening [1073715396] [HTTP] [ 0] INDEX [157916] BUFFER_MAX_LENGTH [ 5618]

Succcessful read of 5618 [HTTP] [ 5618] INDEX [157916] BUFFER_MAX_LENGTH [ 5736] Succcessful read of 5736 [HTTP] [ 11354] INDEX [157916] BUFFER_MAX_LENGTH [ 5736] Succcessful read of 5736 [HTTP] [ 17090] INDEX [157916] BUFFER_MAX_LENGTH [ 2864] Succcessful read of 2864 [HTTP] [ 19954] INDEX [157916] BUFFER_MAX_LENGTH [ 2864] Succcessful read of 2864 [HTTP] [ 22818] INDEX [157916] BUFFER_MAX_LENGTH [ 2864] Succcessful read of 2864 [HTTP] [ 25682] INDEX [157916] BUFFER_MAX_LENGTH [ 5736] Succcessful read of 5736 [HTTP] [ 31418] INDEX [157916] BUFFER_MAX_LENGTH [ 5736] Succcessful read of 5736 [HTTP] [ 37154] INDEX [157916] BUFFER_MAX_LENGTH [ 2864] Succcessful read of 2864 [HTTP] [ 40018] INDEX [157916] BUFFER_MAX_LENGTH [ 1428] Succcessful read of 1428 [Radio silence forever more...]

[zekageri]

Try it with LittleFs instead of SPIFFS

[jordanfuerst]

Are you sure the esp32 isn't crashing during the response and rebooting? I have an issue with the beginChunkedResponse function, but it completely crashes the board, which causes it to not ask for more data...

[jordanfuerst]

Elsewhere, this is the ram struct that contains my data:

struct 
{
  #define histBuffSize (1024*16)
  #define histBufLen (histBuffSize-1)

  int tempTail=0;                       // temp tail used to send data back to ajax
  int tempCount=0;                      // temp count to send data back to ajax
  struct
  {
    int16_t buffer[histBuffSize];         // our circular buffer.. storing floats by dividing by 100 for only 2 decimals
    int tail=0;                           // location of tail
    int count=0;                          // how many readings we have
  }insideTemp,outsideTemp;
}fanHist;

This is the chunked response, I have managed to cut down the crashes by shrinking the buffer by half and subtracting 16 bytes due to the way I pack the buffer... Crashes whole board randomly. Presents by not coming back and asking for more data...

// Serve up our csv file that the graph page asks for
  server.on("/data", HTTP_GET, [](AsyncWebServerRequest *request){
    AsyncWebServerResponse *response = request->beginChunkedResponse("text/plain", [](uint8_t *buffer, size_t maxLen, size_t index)-> size_t
    {
      maxLen = maxLen >> 1;   // some sort of bug with actual size available... shrinki it and doesnt crash as often
      maxLen-=16;             // we have padding in here since we check at the end of ever loop and not during

      if(index==0)  // This is our first time in this code.. ASSUME we have enough space for this whole message: Have never seen maxLen too small.
      {
        len = sprintf((char*)buffer,"Time,Inside,Outside\n%d\n",fanHist.lastEpoch);
        fanHist.tempTail=fanHist.insideTemp.tail;       // Store our tail to use every time we come back into this code in the future
        fanHist.tempCount=fanHist.insideTemp.count;     // store our data count to meter how much we send
      }
      while((len < (maxLen))  && fanHist.tempCount>0)   // while there is room in the buffer and we have not sent it all
      {
        len+=sprintf((char*)buffer+len,"%d,%d,",fanHist.insideTemp.buffer[fanHist.tempTail],fanHist.outsideTemp.buffer[fanHist.tempTail]);
        ++fanHist.tempTail &= histBufLen;
        --fanHist.tempCount;
      }
      return len;
    });
    request->send(response);
  });

[stale[bot]]

[STALE_SET] This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[av4625]

I am seeing similar to the OP with almost identical code. I am using SD instead of SPIFFS. I noticed that every time the callback that is passed to beginChunkedResponse is called the maxLen var always gets smaller by roughly the size of the previous chunk. Then when maxLen gets to or near zero the ESP32 just "freezes" and nothing happens. Almost seems like it isn't sending out the chunked data and it is saving the chunks behind the scene which defeats the purpose. I am able to send smaller files using the same code and I can see the callback is called multiple times.

[stale[bot]]

[STALE_CLR] This issue has been removed from the stale queue. Please ensure activity to keep it openin the future.

[stale[bot]]

[STALE_SET] This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[zyeborm]

Are either of you perchance using I2C in your projects? I have noticed some very weird behaviour as soon as I tell the wire library to connect to an RTC. transfer speed goes down by half and occasionally stalls. I'm sending files from SD card. If I never start I2C it's fine. If I start, then stop I2C it's toast.

[stale[bot]]

[STALE_CLR] This issue has been removed from the stale queue. Please ensure activity to keep it openin the future.

[av4625]

Are either of you perchance using I2C in your projects? I have noticed some very weird behaviour as soon as I tell the wire library to connect to an RTC. transfer speed goes down by half and occasionally stalls. I'm sending files from SD card. If I never start I2C it's fine. If I start, then stop I2C it's toast.

I’m not using any I2C sensors or anything like that. I am using a sensor over serial (gps), a six digit seven segment display, a neopixel with FastLED and a simple push button. I am also using the SD library for reading and writing to a SD card. Not sure what protocols it uses under the hood but it is connected using five or six wires.

Since this I have made the ability to run my project on either a ESP32 or embedded linux and use an OrangePi now as it is much more stable. I use boost asio/beast for the async server on linux.