search

espressif / arduino-esp32

Arduino core for the ESP32
GNU Lesser General Public License v2.1
13.33k stars 7.36k forks source link

SD Card reading --> ESP32 freezes (connection to server/WLAN gets lost) #6846

Closed basementmedia2 closed 2 years ago

basementmedia2 commented 2 years ago

Board

ESP32 (Wemos D1 mini)

Device Description

Custom ESP32, got no photo here sorry (am in the office right now)

Hardware Configuration

SD-Card Shield connected

Version

v2.0.3

IDE Name

Arduino IDE

Operating System

Linux Manjaro

Flash frequency

40

PSRAM enabled

no

Upload speed

115200

Description

I have a sketch which works as follows:

On client-side (webserver) the user selects a date which is then sent (per Ajax) to the ESP. if the date is e.g. "2022-06-08" a file named "2022-06-08.bin" is opened for reading and then read line by line.

With esp core 1.0.6 it works perfectly. Same code with esp core 2.0.2 was very slow (cause there was still a bug i think) Now with esp core 2.0.3 it is really fast again, but after a few date selects (means a few SD-card-Read-cycles) the esp totally freezes (webserver is not available any more).

Sketch

// Just Parts of the Sketch
// I cannot publish complete Sketch here because of copyrights (customer project)

struct data {
  uint16_t value1;
  uint16_t value2;
  uint16_t value3;
  // 10 more values, just an example 
};

void tagesauswertung (char* datum_select, int startpos, int endpos, int tag, int monat, int jahr, int rand_key_sent) {
  struct data Messung;
  DynamicJsonDocument auswertung_arr(4800);

  sprintf(filename_auswertung, "/e/%s.bin", datum_select);
  auswertungenFile = SD.open(filename_auswertung, FILE_READ);
  if (auswertungenFile) {

    int i = 0;
    int alive = 0;
    while (auswertungenFile.available()) {

      auswertungenFile.read((uint8_t *)&Messung, sizeof(Messung));

      if (i >= startpos && i <= endpos) {
        // As the file is very big, the JSON-Array would get to o big
        // So the server sends "startpos" and "endpos" and thus the content is transferred partly to the client
        alive = 1;
        auswertung_arr[i - startpos]["val1"] = Messung.value1;
        auswertung_arr[i - startpos]["val2"] = Messung.value2;
        auswertung_arr[i - startpos]["val3"] = Messung.value3;
        // ten more values, just an example
      }
      i++;
    }

    if (alive == 1) {
      // Still Data left --> Send it to the client
      String jsonString;
      serializeJson(auswertung_arr, jsonString);
      Serial.println(jsonString);
      server.send(200, "application/json", jsonString);
    } else {
        server.send(200, "text/plain", "Done");
    }

  } else {
    server.send(200, "text/plain", "Fehler");
  } 
}

Debug Message

ESP Freezes (Webserver is not available anymore)

Other Steps to Reproduce

No response

I have checked existing issues, online documentation and the Troubleshooting Guide

VojtechBartoska commented 2 years ago

Can you please take a look @P-R-O-C-H-Y?

basementmedia2 commented 2 years ago

Hi @VojtechBartoska

what do you mean? Does @P-R-O-C-H-Y have a solution for this problem? Or should i post my issue in his repo?

Best wishes Daniel

P-R-O-C-H-Y commented 2 years ago

Hi @basementmedia2

I take care of issues about SD lib :) Thats why @VojtechBartoska assigned me and pinged me in comment. I will take a look on you issue soon, no need to repost issue anywhere ;)

P-R-O-C-H-Y commented 2 years ago

Hi @basementmedia2, Can you try it with latest 2.0.4 release? Thanks

basementmedia2 commented 2 years ago

Hi @P-R-O-C-H-Y ,

sure, I will check it these days and come back with my experience. Thank you for your help!

Best wishes Daniel

basementmedia2 commented 2 years ago

Hi @P-R-O-C-H-Y ,

unfortunately updating to 2.0.4 does not fix the problem.

I have created a test sketch, which I think you can use to reproduce the problem yourself.


#include <WiFi.h>
#include <WiFiAP.h>
#include <HTTPClient.h>
#include <WebServer.h>
WebServer server(80);
#include <WiFiClient.h>
#include <WiFiClientSecure.h>

#include <FS.h>
#include <SD.h>
#include <ArduinoJson.h>

char firmware_update_url[150]="some url to json file";
char IPlocal[20];

unsigned long connTimer=0;
unsigned long dauer = 10000;
const char ssid[14] = "MySSID";
const char password[21] = "MyPW";
const char apssid[10] = "MyESP32";
const char appassword[10] = "123456789";

unsigned long previousMillis_reconn = 0;
unsigned long interval_reconn = 30000;

unsigned long previousMillis_fwc = 0;
unsigned long interval_fwc = 20000;

unsigned long previousMillis_sd_loop = 0;
unsigned long interval_sd_loop = 20000;

int counter=0;

static const char indexPage[] PROGMEM = R"=====(
<!doctype html>
<html lang="de">
<body>
Hello World
</body>
</html>
)=====";

struct data {
    uint16_t tag;
    uint16_t stunde;
    float liter;
};

char filename_auswertung[35];
char* datum;
File auswertungenFile;

void display_root() {
    const char * httpType PROGMEM = "text/html";
    server.send_P(200, httpType, indexPage);
}

void setup() {
    delay(2000);
    Serial.begin(115200);
    if(!SD.begin()){
        Serial.println("Card Mount Failed");
    }

    xTaskCreatePinnedToCore(
        keepWiFiAlive,
        "Keep Wifi Alive",
        5000,
        NULL,
        2,
        NULL,
        CONFIG_ARDUINO_RUNNING_CORE
    );

    WiFi.mode(WIFI_AP_STA);
    WiFi.begin(ssid, password);

    connTimer = millis();

    while (WiFi.status() != WL_CONNECTED) {
        Serial.print(".");
        delay(500);
    }
    if (WiFi.status() == WL_CONNECTED) {
        Serial.println("We are connected...");
        strcpy(IPlocal, WiFi.localIP().toString().c_str());
    } else {
        Serial.println("We are not connected");
    }

    WiFi.softAP(apssid, appassword);

    server.on("/", display_root);
    server.on("/auswertung", auswertung);

    server.begin();
}

void loop() {
    server.handleClient();
    unsigned long currentMillis_reconn = millis();
    if ((WiFi.status() != WL_CONNECTED) && (currentMillis_reconn - previousMillis_reconn >=interval_reconn)) {
        Serial.println("Reconnecting to WiFi...");
        WiFi.disconnect();
        WiFi.begin(ssid, password);
        connTimer = millis();
        while (WiFi.status() != WL_CONNECTED && (millis() - connTimer < dauer)) {
            Serial.print(".");
            delay(500);
        }
        previousMillis_reconn = currentMillis_reconn;
    } else {
        if (millis() - previousMillis_fwc > interval_fwc) {
            Serial.println("Firmware Check!");
            check_firmware_version();
            previousMillis_fwc = millis();
        }
    }

    if (millis() - previousMillis_sd_loop > interval_sd_loop) {
        if(SD.begin()){
            auswertung();
            previousMillis_sd_loop = millis();
        }
    }
}

void keepWiFiAlive(void * parameters) {
    for (;;) {
        if (WiFi.status() == WL_CONNECTED) {
            Serial.println("We are connected");
            vTaskDelay(10000 / portTICK_PERIOD_MS);
            check_firmware_version();
            continue;
        }
        WiFi.disconnect();
        WiFi.begin(ssid, password);
        unsigned long startAttemptTime=millis();
        while (WiFi.status() != WL_CONNECTED && millis() - startAttemptTime < 5000) {

        }
        if (WiFi.status() != WL_CONNECTED) {
            Serial.println("Not connected! ");
            continue;
        } else {
            // Again connected
            continue;
        }
    }
}

void auswertung() {
    Serial.println(WiFi.localIP());
    Serial.print("Starting SD-Reading ->");
    datum="2022-05-26";
    for (int i = 0; i < 10; i = i + 1) {
        tagesauswertung(datum, i*16, (i*16)+16, 26, 5, 2022);
    }
    datum="2022-05-27";
    for (int i = 0; i < 10; i = i + 1) {
        tagesauswertung(datum, i*16, (i*16)+16, 27, 5, 2022);
    }
    counter++;
    Serial.println(counter);
}

void tagesauswertung (char* datum_select, int startpos, int endpos, int tag, int monat, int jahr) {
    struct data Messung;
    DynamicJsonDocument auswertung_arr(4800);
    sprintf(filename_auswertung, "/e/%s.bin", datum_select);
    auswertungenFile = SD.open(filename_auswertung, FILE_READ);
    if (auswertungenFile) {
        Serial.print("Opening file ");
        Serial.println(filename_auswertung);
        int i = 0;
        int alive = 0;
        while (auswertungenFile.available()) {
            auswertungenFile.read((uint8_t *)&Messung, sizeof(Messung));
            if (i >= startpos && i <= endpos) {
                alive = 1;
                auswertung_arr[i - startpos]["t"] = Messung.tag;
                auswertung_arr[i - startpos]["s"] = Messung.stunde;
                auswertung_arr[i - startpos]["l"] = Messung.liter;
            }
            i++;
        }

        if (alive == 1) {
            String jsonString;
            serializeJson(auswertung_arr, jsonString);
            Serial.println(jsonString);
            server.send(200, "application/json", jsonString);
        } else {
            // We are finished with reading
            server.send(200, "text/plain", "Fertig");
        }
    } else {
        Serial.println("File could not be opened");
    }
}

void check_firmware_version() {
    if (WiFi.status() == WL_CONNECTED) {
        WiFiClientSecure *client = new WiFiClientSecure;
        if(client) {
            client->setInsecure(); {
                HTTPClient https;
                https.useHTTP10(true);
                if (https.begin(*client, firmware_update_url)) { //Abfrage-URL
                    int httpsCode = https.GET(); //Antwort des Servers abrufen
                    if (httpsCode == 200) {
                        Serial.println("Firmware-File found!");
                    }
                    https.end();
                }
            }
            delete client;
        }
    } else {
        Serial.println("Not connected --> No FW-Check!");
    }
}

Here are the bin-files, used in the function "auswertung"

bin-files.zip

The JSON-File you must pölace somewhere on your server (but i don't know if this part is really relevant).

If you run the sketch with 1.0.6, it runs endlessly correctly through - the function "day evaluation" is started again and again. With 2.0.3 and with 2.0.4 after a few runs (sometimes more, sometimes less) the loop-task is shot down. Then only the separate task "keepWiFiAlive" is running.

Could it be that the heap memory is running full and therefore at some point the void loop can no longer be executed?

I also noticed that any execution of SD.begin() when no SD card is inserted causes the heap to grow until then eventually the memory overflows. No idea if this is intentional or if it's a bug.

You can test this with the following sketch and NOT putting an SD-Card in the SD-Card-Reader:

#include <SD.h> // SD-Kartenleser

unsigned long millisMerker= 0;
unsigned long dauerBisRepeat = 20000;

int sd_card_status =0;

void setup() {
  Serial.begin(115200);
  if (!SD.begin(5)) {
    sd_card_status=0;
  } else {
    sd_card_status=1;
  }
}

void loop() {
  if (!SD.begin(5)) {
    sd_card_status=0;
  } else {
    sd_card_status=1;
  }
  if (millis() - millisMerker > dauerBisRepeat) {
    uint32_t heapSize=ESP.getHeapSize(); //total heap size
    uint32_t freeHeap=ESP.getFreeHeap(); //available heap
    uint32_t minFreeHeap=ESP.getMinFreeHeap(); //lowest level of free heap since boot
    uint32_t maxAllocHeap=ESP.getMaxAllocHeap(); //largest block of heap that can be allocated at once
    Serial.println(" -> Total: " + (String)heapSize + ", Frei: " + (String)freeHeap + " Min: "+ (String)minFreeHeap + ", Max: " + (String)maxAllocHeap) + " SD-Card:" + sd_card_status;
    millisMerker = millis(); 
  }
}

Would be great to get further support.

Best wishes Daniel

P-R-O-C-H-Y commented 2 years ago

Hi @basementmedia2 Thanks for really nice report 👍

Gonna take a look on the issue, and for sure the heap cannot grow. There has to be some bug. I will inform you about the progress :)

P-R-O-C-H-Y commented 2 years ago

About the SD card causing heap to grow. I run your 2nd sketch to test that (thank you for preparing that), but for me there is no leaks or growing. Can you print your output?

 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
 -> Total: 362908, Frei: 337328 Min: 304832, Max: 118772 SD-Card:0
basementmedia2 commented 2 years ago

Hi,

ah sorry: As i cannot use the other Sketch (the first one i posted) with 2.0.4 i still use 1.0.6. Sot the problem with the heap size i tested under 1.0.6. I've been switching back and forth between 1.0.6 and 2.0.3 / 2.0.4 so many times the last few days that I apparently thought the heap problem was with 2.0.4. Sorry for that

Here is the output with 1.0.6:

-> Total: 401380, Frei: 374484 Min: 341524, Max: 126656
 -> Total: 400836, Frei: 373804 Min: 340844, Max: 126656
 -> Total: 400292, Frei: 373124 Min: 340164, Max: 126656
 -> Total: 399748, Frei: 372444 Min: 339484, Max: 126656
 -> Total: 399204, Frei: 371764 Min: 338804, Max: 126656
 -> Total: 398660, Frei: 371084 Min: 338124, Max: 126656
 -> Total: 398116, Frei: 370404 Min: 337444, Max: 126656
 -> Total: 397568, Frei: 369716 Min: 336768, Max: 126656
 -> Total: 397024, Frei: 369036 Min: 336088, Max: 126656
 -> Total: 396480, Frei: 368356 Min: 335408, Max: 126656
 -> Total: 395936, Frei: 367676 Min: 334728, Max: 126656
 -> Total: 395392, Frei: 366996 Min: 334048, Max: 126656
 -> Total: 394848, Frei: 366316 Min: 333368, Max: 126656
 -> Total: 394304, Frei: 365636 Min: 332688, Max: 126656
 -> Total: 393760, Frei: 364956 Min: 332008, Max: 126656
 -> Total: 393216, Frei: 364276 Min: 331328, Max: 126656
 -> Total: 392672, Frei: 363596 Min: 330648, Max: 126656
 -> Total: 392128, Frei: 362916 Min: 329968, Max: 126656
 -> Total: 391584, Frei: 362236 Min: 329288, Max: 126656
 -> Total: 391040, Frei: 361556 Min: 328608, Max: 126656
 -> Total: 390496, Frei: 360876 Min: 327928, Max: 126656
 -> Total: 389952, Frei: 360196 Min: 327248, Max: 126656
 -> Total: 389408, Frei: 359516 Min: 326568, Max: 126656
 -> Total: 388864, Frei: 358836 Min: 325888, Max: 126656
 -> Total: 388320, Frei: 358156 Min: 325208, Max: 126656
 -> Total: 387776, Frei: 357476 Min: 324528, Max: 126656
 -> Total: 387232, Frei: 356796 Min: 323848, Max: 126656
 -> Total: 386688, Frei: 356116 Min: 323168, Max: 126656
 -> Total: 386144, Frei: 355436 Min: 322488, Max: 126656
 -> Total: 385600, Frei: 354756 Min: 321808, Max: 126656
 -> Total: 385056, Frei: 354076 Min: 321128, Max: 126656
 -> Total: 384512, Frei: 353396 Min: 320448, Max: 126656
 -> Total: 383968, Frei: 352716 Min: 319780, Max: 126656
 -> Total: 383432, Frei: 352044 Min: 319096, Max: 126656
 -> Total: 382888, Frei: 351364 Min: 318416, Max: 126656
 -> Total: 382344, Frei: 350684 Min: 317736, Max: 126656
 -> Total: 381800, Frei: 350004 Min: 317056, Max: 126656
 -> Total: 381256, Frei: 349324 Min: 316376, Max: 126656
 -> Total: 380712, Frei: 348644 Min: 315696, Max: 126656
 -> Total: 380168, Frei: 347964 Min: 315016, Max: 126656
 -> Total: 379624, Frei: 347284 Min: 314336, Max: 126656
 -> Total: 379088, Frei: 346612 Min: 314084, Max: 126656
 -> Total: 378552, Frei: 345940 Min: 314084, Max: 126656
 -> Total: 378008, Frei: 345260 Min: 314084, Max: 126656
 -> Total: 377464, Frei: 344580 Min: 314084, Max: 126656
 -> Total: 376920, Frei: 343900 Min: 314084, Max: 126656
 -> Total: 376376, Frei: 343220 Min: 314084, Max: 126656
 -> Total: 375832, Frei: 342540 Min: 314084, Max: 126656
 -> Total: 375288, Frei: 341860 Min: 314084, Max: 126656
 -> Total: 374744, Frei: 341180 Min: 313712, Max: 126656
 -> Total: 374200, Frei: 340500 Min: 313044, Max: 126116

Running for 20 Minutes now. But as this is fixed with version 2.0.4 (maybe also with 2.0.0) we can forget that...but it would be great if we get the first posted sketch running as if not i have still to use 1.0.6 and must live with this but ;-(

P-R-O-C-H-Y commented 2 years ago

Hi @basementmedia2,

Two quick things I found in your sketch.

1) in loop() - why are you calling SD.begin() each loop when it should be ruined one time from setup. You can use SD.exists("/") function for checking if SD card is inserted and started or if available use SD detect pin. Why to reinitialise SD card each time if there is no need to.

2) in tagesauswertung() function you never close the file. Do auswertungenFile.close() after end of read.

I was not able to get more then 2-5 reads. With these changes I am currently on 25th read and still running.

basementmedia2 commented 2 years ago

Hi @P-R-O-C-H-Y ,

You're right on both counts:

Thank you again for your great help Daniel

P-R-O-C-H-Y commented 2 years ago

EDIT: seems I got stuck on 37th read.

P-R-O-C-H-Y commented 2 years ago

The SD.exists() function works now well I think from 2.0.3 version :)

Can you print the heap from your sketch using 2.0.4 after each file open please? To be sure its not the issue.

P-R-O-C-H-Y commented 2 years ago

When it gets stuck you end up here in the while loop:

void tagesauswertung (char* datum_select, int startpos, int endpos, int tag, int monat, int jahr) {
    struct data Messung;
    DynamicJsonDocument auswertung_arr(4800);
    sprintf(filename_auswertung, "/%s.bin", datum_select);
    auswertungenFile = SD.open(filename_auswertung, FILE_READ);
    if (auswertungenFile) {
        Serial.print("Opening file ");
        Serial.println(filename_auswertung);
        int i = 0;
        int alive = 0;

        // ------------
        while (auswertungenFile.available()) {
            auswertungenFile.read((uint8_t *)&Messung, sizeof(Messung));
            if (i >= startpos && i <= endpos) {
                alive = 1;
                auswertung_arr[i - startpos]["t"] = Messung.tag;
                auswertung_arr[i - startpos]["s"] = Messung.stunde;
                auswertung_arr[i - startpos]["l"] = Messung.liter;
            }
            Serial.println(".");
            i++;
        }
        // -------------

        //auswertungenFile.close();

        if (alive == 1) {
            String jsonString;
            serializeJson(auswertung_arr, jsonString);
            //Serial.println(jsonString);
            server.send(200, "application/json", jsonString);
        } else {
            // We are finished with reading
            server.send(200, "text/plain", "Fertig");
        }
    } else {
        Serial.println("File could not be opened");
    }
}

Will do more tests to find out why :)

P-R-O-C-H-Y commented 2 years ago

@basementmedia2 I have added Serial.println(i); to the while where it breaks.

        while (auswertungenFile.available()) {
            auswertungenFile.read((uint8_t *)&Messung, sizeof(Messung));
            if (i >= startpos && i <= endpos) {
                alive = 1;
                auswertung_arr[i - startpos]["t"] = Messung.tag;
                auswertung_arr[i - startpos]["s"] = Messung.stunde;
                auswertung_arr[i - startpos]["l"] = Messung.liter;
            }
            Serial.println(i);
            i++;
        }

Normal output:

Opening file /2022-05-26.bin
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83

When it breaks it continues to count forever.

basementmedia2 commented 2 years ago

Hi @P-R-O-C-H-Y ,

thank you so much for working on it. OK, i will print the heap using 2.0.4 after each file open.

I can't interpret your last post correctly (my English is also not the best ;-) I.e. but that we currently have no solution and you can understand my problem?

You can also test with the core 1.0.6, the sketch runs endlessly. I.e. you would have to check what has changed since 1.0.6 regarding the SD library, right? But that won't be easy, I don't know enough about it ...

P-R-O-C-H-Y commented 2 years ago

Hi @basementmedia2,

Yes I understand your problem and I know where to look at, where is the issue. But it's weird that in 1.0.6 it works. You might take a look on the While function I mentioned. The problem is for sure there.

Huge output log to show you the issue, I have added print at the start to show startpos and endpos you are passing to the function and print "messung" to the part where you actually needs the data and write them to the array : 

auswertung_arr[i - startpos]["t"] = Messung.tag;
auswertung_arr[i - startpos]["s"] = Messung.stunde;
auswertung_arr[i - startpos]["l"] = Messung.liter;

Why you always read whole file, and use just a part of the data? I think using seek you can always jump to the part of file where you need to read and change the while to something like read only the part from startpos to endpos.

Opening file : startpos = 0, endpos = 16/2022-05-27.bin
messung
0
messung
1
messung
2
messung
3
messung
4
messung
5
messung
6
messung
7
messung
8
messung
9
messung
10
messung
11
messung
12
messung
13
messung
14
messung
15
messung
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
Opening file : startpos = 16, endpos = 32/2022-05-27.bin
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
messung
16
messung
17
messung
18
messung
19
messung
20
messung
21
messung
22
messung
23
messung
24
messung
25
messung
26
messung
27
messung
28
messung
29
messung
30
messung
31
messung
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
Opening file : startpos = 32, endpos = 48/2022-05-27.bin
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
messung
32
messung
33
messung
34
messung
35
messung
36
messung
37
messung
38
messung
39
messung
40
messung
41
messung
42
messung
43
messung
44
messung
45
messung
46
messung
47
messung
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
Opening file : startpos = 48, endpos = 64/2022-05-27.bin
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
messung
48
messung
49
messung
50
messung
51
messung
52
messung
53
messung
54
messung
55
messung
56
messung
57
messung
58
messung
59
messung
60
messung
61
messung
62
messung
63
messung
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
Opening file : startpos = 64, endpos = 80/2022-05-27.bin
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
messung
64
messung
65
messung
66
messung
67
messung
68
messung
69
messung
70
messung
71
messung
72
messung
73
messung
74
messung
75
messung
76
messung
77
messung
78
messung
79
messung
80
81
82
83
84
85
86
87
Opening file : startpos = 80, endpos = 96/2022-05-27.bin
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
messung
80
messung
81
messung
82
messung
83
messung
84
messung
85
messung
86
messung
87
Opening file : startpos = 96, endpos = 112/2022-05-27.bin
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
Opening file : startpos = 112, endpos = 128/2022-05-27.bin
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
Opening file : startpos = 128, endpos = 144/2022-05-27.bin
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
[ 52526][D][HTTPClient.cpp:152] begin(): failed to parse protocol
[ 52527][V][ssl_client.cpp:324] stop_ssl_socket(): Cleaning SSL connection.
[ 52531][V][ssl_client.cpp:324] stop_ssl_socket(): Cleaning SSL connection.
We are connected
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
messung
128
messung
129
messung
130
messung
131
messung
132
messung
133
messung
134
messung
135
messung
136
messung
137
messung
138
messung
139
messung
140
messung
141
messung
142
messung
143
messung
144
145
146
147
148
149
150
// NOW it starts to count forever (stuck in loop)
P-R-O-C-H-Y commented 2 years ago

In the log above, you can see that you don't even get the data for position 88 - 128.

basementmedia2 commented 2 years ago

Hi,

for explanation: The file contains measurings of a water meter. For each day there is a separate file. Each file contains a few hundred measurings. At the frontend (HTML) i can select a day and then the measurings of this day should be displayed in a table. Therefore the file (which is on SD-card) is to be read out and then send to the webserver frontend. The measurings i send in JSON format. Because of the high number of measurings, JSON file would get to big to send all in one. So my idea was to read out the file partially. But i understand, that my current way is not really the best (also in performance i think).

Also i don't understand, why 88-128 is not read out??

Do you have a tip, how to do this better? Can you help me optimizing the posted code, would be really great help. I'm still learning every day and not an expert.

Thank you in advance again

P-R-O-C-H-Y commented 2 years ago

Hi, thanks for explanation.

I don't know much about JSON file handling so I cannot help on this side.

For the SD card. For sure close the file ofter finishing reading. I would change the function to read whole file but after you hit some point line now 16 "lines" of data, just send it. And continue reading. Don't split it by closing and opening file again.

After opening file, you can get size. From that you can read multiple bytes at once and just fill the array, that can improve the speed.

About these 88-128 data was not read I am not sure, but it shows that the code is not safe and some faults are happening.

I can try to change that a bit, but like I said, I am not familiar with JSON.

basementmedia2 commented 2 years ago

Hi,

the JSON part is no problem for me. But how to optimize the while loop so that not always the whole file is read althought i just need a part of it - that is more a problem ;-( If you have an idea how to optimize this would be great help.

You wrote

"I would change the function to read whole file but after you hit some point line now 16 "lines" of data, just send it."

Current procedure is:

  1. Webserver (=javascript ajax call) requests startpos 0, endpos 16
  2. tagesauswertung opens file, reads lines 0 to 15, sends JSON and closes file
  3. Webserver gets the JSON result and displays the values in a graphic
  4. Webserver requests startpos 16 and endpos 32
  5. tagesauswertung opens File reads lines 16 to 32 and sends JSON and closes file
  6. Webserver gets the JSON result and displays the values in a graphic
  7. and so on until we have reached endpos

and you mean something like this

  1. Webserver (=javascript ajax call) requests startpos 0, endpos 16
  2. tagesauswertung opens file
  3. tagesauswertung reads lines 0 to 15 and sends JSON BUT NOT CLOSES FILE YET
  4. Webserver gets the JSON result and displays the values in a graphic
  5. Webserver requests startpos 16 and endpos 32
  6. tagesauswertung reads lines 16 to 32 and sends JSON BUT NOT CLOSES FILE YET
  7. Webserver gets the JSON result and displays the values in a graphic
  8. as soon as endpos is reached tagesauswertung closes the file

But if you could demonstrate how to use seek (you mentioned) would be great - so that not always the whole file is read.

Best wishes Daniel

P-R-O-C-H-Y commented 2 years ago

Hi,

I have edited the code a bit, but now I see your comments and don't think it is what you need. I did not know, that server requests just part of the data. I thought that server server request while file, and you send it divided into smaller parts.

Take a look on my change, it never failed for me now. (I am currently on 170th loop) After opening file, I check the file size, divide it by number of bytes in your Message (2+2+4) and after that call read that exact times that are data for in the file. Meanwhile send the JSON when you already read 16 messages into array and send it, then clear it (don't know if needed, but did it that it should be the some now, like you have it before). This send JSON send is happening each 16 messages read and last JSON send as after all lines were read. Then just close the file and send the finish.

void auswertung() {
    Serial.println(WiFi.localIP());
    Serial.print("Starting SD-Reading ->");
    datum="2022-05-26";
    //for (int i = 0; i < 10; i = i + 1) {
        tagesauswertung(datum, 26, 5, 2022);
    //}
    datum="2022-05-27";
    //for (int i = 0; i < 10; i = i + 1) {
        tagesauswertung(datum,27, 5, 2022);
    //}
    counter++;
    Serial.println("");
    Serial.println(counter);
}

void tagesauswertung (char* datum_select, int tag, int monat, int jahr) {
    struct data Messung;
    DynamicJsonDocument auswertung_arr(4800);
    sprintf(filename_auswertung, "/%s.bin", datum_select);
    auswertungenFile = SD.open(filename_auswertung, FILE_READ);
    if (auswertungenFile) {
        uint32_t file_size = auswertungenFile.size();
        uint16_t num_of_reads = file_size / 8; // 8 = bytes per line
        Serial.printf("\n Opening file %s , size = %d , num_of_reads = %d",filename_auswertung,file_size,num_of_reads);
        Serial.println();
        int i = 0;

        while(i < num_of_reads)

        //for (int i = 0; i < num_of_reads; i++)
        {
            auswertungenFile.read((uint8_t *)&Messung, sizeof(Messung));

            Serial.println(i);
            //alive = 1;
            auswertung_arr[i]["t"] = Messung.tag;
            auswertung_arr[i]["s"] = Messung.stunde;
            auswertung_arr[i]["l"] = Messung.liter;

            if ((( i % 16 ) == 0 ) || ( i == ( num_of_reads - 1 )))
            { 
            String jsonString;
            serializeJson(auswertung_arr, jsonString);
            //Serial.println(jsonString);
            server.send(200, "application/json", jsonString);
            Serial.println("JSON send");

            //To make it the same as it was before, clearing the array after 16 lines of data
            auswertung_arr.clear();
            }

            i++;
        }
        auswertungenFile.close();

      //> not sure about this, if it is right, check it!
        // We are finished with reading
        server.send(200, "text/plain", "Fertig");
        Serial.println("finished");

    } 
    else {
        Serial.println("File could not be opened");
    }
}

I did not use seek in the code above, but its easy to use. You just need position where you want to move in the file. The call the seek function to the point. You can find a lot of examples on the internet :)

PS: There are few Serial prints left for better debugging :)

basementmedia2 commented 2 years ago

OK so THANK YOU VERY MUCH for so much help I will try to get the rest (e.g. seek) by myself. We will see if my brain gets a damage on it ;-)

Best wishes and nice weekend

P-R-O-C-H-Y commented 2 years ago

You are welcome :)

If you have any problem, you can write any comment directly to this issue. I will close it, but feel free to comment ;)

Thanks and nice weekend for you too.

lbernstone commented 2 years ago

And, just as background advertising, if you prefer to spend your brain damage learning a data managment tool instead of managing the bits, you can take a look at using https://github.com/lbernstone/rrdtool_ESP32 and just deliver the raw rrd files to the client for presentation.

basementmedia2 commented 2 years ago

Hi @P-R-O-C-H-Y ,

the corrected sketch no works with 2.0.4 as it should, but: As told my sketch works as follows:

Current procedure is:

  1. Webserver (=javascript ajax call) requests startpos 0, endpos 16
  2. tagesauswertung closes auswertungenFile (so that it closed for sure) and then opens file, reads lines 0 to 16, sends JSON
  3. Webserver gets the JSON result and displays the values in a graphic
  4. Webserver requests startpos 16 and endpos 32
  5. tagesauswertung opens File reads lines 16 to 32 and sends JSON
  6. Webserver gets the JSON result and displays the values in a graphic
  7. and so on until we have reached endpos

This works fine if WiFi-Connection is good. But with 2.0.4 if WiFi-connection is not so good it happens sometime, that some JSON-values seem not to "reach" the webserver which results in incomplete statistic (i show the values in a graphic). WIth 1.0.6 it works also perfect when WIFi-Connection is not so good. It then takes longer time but in the end ALL values reach the webserver.

So two questions left:

Best wishes Daniel

lbernstone commented 2 years ago

There is explicitly no warranty for the code in this repository. There is also no SLA on this forum, so any deadlines imposed, are yours to manage. If you are unable to make 2.0.4 work for you, then I would recommend you stick with 1.0.6. There will be no new features added to 1.0.6, but if you have a static product, you should choose to stick with what is working.

basementmedia2 commented 2 years ago

OK, But do you have an idea, why 1.0.6 has another behaviour in my case than 2.0.4. And do you think that i will anyway be kind of forced to update to 2.0.4 in the near future. Sorry for this maybe stupid question but i try to find out, if it is better to getting 2.0.4 working or go with 1.0.6.