Buffer on SPFFIS or SD Card

[gomez100]

Hi Tobias, it should be good having the option to store buffer in SPFFIS or SD Card for data sampled each minute or more, so it's possible to put ESP in deepsleep without losing buffered data. Regards, Andrea

[tobiasschuerg]

Hi @gomez100 ,

this is a good idea - however, I' also not sure if this should be in the scope of this library. Maybe as an extension?

With what kind of data are you dealing with? Most (arduino) applications I've seen so far which use this library "just" collect sensor data or simple metrics metrics for whose it is not critical if there are some data points missing. Many of them even don't have any storage.

Therefore I would suggest to either save the points in the buffer manually if writing fails and you want the device to go to deep sleep or not to use buffering of this library but to implement you own buffering strategy.

Still, feel free to create a PR. :-)

[gomez100]

Hi Tobias, I'm dealing with an esp32 with sim800 gprs modem with solar charge and deep sleep to save more energy as possibile. The idea is to wake the esp every one or five minute, store the data, then each hour ( or on event ) send a data batch by GPRS connection. GPRS is very energy angry and each connection take around 20 sec ( network registration, modem power on etc. ). As solution, I was thinking to buffer manually the string returned from your ".toLineProtocol()" function, save in a file somewhere, and when necessary iterate the string saved in the file as send them as raw inline protocol. But I'm unable to send raw data with your library to Influx. Is it possibile? About the PR....I'd like to...but I'm a really bad coder, my code are just ( sometimes ) working patches! :-D

[vlastahajek]

@gomez100, use InfluxDBClient::writeRecord() to write line protocol directly. Be sure to sync time and use Point::setTime to set correct time to your saved data.

However, line protocol is not good for saving space. I would suggest storing your data in a raw form (a binary) with timestamp, if possible, a then create data Point at the time of writing.

When using SPFFIS, consider the fact, that this is flash memory with limited number of writes.

[gomez100]

@vlastahajek I totally agree with you about the SPFFIS, and SD card seems to be the best solution even when much power consumption ( for writes cycle and available space). Your suggestion is:

• create point
• add time
• store inlineprotocol in a file ( SD o SPFFIS )
• use writeRecord as client.writeRecord("inlineprotocol read from file") Thanks

[marmat123]

After a long search and the same idea, I ended up here.

The inline protocol can also be created using a simple string addition. The generated file can be read out here without any problems:

SPIFFS.begin();
File f;
f = SPIFFS.open(POINTS_FILE, "r");

while (f.available()) {
    char buff[256] = { 0 };
    f.readBytes(buff, 256);
    Serial.print(&buff[0]);
    client.writeRecord(&buff[0]);
}

Only the client.writeRecord-command doesn't work because it doesn't pass the file as a stream - I get error messages, each with 256-byte pieces - and it's slow because the function used in the ESP8266http client passes the whole file byte by byte (searching for "\r")².

influxDB supports sending files³:

curl -i -XPOST 'http://localhost:8086/write?db=mydb' --data-binary @cpu_data.txt`

and it worked here and --data-binary indicates that these are not line by line excerpts from text files.

² ESP8266HTTPClient takes very long time to send 1MB file(SPIFFS) ³ Write several points to the database mydb from the file

I would also be very grateful for ideas. In the moment, I'm refuelling myself through the ESP8266httpClient.cpp, hoping to solve the whole thing without the Influx library - since I don't need a general solution, but only a specialized one.

[szerwi]

@marmat123 have you worked out the bast way to save and then send points from SPIFFS?

[JKoss2]

I'm gonna say +1 for this feature to be able to choose where the program stores the buffer, if that's possible. In my case I am making a portable setup which will remain at home a lot of the time and it will be logging data from various water sensors (flow, psi, TC, etc). However, when I leave home with the setup I want the ESP to continue saving the data points to the SD card, then once I get back home, have it automatically continue uploading the data from the SD card to Influx. I'm relatively new to programming ESP's/Arduino's so making custom code that hasn't been done before is not easy for me, although it seems like I will have to try in this case because this feature request is nearing 2 years in age.