Slow write speed to local influxdb

[tomalex-v]

I'm trying to capture vibration data from an MPU9250 sensor connected to an ESP32. The system samples vibration data (ax, ay, az) at 4 kHz and pushes it to a local InfluxDB (OSS v2) instance on my LAN for spectral analysis. The code I have used is based on the example SecureBatchWrite but I am not using secure connection for the time being. I am able to get the vibration data properly with correct sampling rate from the IMU. I am also capturing the timestamp as a field using micros(). I have tried keeping MAX_BATCH_SIZE to 4000 and WRITE_BUFFER_SIZE to 8000. But in that case the ESP32 is getting restarted (probably due to RAM getting exhausted). So, I changed the MAX_BATCH_SIZE to a low value (ex:10) while keeping the same WRITE_BUFFER_SIZE. I am using client.writePoint() within the loop which also collects data from the IMU. This is making the loop run slow and hence I am not able to capture the vibration data with 4kHz sampling rate.

I have tried to cache the vibration data to EEPROM to maintain 4kHz sampling rate and push the data from EEPROM to influxdb using batchwrite. Since the data push rate to influxdb is very low I am not able to use the method.

Is there any way to make the push rate to influxdb faster? Or is there a better way to do it?

[vlastahajek]

@tomalex-v, I'm not sure if ESPxxxx devices are capable of sending string data over TCP/IP so fast.

To speed up very frequently writing you definitively need to turn on connection reuse:

client.setHTTPOptions(HTTPOptions().connectionReuse(true));

Data are stored and sent as strings. 4000 points are usually too much to keep in RAM. Of course, it depends on the data (field names, tag names and tag values). My older test has shown it was able to hold 2048 lines of length 71chars. But it depends on the ESP32 Arduino Core version, cause, at that time, each new version was more memory-hungry.

You also don't need to keep writeBuffer too big, 2-3x batch size is ok. If you don't need retrying (resending data on write failure), set retry-interval to zero and write buffer size the same as batch size.

client.setWriteOptions(WriteOptions().batchSize(1000).bufferSize(1000).retryInterval(0));

You can also save RAM for more points if you use stream-writing. But it is a bit slower. You need to try it:

client.setStreamWrite(true);

In the case the above advices will not help you, I would recommend trying changing the data pipeline. You can use binary messages and send them using MQTT to a server, where a collector (Telegraf) can read them and push them to InfluxDB.

[tomalex-v]

@vlastahajek Thank you for your response. I will try out the options you have recommended and update.

[tomalex-v]

@vlastahajek The options you mentioned helped a bit in reducing the time. But it is still not sufficient to capture the data in real-time. I think this is mostly attributed to my network speed. Hence, I am closing the ticket as solved.