mblackstock / node-red-contrib-influxdb

Node-RED nodes to save and query data from an influxdb time series database
Other
64 stars 46 forks source link

InfluxDB out Node RequestTimedOutError on InfluxDB 2 online #125

Closed sbieri1 closed 10 months ago

sbieri1 commented 2 years ago

Dear Authors

I am using a local dockered node-red instance (switzerland) aquiring data from an azure IoT-Hub and pushing data into an influxDB 2 online instance (deployed westeurope, ireland). I am using the influxdb out node (from node-red-contrib-influxdb, version 0.6.1) and writing every 0.5s. After some days of running perfectly fine, the whole node-red instance stops working. In the logs i get the following errors:

WARN: Write to InfluxDB failed (attempt: 1). S [RequestTimedOutError]: Request timed out
    at ClientRequest.<anonymous> (/data/node_modules/@influxdata/influxdb-client/dist/index.js:16:4142)
    at ClientRequest.emit (events.js:400:28)
    at TLSSocket.emitRequestTimeout (_http_client.js:790:9)
    at Object.onceWrapper (events.js:519:28)
    at TLSSocket.emit (events.js:412:35)
    at TLSSocket.Socket._onTimeout (net.js:495:8)
    at listOnTimeout (internal/timers.js:557:17)
    at processTimers (internal/timers.js:500:7)
20 Jul 07:31:25 - [error] [influxdb out:Influx CtxGamma] RequestTimedOutError: Request timed out

This message is repeated (thousands of times) for every trigger event until the node-red stops trying. After i restart the node (redeploy, not restart container), it continues working for maybe 4-5 days. On node-red there are no alarms, errors or what so ever. Can there be a way to either handle or configure timeouts?

manelmg commented 1 year ago

I have the same situation. I get these annoyings errors. In my case I write into influxdb every 1 second. Any possible solution to fix it or how to debug it?

ChenChiang commented 1 year ago

I am writing every 5s and receiving thousands messages of this error. Dose anyone have a solution?

LangeWiese commented 1 year ago

Since some weeks, I have the same problem with my communication between NodeRed and influxDB. NodeRed is running on a raspberryPi4 and influxDB in a docker container on my Synology. No one can help ? Best Regards IMG_3390

tinooo commented 11 months ago

having a very similar (or same) issue. After a few days, I'm receiving many errors.

However, the infuxDB is receiving and logging the values properly. So for me everything seems to work fine, aside from many errors triggering.

InfluxDB is running locally on the same machine. I've tried to decrease the amount of messages, but without changes in the behavior.

mblackstock commented 11 months ago

Since this issue would be difficult to reproduce, requiring days of testing, and based on the report of a request timeout error, I decided to try adding a connection timeout feature (0.7.0) so that it's possible to increase the connection timeout. Have you tried the latest version 0.7.0 and increasing the timeout to see if this addresses the problem in your setup?

mblackstock commented 10 months ago

Same as #134 assuming connection timeout feature fixes this unless I hear otherwise.

tinooo commented 10 months ago

Hi, I've meanwhile tested the new feature, but it doesn't solve the issue. What I wan't to highlight, the data is being successfully written to the InfluxDB, no matter if I get the error or not.

And as mentioned, the error only starts to come after a few hours/days.

However what is new with 0.7.0: There is a new Error: image

Both of these errors do come in bulk. And after some time, there is a bulk of the following errors: image

Then it starts over.

Seems like the timeout does not have any effect. Please let me know if I can provide any further details or logs to help support you.

mblackstock commented 10 months ago

Hmm. It's not clear to me this is a node/client problem, but if I have time I will try looking into it.

Can you characterize the load and tell me what sort of machine you are running InfluxDb on? i.e. how many writes, how much is written, what sorts of queries? Does the load change before this happens? What version of InfluxDb are you using (2.7.x)? Perhaps you just need more resources (CPU, Memory) for the db or an upgrade to a newer version?

tinooo commented 10 months ago

okay, so here some infos:

Everything running on a Synology NAS (DS220+ (Intel Celeron J4025 2-Core 2,0 GHz, Turbo bis 2,9 GHz ; 6GB RAM)) Setup everything using Docker on the NAS. Overall ressource consumption of the NAS itself is quite low. image Also as mentioned, the values seem to be received by InfluxDB. I don't find any values missing there.

My Requests are roughly: ~25 every 1 minute ~4 every 10 seconds ~30 every 15 seconds

So it is always a bulk that is triggered. e.g. with the 25 every minute, it roughly looks like this, and the result goes to the influxDb node. image

The data per request is always similar. Some sensor data, with some tags. A typical message looks like this: { payload: [{ // _values here: temp: msg.payload*msg.topic.faktor, }, { //Tags here: prozesswert: msg.topic.name, register: msg.topic.register, einheit: msg.topic.einheit }] };

everything either strings with 5-15 characters or some temperature, voltage, etc. values.

However, I've also tried with a delay node to reduce message load to 5msg/second. without change. image

thanks for your efforts. Let me know if I can be any more of a help.