search

pinojs / pino-elasticsearch

🌲 load pino logs into Elasticsearch
MIT License
179 stars 67 forks source link

Connection closing #102

Closed Allain55 closed 3 years ago

Allain55 commented 3 years ago

Am I right the the ElasticSearch connections are never closed? I see lots of active clients at ElasticSearch (value of nodes.http.current_open) and no trace that connections are closed using pino-elasticsearch. Since the client is not accessible outside it doesn't seem possible to close a connection by myself so how should they get closed?

mcollina commented 3 years ago

cc @delvedor

delvedor commented 3 years ago

The client relies on the internal http agent to handle connections. If a connection it's not used, after a while it will be closed automatically.

Allain55 commented 3 years ago

I have connections active for hours now. Wouldn't be better to close connections sooner? I don't know how important that is when using ElasticSearch though.

delvedor commented 3 years ago

You can configure for how much time an idle socket should be kept alive. But if you are sending data continuously it won't be closed. It a good thing that connections are not closed often tho, opening a new one is costly. Reusing old connections is more performant.

If you have a huge number of connections that you aren't expecting (by default the client opens up to 512 sockets), it might be worth investigating how many sources are connected to Elasticsearch (which can handle thousands of connections at the same time).

Allain55 commented 3 years ago

I checked and the connections (88) are active for 3 days now . I haven't changed any Elasticsearch setting related to this so this is by default. These were created by me when implementing pino-elasticsearch. I'll reset these manually

How could I configure idle sockets keep alive? Maybe you mean the network.tcp.keep_idle ElasticSearch setting?

delvedor commented 3 years ago

The sockets are being kept alive by default by the client.

Allain55 commented 3 years ago

By client you mean the @elastic/elasticsearch client, right? Since Friday I've even restarted the OS running the app with pino-elasticsearch. Sorry but it's not clear to me how are these connections still alive and how to close them after some time (like 1 day)

Allain55 commented 3 years ago

I'm going in a different direction. Thank you!