benthosdev / benthos

Fancy stream processing made operationally mundane
https://www.benthos.dev
MIT License
7.68k stars 752 forks source link

Nats Connection Pooling #2554

Open calmera opened 2 weeks ago

calmera commented 2 weeks ago

Use a singleton pool to track connections being made to nats.

Connections are tracked based on an advanced pool_key parameter. Components which share the same value for this parameter will also share the same connection.

The pool will do reference tracking; for this each component has been equiped with a pcid field which holds a UUID upon instantiation. When a component retrieves a connection from the pool, its picd is being added to the references. Components are required to release their connections and code has been adapted for them to do so. When releasing a connection, the reference will be removed from the connection and if no references are left, the connection will be closed and removed from the pool.