depyronick / clickhouse-client

ClickHouse Client for NodeJS
https://clickhouse.js.org
MIT License
49 stars 11 forks source link

Crucial memory leak Investigation: requesting author's expertise for help in resolving the issue #28

Open calebeaires opened 1 year ago

calebeaires commented 1 year ago

Using this library in an application where a single connection is created to perform a Clickhouse SELECT query to retrieve a response, followed by another SELECT query and subsequently repeating the entire process multiple times, the Axios HTTP library is not releasing memory.

I have provided a simple and practical example below to simulate what I have been doing. The number of example calls is for illustration purposes only. The objective is to demonstrate that a heapsnapshot reveals that memory in multiple requests leads to a memory leak. After running for 2 minutes, the memory usage increases from 70MB to 90MB and continues to rise progressively.

I have created this issue to discuss how to resolve a potential memory leak when making multiple requests. I have considered disabling KeepAlive or implementing better socket control.

Do you have any suggestions?

import { ClickHouseClient } from '@depyronick/clickhouse-client';

let a = 1
const client = new ClickHouseClient({
  host: 'localhost',
  password: 'mysecret',
  settings: {
    wait_end_of_query: 1
  }
});

const init = async () => {
  await client.queryPromise('SELECT * from some_database.some_table limit 1')
  a = a +1
  console.log(a);
}

const loopIterations = 1000000;

const runLoop = async () => {
  for (let i = 0; i < loopIterations; i++) {
    await init();
  }
}

runLoop();

We know that the issue does not lie with ClickHouse, but rather in the inability of HTTP/Axios to release memory in subsequent requests. This is the key aspect where we can attempt to find a solution.