codenotary / immudb

immudb - immutable database based on zero trust, SQL/Key-Value/Document model, tamperproof, data change history
https://immudb.io
Other
8.57k stars 344 forks source link

Load testing immudb hits `max active snapshots` error #1998

Closed sleepycat closed 1 month ago

sleepycat commented 3 months ago

What happened When load testing immudb, a 1 second test will succeed, but a 5 second test will eventually will eventually return ERRO[0001] GoError: pq: tbtree: max active snapshots limit reached and after this error immudb cannot be used until it is restarted.

What you expected to happen I expected the tests to complete regardless of the duration.

How to reproduce it (as minimally and precisely as possible) Install the xk6 binary for building custom versions of k6. go install go.k6.io/xk6/cmd/xk6@latest Build a version of k6 with sql support in the current directory. xk6 build v0.52.0 --with github.com/grafana/xk6-sql Add the demo data to a fresh copy of immudb configured with maxActiveSnapshots = 1200: image

Using the k6 binary built above, run ./k6 run sqltest.js --duration 1s with the following script:

import sql from 'k6/x/sql';

export function setup() {}

export function teardown() {}

export default function () {
  // The second argument is a PostgreSQL connection string, e.g.
  // postgres://myuser:mypass@127.0.0.1:5432/postgres?sslmode=disable
  const db = sql.open('postgres', 'postgres://immudb:immudb@localhost:5432/defaultdb?sslmode=disable');
  let results = sql.query(db, `SELECT * FROM orders JOIN customers ON orders.customerid = customers.id WHERE orders.productid = $1;`, 1);
  console.log({results})
  for (const row of results) {
    console.log(`key: ${row.key}, value: ${row.value}`);
  }
  db.close();
}

Running ./k6 run sqltest.js --duration 1s will succeed. Running ./k6 run sqltest.js --duration 5s will fail with tbtree: max active snapshots limit reached All subsequent tests of any duration will will fail with tbtree: max active snapshots limit reached Restart immudb. Running ./k6 run sqltest.js --duration 1s will succeed.

Environment

# run "immu* version" and copy/paste the output here
[mike@kijimi immudb-deployment]$ immudb version
immudb 1.9DOM.2
Commit  : d422d9f828c6ccfce6ce4af6c565946f764a15ca
Built by: makepkg
[mike@kijimi immudb-deployment]$ immuclient version
immuclient 1.9DOM.2
Commit  : d422d9f828c6ccfce6ce4af6c565946f764a15ca
Built by: makepkg
[mike@kijimi immudb-deployment]$ immuadmin version
immuadmin 1.9DOM.2
Commit  : d422d9f828c6ccfce6ce4af6c565946f764a15ca
Built by: makepkg

Additional info (any other context about the problem)