search

louislam / uptime-kuma

A fancy self-hosted monitoring tool
https://uptime.kuma.pet
MIT License
59.76k stars 5.34k forks source link

The pool is probably full. Are you missing a .transacting(trx) call? #2923

Closed charlieporth1 closed 1 year ago

charlieporth1 commented 1 year ago

⚠️ Please verify that this bug has NOT been raised before.

🛡️ Security Policy

Description

Web sockets disconnect for a brief moment and then this happens. Also in order to get my monitors to appear when loading the webpage I have to log in and restart docker which might be the reason for this. I've got 275 monitors

This is on Nvidia Jetso nano with a 256 Samsung EVO sdcard

👟 Reproduction steps

Step 1.) Start Uptime Kuma Docker any version from 1.21.0-beta.0 to 1.20.0 Step 2.) Login to uptime kuma Step 3.) before monitors load (Which they never load I have waited an hour plus) restart the docker instance Step 4.) Find the monitors load Step 5.) Uptime kuma disonnects briefly 1 time Step 6.) Uptime kuma reconnects briefly Step 7.) Uptime kuma disonnects again second time Step 8.) Find log output below Step 9.) All monitors go offline Step 10.) All monitors go gray

👀 Expected behavior

Load montiors on login

😓 Actual Behavior

Crash

🐻 Uptime-Kuma Version

121.0-beta.0, 1.20.2, 1.20.1

💻 Operating System and Arch

Ubuntu 18.04

🌐 Browser

Chrome Version 111.0.5563.64 (Official Build) (x86_64)

🐋 Docker Version

Docker version 20.10.12, build 20.10.12-0ubuntu2~18.04.1

🟩 NodeJS Version

No response

📝 Relevant log output

2023-03-12T10:52:59-05:00 [MONITOR] WARN: Monitor #17 'galaxy-tab-a7-lite-1': Failing: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call? | Interval: 60 seconds | Type: ping | Down Count: 0 | Resend Interval: 0
Trace: KnexTimeoutError: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
    at Client_SQLite3.acquireConnection (/app/node_modules/knex/lib/client.js:312:26)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at listOnTimeout (node:internal/timers:528:9)
    at processTimers (node:internal/timers:502:7)
    at async Runner.ensureConnection (/app/node_modules/knex/lib/execution/runner.js:287:28)
    at async Runner.run (/app/node_modules/knex/lib/execution/runner.js:30:19)
    at async RedBeanNode.normalizeRaw (/app/node_modules/redbean-node/dist/redbean-node.js:569:22)
    at async RedBeanNode.getRow (/app/node_modules/redbean-node/dist/redbean-node.js:555:22)
    at async RedBeanNode.getCell (/app/node_modules/redbean-node/dist/redbean-node.js:590:19)
    at async Function.sendAvgPing (/app/server/model/monitor.js:962:32) {
  sql: '\n' +
    '            SELECT AVG(ping)\n' +
    '            FROM heartbeat\n' +
    "            WHERE time > DATETIME('now', ? || ' hours')\n" +
    '            AND ping IS NOT NULL\n' +
    '            AND monitor_id = ?  limit ?',
  bindings: [ -24, 121, 1 ]
}
    at process.<anonymous> (/app/server/server.js:1797:13)
    at process.emit (node:events:513:28)
    at emit (node:internal/process/promises:140:20)
    at processPromiseRejections (node:internal/process/promises:274:27)
    at processTicksAndRejections (node:internal/process/task_queues:97:32)
    at runNextTicks (node:internal/process/task_queues:65:3)
    at listOnTimeout (node:internal/timers:528:9)
    at processTimers (node:internal/timers:502:7)
If you keep encountering errors, please report to https://github.com/louislam/uptime-kuma/issues
Trace: KnexTimeoutError: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
    at Client_SQLite3.acquireConnection (/app/node_modules/knex/lib/client.js:312:26)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at listOnTimeout (node:internal/timers:528:9)
    at processTimers (node:internal/timers:502:7)
    at async Runner.ensureConnection (/app/node_modules/knex/lib/execution/runner.js:287:28)
    at async Runner.run (/app/node_modules/knex/lib/execution/runner.js:30:19)
    at async RedBeanNode.storeCore (/app/node_modules/redbean-node/dist/redbean-node.js:147:26)
    at async RedBeanNode.store (/app/node_modules/redbean-node/dist/redbean-node.js:107:20)
    at async beat (/app/server/model/monitor.js:778:13)
    at async Timeout.safeBeat [as _onTimeout] (/app/server/model/monitor.js:797:17) {
  sql: undefined,
  bindings: undefined
}
    at Timeout.safeBeat [as _onTimeout] (/app/server/model/monitor.js:799:25)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at listOnTimeout (node:internal/timers:528:9)
    at processTimers (node:internal/timers:502:7)
2023-03-12T10:52:59-05:00 [MONITOR] INFO: Try to restart the monitor
2023-03-12T10:52:59-05:00 [MONITOR] ERROR: Please report to https://github.com/louislam/uptime-kuma/issues
2023-03-12T10:52:59-05:00 [MONITOR] WARN: Monitor #7 'charless-tab-s7-1': Pending: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call? | Max retries: 2 | Retry: 1 | Retry Interval: 60 seconds | Type: ping
Trace: KnexTimeoutError: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
    at Client_SQLite3.acquireConnection (/app/node_modules/knex/lib/client.js:312:26)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at listOnTimeout (node:internal/timers:528:9)
    at processTimers (node:internal/timers:502:7)
    at async Runner.ensureConnection (/app/node_modules/knex/lib/execution/runner.js:287:28)
    at async Runner.run (/app/node_modules/knex/lib/execution/runner.js:30:19)
    at async RedBeanNode.normalizeRaw (/app/node_modules/redbean-node/dist/redbean-node.js:569:22)
    at async RedBeanNode.getRow (/app/node_modules/redbean-node/dist/redbean-node.js:555:22)
    at async RedBeanNode.getCell (/app/node_modules/redbean-node/dist/redbean-node.js:590:19)
    at async Function.sendAvgPing (/app/server/model/monitor.js:962:32) {
  sql: '\n' +
    '            SELECT AVG(ping)\n' +
    '            FROM heartbeat\n' +
    "            WHERE time > DATETIME('now', ? || ' hours')\n" +
    '            AND ping IS NOT NULL\n' +
    '            AND monitor_id = ?  limit ?',
  bindings: [ -24, 62, 1 ]
}
    at process.<anonymous> (/app/server/server.js:1797:13)
    at process.emit (node:events:513:28)
    at emit (node:internal/process/promises:140:20)
    at processPromiseRejections (node:internal/process/promises:274:27)
    at processTicksAndRejections (node:internal/process/task_queues:97:32)
    at runNextTicks (node:internal/process/task_queues:65:3)
    at listOnTimeout (node:internal/timers:528:9)
    at processTimers (node:internal/timers:502:7)
If you keep encountering errors, please report to https://github.com/louislam/uptime-kuma/issues
Trace: KnexTimeoutError: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
    at Client_SQLite3.acquireConnection (/app/node_modules/knex/lib/client.js:312:26)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at listOnTimeout (node:internal/timers:528:9)
    at processTimers (node:internal/timers:502:7)
    at async Runner.ensureConnection (/app/node_modules/knex/lib/execution/runner.js:287:28)
    at async Runner.run (/app/node_modules/knex/lib/execution/runner.js:30:19)
    at async RedBeanNode.storeCore (/app/node_modules/redbean-node/dist/redbean-node.js:147:26)
    at async RedBeanNode.store (/app/node_modules/redbean-node/dist/redbean-node.js:107:20)
    at async beat (/app/server/model/monitor.js:778:13)
    at async Timeout.safeBeat [as _onTimeout] (/app/server/model/monitor.js:797:17) {
  sql: undefined,
  bindings: undefined
}
    at Timeout.safeBeat [as _onTimeout] (/app/server/model/monitor.js:799:25)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at listOnTimeout (node:internal/timers:528:9)
    at processTimers (node:internal/timers:502:7)
2023-03-12T10:52:59-05:00 [MONITOR] ERROR: Please report to https://github.com/louislam/uptime-kuma/issues
2023-03-12T10:52:59-05:00 [MONITOR] INFO: Try to restart the monitor
2023-03-12T10:52:59-05:00 [MONITOR] WARN: Monitor #8 'ctp-vpn': Pending: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call? | Max retries: 2 | Retry: 1 | Retry Interval: 60 seconds | Type: ping
Trace: KnexTimeoutError: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
    at Client_SQLite3.acquireConnection (/app/node_modules/knex/lib/client.js:312:26)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at listOnTimeout (node:internal/timers:528:9)
    at processTimers (node:internal/timers:502:7)
    at async Runner.ensureConnection (/app/node_modules/knex/lib/execution/runner.js:287:28)
    at async Runner.run (/app/node_modules/knex/lib/execution/runner.js:30:19)
    at async RedBeanNode.normalizeRaw (/app/node_modules/redbean-node/dist/redbean-node.js:569:22)
    at async RedBeanNode.getRow (/app/node_modules/redbean-node/dist/redbean-node.js:555:22)
    at async RedBeanNode.getCell (/app/node_modules/redbean-node/dist/redbean-node.js:590:19)
    at async Function.sendAvgPing (/app/server/model/monitor.js:962:32) {
  sql: '\n' +
    '            SELECT AVG(ping)\n' +
    '            FROM heartbeat\n' +
    "            WHERE time > DATETIME('now', ? || ' hours')\n" +
    '            AND ping IS NOT NULL\n' +
    '            AND monitor_id = ?  limit ?',
  bindings: [ -24, 163, 1 ]
}
    at process.<anonymous> (/app/server/server.js:1797:13)
    at process.emit (node:events:513:28)
    at emit (node:internal/process/promises:140:20)
    at processPromiseRejections (node:internal/process/promises:274:27)
    at processTicksAndRejections (node:internal/process/task_queues:97:32)
    at runNextTicks (node:internal/process/task_queues:65:3)
    at listOnTimeout (node:internal/timers:528:9)
    at processTimers (node:internal/timers:502:7)
2023-03-12T10:52:59-05:00 [MONITOR] INFO: Try to restart the monitor
If you keep encountering errors, please report to https://github.com/louislam/uptime-kuma/issues
Trace: KnexTimeoutError: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
    at Client_SQLite3.acquireConnection (/app/node_modules/knex/lib/client.js:312:26)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at listOnTimeout (node:internal/timers:528:9)
    at processTimers (node:internal/timers:502:7)
    at async Runner.ensureConnection (/app/node_modules/knex/lib/execution/runner.js:287:28)
    at async Runner.run (/app/node_modules/knex/lib/execution/runner.js:30:19)
    at async RedBeanNode.storeCore (/app/node_modules/redbean-node/dist/redbean-node.js:147:26)
    at async RedBeanNode.store (/app/node_modules/redbean-node/dist/redbean-node.js:107:20)
    at async beat (/app/server/model/monitor.js:778:13)
    at async Timeout.safeBeat [as _onTimeout] (/app/server/model/monitor.js:797:17) {
  sql: undefined,
  bindings: undefined
}
    at Timeout.safeBeat [as _onTimeout] (/app/server/model/monitor.js:799:25)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at listOnTimeout (node:internal/timers:528:9)
    at processTimers (node:internal/timers:502:7)
2023-03-12T10:52:59-05:00 [MONITOR] ERROR: Please report to https://github.com/louislam/uptime-kuma/issues
Trace: KnexTimeoutError: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
    at Client_SQLite3.acquireConnection (/app/node_modules/knex/lib/client.js:312:26)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at listOnTimeout (node:internal/timers:528:9)
    at processTimers (node:internal/timers:502:7)
    at async Runner.ensureConnection (/app/node_modules/knex/lib/execution/runner.js:287:28)
    at async Runner.run (/app/node_modules/knex/lib/execution/runner.js:30:19)
    at async RedBeanNode.findOne (/app/node_modules/redbean-node/dist/redbean-node.js:496:19)
    at async Function.sendCertInfo (/app/server/model/monitor.js:984:23)
    at async Function.sendStats (/app/server/model/monitor.js:949:13) {
  sql: undefined,
  bindings: undefined
}
    at process.<anonymous> (/app/server/server.js:1797:13)
    at process.emit (node:events:513:28)
    at emit (node:internal/process/promises:140:20)
    at processPromiseRejections (node:internal/process/promises:274:27)
    at processTicksAndRejections (node:internal/process/task_queues:97:32)
    at runNextTicks (node:internal/process/task_queues:65:3)
    at listOnTimeout (node:internal/timers:528:9)
    at processTimers (node:internal/timers:502:7)
If you keep encountering errors, please report to https://github.com/louislam/uptime-kuma/issues
2023-03-12T10:52:59-05:00 [MONITOR] WARN: Monitor #10 'desktop-veahae7': Pending: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call? | Max retries: 2 | Retry: 1 | Retry Interval: 60 seconds | Type: ping
Trace: KnexTimeoutError: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
    at Client_SQLite3.acquireConnection (/app/node_modules/knex/lib/client.js:312:26)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at listOnTimeout (node:internal/timers:528:9)
    at processTimers (node:internal/timers:502:7)
    at async Runner.ensureConnection (/app/node_modules/knex/lib/execution/runner.js:287:28)
    at async Runner.run (/app/node_modules/knex/lib/execution/runner.js:30:19)
    at async RedBeanNode.findOne (/app/node_modules/redbean-node/dist/redbean-node.js:496:19)
    at async Function.sendCertInfo (/app/server/model/monitor.js:984:23)
    at async Function.sendStats (/app/server/model/monitor.js:949:13) {
  sql: undefined,
  bindings: undefined
}
    at process.<anonymous> (/app/server/server.js:1797:13)
    at process.emit (node:events:513:28)
    at emit (node:internal/process/promises:140:20)
    at processPromiseRejections (node:internal/process/promises:274:27)
    at processTicksAndRejections (node:internal/process/task_queues:97:32)
    at runNextTicks (node:internal/process/task_queues:65:3)
    at listOnTimeout (node:internal/timers:528:9)
    at processTimers (node:internal/timers:502:7)
If you keep encountering errors, please report to https://github.com/louislam/uptime-kuma/issues
bytes of
nh4ttruong commented 1 year ago

Excuse me, any fix for that issues?

csakaszamok commented 1 year ago

The error is still existing. I get a lot of false positive alarm: "Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?"

Noschvie commented 9 months ago

Please reopen this ticket, still isn't fixed, thanks.

CommanderStorm commented 9 months ago

A lot of performance improvements (using aggregated vs non-aggregated tables to store heartbeats, enabling users to choose mariadb as a db-backend, pagination of important events) have been made in v2.0 resolving this problem-area. => I'm going to keep it closed

Meanwhile (the issue is with sqlite not reading data fast enough to keep up):

Noschvie commented 9 months ago

Is there a v2.0 test image available?

CommanderStorm commented 9 months ago

You can subscribe to our releases and get notified when a new release (such as v2.0-beta.0) gets made. See https://github.com/louislam/uptime-kuma/pull/4171 for the bugs that need adressing before that can happen.