`renovate-ce:7.5.0-full` breaks after a few days

[meenzen]

Deleting the /tmp volume temporarily fixes the issue.

Error message: Failure initializing Renovate community edition - exiting...

Stacktrace:

Error: The module '/usr/src/app/node_modules/better-sqlite3/build/Release/better_sqlite3.node'
was compiled against a different Node.js version using
NODE_MODULE_VERSION 108. This version of Node.js requires
NODE_MODULE_VERSION 115. Please try re-compiling or re-installing
the module (for instance, using `npm rebuild` or `npm install`).
    at Module._extensions..node (node:internal/modules/cjs/loader:1454:18)
    at Module.load (node:internal/modules/cjs/loader:1208:32)
    at Module._load (node:internal/modules/cjs/loader:1024:12)
    at Module.require (node:internal/modules/cjs/loader:1233:19)
    at require (node:internal/modules/helpers:179:18)
    at bindings (/usr/src/app/node_modules/bindings/bindings.js:112:48)
    at new Database (/usr/src/app/node_modules/better-sqlite3/lib/database.js:48:64)
    at new t.SQLiteDataHandler (/usr/src/app/src/community.js:3:37753)
    at t.dataHandlerFactory (/usr/src/app/src/community.js:1:59655)
    at /usr/src/app/src/community.js:5:111892
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /usr/src/app/src/community.js:5:111397
    at async /usr/src/app/src/community.js:5:111033

[rarkins]

Has your CE restarted in this time?

[meenzen]

I don't think so, my docker host doesn't restart often. Everything seems to work fine until it ends up in the broken state, only showing the Failure initializing Renovate community edition - exiting... error.

Without manual intervention it is then stuck restarting every 2 minutes.

[jgraichen]

We have this immediately after start, as well as after restarts:

Jul 31 17:04:23 renovate 8c8f9a241c27[819]:  WARN: MEND_RNV_CRON_JOB_SCHEDULER is deprecated, please use MEND_RNV_CRON_JOB_SCHEDULER_ALL instead
Jul 31 17:04:23 renovate 8c8f9a241c27[819]:  INFO: Validating terms of service agreement
Jul 31 17:04:23 renovate 8c8f9a241c27[819]:        "input": "y"
Jul 31 17:04:23 renovate 8c8f9a241c27[819]:  INFO: Validating Mend license key
Jul 31 17:04:24 renovate 8c8f9a241c27[819]:  INFO: Merge Confidence is off; skipping Merge Confidence validation
Jul 31 17:04:24 renovate 8c8f9a241c27[819]: FATAL: Failure initializing Renovate community edition - exiting...
Jul 31 17:04:24 renovate 8c8f9a241c27[819]:        "err": {
Jul 31 17:04:24 renovate 8c8f9a241c27[819]:          "code": "ERR_DLOPEN_FAILED",
Jul 31 17:04:24 renovate 8c8f9a241c27[819]:          "message": "The module '/usr/src/app/node_modules/better-sqlite3/build/Release/better_sqlite3.node'\nwas compiled against a different Node.js version using\nNODE_MODULE_VERSION 108. This version of Node.js requires\nNODE_MODULE_VERSION 115. Please try re-compiling or re-installing\nthe module (for instance, using `npm rebuild` or `npm install`).",
Jul 31 17:04:24 renovate 8c8f9a241c27[819]:          "stack": "Error: The module '/usr/src/app/node_modules/better-sqlite3/build/Release/better_sqlite3.node'\nwas compiled against a different Node.js version using\nNODE_MODULE_VERSION 108. This version of Node.js requires\nNODE_MODULE_VERSION 115. Please try re-compiling or re-installing\nthe module (for instance, using `npm rebuild` or `npm install`).\n    at Module._extensions..node (node:internal/modules/cjs/loader:1454:18)\n    at Module.load (node:internal/modules/cjs/loader:1208:32)\n    at Module._load (node:internal/modules/cjs/loader:1024:12)\n    at Module.require (node:internal/modules/cjs/loader:1233:19)\n    at require (node:internal/modules/helpers:179:18)\n    at bindings (/usr/src/app/node_modules/bindings/bindings.js:112:48)\n    at new Database (/usr/src/app/node_modules/better-sqlite3/lib/database.js:48:64)\n    at new t.SQLiteDataHandler (/usr/src/app/src/community.js:3:37753)\n    at t.dataHandlerFactory (/usr/src/app/src/community.js:1:59655)\n    at /usr/src/app/src/community.js:5:111892\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async /usr/src/app/src/community.js:5:111397\n    at async /usr/src/app/src/community.js:5:111033"
Jul 31 17:04:24 renovate 8c8f9a241c27[819]:        }
Jul 31 17:04:24 renovate 8c8f9a241c27[819]: FATAL: Process will exit after 2 minutes

All our on-prem renovatebots ain't starting up anymore.

[rarkins]

@jgraichen this seems like a different problem, please create a separate issue. Also please set LOG_LEVEL=debug in env for increased logs

[jgraichen]

@rarkins It does say the same (added line breaks):

"message": "The module '/usr/src/app/node_modules/better-sqlite3/
build/Release/better_sqlite3.node'\nwas compiled against a different 
Node.js version using\nNODE_MODULE_VERSION 108. This version of 
Node.js requires\nNODE_MODULE_VERSION 115. Please try re-compiling 
or re-installing\nthe module (for instance, using `npm rebuild` 
or `npm install`).",

The image is ghcr.io/mend/renovate-ce:7.6.1-full now, and the errors start, at least, immediate after the container is restarted. Since the database file is stored in /tmp I have to destroy and recreate the container to have it work again.

Here with debug:

Aug 06 10:08:33 renovate 963c567a23e9[833]:  WARN: MEND_RNV_CRON_JOB_SCHEDULER is deprecated, please use MEND_RNV_CRON_JOB_SCHEDULER_ALL instead
Aug 06 10:08:33 renovate 963c567a23e9[833]: DEBUG: Resolved configuration
Aug 06 10:08:33 renovate 963c567a23e9[833]:        "resolvedConfig": {
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "cronJobSchedulerAll": "5/15 * * * *",
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "licenseKey": "***********",
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "acceptTOS": "y",
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "versionMismatchPrintInterval": 60,
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "dataHandlerType": "sqlite",
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "sqlTraceMode": false,
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "sqliteFilePath": "/tmp/database.sqlite",
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "sqliteVerbose": false,
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "sqliteExtensionsPaths": ["/usr/src/app/resources/sqlite/uuid.so"],
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "serverPort": "8080",
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "userAgent": "mend-renovate",
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "syncMode": "bulk",
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "syncOnStartup": true,
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "serverApiRoot": "/api",
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "adminApiEnabled": false,
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "cronAppSync": "0 0,4,8,12,16,20 * * *",
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "isReportingEnabled": false,
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "enqueueJobsOnStartup": "discovered",
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "webhookBaseBranches": ["master", "main", "develop"],
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "webhookBranchPrefixes": ["renovate/"],
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "logHistoryDir": "/tmp/renovate/logs",
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "logHistoryTTLDays": 30,
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "logHistoryCleanupCron": "0 0 * * *",
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "workerExecutionTimeout": 3600000,
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "workerHealthPort": "8080",
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "workerCleanup": "off",
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "workerCleanupDirs": [
Aug 06 10:08:33 renovate 963c567a23e9[833]:            "/opt/containerbase",
Aug 06 10:08:33 renovate 963c567a23e9[833]:            "/tmp/renovate/cache",
Aug 06 10:08:33 renovate 963c567a23e9[833]:            "/tmp/renovate/repos",
Aug 06 10:08:33 renovate 963c567a23e9[833]:            "/home/ubuntu"
Aug 06 10:08:33 renovate 963c567a23e9[833]:          ],
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "platform": "gitlab",
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "gitlabEndpoint": "***********",
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "gitlabPat": "***********",
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "gitlabWebhookSecret": "***********"
Aug 06 10:08:33 renovate 963c567a23e9[833]:        }
Aug 06 10:08:33 renovate 963c567a23e9[833]:  INFO: Validating terms of service agreement
Aug 06 10:08:33 renovate 963c567a23e9[833]:        "input": "y"
Aug 06 10:08:33 renovate 963c567a23e9[833]:  INFO: Validating Mend license key
Aug 06 10:08:33 renovate 963c567a23e9[833]:  INFO: This is a CORPORATE Mend Renovate license, registered to: Hasso-Plattner-Institut gGmbH. It was registered by Jan Graichen (jan.graichen@hpi.de)
Aug 06 10:08:33 renovate 963c567a23e9[833]: DEBUG: Initializing...
Aug 06 10:08:33 renovate 963c567a23e9[833]:  INFO: Merge Confidence is off; skipping Merge Confidence validation
Aug 06 10:08:33 renovate 963c567a23e9[833]: DEBUG: Initializing sqlite
Aug 06 10:08:33 renovate 963c567a23e9[833]: FATAL: Failure initializing Renovate community edition - exiting...
Aug 06 10:08:33 renovate 963c567a23e9[833]:        "err": {
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "code": "ERR_DLOPEN_FAILED",
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "message": "The module '/usr/src/app/node_modules/better-sqlite3/build/Release/better_sqlite3.node'\nwas compiled against a different Node.js version using\nNODE_MODULE_VERSION 108. This version of Node.js requires\nNODE_MODULE_VE>
Aug 06 10:08:33 renovate 963c567a23e9[833]:          "stack": "Error: The module '/usr/src/app/node_modules/better-sqlite3/build/Release/better_sqlite3.node'\nwas compiled against a different Node.js version using\nNODE_MODULE_VERSION 108. This version of Node.js requires\nNODE_MODU>
Aug 06 10:08:33 renovate 963c567a23e9[833]:        }
Aug 06 10:08:33 renovate 963c567a23e9[833]: FATAL: Process will exit after 2 minutes

[Dresdn]

I've been encountering this issue for a few weeks now, and my current workaround has been to re-create the container each time. Running with LOG_LEVEL=debug hasn't provided any useful insights, even when the startup error occurs.

I suspect that the Node.js version used to run Renovate might be getting replaced or altered by a dependency from a repository during runtime. Here's what I observed on a container that's been running for a week and was restarted this morning as part of a routine backup/maintenance process.

I'm mounting volumes for /tmp/ and /db/, and clearing both doesn't resolve the issue, contrary to what @meenzen suggested. The only consistent fix I've found is re-creating the container.

This behavior has been consistent across both the -full and non-full images.

# docker diff renovate-ce | grep -vE '/home/ubuntu/|/opt/containerbase'
C /opt
C /usr
C /usr/local
C /usr/local/bin
A /usr/local/bin/python3.12
C /usr/local/bin/yarn
C /usr/local/bin/npx
C /usr/local/bin/pnpx
A /usr/local/bin/ruby
A /usr/local/bin/pip
A /usr/local/bin/pip3
A /usr/local/bin/pip3.12
A /usr/local/bin/python3
A /usr/local/bin/gem
A /usr/local/bin/pip3.11
A /usr/local/bin/python
A /usr/local/bin/python3.11
C /usr/local/bin/npm
C /usr/local/bin/corepack
C /usr/local/bin/node
C /usr/local/bin/pnpm
C /usr/local/bin/yarnpkg
A /usr/local/bin/bundle
A /usr/local/bin/bundler
A /usr/local/bin/poetry
C /home
C /home/ubuntu
A /db

[Gabriel-Ladzaretti]

A fix will be available soon. In the meantime, you can use the following workaround:

Start the container with the following command: node_default src/community.js.

[Gabriel-Ladzaretti]

Currently, in our docker image, we have: CMD [ "node", "src/community.js" ]. This means that if the node version is changed at runtime, it will affect the version used resuming a container.

The fix is to use node_default, which resolves to a build time set symbolic link to the node runtime present during the build.

[nabeelsaabna]

Handled in release 8.1.0