search

nrwl / nx

Smart Monorepos · Fast CI
https://nx.dev
MIT License
23.78k stars 2.37k forks source link

Unable to set journal_mode: SqliteFailure(Error { code: CannotOpen, extended_code: 14 }, Some("unable to open database file")) #28640

Open wereHamster opened 1 month ago

wereHamster commented 1 month ago

Current Behavior

nx fails to run on my self-hosted github runners when trying to upgrade from 20.0.3 to 20.0.5 with the error in $SUBJECT. No additional information is printed into the console. Any nx command seems to trigger that, such as nx run-many --target build.

Expected Behavior

Nx works, or at least prints a more useful message (eg. path to the database it's trying to access).

GitHub Repo

No response

Steps to Reproduce

It's a private repository, but I can try to reproduce if you think it'd help. I very much suspect it only affects certain type of self-hosted github runners, as the new version works on my local machine (macOS).

Nx Report

Node           : 20.15.1
OS             : linux-x64
Native Target  : x86_64-linux
pnpm           : 9.6.0

nx                 : 20.0.5
@nx/js             : 20.0.5
@nx/eslint         : 20.0.5
@nx/workspace      : 20.0.5
@nx/devkit         : 20.0.5
@nx/esbuild        : 20.0.5
@nx/eslint-plugin  : 20.0.5
@nx/next           : 20.0.5
@nx/react          : 20.0.5
@nx/web            : 20.0.5
@nx/webpack        : 20.0.5
typescript         : 5.6.3
---------------------------------------
Registered Plugins:
@nx/eslint/plugin
@nx/next/plugin

Failure Logs

Run npx nx run-many --target build

NX Unable to set journal_mode: SqliteFailure(Error { code: CannotOpen, extended_code: 14 }, Some("unable to open database file"))

Pass --verbose to see the stacktrace.

Error: Process completed with exit code 1.

Package Manager Version

No response

Operating System

Additional Information

No response

njsokol commented 1 month ago

Getting the same thing on Linux. Note that this looks similar to #28608 error

wereHamster commented 1 month ago

In my case, each github runner has its own workspace and even running under its own user account, so there should not be any cross-workspace conflicts.

wereHamster commented 1 month ago

The regression was introduced either in 20.0.4 or 20.0.5. I can try to narrow down which one it was. Looking at the changelog, nothing stands out (but there may have been changes not reflected in the changelog…).

Image

wereHamster commented 1 month ago

Alright, regression was introduced in 20.0.4.

bjornharvold commented 4 weeks ago

Seeing this too

enchorb commented 4 weeks ago

Was getting this on an M2 Mac Unable to set journal_mode: SqliteFailure(Error { code: SystemIoFailure, extended_code: 522 }, Some("disk I/O error")) after upgrading to 20.0.6 from 20.0.0. After deleting the .nx folder the error went away as per the last comment here https://github.com/nrwl/nx/issues/28424

Cammisuli commented 4 weeks ago

@wereHamster if you run nx with the env NX_NATIVE_LOGGING=nx::native:db set, you should see where we're trying to open the database from.

Also, if you don't mind pasting the logs with NX_NATIVE_LOGGING here so that I can see the flow. Thank you.

cpierceworld commented 4 weeks ago

I'm getting a SqliteFailure "DatabaseBusy" error for the same command (after upgrading to NX 20, running on docker/linux). I don't know if it is related... I made a separate ticket since it is a different error code from the OP. https://github.com/nrwl/nx/issues/28665

minijus commented 3 weeks ago

@wereHamster if you run nx with the env NX_NATIVE_LOGGING=nx::native:db set, you should see where we're trying to open the database from.

Also, if you don't mind pasting the logs with NX_NATIVE_LOGGING here so that I can see the flow. Thank you.

Setting NX_NATIVE_LOGGING=nx::native:db did not produce any additional logs (running on self hosted Nx Cloud).

We are getting the same error:

NX   Unable to set journal_mode: SqliteFailure(Error { code: DatabaseBusy, extended_code: 5 }, Some("database is locked"))

"useLegacyCache": true is set in nx.json.

Cammisuli commented 3 weeks ago

@minijus make sure to set NX_NATIVE_LOGGING before doing start-ci-run for Nx Cloud. The start command will be the step that collects the env vars to send to agents, not specific nx runs

minijus commented 3 weeks ago

It seems the issue was addressed with v20.0.7 release.

wereHamster commented 3 weeks ago

@Cammisuli setting that env. variable does not show any logs. I set it via env: top-level option in the github workflow so it's set when any job runs. And issue is still present in 20.0.7.

DaSchTour commented 3 weeks ago

The issue is still present in 20.0.7 and deleting .nx didn't help :(

Cammisuli commented 3 weeks ago

@wereHamster are you using Nx Agents?

saurabh-cimpress commented 2 weeks ago

For me deleting the .nx folder did help and then command went ahead normally

raulgaf commented 2 weeks ago

Not sure if this is a fix, but for us the issue was on the yarnrc.yml file.

We recently migrated to yarn berry (3.6.4) for one of our projects to use yarn workspaces & Lerna. But sometimes we need to go back to yarn classic since other projects do not support that version yet.

Anyways, we noticed that switching between versions will add the yarnPath config to the.yarnrc.yml file. After we removed that config lerna was working as expected. 🤷‍♂

Not sure if this is related with lerna or nx but hopefully this helps someone with the same issue 😅

djohnson-aperture commented 2 weeks ago

Chiming in. I was using a clean clone of a repo. I'd ask if anyone here is using the VS Code Nx Console (nrwl.angular-console) extension? I think it does something to mess up the cache during that initial loading of the workspace in VS Code.

Doing the clone and initial setup in a basic terminal seemed to get me past the point where things could get mangled.

wereHamster commented 2 weeks ago

@wereHamster are you using Nx Agents?

No. I'm running Nx on self-hosted github action runners.

Erbenos commented 2 weeks ago

Deleting the .nx folder does help, but its merely a temporary solution, error happens again eventually, and then you need to delete .nx folder again.

LeLunZ commented 2 weeks ago

Same issue here (on a Mac) after migrating from angular 17 to 18. and nx is on version 20.0.11. Deleting .nx folder or calling nx reset or nx repair isn't helping

deleting in web storm didn't work. After retrying with rm -rf .nx its working

Cammisuli commented 2 weeks ago

Folks here, can we try the latest version of Nx (20.0.12) to see if this is still reproducible?

kaylesworth commented 2 weeks ago

Folks here, can we try the latest version of Nx (20.0.12) to see if this is still reproducible?

Upgraded to 20.0.12 today and using github actions with namespace runners I am still getting the error

Full stack trace with --verbose

Run pnpm nx affected -t=test --base=origin/main --head=HEAD --exclude='tag:type:integration' --shard=2/2 --bail=1 --silent=true --verbose

Error: Unable to set journal_mode: DB pragma update error: SqliteFailure(Error { code: CannotOpen, extended_code: 14 }, Some("unable to open database file"))
    at /home/runner/work/XXX/frontendnx/node_modules/.pnpm/nx@20.0.12_@swc-node+register@1.10.9_@swc+core@1.7.28_@swc+helpers@0.5.13__@swc+types@0.1.12__xbjtr5ucvm2hspey2expj3735e/node_modules/nx/src/utils/db-connection.js:12:93
    at getEntryOrSet (/home/runner/work/XXX/frontendnx/node_modules/.pnpm/nx@20.0.12_@swc-node+register@1.10.9_@swc+core@1.7.28_@swc+helpers@0.5.13__@swc+types@0.1.12__xbjtr5ucvm2hspey2expj3735e/node_modules/nx/src/utils/db-connection.js:26:17)
    at getDbConnection (/home/runner/work/XXX/frontendnx/node_modules/.pnpm/nx@20.0.12_@swc-node+register@1.10.9_@swc+core@1.7.28_@swc+helpers@0.5.13__@swc+types@0.1.12__xbjtr5ucvm2hspey2expj3735e/node_modules/nx/src/utils/db-connection.js:12:24)
    at getTaskDetails (/home/runner/work/XXX/frontendnx/node_modules/.pnpm/nx@20.0.12_@swc-node+register@1.10.9_@swc+core@1.7.28_@swc+helpers@0.5.13__@swc+types@0.1.12__xbjtr5ucvm2hspey2expj3735e/node_modules/nx/src/hasher/hash-task.js:19:84)
    at invokeTasksRunner (/home/runner/work/XXX/frontendnx/node_modules/.pnpm/nx@20.0.12_@swc-node+register@1.10.9_@swc+core@1.7.28_@swc+helpers@0.5.13__@swc+types@0.1.12__xbjtr5ucvm2hspey2expj3735e/node_modules/nx/src/tasks-runner/run-command.js:367:56)
    at runCommandForTasks (/home/runner/work/XXX/frontendnx/node_modules/.pnpm/nx@20.0.12_@swc-node+register@1.10.9_@swc+core@1.7.28_@swc+helpers@0.5.13__@swc+types@0.1.12__xbjtr5ucvm2hspey2expj3735e/node_modules/nx/src/tasks-runner/run-command.js:117:31)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /home/runner/work/XXX/frontendnx/node_modules/.pnpm/nx@20.0.12_@swc-node+register@1.10.9_@swc+core@1.7.28_@swc+helpers@0.5.13__@swc+types@0.1.12__xbjtr5ucvm2hspey2expj3735e/node_modules/nx/src/tasks-runner/run-command.js:105:29
    at async handleErrors (/home/runner/work/XXX/frontendnx/node_modules/.pnpm/nx@20.0.12_@swc-node+register@1.10.9_@swc+core@1.7.28_@swc+helpers@0.5.13__@swc+types@0.1.12__xbjtr5ucvm2hspey2expj3735e/node_modules/nx/src/utils/handle-errors.js:9:24)
    at async runCommand (/home/runner/work/XXX/frontendnx/node_modules/.pnpm/nx@20.0.12_@swc-node+register@1.10.9_@swc+core@1.7.28_@swc+helpers@0.5.13__@swc+types@0.1.12__xbjtr5ucvm2hspey2expj3735e/node_modules/nx/src/tasks-runner/run-command.js:104:20)
Cammisuli commented 1 week ago

@kaylesworth lets try 20.0.13 now 🙂 thanks for your help!

kaylesworth commented 1 week ago

@kaylesworth lets try 20.0.13 now 🙂 thanks for your help!

@Cammisuli can confirm that it works in 20.0.13!

tFaster commented 1 week ago

We updated to the latest Nx v20.1.0 and still get an SqliteFailureError in our build: NX DB transaction operation error: SqliteFailure(Error { code: DatabaseBusy, extended_code: 5 }, Some("database is locked")) These errors started to occurr for us since Nx v20.0.4

Cammisuli commented 1 week ago

@tFaster where is this happening for you? Local dev machine? (macOS, Linux, Windows?). If locally do you have Nx Console installed? Or was this happening CI? Can you give me more details on what was happening before?

mruknowme commented 1 week ago

For me it works on v20.1.0

ImSingee commented 1 week ago

v20.1.0 not works for me, in macOS, without nx console (well I even do not know what’s it).

I’m off computer, I will provide more details later

ImSingee commented 1 week ago

my nx.json:

{
  "useDaemonProcess": true,
  "targetDefaults": {
    "dev": {
      "dependsOn": ["^build"]
    },
    "build": {
      "dependsOn": ["^build", "check"]
    },
    "deploy": {
      "dependsOn": ["build", "^build", "check"]
    },
    "start": {
      "dependsOn": ["build", "^build"]
    },
    "check": {
      "dependsOn": ["^build"]
    }
  }
}

run nx, got the following error:

nx dev xxx

 NX   Running target dev for project xxx and 8 tasks it depends on:

——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

 NX   Daemon process terminated and closed the connection

Please rerun the command, which will restart the daemon.
If you get this error again, check for any errors in the daemon process logs found in: /xxx/.nx/workspace-data/d/daemon.log

 ELIFECYCLE  Command failed with exit code 1.

Process finished with exit code 1

daemon.log:

[NX Daemon Server] - 2024-11-13T16:00:47.171Z - Handled HASH_TASKS. Handling time: 16. Response time: 0.
<my project dir>/node_modules/.pnpm/nx@20.0.11_@swc+core@1.7.26_@swc+helpers@0.5.13_/node_modules/nx/src/utils/db-connection.js:12
    const connection = getEntryOrSet(dbConnectionMap, key, () => (0, native_1.connectToNxDb)(opts.directory, package_json_1.version, opts.dbName));
                                                                                            ^

Error: Unable to set journal_mode: DB pragma update error: SqliteFailure(Error { code: CannotOpen, extended_code: 14 }, Some("unable to open database file"))
    at <my project dir>/node_modules/.pnpm/nx@20.0.11_@swc+core@1.7.26_@swc+helpers@0.5.13_/node_modules/nx/src/utils/db-connection.js:12:93
    at getEntryOrSet (<my project dir>/node_modules/.pnpm/nx@20.0.11_@swc+core@1.7.26_@swc+helpers@0.5.13_/node_modules/nx/src/utils/db-connection.js:26:17)
    at getDbConnection (<my project dir>/node_modules/.pnpm/nx@20.0.11_@swc+core@1.7.26_@swc+helpers@0.5.13_/node_modules/nx/src/utils/db-connection.js:12:24)
    at new TaskHistory (<my project dir>/node_modules/.pnpm/nx@20.0.11_@swc+core@1.7.26_@swc+helpers@0.5.13_/node_modules/nx/src/utils/task-history.js:11:91)
    at getTaskHistory (<my project dir>/node_modules/.pnpm/nx@20.0.11_@swc+core@1.7.26_@swc+helpers@0.5.13_/node_modules/nx/src/utils/task-history.js:48:23)
    at handleGetEstimatedTaskTimings (<my project dir>/node_modules/.pnpm/nx@20.0.11_@swc+core@1.7.26_@swc+helpers@0.5.13_/node_modules/nx/src/daemon/server/handle-task-history.js:24:59)
    at <my project dir>/node_modules/.pnpm/nx@20.0.11_@swc+core@1.7.26_@swc+helpers@0.5.13_/node_modules/nx/src/daemon/server/server.js:146:141
    at handleResult (<my project dir>/node_modules/.pnpm/nx@20.0.11_@swc+core@1.7.26_@swc+helpers@0.5.13_/node_modules/nx/src/daemon/server/server.js:172:22)
    at handleMessage (<my project dir>/node_modules/.pnpm/nx@20.0.11_@swc+core@1.7.26_@swc+helpers@0.5.13_/node_modules/nx/src/daemon/server/server.js:146:15)
    at <my project dir>/node_modules/.pnpm/nx@20.0.11_@swc+core@1.7.26_@swc+helpers@0.5.13_/node_modules/nx/src/daemon/server/server.js:72:15 {
  code: 'GenericFailure'
}
jrista commented 1 week ago

Getting the same. Intel Mac ATM. NMX 19 and 20. Quite a persistent error. Usually have to run commands exactly three times to resolve the issue. Not sure why exactly three. The first two fail, third succeeds. Does not seem to matter how much time passes between runs. I can run three times in a row as fast as I can, or wait minutes between each. Always fails the first two, succeeds the third.

kohsine commented 1 week ago

Getting the same. Intel Mac ATM. NMX 19 and 20. Quite a persistent error. Usually have to run commands exactly three times to resolve the issue. Not sure why exactly three. The first two fail, third succeeds. Does not seem to matter how much time passes between runs. I can run three times in a row as fast as I can, or wait minutes between each. Always fails the first two, succeeds the third.

On Linux and yes I get this exact behavior where I have to run the command three times. It seems to happen consistently on first launch after turning on my PC then for the rest of the day it will (mostly) work on the first try.

novascreen commented 1 week ago

I am using nx-remotecache-gcs and noticed that via peer dependencies it installed a different version of nx. So I had multiple versions of nx. I ended up installing the right version of the peer dependency explicitly to get the versions in line.

Maybe have a look at your dependency lock files and see if you have multiple nx versions installed for some reason.

jrista commented 1 week ago

Getting the same. Intel Mac ATM. NMX 19 and 20. Quite a persistent error. Usually have to run commands exactly three times to resolve the issue. Not sure why exactly three. The first two fail, third succeeds. Does not seem to matter how much time passes between runs. I can run three times in a row as fast as I can, or wait minutes between each. Always fails the first two, succeeds the third.

On Linux and yes I get this exact behavior where I have to run the command three times. It seems to happen consistently on first launch after turning on my PC then for the rest of the day it will (mostly) work on the first try.

Interesting. In my case, I always have to run three times, every time. It never seems to resolve itself such that I can run just once.

I also have a tail going on the log. I've never seen any errors in the log that would indicate the issue...the daemon just dies, and running three times works, but there are never any log entries indicating what the problem might be.

kohsine commented 1 week ago

On v20.0.12 and it seems to be fixed for me at least on my local linux machine.

Cammisuli commented 1 week ago

Yea, this is an elusive issue that doesnt happen to everyone.

@jrista does this happen locally for you or on CI as well?

WSINTRA commented 1 week ago

Had same, issue. When running 


$nx run-many --target=typecheck

Error: Unable to set journal_mode: DB pragma update error: SqliteFailure(Error { code: SystemIoFailure, extended_code: 522 }, Some("disk I/O error"))

// Fixed with
$nx reset
jrista commented 6 days ago

Yea, this is an elusive issue that doesnt happen to everyone.

@jrista does this happen locally for you or on CI as well?

  • Can you give me everything you can think of that would make your system unique/different from a base ubuntu lts image (if running on Linux)?
  • Versions of OS's, repos, versions of nx with nx report.
  • Does this usually happen if you run without the daemon with NX_DAEMON=false?

  • Do you have Nx Console installed (vscode, intellij)

    • Is it updated to the latest version?
    • Does the issue go away if you disable nx console?
  • Also can you run the commands with NX_NATIVE_LOGGIN=nx::native::nx and share those outputs? (The outputs will be in the daemon.log file found with nx daemon, or in terminal output while running with NX_DAEMON=false)

I don't have CI set up on these. I'm currently doing a series of POCs to prepare for an upcoming contract. So its just local.

Here is the Nx report (macos x):

➜ nx report

 NX   Report complete - copy this into the issue template

Node           : 20.18.0
OS             : darwin-x64
Native Target  : x86_64-macos
npm            : 10.8.2

nx                 : 19.8.10
@nx/js             : 19.8.10
@nx/jest           : 19.8.10
@nx/linter         : 19.8.10
@nx/eslint         : 19.8.10
@nx/workspace      : 19.8.10
@nx/angular        : 19.8.10
@nx/devkit         : 19.8.10
@nx/eslint-plugin  : 19.8.10
@nx/playwright     : 19.8.10
@nrwl/tao          : 19.8.10
@nx/vite           : 19.8.10
@nx/web            : 19.8.10
@nx/webpack        : 19.8.10
typescript         : 5.5.4
---------------------------------------
Registered Plugins:
@nx/playwright/plugin
@nx/eslint/plugin
---------------------------------------
Community plugins:
@auth0/auth0-angular : 2.2.3
@ionic/angular       : 8.4.0
@ngrx/effects        : 18.1.1
@ngrx/router-store   : 18.1.1
@ngrx/store          : 18.1.1
@ngrx/store-devtools : 18.1.1
@nxext/ionic-angular : 19.1.2

I have been having some other issues with 20, so I'm currently using 19.8.

FWIW, I AM actually using Nx Console in Webstorm. I hadn't considered that that might be part of the problem. So, I uninstalled it, closed webstorm, and restarted. ATM, I am not having issues. I reinstalled Nx Console, and am still not having issues...

SO, that makes me wonder if somehow that WAS interfereing with it. Thing is, that's how it originally was as well. When I first set up these projects, everything was fine. Then one day, they weren't, and it was persistently a problem from that day on. So I don't know if the issue will reoccur. If it does, then I'll uninstall Nx Console and leave it uninstalled, and see if the issue still reoccurs.

jrista commented 1 day ago

I think that Nx Console is a part of the problem. I don't think it is the entire problem, but removing Nx Console did eliminate the need for me to run every command three times all the time. I still run into the issue, usually first thing in the day, first time I run an Nx command. Sometimes it will occur at other times during the day, but it is not as persistent as it used to be. Adding Nx Console back, does not immediately result in the issue, however over time, the issue gets worse until I am once again having to run every command three times in a row.

Not sure what it is, but there does seem to be an issue with Nx Console and however it is interacting with Nx. Will try to contact the author and let them know about this issue.