search

nrwl / nx

Smart Monorepos · Fast CI
https://nx.dev
MIT License
23.78k stars 2.37k forks source link

NX parallel tasks fail with the errors 'database disk image is malformed' and 'NX DB transaction operation error: SqliteFailure(Error { code: ConstraintViolation, extended_code: 1555 }, Some("UNIQUE constraint failed: task_details.hash"))' #28772

Open ytchenak opened 3 weeks ago

ytchenak commented 3 weeks ago

Current Behavior

NX parallel tasks fail with the errors 'database disk image is malformed' and 'NX DB transaction operation error: SqliteFailure(Error { code: ConstraintViolation, extended_code: 1555 }, Some("UNIQUE constraint failed: task_details.hash"))'

Expected Behavior

Parallel tests have to be finished without error

GitHub Repo

No response

Steps to Reproduce

Run two or more jobs that execute non-overlapping sets of NX tasks

Nx Report

Node : 20.18.0 OS : win32-x64 Native Target : x86_64-windows pnpm : 9.12.1

nx (global) : 19.8.4 nx : 19.8.8 @nx/js : 19.8.8 @nx/jest : 19.8.8 @nx/linter : 19.8.8 @nx/eslint : 19.8.8 @nx/workspace : 19.8.8 @nx/angular : 19.8.8 @nx/eslint-plugin : 19.8.8 typescript : 5.5.4

Community plugins: @ngneat/spectator : 19.0.0 ng-mocks : 14.13.1 ngx-bootstrap : 18.0.2 nx-stylelint : 17.1.6

Failure Logs

first job:

NX   database disk image is malformed
[2024-11-03T08:02:14.318Z] 
[2024-11-03T08:02:14.318Z] Error: database disk image is malformed
[2024-11-03T08:02:14.318Z]     at TaskHistory.getEstimatedTaskTimings (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1454/frontend/node_modules/.pnpm/nx@19.8.9_@swc-node+register@1.10.9_@swc+core@1.7.42_@swc+helpers@0.5.13__@swc+types@0.1.13_t_7idvw5wocp6v2fkns4phtyxxzi/node_modules/nx/src/utils/task-history.js:20:37)
[2024-11-03T08:02:14.318Z]     at TasksSchedule.init (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1454/frontend/node_modules/.pnpm/nx@19.8.9_@swc-node+register@1.10.9_@swc+core@1.7.42_@swc+helpers@0.5.13__@swc+types@0.1.13_t_7idvw5wocp6v2fkns4phtyxxzi/node_modules/nx/src/tasks-runner/tasks-schedule.js:28:40)
[2024-11-03T08:02:14.318Z]     at TaskOrchestrator.run (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1454/frontend/node_modules/.pnpm/nx@19.8.9_@swc-node+register@1.10.9_@swc+core@1.7.42_@swc+helpers@0.5.13__@swc+types@0.1.13_t_7idvw5wocp6v2fkns4phtyxxzi/node_modules/nx/src/tasks-runner/task-orchestrator.js:48:32)
[2024-11-03T08:02:14.318Z]     at runAllTasks (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1454/frontend/node_modules/.pnpm/nx@19.8.9_@swc-node+register@1.10.9_@swc+core@1.7.42_@swc+helpers@0.5.13__@swc+types@0.1.13_t_7idvw5wocp6v2fkns4phtyxxzi/node_modules/nx/src/tasks-runner/default-tasks-runner.js:60:25)
[2024-11-03T08:02:14.318Z]     at defaultTasksRunner (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1454/frontend/node_modules/.pnpm/nx@19.8.9_@swc-node+register@1.10.9_@swc+core@1.7.42_@swc+helpers@0.5.13__@swc+types@0.1.13_t_7idvw5wocp6v2fkns4phtyxxzi/node_modules/nx/src/tasks-runner/default-tasks-runner.js:51:22)
[2024-11-03T08:02:14.318Z]     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[2024-11-03T08:02:14.318Z]     at async invokeTasksRunner (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1454/frontend/node_modules/.pnpm/nx@19.8.9_@swc-node+register@1.10.9_@swc+core@1.7.42_@swc+helpers@0.5.13__@swc+types@0.1.13_t_7idvw5wocp6v2fkns4phtyxxzi/node_modules/nx/src/tasks-runner/run-command.js:441:5)
[2024-11-03T08:02:14.318Z]     at async runCommandForTasks (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1454/frontend/node_modules/.pnpm/nx@19.8.9_@swc-node+register@1.10.9_@swc+core@1.7.42_@swc+helpers@0.5.13__@swc+types@0.1.13_t_7idvw5wocp6v2fkns4phtyxxzi/node_modules/nx/src/tasks-runner/run-command.js:117:25)
[2024-11-03T08:02:14.318Z]     at async /h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1454/frontend/node_modules/.pnpm/nx@19.8.9_@swc-node+register@1.10.9_@swc+core@1.7.42_@swc+helpers@0.5.13__@swc+types@0.1.13_t_7idvw5wocp6v2fkns4phtyxxzi/node_modules/nx/src/tasks-runner/run-command.js:105:29
[2024-11-03T08:02:14.318Z]     at async handleErrors (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1454/frontend/node_modules/.pnpm/nx@19.8.9_@swc-node+register@1.10.9_@swc+core@1.7.42_@swc+helpers@0.5.13__@swc+types@0.1.13_t_7idvw5wocp6v2fkns4phtyxxzi/node_modules/nx/src/utils/handle-errors.js:9:24)

second job:

|NX   DB transaction operation error: SqliteFailure(Error { code: ConstraintViolation, extended_code: 1555 }, Some("UNIQUE constraint failed: task_details.hash"))
[2024-11-03T08:02:18.848Z] 
[2024-11-03T08:02:18.848Z] Error: DB transaction operation error: SqliteFailure(Error { code: ConstraintViolation, extended_code: 1555 }, Some("UNIQUE constraint failed: task_details.hash"))
[2024-11-03T08:02:18.848Z]     at hashTasksThatDoNotDependOnOutputsOfOtherTasks (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1454/frontend/node_modules/.pnpm/nx@19.8.9_@swc-node+register@1.10.9_@swc+core@1.7.42_@swc+helpers@0.5.13__@swc+types@0.1.13_t_7idvw5wocp6v2fkns4phtyxxzi/node_modules/nx/src/hasher/hash-task.js:46:22)
[2024-11-03T08:02:18.848Z]     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[2024-11-03T08:02:18.848Z]     at async invokeTasksRunner (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1454/frontend/node_modules/.pnpm/nx@19.8.9_@swc-node+register@1.10.9_@swc+core@1.7.42_@swc+helpers@0.5.13__@swc+types@0.1.13_t_7idvw5wocp6v2fkns4phtyxxzi/node_modules/nx/src/tasks-runner/run-command.js:373:5)
[2024-11-03T08:02:18.848Z]     at async runCommandForTasks (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1454/frontend/node_modules/.pnpm/nx@19.8.9_@swc-node+register@1.10.9_@swc+core@1.7.42_@swc+helpers@0.5.13__@swc+types@0.1.13_t_7idvw5wocp6v2fkns4phtyxxzi/node_modules/nx/src/tasks-runner/run-command.js:117:25)
[2024-11-03T08:02:18.848Z]     at async /h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1454/frontend/node_modules/.pnpm/nx@19.8.9_@swc-node+register@1.10.9_@swc+core@1.7.42_@swc+helpers@0.5.13__@swc+types@0.1.13_t_7idvw5wocp6v2fkns4phtyxxzi/node_modules/nx/src/tasks-runner/run-command.js:105:29
[2024-11-03T08:02:18.848Z]     at async handleErrors (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1454/frontend/node_modules/.pnpm/nx@19.8.9_@swc-node+register@1.10.9_@swc+core@1.7.42_@swc+helpers@0.5.13__@swc+types@0.1.13_t_7idvw5wocp6v2fkns4phtyxxzi/node_modules/nx/src/utils/handle-errors.js:9:24)
[2024-11-03T08:02:18.848Z]     at async runCommand (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1454/frontend/node_modules/.pnpm/nx@19.8.9_@swc-node+register@1.10.9_@swc+core@1.7.42_@swc+helpers@0.5.13__@swc+types@0.1.13_t_7idvw5wocp6v2fkns4phtyxxzi/node_modules/nx/src/tasks-runner/run-command.js:104:20)
[2024-11-03T08:02:18.848Z]     at async Object.runMany (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1454/frontend/node_modules/.pnpm/nx@19.8.9_@swc-node+register@1.10.9_@swc+core@1.7.42_@swc+helpers@0.5.13__@swc+types@0.1.13_t_7idvw5wocp6v2fkns4phtyxxzi/node_modules/nx/src/command-line/run-many/run-many.js:46:24)
[2024-11-03T08:02:18.848Z]     at async /h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1454/frontend/node_modules/.pnpm/nx@19.8.9_@swc-node+register@1.10.9_@swc+core@1.7.42_@swc+helpers@0.5.13__@swc+types@0.1.13_t_7idvw5wocp6v2fkns4phtyxxzi/node_modules/nx/src/command-line/run-many/command-object.js:13:13
[2024-11-03T08:02:18.848Z]     at async handleErrors (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1454/frontend/node_modules/.pnpm/nx@19.8.9_@swc-node+register@1.10.9_@swc+core@1.7.42_@swc+helpers@0.5.13__@swc+types@0.1.13_t_7idvw5wocp6v2fkns4phtyxxzi/node_modules/nx/src/utils/handle-errors.js:9:24)

Package Manager Version

No response

Operating System

Additional Information

It seems that the original issue https://github.com/nrwl/nx/issues/28424 was not completely resolved. The issue persists in version 19.8.8, even after the fix https://github.com/nrwl/nx/pull/28544. It appears that the SQLite files are still not adequately protected against concurrent write access, despite the presence of the lockfile.hash file in the directory.

DaSchTour commented 2 weeks ago

Maybe related to #28640

ytchenak commented 2 weeks ago

I believe this issue is related to multiple NX processes running concurrently and using the same workspace, which leads to conflicts in the cache metadata SQLite database. I tried upgrading to the latest version (20.0.10) but still encountered an SQLite error.


 NX   database disk image is malformed

Error: database disk image is malformed
    at TaskHistory.getEstimatedTaskTimings (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1523/frontend/node_modules/.pnpm/nx@20.0.10_@swc-node+register@1.10.9_@swc+core@1.9.1_@swc+helpers@0.5.13__@swc+types@0.1.14_t_yc76nam4t7wmmxixmaewf25aca/node_modules/nx/src/utils/task-history.js:20:37)
    at TasksSchedule.init (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1523/frontend/node_modules/.pnpm/nx@20.0.10_@swc-node+register@1.10.9_@swc+core@1.9.1_@swc+helpers@0.5.13__@swc+types@0.1.14_t_yc76nam4t7wmmxixmaewf25aca/node_modules/nx/src/tasks-runner/tasks-schedule.js:28:40)
    at TaskOrchestrator.run (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1523/frontend/node_modules/.pnpm/nx@20.0.10_@swc-node+register@1.10.9_@swc+core@1.9.1_@swc+helpers@0.5.13__@swc+types@0.1.14_t_yc76nam4t7wmmxixmaewf25aca/node_modules/nx/src/tasks-runner/task-orchestrator.js:48:32)
    at runAllTasks (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1523/frontend/node_modules/.pnpm/nx@20.0.10_@swc-node+register@1.10.9_@swc+core@1.9.1_@swc+helpers@0.5.13__@swc+types@0.1.14_t_yc76nam4t7wmmxixmaewf25aca/node_modules/nx/src/tasks-runner/default-tasks-runner.js:60:25)
    at defaultTasksRunner (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1523/frontend/node_modules/.pnpm/nx@20.0.10_@swc-node+register@1.10.9_@swc+core@1.9.1_@swc+helpers@0.5.13__@swc+types@0.1.14_t_yc76nam4t7wmmxixmaewf25aca/node_modules/nx/src/tasks-runner/default-tasks-runner.js:51:22)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async invokeTasksRunner (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1523/frontend/node_modules/.pnpm/nx@20.0.10_@swc-node+register@1.10.9_@swc+core@1.9.1_@swc+helpers@0.5.13__@swc+types@0.1.14_t_yc76nam4t7wmmxixmaewf25aca/node_modules/nx/src/tasks-runner/run-command.js:441:5)
    at async runCommandForTasks (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1523/frontend/node_modules/.pnpm/nx@20.0.10_@swc-node+register@1.10.9_@swc+core@1.9.1_@swc+helpers@0.5.13__@swc+types@0.1.14_t_yc76nam4t7wmmxixmaewf25aca/node_modules/nx/src/tasks-runner/run-command.js:117:25)
    at async /h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1523/frontend/node_modules/.pnpm/nx@20.0.10_@swc-node+register@1.10.9_@swc+core@1.9.1_@swc+helpers@0.5.13__@swc+types@0.1.14_t_yc76nam4t7wmmxixmaewf25aca/node_modules/nx/src/tasks-runner/run-command.js:105:29
    at async handleErrors (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1523/frontend/node_modules/.pnpm/nx@20.0.10_@swc-node+register@1.10.9_@swc+core@1.9.1_@swc+helpers@0.5.13__@swc+types@0.1.14_t_yc76nam4t7wmmxixmaewf25aca/node_modules/nx/src/utils/handle-errors.js:9:24)```
gregroyal commented 1 week ago

I'm seeing this issue as well 20.0.7

specifically when i add {"dependentTasksOutputFiles":} to a task input

ytchenak commented 6 days ago

I tested the latest version (20.1.2), and the issue is still persistent. When two concurrent NX processes run on the same workspace, one of the jobs fails with a SQLite error.

Is there any progress on this? This issue is preventing us from upgrading to the latest NX versions. I am currently forced to use the last NX version that does not rely on SQLite.

@vsavkin - FYI

[2024-11-19T11:42:48.653Z]  NX   DB transaction operation error: SqliteFailure(Error { code: NotADatabase, extended_code: 26 }, Some("file is not a database"))
[2024-11-19T11:42:48.653Z] 
[2024-11-19T11:42:48.653Z] Error: DB transaction operation error: SqliteFailure(Error { code: NotADatabase, extended_code: 26 }, Some("file is not a database"))
[2024-11-19T11:42:48.653Z]     at TaskHistory.recordTaskRuns (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1671/frontend/node_modules/.pnpm/nx@20.1.2_@swc-node+register@1.10.9_@swc+core@1.7.42_@swc+helpers@0.5.13__@swc+types@0.1.13_t_hvi7kj5btq7ix24w7wn7bi43b4/node_modules/nx/src/utils/task-history.js:32:37)
[2024-11-19T11:42:48.653Z]     at TaskHistoryLifeCycle.endCommand (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1671/frontend/node_modules/.pnpm/nx@20.1.2_@swc-node+register@1.10.9_@swc+core@1.7.42_@swc+helpers@0.5.13__@swc+types@0.1.13_t_hvi7kj5btq7ix24w7wn7bi43b4/node_modules/nx/src/tasks-runner/life-cycles/task-history-life-cycle.js:37:32)
[2024-11-19T11:42:48.653Z]     at CompositeLifeCycle.endCommand (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1671/frontend/node_modules/.pnpm/nx@20.1.2_@swc-node+register@1.10.9_@swc+core@1.7.42_@swc+helpers@0.5.13__@swc+types@0.1.13_t_hvi7kj5btq7ix24w7wn7bi43b4/node_modules/nx/src/tasks-runner/life-cycle.js:18:25)
[2024-11-19T11:42:48.653Z]     at async defaultTasksRunner (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1671/frontend/node_modules/.pnpm/nx@20.1.2_@swc-node+register@1.10.9_@swc+core@1.7.42_@swc+helpers@0.5.13__@swc+types@0.1.13_t_hvi7kj5btq7ix24w7wn7bi43b4/node_modules/nx/src/tasks-runner/default-tasks-runner.js:54:9)
[2024-11-19T11:42:48.653Z]     at async invokeTasksRunner (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1671/frontend/node_modules/.pnpm/nx@20.1.2_@swc-node+register@1.10.9_@swc+core@1.7.42_@swc+helpers@0.5.13__@swc+types@0.1.13_t_hvi7kj5btq7ix24w7wn7bi43b4/node_modules/nx/src/tasks-runner/run-command.js:441:5)
[2024-11-19T11:42:48.653Z]     at async runCommandForTasks (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1671/frontend/node_modules/.pnpm/nx@20.1.2_@swc-node+register@1.10.9_@swc+core@1.7.42_@swc+helpers@0.5.13__@swc+types@0.1.13_t_hvi7kj5btq7ix24w7wn7bi43b4/node_modules/nx/src/tasks-runner/run-command.js:117:25)
[2024-11-19T11:42:48.653Z]     at async /h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1671/frontend/node_modules/.pnpm/nx@20.1.2_@swc-node+register@1.10.9_@swc+core@1.7.42_@swc+helpers@0.5.13__@swc+types@0.1.13_t_hvi7kj5btq7ix24w7wn7bi43b4/node_modules/nx/src/tasks-runner/run-command.js:105:29
[2024-11-19T11:42:48.653Z]     at async handleErrors (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1671/frontend/node_modules/.pnpm/nx@20.1.2_@swc-node+register@1.10.9_@swc+core@1.7.42_@swc+helpers@0.5.13__@swc+types@0.1.13_t_hvi7kj5btq7ix24w7wn7bi43b4/node_modules/nx/src/utils/handle-errors.js:8:24)
[2024-11-19T11:42:48.653Z]     at async runCommand (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1671/frontend/node_modules/.pnpm/nx@20.1.2_@swc-node+register@1.10.9_@swc+core@1.7.42_@swc+helpers@0.5.13__@swc+types@0.1.13_t_hvi7kj5btq7ix24w7wn7bi43b4/node_modules/nx/src/tasks-runner/run-command.js:104:20)
[2024-11-19T11:42:48.653Z]     at async Object.runMany (/h/bldrtlv/jenkins/git_repo/prod/emweb_PR-1671/frontend/node_modules/.pnpm/nx@20.1.2_@swc-node+register@1.10.9_@swc+core@1.7.42_@swc+helpers@0.5.13__@swc+types@0.1.13_t_hvi7kj5btq7ix24w7wn7bi43b4/node_modules/nx/src/command-line/run-many/run-many.js:43:24)