Throw error 'error Command failed with signal "SIGKILL".' in gitlab CI

a1245582339 commented 2 weeks ago

Verify canary release

[X] I verified that the issue exists in the latest Turborepo canary release.

Link to code that reproduces this issue

N/A

What package manager are you using / does the bug impact?

Yarn v2/v3/v4 (node_modules linker only)

What operating system are you using?

Linux

Which canary version will you have in your reproduction?

turbo@2.1.2

Describe the Bug

In my gitlab CI, the "turbo run type-check" command throwed several errors: error Command failed with signal "SIGKILL". Intriguingly, this error will only appear when more then 3 packages need to be check. When the number of packages that need to execute 'tsc -- noEmit' is less than or equal to 3, it can be executed successfully. And no matter how many packages need to be checked, it will success on my local machine.

I have tried to run turbo run type-check --concurrency 1. But the error is the same.

I have checked my memory of my CI devops. I think memory is enough. And the NODE_OPTIONS: --max-old-space-size=4096 has also be added to .gitlab-ci.yml

free -m
                  total        used        free      shared  buff/cache   available
Mem:          128625        8252       67182          25       53190      119276
Swap:              0           0           0

My turbo config:

{
  "$schema": "https://turbo.build/schema.json",
  "tasks": {
    "type-check": {
      "dependsOn": ["^type-check"]
    },
  }
}

The "type-check" in each packages are the same.

# package.json
"scripts": {
    "dev": "vite --host 0.0.0.0 --mode=development-test",
    "build": "vite build",
    "type-check": "tsc --noEmit"
  },

Expected Behavior

Expected to happen: It will success in gitlab CI.

I want to know which step might have gone wrong, yarn or turbo, or dev mechine? Thank you for your help.

To Reproduce

Sorry, it is in a internal gitlab repo of a company. I can not provide the reproduce behavior. But if you want to know more detail of this situation, you can leave me a comment

Additional context

No response

chris-olszewski commented 2 weeks ago

Can you provide the full error message as it appears in the logs? I believe error Command failed with signal "SIGKILL" is a yarn error message, but there might be some prelude that indicates whether turbo received a SIGKILL and is passing on the message.

a1245582339 commented 2 weeks ago

Can you provide the full error message as it appears in the logs? I believe error Command failed with signal "SIGKILL" is a yarn error message, but there might be some prelude that indicates whether turbo received a SIGKILL and is passing on the message.

Hi Chris, thank you for your reply and attention. There is the whole log in my gitlab-ci. I have mask some sensitive infomation, such as package name.

$ npx turbo run type-check --force
Attention:
Turborepo now collects completely anonymous telemetry regarding usage.
This information is used to shape the Turborepo roadmap and prioritize features.
You can learn more, including how to opt-out if you'd not like to participate in this anonymous program, by visiting the following URL:
https://turbo.build/repo/docs/telemetry
• Packages in scope: @repo/brick, @repo/brick-field-auto-complete, @repo/brick-field-cascader, @repo/brick-field-checkbox, @repo/brick-field-datetime-picker, @repo/brick-field-datetime-range-picker, @repo/brick-field-editor, @repo/brick-field-input-range, @repo/brick-field-money, @repo/brick-field-multiple-cascader, @repo/brick-field-multiple-select, @repo/brick-field-multiple-uploader, @repo/brick-field-number, @repo/brick-field-number-range, @repo/brick-field-radio, @repo/brick-field-select, @repo/brick-field-switch, @repo/brick-field-table, @repo/brick-field-tag-select, @repo/brick-field-text, @repo/brick-field-textarea, @repo/brick-field-tree-panel, @repo/brick-field-uploader, @repo/chart, @repo/components, @repo/constant, @repo/diff-editor, @repo/feishu-timeline, @repo/flow, @repo/flow-action-delete, @repo/flow-action-form, @repo/flow-action-link, @repo/flow-action-update, @repo/flow-condition, @repo/flow-field-boolean, @repo/flow-field-datetime, @repo/flow-field-enum, @repo/flow-field-foreign, @repo/flow-field-number, @repo/flow-field-table, @repo/flow-field-text, @repo/flow-field-url, @repo/flow-meta, @repo/flow-mock, @repo/flow-plugin-auto-fill, @repo/flow-plugin-data-filter, @repo/flow-plugin-hidden, @repo/flow-plugin-textarea-code, @repo/flow-requester, @repo/flow-requester-auth, @repo/flow-requester-idb, @repo/flow-requester-server, @repo/flow-requester-server-idb, @repo/flow3, @repo/flow3-template, @repo/interpolation, @repo/lark-feedback, @repo/memory-file-manager, @repo/memory-file-node, @repo/xxx-components, @repo/xxx-styles, @repo/nexus-rpc, @repo/project, @repo/protobuf, @repo/schema-manager, @repo/ts-live-bundle, @repo/ts-live-editor, @repo/ts-live-previewer, @repo/types, @repo/typescript-config, @repo/typography, @repo/util, @repo/web-socket, d-platform, b-platform, g-platform, bd-platform, flow, f-platform, c-platform, meta-fe-cra, h-platform, e-platform, a-platform, i-platform
• Running type-check in 85 packages
• Remote caching disabled
a-platform:type-check: cache bypass, force executing fb64dff8e[111](https://gitlab.corp.xxx.com/xxx/trading-infra/a-platform/-/jobs/3497455#L111)5ab4
b-platform:type-check: cache bypass, force executing 48c116897d1de171
c-platform:type-check: cache bypass, force executing c5c582a1568a1e7c
d-platform:type-check: cache bypass, force executing c9873c6a6d31e4aa
e-platform:type-check: cache bypass, force executing 3fa0a51eecc91cf6
f-platform:type-check: cache bypass, force executing 139ae9980d2f1296
g-platform:type-check: cache bypass, force executing ce155f50f093cbe3
h-platform:type-check: cache bypass, force executing 81e1dc167a1f7ba5
j-platform:type-check: cache bypass, force executing 6d070ccf3c6a86c0
i-platform:type-check: cache bypass, force executing 958ce601f[112](https://gitlab.corp.xxx.com/xxx/trading-infra/a-platform/-/jobs/3497455#L112)4ee4
a-platform:type-check: yarn run v1.22.22
c-platform:type-check: yarn run v1.22.22
b-platform:type-check: yarn run v1.22.22
d-platform:type-check: yarn run v1.22.22
a-platform:type-check: $ tsc --noEmit
c-platform:type-check: $ tsc --noEmit
e-platform:type-check: yarn run v1.22.22
b-platform:type-check: $ tsc --noEmit
f-platform:type-check: yarn run v1.22.22
g-platform:type-check: yarn run v1.22.22
d-platform:type-check: $ tsc --noEmit
e-platform:type-check: $ tsc --noEmit
h-platform:type-check: yarn run v1.22.22
i-platform:type-check: yarn run v1.22.22
g-platform:type-check: $ tsc --noEmit
j-platform:type-check: yarn run v1.22.22
f-platform:type-check: $ tsc --noEmit
i-platform:type-check: $ tsc --noEmit
h-platform:type-check: $ tsc --noEmit
j-platform:type-check: $ tsc --noEmit
j-platform:type-check: error Command failed with signal "SIGKILL".
c-platform:type-check: info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
c-platform:type-check: error Command failed with signal "SIGKILL".
j-platform:type-check: info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
e-platform:type-check: info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
e-platform:type-check: error Command failed with signal "SIGKILL".
c-platform:type-check: ERROR: command finished with error: command (/builds/giJtT4Lj/0/xxx/trading-infra/a-platform/workspace/c-platform) /builds/giJtT4Lj/0/xxx/trading-infra/a-platform/node_modules/.bin/yarn run type-check exited (1)
j-platform:type-check: ERROR: command finished with error: command (/builds/giJtT4Lj/0/xxx/trading-infra/a-platform/workspace/j-platform) /builds/giJtT4Lj/0/xxx/trading-infra/a-platform/node_modules/.bin/yarn run type-check exited (1)
e-platform:type-check: ERROR: command finished with error: command (/builds/giJtT4Lj/0/xxx/trading-infra/a-platform/workspace/e-platform) /builds/giJtT4Lj/0/xxx/trading-infra/a-platform/node_modules/.bin/yarn run type-check exited (1)
bd-platform:type-check: cache bypass, force executing 7cb5946f53d2dc76
c-platform#type-check: command (/builds/giJtT4Lj/0/xxx/trading-infra/a-platform/workspace/c-platform) /builds/giJtT4Lj/0/xxx/trading-infra/a-platform/node_modules/.bin/yarn run type-check exited (1)
j-platform#type-check: command (/builds/giJtT4Lj/0/xxx/trading-infra/a-platform/workspace/j-platform) /builds/giJtT4Lj/0/xxx/trading-infra/a-platform/node_modules/.bin/yarn run type-check exited (1)
e-platform#type-check: command (/builds/giJtT4Lj/0/xxx/trading-infra/a-platform/workspace/e-platform) /builds/giJtT4Lj/0/xxx/trading-infra/a-platform/node_modules/.bin/yarn run type-check exited (1)
 Tasks:    0 successful, 11 total
Cached:    0 cached, 11 total
  Time:    9.56s 
Failed:    j-platform#type-check, c-platform#type-check, e-platform#type-check
 ERROR  run failed: command  exited (1)
Cleaning up project directory and file based variables
00:02
Removing .turbo/
Removing node_modules/
Removing packages/memory-file-node/node_modules/
Removing packages/xxx-components/node_modules/
Removing packages/protobuf/node_modules/
Removing packages/ts-live-bundle/node_modules/
Removing workspace/d-platform/.turbo/
Removing workspace/b-platform/.turbo/
Removing workspace/g-platform/.turbo/
Removing workspace/j-platform/.turbo/
Removing workspace/f-platform/.turbo/
Removing workspace/c-platform/.turbo/
Removing workspace/c-platform/node_modules/
Removing workspace/h-platform/.turbo/
Removing workspace/h-platform/node_modules/
Removing workspace/e-platform/.turbo/
Removing workspace/a-platform/.turbo/
Removing workspace/a-platform/node_modules/
Removing workspace/i-platform/.turbo/
HEAD is now at 0c708ad6 test: add memo
ERROR: Job failed: command terminated with exit code 1

chris-olszewski commented 2 weeks ago

Interesting it looks like something on the machine is killing the type-check processes out from under turbo. There's not much turbo can do since the signals are being sent directly to the processes.

I'm not super familiar with Gitlab CI so I'm not sure if there's some limit that's getting hit here. You're correct that this isn't related to Node running out of memory as that wouldn't result in a SIGKILL getting sent. It could be an overall memory usage limit imposed by Gitlab that isn't displayed by free.

[111](https://gitlab.corp.xxx.com/xxx/trading-infra/a-platform/-/jobs/3497455#L111)

I notice a few logs like these interspersed with the turbo output, this might be Gitlab trying to display some related info.

If you would be willing, running turbo with -vv would produce a lot more logs that might shed some light. They will probably need to be heavily redacted. You can share them with me here or at chris.olszewski@vercel.com

a1245582339 commented 2 weeks ago

Interesting it looks like something on the machine is killing the type-check processes out from under turbo. There's not much turbo can do since the signals are being sent directly to the processes.

I'm not super familiar with Gitlab CI so I'm not sure if there's some limit that's getting hit here. You're correct that this isn't related to Node running out of memory as that wouldn't result in a SIGKILL getting sent. It could be an overall memory usage limit imposed by Gitlab that isn't displayed by free.

111

I notice a few logs like these interspersed with the turbo output, this might be Gitlab trying to display some related info.

If you would be willing, running turbo with -vv would produce a lot more logs that might shed some light. They will probably need to be heavily redacted. You can share them with me here or at chris.olszewski@vercel.com

Hi Chris, I have sent the log with -vv by email. Thank you for your attention again.

And I have noticed another thing which is I tried to change the package management tool to pnpm and run turbo type-check with --concurrency=1, the gitlab-ci can be success. So I guess the problem of this issue is related to yarn probably.

chris-olszewski commented 2 weeks ago

Thanks for sending the logs. They do indicate something else on the system is killing the process out from under turbo. If switching to pnpm results in a success that does indicate something with how yarn is invoking tsc might be at fault.

I'm closing this as I don't think as it stands there's anything actionable from the turbo side. Please reopen if you have any new information or have any further questions!

vercel / turborepo