search

nrwl / nx

Smart Monorepos · Fast CI
https://nx.dev
MIT License
23.54k stars 2.35k forks source link

Zombie heartbeat-process.js #22867

Open jeffora opened 6 months ago

jeffora commented 6 months ago

Current Behavior

We have some build machines that run a portion of our CI process (on-premise Mac machines for running xcode / iOS builds). Our app is an ionic/cordova app in a monorepo with nx, so the build process on these machines runs nx tasks (via npx nx run ...). The repository is connected to nx cloud, however we are not calling start-ci-run or similar from these machines.

We are seeing a behaviour on these machines that a heartbeat process is started from nx cloud, and it never exits. Individually this doesn't seem to be a big issue, but the process appears to be orphaned or become unresponsive, and over time we end up with hundreds of these processes running:

$ pgrep -lf heartbeat
...
84272 /project/.nx/cache/cloud/2404.05.9.hotfix2/lib/heartbeat/heartbeat-process.js
84300 /project/.nx/cache/cloud/2404.05.9.hotfix2/lib/heartbeat/heartbeat-process.js
84333 /project/.nx/cache/cloud/2404.05.9.hotfix2/lib/heartbeat/heartbeat-process.js

We have seen this grow up to 250 processes in a relatively short time (few hours), and they don't seem to have any impact (i.e. killing them all doesn't cause anything to fail).

Is there a way to disable this process or ensure it properly cleans itself up? I'm not sure where to start in diagnosing why they are not exiting.

Expected Behavior

The heartbeat process should automatically exit, or not start in the first place, if it is not needed.

GitHub Repo

No response

Steps to Reproduce

  1. Unknown

Nx Report

NX  Falling back to ts-node for local typescript execution. This may be a little slower.
  - To fix this, ensure @swc-node/register and @swc/core have been installed

 NX   Report complete - copy this into the issue template

Node   : 18.19.0
OS     : darwin-arm64
npm    : 10.2.3

nx                 : 18.1.2
lerna              : 2.10.2
@nx/js             : 18.1.2
@nx/jest           : 18.1.2
@nx/linter         : 18.1.2
@nx/eslint         : 18.1.2
@nx/workspace      : 18.1.2
@nx/angular        : 18.1.2
@nx/cypress        : 18.1.2
@nx/devkit         : 18.1.2
@nx/eslint-plugin  : 18.1.2
@nx/node           : 18.1.2
@nx/storybook      : 18.1.2
@nrwl/tao          : 18.1.2
@nx/web            : 18.1.2
@nx/webpack        : 18.1.2
typescript         : 4.9.5
---------------------------------------
Community plugins:
@agriwebb/es-nx-tools    : 10.0.152
@compodoc/compodoc       : 1.1.23
@ionic/angular           : 6.7.5
@ionic/angular-toolkit   : 9.0.0
@ionic/cordova-builders  : 9.0.0
@ngneat/tailwind         : 7.0.3
@ngrx/component          : 15.4.0
@ngrx/data               : 15.4.0
@ngrx/effects            : 15.4.0
@ngrx/entity             : 15.4.0
@ngrx/store              : 15.4.0
@ngrx/store-devtools     : 15.4.0
@ngx-formly/schematics   : 6.3.0
@storybook/angular       : 8.0.4
@testing-library/angular : 14.5.0
@twittwer/compodoc       : 1.5.3
nx-stylelint             : 15.0.0

Failure Logs

No response

Package Manager Version

No response

Operating System

Additional Information

No response

github-actions[bot] commented 5 days ago

This issue has been automatically marked as stale because it hasn't had any activity for 6 months. Many things may have changed within this time. The issue may have already been fixed or it may not be relevant anymore. If at this point, this is still an issue, please respond with updated information. It will be closed in 21 days if no further activity occurs. Thanks for being a part of the Nx community! 🙏