Nx Cloud Runners Constantly Getting SIGTERM Cancellation

[JackLazenbyZigzag]

Current Behavior

We have three Nx Cloud Runners in our CI/CD setup in Azure Pipelines. Since upgrading to the latest version of Nx (16, regardless of sub version), our runners are consistently failing when doing larger builds with Agent exited with the error: Agent was terminated via SIGTERM.

We do not host our own agents, they are the Microsoft provided ones.

Our Nx commands, as far as I'm aware (following checks with Nx directly) are best practice and this issue has only started occuring since upgrading to the latest version of Nx from version 14 (where we had no issues).

Here are the Nx packages we are using.

"@nx/cypress": "16.10.0",
        "@nx/eslint-plugin": "16.10.0",
        "@nx/jest": "16.10.0",
        "@nx/linter": "16.10.0",
        "@nx/nest": "16.10.0",
        "@nx/node": "16.10.0",
        "@nx/webpack": "^16.10",
        "@nx/workspace": "16.10.0",
        "@nxext/capacitor": "16.7.0",
        "@nxext/ionic-angular": "16.2.0",
"nx": "16.9.1",
        "nx-cloud": "16.4.0",

"optionalDependencies": {
        "@nx/nx-darwin-arm64": "16.5.1",
        "@nx/nx-darwin-x64": "16.5.1",
        "@nx/nx-linux-x64-gnu": "16.5.1",
        "@nx/nx-win32-x64-msvc": "16.5.1"
    }

Expected Behavior

The Nx Runners should not be shut down by SIGTERM.

GitHub Repo

No response

Steps to Reproduce

1.

Nx Report

Report complete - copy this into the issue template

   Node   : 18.17.1
   OS     : darwin-arm64
   npm    : 9.6.7

   nx                 : 16.10.0
   @nx/js             : 16.9.1
   @nx/jest           : 16.10.0
   @nx/linter         : 16.10.0
   @nx/workspace      : 16.10.0
   @nx/angular        : 16.9.1
   @nx/cypress        : 16.10.0
   @nx/devkit         : 16.9.1
   @nx/eslint-plugin  : 16.10.0
   @nx/nest           : 16.10.0
   @nx/node           : 16.10.0
   @nrwl/tao          : 16.10.0
   @nx/web            : 16.9.1
   @nx/webpack        : 16.10.0
   nx-cloud           : 16.5.1
   typescript         : 4.8.2
   ---------------------------------------
   Community plugins:
   @ionic/angular         : 6.7.5
   @ionic/angular-toolkit : 10.0.0
   @ngneat/spectator      : 14.0.0
   @ngrx/effects          : 15.4.0
   @ngrx/eslint-plugin    : 15.4.0
   @ngrx/router-store     : 15.4.0
   @ngrx/schematics       : 15.4.0
   @ngrx/store            : 15.4.0
   @ngrx/store-devtools   : 15.4.0
   @nxext/capacitor       : 16.7.0
   @nxext/ionic-angular   : 16.2.0
   ng-mocks               : 14.11.0
   ---------------------------------------
   The following packages should match the installed version of nx
     - @nx/js@16.9.1
     - @nrwl/js@16.9.1
     - @nx/angular@16.9.1
     - @nx/devkit@16.9.1
     - @nrwl/devkit@16.9.1
     - @nx/web@16.9.1
     - @nrwl/web@16.9.1

   To fix this, run `nx migrate nx@16.10.0`

Failure Logs

No response

Package Manager Version

No response

Operating System

• [ ] macOS
• [X] Linux
• [ ] Windows
• [ ] Other (Please specify)

Additional Information

No response

[MartinKPal]

Same for us. It is really a pain. @JackLazenbyZigzag some time passed. Did you find a way to deal with this?

[JackLazenbyZigzag]

Hi @MartinKPal! Unfortunately not, no. We still get it but it's been a lot more intermittent for us over the last month. It really does depend on what agent Azure assigns to us but recently we've not suffered too much.

We did start to move over to Github Actions from Azure Pipelines as we didn't have the same issues there but now that it doesn't happen as much for us we popped a pin in that.

If you or anyone else finds a permanent fix for this I'd love to know about it!

[FrozenPandaz]

Is this issue still happening?

[MartinKPal]

Hi Jason, Not anymore.

Martin Kollár

PALDESK | Core Services Product Owner / Team Lead

---

From: Jason Jean @.> Sent: Wednesday, May 15, 2024 11:34 PM To: nrwl/nx @.> Cc: Kollar Martin @.>; Mention @.> Subject: [EXT]:Re: [nrwl/nx] Nx Cloud Runners Constantly Getting SIGTERM Cancellation (Issue #19558)

Is this issue still happening?

— Reply to this email directly, view it on GitHubhttps://ddec1-0-en-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=https%3a%2f%2fgithub.com%2fnrwl%2fnx%2fissues%2f19558%23issuecomment%2d2113492804&umid=c2c6a1ea-a087-4d44-b2ad-c58bd31da7ed&auth=4dd45f74745176e3147efb4649e8aa8dc3cbe8c8-37b9c2de3634bc0871d34377a21d7f5f6f7e792c, or unsubscribehttps://ddec1-0-en-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=https%3a%2f%2fgithub.com%2fnotifications%2funsubscribe%2dauth%2fA5UU5RDWUMLEP4PIU2G3M5LZCPIHVAVCNFSM6AAAAAA532MWAKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJTGQ4TEOBQGQ&umid=c2c6a1ea-a087-4d44-b2ad-c58bd31da7ed&auth=4dd45f74745176e3147efb4649e8aa8dc3cbe8c8-35c6679214affb7c878c93c07e65df3cafee2e7f. You are receiving this because you were mentioned.Message ID: @.***>

[github-actions[bot]]

This issue has been automatically marked as stale because no results of retrying on the latest version of Nx was provided within 7 days. It will be closed in 21 days if no results are provided. If the issue is still present, please reply to keep it active. If the issue was not present, please close this issue. Thanks for being a part of the Nx community! 🙏

[github-actions[bot]]

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.