Nx Cloud Deployment Hangs on Subsequent Attempts with Error in Copying Dependency afterwards

aramfe commented 2 months ago

Current Behavior

When running squential commands using Nx Cloud through GitHub Actions, the command works correctly for the first attempt differentiated solely by a --args="--deployment=preproduction parameter. However, upon attempting a second deployment, especially targeting the --deployment=production environment, the process enters an infinite retry loop and can't continue folloing processes.

This behavior does not result in an error that would terminate the shell execution but instead leads to GitHub cancelling the run due to exceeding the 1-hour timeout limit. Upon inspection within the Nx Cloud interface, the problematic command appears to execute successfully (indicated by a green status), suggesting the issue arises post-execution, likely involving the Nx remote cache.

Expected Behavior

Successful completion of the deployment process for all specified environments without any hang-ups or indefinite retries.
No infinite retry loops or extended execution times beyond what is reasonable for the task at hand.
Proper handling of the Nx remote cache to prevent issues that arise after successful command execution.

GitHub Repo

No response

Steps to Reproduce

This is how it is invoked:

- run: yarn run-many -t deploy --args="--deployment=preproduction"
- run: yarn run-many -t deploy --args="--deployment=production"

Basically run a command twice in one single npx nx-cloud start-ci-run run.

Nx Report

Node   : 18.19.0
OS     : linux-x64
yarn   : 3.5.1

nx              : 19.2.3
@nx/js          : 19.2.3
@nx/jest        : 19.2.3
@nx/linter      : 19.2.3
@nx/eslint      : 19.2.3
@nx/workspace   : 19.2.3
@nx/devkit      : 19.2.3
@nx/playwright  : 19.2.3
@nrwl/tao       : 19.2.3
typescript      : 4.9.4
---------------------------------------
Registered Plugins:
@nxrocks/nx-spring-boot
@nx/playwright/plugin
---------------------------------------
Community plugins:
@nx-tools/nx-container  : 6.0.1
@nxrocks/nx-spring-boot : 10.0.2

Failure Logs

Error: Cannot copy '.nitro/css-tree@2.3.1' to a subdirectory of itself, '.nitro/css-tree@2.3.1'.
    at sM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1745)
    at async Promise.all (index 33)
    at async rM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1225)
    at async Promise.all (index 2)
    at async rM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1225)
    at async Promise.all (index 2)
Error: Cannot copy '.nitro/css-tree@2.3.1' to a subdirectory of itself, '.nitro/css-tree@2.3.1'.
    at sM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1745)
    at async Promise.all (index 33)
    at async rM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1225)
    at async Promise.all (index 2)
    at async rM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1225)
    at async Promise.all (index 2)
    at async rM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1225)
    at async Promise.all (index 1)
    at async rM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1225)
    at async Promise.all (index 0)
    at async rM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1225)
    at async Promise.all (index 0)
    at async rM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1225)
    at async Promise.all (index 0)
    at async rM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1225)
    at async Promise.all (index 0)
Error: Cannot copy '.nitro/css-tree@2.3.1' to a subdirectory of itself, '.nitro/css-tree@2.3.1'.
    at sM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1745)
    at async Promise.all (index 33)
    at async rM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1225)
    at async Promise.all (index 2)
    at async rM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1225)
    at async Promise.all (index 2)
    at async rM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1225)
    at async Promise.all (index 1)
    at async rM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1225)
    at async Promise.all (index 0)
    at async rM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1225)
    at async Promise.all (index 0)
    at async rM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1225)
    at async Promise.all (index 0)
    at async rM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1225)
    at async Promise.all (index 0)
Error: Cannot copy '.nitro/css-tree@2.3.1' to a subdirectory of itself, '.nitro/css-tree@2.3.1'.
    at sM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1745)
    at async Promise.all (index 33)
    at async rM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1225)
    at async Promise.all (index 2)
    at async rM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1225)
    at async Promise.all (index 2)
    at async rM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1225)
    at async Promise.all (index 1)
    at async rM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1225)
    at async Promise.all (index 0)
    at async rM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1225)
    at async Promise.all (index 0)
    at async rM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1225)
    at async Promise.all (index 0)
    at async rM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1225)
    at async Promise.all (index 0)
    at async rM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1225)
    at async Promise.all (index 1)
    at async rM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1225)
    at async Promise.all (index 0)
    at async rM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1225)
    at async Promise.all (index 0)
    at async rM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1225)
    at async Promise.all (index 0)
    at async rM (/home/runner/actions-runner/_work/sde/sde/.nx/cache/cloud/2406.13.5.hotfix2/index.js:30:1225)
    at async Promise.all (index 0)
...

Package Manager Version

Yarn: 3.5.1 OS: Github Ubuntu Runner

Operating System

[ ] macOS
[X] Linux
[ ] Windows
[ ] Other (Please specify)

Additional Information

No response

aramfe commented 2 months ago

It's currently a CRITICAL issue and makes our runs completely non-functionable.

aramfe commented 2 months ago

After some digging, I suppose it's because the locally generated files DO NOT get delete after a run...

Is there a way to "reset" it on the DTE agent machines and main when the nx cloud run already started in the middle of the execution?

Something like this would help here:

- run: npx nx-cloud start-ci-run
- run: yarn nx run-many -t foo, bar --args="--baz=qux"
- run: |
   npx nx-cloud propagate-command-to-agents -s "yarn clean"
   NX_CLOUD_DISTRIBUTED_EXECUTION=false yarn clean
- run: yarn nx run-many

Would love to hear from you

nrwl / nx