Open StefanLobbenmeierObjego opened 1 month ago
I am currently checking the nx source code for this:
The worker is created here:
function createWorkerPool(numberOfWorkers) {
const res = [];
for (let i = 0; i < numberOfWorkers; ++i) {
res.push(new (require('worker_threads').Worker)((0, path_1.join)(__dirname, './project-graph-worker.js'), {
env: process.env,
}));
}
return res;
}
Looking up the stack we see this:
function buildExplicitDependenciesUsingWorkers(jsPluginConfig, ctx, totalNumOfFilesToProcess, builder) {
const numberOfWorkers = Math.min(totalNumOfFilesToProcess, getNumberOfWorkers());
const bins = splitFilesIntoBins(ctx, totalNumOfFilesToProcess, numberOfWorkers);
const workers = createWorkerPool(numberOfWorkers);
let numberOfExpectedResponses = bins.length;
// ...
function buildExplicitDependencies(jsPluginConfig, ctx, builder) {
let totalNumOfFilesToProcess = totalNumberOfFilesToProcess(ctx);
// using workers has an overhead, so we only do it when the number of
// files we need to process is >= 100 and there are more than 2 CPUs
// to be able to use at least 2 workers (1 worker per CPU and
// 1 CPU for the main thread)
if (totalNumOfFilesToProcess < 100 || getNumberOfWorkers() <= 2) {
return buildExplicitDependenciesWithoutWorkers(jsPluginConfig, ctx, builder);
}
else {
return buildExplicitDependenciesUsingWorkers(jsPluginConfig, ctx, totalNumOfFilesToProcess, builder);
}
}
it seems when there are 2 or less threads, nx does not use workers, so it never runs into that path.
Because the file is an external file, it seems webpack is not aware of this. This is the resulting .js after compilation:
const numberOfWorkers = Math.min(totalNumOfFilesToProcess, getNumberOfWorkers()), bins = function(ctx, totalNumOfFilesToProcess, numberOfWorkers) {
const filesPerBin = Math.round(totalNumOfFilesToProcess / numberOfWorkers / 5) + 1, bins = [];
let currentProjectFileMap = {}, currentNumberOfFiles = 0;
for (const source of Object.keys(ctx.filesToProcess)) for (const f of Object.values(ctx.filesToProcess[source])) currentProjectFileMap[source] || (currentProjectFileMap[source] = []), currentProjectFileMap[source].push(f), currentNumberOfFiles++, currentNumberOfFiles >= filesPerBin && (bins.push(currentProjectFileMap), currentProjectFileMap = {}, currentNumberOfFiles = 0);
return bins.push(currentProjectFileMap), bins;
}(ctx, totalNumOfFilesToProcess, numberOfWorkers), workers = function(numberOfWorkers) {
const res = [];
for (let i = 0; i < numberOfWorkers; ++i) res.push(new (__webpack_require__(71267).Worker)((0, path_1.join)(__dirname, "./project-graph-worker.js"), { env: process.env }));
return res;
}(numberOfWorkers);
So you can see here that still the name of the file is used. Not sure if there is a webpack setting to correct this. Or maybe we can just turn off nx workers with some other option.
The project-graph-worker.js was removed anyway in 16.8.0, see this PR: https://github.com/nrwl/nx/pull/18752, mentioned in https://github.com/nrwl/nx/releases/tag/16.8.0
So this might also go away by upgrading the bundled nx version.
For now as a workaround, I added the env variable to limit the nx workers for the project graph using NX_PROJECT_GRAPH_MAX_WORKERS: '2'
:
- name: Execute
uses: e-square-io/nx-distributed-task@v2
id: execute
env:
NX_PROJECT_GRAPH_MAX_WORKERS: '2'
with:
target: ${{ matrix.target }}
projects: ${{ matrix.projects }}
uploadOutputs: true
nxCloud: false
args: ''
Hi @StefanLobbenmeierObjego, thank you for reporting the issue. Unfortunately we're not maintaining the repo anymore and I strongly suggest to look for alternatives, one that I can recommend is nx-aws-cache
I see, thank you for your suggestion, will look into that.
I'm submitting a...
Current behavior
When starting on ubuntu-latest-4-cores, the nx-distributed-task step fails with this error:
Cannot find module '/home/runner/work/_actions/e-square-io/nx-distributed-task/v2/project-graph-worker.js'
Expected behavior
No error, just like on ubuntu-latest
Minimal reproduction of the problem with instructions
What is the motivation / use case for changing the behavior?
The nx project I use has one big project and many small ones. I would like to run the big project on a bigger runner to speed up CI. So I use 4 cores for that one, but unfortunately this seems to be incompatible with this project
Environment
Logs