Closed nkadis closed 1 year ago
Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.
I also received this error. However, I also get it when running sfdx force:source:tracking:reset
. The cli was recently upgraded from 7.143 to 7.161. After doing this, I went to follow the instructions of migrating to use the newer commands, by running legacy:push, and then clearing legacy tracking. After this, I attempt to run tracking:reset and get the error as described above.
Can confirm that the error also occurs with sfdx force:source:tracking:reset
as @nBray-tLP described.
Glad this is not just me, can confirm I also receive this error. Package contains a total of 5000 components.
These pushes are to a scratch org, if that helps at all. Have attempted to move scratch org, re-install VS code, cli, git ETC.
"code": "NotFoundError", "data": { "what": "force-app/main/default/workflows/Bookingc.workflow-meta.xml" }, "caller": "", "message": "Could not find force-app/main/default/workflows/Bookingc.workflow-meta.xml.", "stack": "NotFoundError: Could not find force-app/main/default/workflows/Bookingc.workflow-meta.xml.\n at /usr/local/lib/sfdx/node_modules/isomorphic-git/index.cjs:4518:34" }, { "code": "NotFoundError", "data": { "what": "force-app/main/default/workflows/List_Outcomec.workflow-meta.xml" }, "caller": "", "message": "Could not find force-app/main/default/workflows/List_Outcome__c.workflow-meta.xml.", "stack": "NotFoundError: Could not find force-app/main/default/workflows/List_Outcome__c.workflow-meta.xml.\n at /usr/local/lib/sfdx/node_modules/isomorphic-git/index.cjs:4518:34" } ], "stack": "MultipleGit: There are multiple errors that were thrown by the method. Please refer to the \"errors\" property to see more\n at ShadowRepo.commitChanges (/usr/local/lib/sfdx/node_modules/@salesforce/source-tracking/lib/shared/localShadowRepo.js:194:39)\n at async SourceTracking.updateLocalTracking (/usr/local/lib/sfdx/node_modules/@salesforce/source-tracking/lib/sourceTracking.js:267:9)\n at async Promise.all (index 0)\n at async updateTracking (/usr/local/lib/sfdx/node_modules/@salesforce/plugin-source/lib/trackingFunctions.js:74:5)\n at async Push.updateTracking (/usr/local/lib/sfdx/node_modules/@salesforce/plugin-source/lib/commands/force/source/push.js:102:9)\n at async Push.run (/usr/local/lib/sfdx/node_modules/@salesforce/plugin-source/lib/commands/force/source/push.js:31:9)\n at async Push._run (/usr/local/lib/sfdx/node_modules/@salesforce/command/lib/sfdxCommand.js:102:40)\n at async Config.runCommand (/usr/local/lib/sfdx/node_modules/@oclif/core/lib/config/config.js:272:25)\n at async run (/usr/local/lib/sfdx/node_modules/@oclif/core/lib/main.js:76:5)", "warnings": [], "commandName": "Push" }
Hi @nkadis @booknownick @nBray-tLP sorry for the delayed response.
I'm assuming this issue is still happening. As a workaround you can try the force:source:deploy
command with --tracksource
which should be equivalent to a source:push
. Additionally, if that doesn't work, you can run the source:deploy
command, and then run force:source:tracking:reset
command, to get you in a clean source-tracking state.
Please let me know if either of those work
Hey @WillieRuemmele Can confirm issue is still present, although all of our devs have switched over to using the deploy command for the meantime, as cumbersome as it is! Just to confirm, --tracksource will only push changes made if we use force:source:deploy with the path as 'force-app/main/default' ?
Many thanks!
--tracksource
should update the source tracking for whatever was in that deployment regardless of how you specify it
Okay, so we still have to push individual files in the interim, until we have a fix for this issue
After some fiddling around, I believe we found the issue being caused by forcedotcom/source-tracking/src/shared/localShadowRepo.ts:62.
this.maxFileAdd = this.isWindows ? 8000 : 15000;
This is used to form the chunks/batches to be tracked (using git.add
). On Windows, the max chunk size is set to 8000, but for anything else, its set to 15000. If you change the value to be 8000 for all other platforms too, then the error stops getting thrown.
We've tested this on Mac, so I can't say if this issue also occurs on Linux etc. and we haven't tested if this would work with any other value between 8000 and 15000, but at least we have a temporary fix.
Hello friends! And thank you for your help!
Can you please post the results of the following commands on the offending system:
ulimit -n
ulimit -Sn
ulimit -Hn
For context, we believe our batch size (maxFileAdd
) during source track updates are higher than hard open file limits on some systems.
we are getting the same source-tracking errors, we are on sfdx-cli/7.182.1 linux-x64 node-v16.13.1 and the max file open limit currently is 4096
Thanks @rmathewsbeyond
@cromwellryan was finally able to replicate this issue! Big thanks to @nkadis for determining where this issue was originating!
It appears to be because the number of files that source tracking is batching is exceeding your user specific file limits.
Try one of the following:
ulimit -Hn 20000
prerelease
of plugin-source
with sfdx plugins:install @salesforce/plugin-source@dev
and then set the SFDX_SOURCE_TRACKING_BATCH_SIZE
environment variable to a value lower than the output of ulimit -Hn
ulimit -Hn
or your system will still hit that limit. If you continue to get the error, lower the value of the env var some more.this fix worked, thanks.
Thanks @rmathewsbeyond
@cromwellryan was finally able to replicate this issue! Big thanks to @nkadis for determining where this issue was originating!
It appears to be because the number of files that source tracking is batching is exceeding your user specific file limits.
Try one of the following:
- Increase your hard file limit in the same session with
ulimit -Hn 20000
Or install the
prerelease
ofplugin-source
withsfdx plugins:install @salesforce/plugin-source@dev
and then set theSFDX_SOURCE_TRACKING_BATCH_SIZE
environment variable to a value lower than the output ofulimit -Hn
A few notes:
Don’t set the env var too close to the output of
ulimit -Hn
or your system will still hit that limit. If you continue to get the error, lower the value of the env var some more.We will be releasing these changes in next week's RC
Getting this error Pipelinel echo 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 "status": 1, "result": { "done": true, "errorMessage": "LIMIT_EXCEEDED: Too many files in retrieve call, limit is: 10000", "errorStatusCode": "LIMIT_EXCEEDED" , "status": "Failed", "success": false, "zipFile": { "$":{ "xi:nil": "true" 20:01:30 」 20:01:30 } 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 "MdapiRetrieveCommand", 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 "name" "mdapiRetrieveFailed", "message": "The metadata retrieve operation failed: LIMIT_EXCEEDED: Too many files in retrieve ca. "exitCode": 1, "commandName" : "data": { "done": true, "errorMessage": "LIMIT_EXCEEDED: Too many files in retrieve call, limit is: 10000", "errorStatusCode": "LIMIT_EXCEEDED", "status": "Failed", "success": false, "zipFile": { "$": { "xsi:nil": "true"
Summary
There appears to be an issue with source tracking when deploying a large project (tested with ~13k components). Deployments succeeded after the fix to source deploy for ~9k components, but at ~13k it fails to update source tracking.
Steps To Reproduce:
Repository to reproduce: Can't reproduce in dreamhouse-lwc, but any large project should do (mine pushes 13074 components)
sfdx force:source:push -w 90 --json
to deploy a large component set (wait time added due to amount of components being deployed).Expected result
Deployment completes successfully and source tracking updated.
Actual result
Deployment completes successfully, but source tracking fails.
System Information
Which shell/terminal are you using? (e.g. bash, zsh, powershell 5, powershell 7, cmd.exe, etc.)
zsh
sfdx version --verbose --json
:Additional information
Deployment status succeeded:
Output from command:
Note: truncated result list since all errors are the same (all ~9580 of them) and prettified stacks.