Closed erocheleau closed 9 months ago
github-actions[bot]
commented
1 year ago Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.
erocheleau
commented
1 year ago If I set some retries in the test run, the same commands that fail eventually work.
I thought it may have been a problem with having multiple concurrent workers trying to read the same file maybe? So I lowered the number of workers to 1, but it still doesn't work after that.
I tried looking at the code in the stack trace, it's this line that's always where the fail happens:
/**
* Read the config file and set the config contents. Returns the config contents of the config file. As an
* optimization, files are only read once per process and updated in memory and via `write()`. To force
* a read from the filesystem pass `force=true`.
* **Throws** *{@link SfError}{ name: 'UnexpectedJsonFileFormat' }* There was a problem reading or parsing the file.
*
* @param [throwOnNotFound = false] Optionally indicate if a throw should occur on file read.
* @param [force = false] Optionally force the file to be read from disk even when already read within the process.
*/
async read(throwOnNotFound = false, force = false) {
...
const obj = (0, kit_1.parseJsonMap)(await fs.promises.readFile(this.getPath(), 'utf8'), this.getPath());I tried printing in the console what was causing the JSONParseError when it would happen but the output doesn't make sense to me to be able to debug it, it seems it's just trying to parse regular valid json...
shetzel
commented
1 year ago It's an issue with concurrency of config files. We have it on our radar to fix but it's not a small change and we'll need to do some planning for how to roll it out.
mshanemc
commented
11 months ago @erocheleau that warning indicates that the file isn't good json.
Like @shetzel said, it's probably getting corrupted at some point, probably during your authorization or scratch org creation process. Then when your data or open command goes to use it, it's bad.
If you've got any parallel work that might be touching that file, try to eliminate that.
mshanemc
commented
11 months ago @W-14085765@
erocheleau
commented
11 months ago @shetzel wow thank you for confirming this at least I know I'm not crazy and I pinpointed the issue correctly š
@mshanemc The parallel work is the point of what I'm trying to do, for example to run e2e tests with multiple workers in a CI pipeline, running queries to Salesforce using the CLI.
So IMO the concurrency is an issue that should be fixed. But I understand that it's not a small fix and I can work around it now that I know it's confirmed that this is the real cause of the issue.
It's funny that it worked without any issue on windows + wsl but on Mac it doesn't work even once.
Thanks the replies š
mshanemc
commented
11 months ago yeah, there's some ways to deal with that. We massively parallelize e2e tests on orgs by giving them their own isolated directories (see https://github.com/salesforcecli/cli-plugins-testkit).
It's funny that it worked without any issue on windows + wsl but on Mac it doesn't work even once.
one theory is that it relates to the keychain stuff on mac. You could experiment with SF_USE_GENERIC_UNIX_KEYCHAIN on mac to see if that helps.
erocheleau
commented
10 months ago To me it really looks like it's because most commands will constantly re-write the json config file for your scratch org instead of leaving it alone? (I could be totally wrong here) But I can log every time there is this error, and every time right now it's when trying to read this file:
/Users/<username>/.sfdx/<scratchorg>@example.com.json
Trying to print the content of the file on error returns completely empty. (it could be because the file cannot be opened to read too I guess because something else is still writing in it?)
'ERROR Reading config file: /Users/<username>/.sfdx/<scratchorg>@example.com.json\n' +
'ERROR Reading config file: <- file content.\n' +
'{\n' +
' "code": 1,\n' +
' "context": "OrgOpenCommand",\n' +
' "commandName": "OrgOpenCommand",\n' +
' "message": "Parsing --target-org \\n\\tUnexpected end of JSON input\\nSee more help with --help",\n' +
' "name": "JsonParseError",\n' +
[...]I added the <- file content after logging the content of the file read, to make sure it wasn't just an empty line break or something :)
I changed the code of the read method of configFile.js to
fileContent = await fs.promises.readFile(this.getPath(), 'utf8');
const obj = (0, kit_1.parseJsonMap)(fileContent, this.getPath());
[...]
catch(err) {
console.log(`ERROR Reading config file: ${this.getPath()}`);
console.log(`ERROR Reading config file: ${fileContent} <- file content.`);
}And it completely prints as empty every time this error shows up. (I still have the error show up now with even just 1 working while running playwright tests.
erocheleau
commented
10 months ago async write(newContents) {
if (newContents) {
this.setContents(newContents);
}
try {
await fs.promises.mkdir((0, path_1.dirname)(this.getPath()), { recursive: true });
}
catch (err) {
throw sfError_1.SfError.wrap(err);
}
this.logger.info(`Writing to config file: ${this.getPath()}`);
// await fs.promises.writeFile(this.getPath(), JSON.stringify(this.toObject(), null, 2));
return this.getContents();
}It really seems to be tied to constantly re-writing the same config file, because if I comment out the line showcased here, I stop having issues altogether with the JSON parsing of the file...
Also from the node.fs documentation on this function:
- It is unsafe to use
fsPromises.writeFile()multiple times on the same file- without waiting for the promise to be settled.
- Similarly to
fsPromises.readFile-fsPromises.writeFileis a convenience- method that performs multiple
writecalls internally to write the buffer- passed to it. For performance sensitive code consider using
fs.createWriteStream()orfilehandle.createWriteStream().
I would assume this is exactly what is going on here, multiple calls to sfdx methods executed at the same time (by the test runners, even with 1 runner I am encountering this issue) would all try to call fsPromises.readFile and fsPromises.writeFile() at the same time.
It also looks like the content of the file is being re-written with the exact same content, not re-writing the same content could potentially help???, but I could be wrong it's hard to check what's going on during the execution of these commands.
Hopefully this information is helpful to try to resolve the issue? :)
mshanemc
commented
9 months ago @erocheleau we shipped a really large change to how configFiles work in today's nightly. Would you be able to try your replication with the nightly sf and see if it fixed it?
erocheleau
commented
7 months ago @mshanemc I completely missed this update but just saw the version description of 2.19.7 thank you so much!
I will try it right away!
erocheleau
commented
6 months ago I can confirm it seems to be working now!
MarcTCruz
commented
5 months ago @mshanemc Last week I tried some "sf project deploy start --source-dir ... --json" in parallel and I've got the jsonParseError.
My sf version is usually up to date, last week it was at least @salesforce/cli/2.31.x, today it is @salesforce/cli/2.32.8.
So such errors still happens.
mshanemc
commented
5 months ago @MarcTCruz can you open a new issue with all the info about it. Can you also include what kind of org you're deploying to?
It may or may not be the same file causing the problem.
Summary
I am using the
@salesforce/cliinstalled through npm. I am using it through a nodejs application (automated tests with playwright, and using the nodeexeccommand in the test setup phase) Everything is running smoothly on windows (using wsl2) and on CI (using ubuntu runners). But on a Macbook with a M2 chip (dunno if it matters), I am experiencing very flaky commands. More specifically, when runningsf data record get --sobject Case --where "where clause" --target-org random_alias --jsonI am randomly getting this error:I am getting the same error:
When running also
sf data get record --target-org random_alias --sobject Case --where "where clause" --jsonAnd also:sf org open -p "path/to/record" --urlonly --target-org random_alias --jsonExpected result
I expect these commands to not fail?
Actual result
These commands randomly fail
System Information
Additional information