znewton
commented
3 weeks ago Since the implementation of readBlob there appropriately uses async/await, I actually doubt that the issue is within readBlob itself. We would expect readBlob to throw its errors, and catching those without throwing would possibly cause problems in some areas that should be retrying at a higher level.
Instead, I suspect the problem is within PrefetchDocumentStorageService, which attempts to pre-load the snapshot as soon as the version is requested, potentially saving a couple seconds on load. Within PrefetchDocumentStorageService, there are calls to readBlob that use the void prefix, like so:
private prefetchTree(tree: ISnapshotTree) {
const secondary: string[] = [];
this.prefetchTreeCore(tree, secondary);
for (const blob of secondary) {
// We don't care if the prefetch succeeds
void this.cachedRead(blob);
}
}While the comment is correct, we don't care if the prefetch succeeds, an error there would throw an UncaughtException.
Proof Script:
/**
* Creates a rejected promise.
*/
async function throwErrorInPromise(id: string): Promise<never> {
throw new Error(`Error - ${id}`);
}
(async function main(): Promise<void> {
process.on("uncaughtException", (error: Error) => {
// Log, instead of die, on uncaught exceptions.
console.error(`Uncaught exception: ${error.message}`);
});
process.on("unhandledRejection", (error: Error) => {
// Log, instead of die, on uncaught rejections.
console.error(`Unhandled rejection: ${error.message}`);
});
// This should result in an Unhandled Rejection
// which will be caught by the process.on("unhandledRejection") handler
// and logged as "Unhandled rejection: Error - void call"
void throwErrorInPromise("void call");
// This should just log the error as "Caught error: Error - catch call"
throwErrorInPromise("catch call").catch((error: Error) => {
console.error(`Caught error: ${error.message}`);
});
// This should just log the error as "Try..catch error: Error - await call"
try {
await throwErrorInPromise("await call");
} catch (error: unknown) {
console.error(
`Try..catch error: ${error instanceof Error ? error.message : error}`,
);
}
})();Output
Caught error: Error - catch call
Try..catch error: Error - await call
Unhandled rejection: Error - void call
Describe the bug
At certain condition when there is a network issue between fluid client and Routelicious, It result in unhandled promise and as a result node process exits.
Exception that occur is ( GenericNetworkError(errorMessage, errorMessage.startsWith("NetworkError")) 2024-07-31T13:19:35.839Z 377aa069-abf3-4366-8300-be03a126d813 ERROR Unhandled Promise Rejection { "errorType": "Runtime.UnhandledPromiseRejection", "errorMessage": "Error: R11s fetch error: \r\n
502 Bad Gateway
502 Bad Gateway
502 Bad Gateway
502 Bad Gateway
To Reproduce
I am able to reproduce the error while using wifi as well i stop the wifi during the collaboration and then when fluid client is trying to create Steps to reproduce the behavior: I am able to reproduce though it require quite a bit of attempt, when i turn of wifi in mid of collaboration then also some times I am able to see this error Its very random I have observed 1 or 2 such issue on daily basis. Since one of the user is running fluid client on top of lambda as bot, unhandle rejection is causing it to crash. restWrapper.mjs:101 Uncaught (in promise) Error: NetworkError: Network request failed at restWrapper.mjs:101:1 at async VM41070 main.03c9eba.4219a8761b37ae28a6d6.js:207786:28 at async RouterliciousStorageRestWrapper.request (VM41070 main.03c9eba.4219a8761b37ae28a6d6.js:207784:21) at async VM41070 main.03c9eba.4219a8761b37ae28a6d6.js:208204:31 at async PerformanceEvent.timedExecAsync (VM41070 main.03c9eba.4219a8761b37ae28a6d6.js:48395:25) at async ShreddedSummaryDocumentStorageService.readBlob (VM41070 main.03c9eba.4219a8761b37ae28a6d6.js:208199:23)
Expected behavior
when I dig little bit I notice that catch is missing in ShreddedSummaryDocumentStorageService.readBlob function
public async readBlob(blobId: string): Promise {
const cachedBlob = await this.blobCache?.get(this.getCacheKey(blobId));
if (cachedBlob) {
return cachedBlob;
}
Logs