chrisulloa
commented
3 months ago I was able to collect a little bit more information here and was able to recreate the issue locally.
I created a fresh Apollo Gateway using minimal dependencies above. I created a test server that returns the supergraph sdl successfully on first call, then times out on subsequent calls.
let called = false;
app.use("/uplink-test", async (req, res) => {
if (!called) {
called = true;
console.log("Sending valid response");
const data = await fs.readFile(
"/Users/user/test-server/source/supergraph.graphql",
{
encoding: "utf-8",
}
);
res.status(200);
res.send({
data: {
__typename: "Query",
routerConfig: {
__typename: "RouterConfigResult",
id: "123",
supergraphSdl: data,
},
},
});
} else {
console.log("Starting timeout");
await timeout(60000);
res.status(200);
res.send({});
}
});My Apollo gateway starts up successfully on first call, but on subsequent calls it breaks on timeout.
import { fetch } from "undici";
const gateway = new ApolloGateway({
uplinkEndpoints: ["http://localhost:8111/uplink-test"],
debug: true,
fetcher: fetch
});/node_modules/undici/lib/web/fetch/request.js:57
this.removeEventListener('abort', abort)
^
TypeError: this.removeEventListener is not a function
at EventEmitter.abort (/node_modules/undici/lib/web/fetch/request.js:57:12)
at EventEmitter.emit (node:events:519:28)
at EventEmitter.emit (node:domain:488:12)
at AbortSignal.dispatchEvent (/node_modules/node-abort-controller/index.js:28:23)
at AbortController.abort (/node_modules/node-abort-controller/index.js:58:17)
at Timeout._onTimeout (/node_modules/@apollo/gateway/src/supergraphManagers/UplinkSupergraphManager/loadSupergraphSdlFromStorage.ts:130:16)
at listOnTimeout (node:internal/timers:573:17)
at processTimers (node:internal/timers:514:7)When I remove undici the server starts up correctly, and on subsequent calls to the endpoint it times out and does not crash
const gateway = new ApolloGateway({
uplinkEndpoints: ["http://localhost:8111/uplink-test"],
debug: true
});2024-08-01 11:38:01.064 ERROR - UplinkSupergraphManager failed to update supergraph with the following error: An error occurred while fetching your schema from Apollo: The user aborted a request.I think it might be fair to say that Undici is broken as a fetcher for Apollo Gateway due to the removeEventListener missing. If the uplink begins to timeout for whatever reason it will crash the gateway if Undici is used as a fetcher.
squirrelchew
Issue Description
Apollo outage on July 3rd caused issues communicating with uplink, forcing server crash.
We use undici as a primary fetch library for Apollo federation. We are in the process of integrating with Apollo enterprise, in our lower environment our server began to crash during a reported outage. The services recovered but our gateway crashed and impacted some testing in our lower environments. If we were using Apollo in production I would anticipate this would be a major outage for us.
Mainly asking to see if this is a known issue that has occurred for others, or if there's potentially an issue with using undici as a fetcher library.
Node version 18
Looking at the error source this appears to be related to a request timeout, which calls
node-abort-controllerpolyfill which in turn tries to call the undici removeEventListener function which doesn't exist.https://github.com/apollographql/federation/blob/38beaa2f753eaf31be74b272ea62f3c1948c854f/gateway-js/src/supergraphManagers/UplinkSupergraphManager/loadSupergraphSdlFromStorage.ts#L130
https://github.com/southpolesteve/node-abort-controller/blob/10e0cea66a069d9319f948d055621e1d37aea5db/index.js#L16
Link to Reproduction
N/A
Reproduction Steps
Service Dependencies:
Gateway init:
Node version: 18