Federated Traces: Possible EventEmitter memory leak detected.

maurocolella commented 4 years ago

apollo-server-core@^2.18.2

https://github.com/apollographql/apollo-server/blob/main/packages/apollo-server-core/src/ApolloServer.ts#L358

Background: we are using apollo federation/ typescript to run multiple services on express.

"@apollo/federation": "^0.16.0"
"@apollo/gateway": "^0.16.0"
"apollo-server-express": "^2.13.1",
"apollo-server-types": "^0.4.0",

Typescript requested version is 3.9.2 (actual 3.9.3 to 3.9.7).

Expected behavior: no leak.

Actual behavior: At startup, the application reports possible memory leaks.

I am not sure how to create a "simple" test case, as the application is large. I did my best to trace things, see:

yarn run v1.22.5
$ ts-node-dev --inspect -r tsconfig-paths/register --respawn src/gateway/index.ts
ts-node-dev ver. 1.0.0-pre.63 (using ts-node ver. 8.10.2, typescript ver. 3.9.7)
Debugger listening on ws://127.0.0.1:9229/37d49f58-21ef-4951-aa19-c6b7de71e5fd
For help, see: https://nodejs.org/en/docs/inspector
Enabling inline tracing for this federated service. To disable, use ApolloServerPluginInlineTraceDisabled.
Enabling inline tracing for this federated service. To disable, use ApolloServerPluginInlineTraceDisabled.
Enabling inline tracing for this federated service. To disable, use ApolloServerPluginInlineTraceDisabled.
Enabling inline tracing for this federated service. To disable, use ApolloServerPluginInlineTraceDisabled.
Enabling inline tracing for this federated service. To disable, use ApolloServerPluginInlineTraceDisabled.
Enabling inline tracing for this federated service. To disable, use ApolloServerPluginInlineTraceDisabled.
Enabling inline tracing for this federated service. To disable, use ApolloServerPluginInlineTraceDisabled.
Enabling inline tracing for this federated service. To disable, use ApolloServerPluginInlineTraceDisabled.
Enabling inline tracing for this federated service. To disable, use ApolloServerPluginInlineTraceDisabled.
Enabling inline tracing for this federated service. To disable, use ApolloServerPluginInlineTraceDisabled.
Enabling inline tracing for this federated service. To disable, use ApolloServerPluginInlineTraceDisabled.
Enabling inline tracing for this federated service. To disable, use ApolloServerPluginInlineTraceDisabled.
Enabling inline tracing for this federated service. To disable, use ApolloServerPluginInlineTraceDisabled.
Enabling inline tracing for this federated service. To disable, use ApolloServerPluginInlineTraceDisabled.
Enabling inline tracing for this federated service. To disable, use ApolloServerPluginInlineTraceDisabled.
Enabling inline tracing for this federated service. To disable, use ApolloServerPluginInlineTraceDisabled.
Enabling inline tracing for this federated service. To disable, use ApolloServerPluginInlineTraceDisabled.
Enabling inline tracing for this federated service. To disable, use ApolloServerPluginInlineTraceDisabled.
Allowing requests from: http://localhost:8000
Allowing requests from: http://localhost:3000
Allowing requests from: http://localhost:4000
(node:18568) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGTERM listeners added to [process]. Use emitter.setMaxListeners() to increase limit
MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGTERM listeners added to [process]. Use emitter.setMaxListeners() to increase limit
    at _addListener (events.js:389:17)
    at process.addListener (events.js:405:10)
    at process.once (events.js:436:8)
    at D:\dev\<app>\data-warehouse\node_modules\apollo-server-core\src\ApolloServer.ts:366:17
    at Array.forEach (<anonymous>)
    at new ApolloServerBase (D:\dev\<app>\data-warehouse\node_modules\apollo-server-core\src\ApolloServer.ts:359:15)
    at new ApolloServer (D:\dev\<app>\data-warehouse\node_modules\apollo-server-express\src\ApolloServer.ts:88:5)
    at Object.exports.createServer (D:\dev\<app>\data-warehouse\src\utils\createServer.ts:15:92)
    at Component.get server [as server] (D:\dev\<app>\data-warehouse\src\utils\Component.ts:37:12)
    at ComponentGraphQlDataSource.process (D:\dev\<app>\data-warehouse\src\gateway\ComponentGraphQlDataSource.ts:20:25)
    at D:\dev\<app>\data-warehouse\node_modules\@apollo\gateway\src\loadServicesFromRemoteEndpoint.ts:45:8
    at Array.map (<anonymous>)
    at Object.getServiceDefinitionsFromRemoteEndpoint (D:\dev\<app>\data-warehouse\node_modules\@apollo\gateway\src\loadServicesFromRemoteEndpoint.ts:29:44)
    at ApolloGateway.loadServiceDefinitions (D:\dev\<app>\data-warehouse\node_modules\@apollo\gateway\src\index.ts:610:14)
    at ApolloGateway.updateComposition (D:\dev\<app>\data-warehouse\node_modules\@apollo\gateway\src\index.ts:340:27)
    at ApolloGateway.load (D:\dev\<app>\data-warehouse\node_modules\@apollo\gateway\src\index.ts:303:16)
(node:18568) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGINT listeners added to [process]. Use emitter.setMaxListeners() to increase limit
MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGINT listeners added to [process]. Use emitter.setMaxListeners() to increase limit
    at _addListener (events.js:389:17)
    at process.addListener (events.js:405:10)
    at process.once (events.js:436:8)
    at D:\dev\<app>\data-warehouse\node_modules\apollo-server-core\src\ApolloServer.ts:366:17
    at Array.forEach (<anonymous>)
    at new ApolloServerBase (D:\dev\<app>\data-warehouse\node_modules\apollo-server-core\src\ApolloServer.ts:359:15)
    at new ApolloServer (D:\dev\<app>\data-warehouse\node_modules\apollo-server-express\src\ApolloServer.ts:88:5)
    at Object.exports.createServer (D:\dev\<app>\data-warehouse\src\utils\createServer.ts:15:92)
    at Component.get server [as server] (D:\dev\<app>\data-warehouse\src\utils\Component.ts:37:12)
    at ComponentGraphQlDataSource.process (D:\dev\<app>\data-warehouse\src\gateway\ComponentGraphQlDataSource.ts:20:25)
    at D:\dev\<app>\data-warehouse\node_modules\@apollo\gateway\src\loadServicesFromRemoteEndpoint.ts:45:8
    at Array.map (<anonymous>)
    at Object.getServiceDefinitionsFromRemoteEndpoint (D:\dev\<app>\data-warehouse\node_modules\@apollo\gateway\src\loadServicesFromRemoteEndpoint.ts:29:44)
    at ApolloGateway.loadServiceDefinitions (D:\dev\<app>\data-warehouse\node_modules\@apollo\gateway\src\index.ts:610:14)
    at ApolloGateway.updateComposition (D:\dev\<app>\data-warehouse\node_modules\@apollo\gateway\src\index.ts:340:27)
    at ApolloGateway.load (D:\dev\<app>\data-warehouse\node_modules\@apollo\gateway\src\index.ts:303:16)
� Server ready at http://localhost:4000/graphql

It appears that the issue originates in: https://github.com/apollographql/apollo-server/blob/main/packages/apollo-server-core/src/ApolloServer.ts#L358

Thank you for your help with this.

maurocolella commented 4 years ago

In our case: I worked around this by allowing the correct amount of listeners manually, according to the logic hereafter. Kindly note: I have verified all the dependencies, and I believe that they are consistent across - we're not using any "legacy" middleware, or skirting Apollo lifecycle events in any "visible" way.

our server building code initializes an ApolloGateway from the @apollo/gateway module, version ^0.21.0 at the time of this writing.
it passes in a service list, serviceList: [...componentServices]
Before that, I have allowed componentServices 2 listeners to be registered, `process.setMaxListeners(componentServices.length 2)<- x2 forSIGINTandSIGTERM` with each service.
the gateway is then passed to an ApolloServer, (apollo-server-express v. ^2.18.2)
```
export const server = new ApolloServer({
gateway,
context,
...
```
This is then applied as middleware to our express server via
```
server.applyMiddleware({
app,
cors: cors(),
})
```

And so: using the manual limit, I can get rid of the error message above.

pmirandaarias commented 3 years ago

@maurocolella any news with this? My case is very simple. I'm trying to console.log the output of a DocumentClient.getInstance() .get() method, and then in cloudwatch I got that same error than you:

undefined INFO Enabling inline tracing for this federated service. To disable, use ApolloServerPluginInlineTraceDisabled

No idea what is happening. If I remove that console.log everything goes ok.

The function doesn't get corrupted, it all works ok, but I can't do a console.log, I don't get it.

Plus, I hade to downgrade Apollo Federation and Gateway today, from latest (both) to 19.0 (both), I was having another issue reported here https://github.com/apollographql/federation/issues/211

maurocolella commented 3 years ago

@pmirandaarias this is just a warning. I think inline tracing is enabled automatically if an Apollo Studio key is discovered in the environment.

Should you encounter the issue above (memory leak), my second post also discusses the solution.

maurocolella commented 3 years ago

For the record: I am seeing an adjacent issue, with the same warning, due to the introspection requests fired by the graphql playground.

I think eventemitters should probably be disposed of asap; alternatively, might I suggest an implementation like this? https://www.npmjs.com/package/eventemitter3

If it works, it could potentially help to resolve the issue.

apollographql / federation

Federated Traces: Possible EventEmitter memory leak detected. #417