search

backstage / backstage

Backstage is an open framework for building developer portals
https://backstage.io/
Apache License 2.0
26.61k stars 5.46k forks source link

🐛 Bug Report: MaxListenersExceededWarning from TechDocs #24583

Open drodil opened 2 weeks ago

drodil commented 2 weeks ago

📜 Description

Error and possible memory leak detected in the plugin-techdocs-backend. This has started to happen after updating from 1.25.2 to 1.26.4.

MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. Use emitter.setMaxListeners() to increase limit

We are using the AWS S3 Publisher for TechDocs in production but local for local development, and it's failing there as well.

👍 Expected behavior

No error

👎 Actual Behavior with Screenshots

[0] (node:93627) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. Use emitter.setMaxListeners() to increase limit
[0]     at genericNodeError (node:internal/errors:984:15)
[0]     at wrappedFn (node:internal/errors:538:14)
[0]     at _addListener (node:events:592:17)
[0]     at Socket.addListener (node:events:610:10)
[0]     at Readable.on (node:internal/streams/readable:1126:35)
[0]     at Socket.socketListenerWrap [as on] (node:_http_server:1188:54)
[0]     at createEventStream (/backstage/node_modules/@backstage/plugin-techdocs-backend/src/service/router.ts:346:15)
[0]     at <anonymous> (/backstage/node_modules/@backstage/plugin-techdocs-backend/src/service/router.ts:243:55)
[0]     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

👟 Reproduction steps

Just run the backend for a while and click around the UI. Good to run with to get the backtrace:

export NODE_OPTIONS='--trace-warnings'

📃 Provide the context for the Bug.

No response

🖥️ Your Environment

OS: Darwin 23.4.0 - darwin/arm64 node: v18.19.0 yarn: 1.22.19 cli: 0.26.4 (installed) backstage: 1.26.4

Dependencies: @backstage/app-defaults 1.5.4 @backstage/backend-app-api 0.6.2, 0.7.1 @backstage/backend-common 0.21.7 @backstage/backend-defaults 0.2.17 @backstage/backend-dev-utils 0.1.4 @backstage/backend-openapi-utils 0.1.10 @backstage/backend-plugin-api 0.6.17 @backstage/backend-tasks 0.5.22 @backstage/backend-test-utils 0.3.7 @backstage/catalog-client 1.6.4 @backstage/catalog-model 1.4.5 @backstage/cli-common 0.1.13 @backstage/cli-node 0.2.5 @backstage/cli 0.26.4 @backstage/config-loader 1.8.0 @backstage/config 1.2.0 @backstage/core-app-api 1.12.4 @backstage/core-compat-api 0.2.4 @backstage/core-components 0.14.5 @backstage/core-plugin-api 1.9.2 @backstage/dev-utils 1.0.31 @backstage/e2e-test-utils 0.1.1 @backstage/errors 1.2.4 @backstage/eslint-plugin 0.1.7 @backstage/frontend-plugin-api 0.6.4 @backstage/integration-aws-node 0.1.12 @backstage/integration-react 1.1.26 @backstage/integration 1.10.0 @backstage/plugin-api-docs 0.11.4 @backstage/plugin-app-backend 0.3.65 @backstage/plugin-app-node 0.1.17 @backstage/plugin-auth-backend-module-atlassian-provider 0.1.9 @backstage/plugin-auth-backend-module-aws-alb-provider 0.1.9 @backstage/plugin-auth-backend-module-azure-easyauth-provider 0.1.0 @backstage/plugin-auth-backend-module-bitbucket-provider 0.1.0 @backstage/plugin-auth-backend-module-cloudflare-access-provider 0.1.0 @backstage/plugin-auth-backend-module-gcp-iap-provider 0.2.12 @backstage/plugin-auth-backend-module-github-provider 0.1.14 @backstage/plugin-auth-backend-module-gitlab-provider 0.1.14 @backstage/plugin-auth-backend-module-google-provider 0.1.14 @backstage/plugin-auth-backend-module-guest-provider 0.1.3 @backstage/plugin-auth-backend-module-microsoft-provider 0.1.12 @backstage/plugin-auth-backend-module-oauth2-provider 0.1.14 @backstage/plugin-auth-backend-module-oauth2-proxy-provider 0.1.10 @backstage/plugin-auth-backend-module-oidc-provider 0.1.8 @backstage/plugin-auth-backend-module-okta-provider 0.0.10 @backstage/plugin-auth-backend 0.22.4 @backstage/plugin-auth-node 0.4.12 @backstage/plugin-auth-react 0.1.1 @backstage/plugin-catalog-backend-module-aws 0.3.12 @backstage/plugin-catalog-backend-module-msgraph 0.5.25 @backstage/plugin-catalog-backend-module-scaffolder-entity-model 0.1.15 @backstage/plugin-catalog-backend-module-unprocessed 0.4.4 @backstage/plugin-catalog-backend 1.21.1 @backstage/plugin-catalog-common 1.0.22 @backstage/plugin-catalog-graph 0.4.4 @backstage/plugin-catalog-node 1.11.1 @backstage/plugin-catalog-react 1.11.3 @backstage/plugin-catalog-unprocessed-entities-common 0.0.1 @backstage/plugin-catalog-unprocessed-entities 0.2.3 @backstage/plugin-catalog 1.19.0 @backstage/plugin-devtools-backend 0.3.3 @backstage/plugin-devtools-common 0.1.9 @backstage/plugin-devtools 0.1.13 @backstage/plugin-events-backend 0.3.4 @backstage/plugin-events-node 0.3.3 @backstage/plugin-home-react 0.1.12 @backstage/plugin-home 0.7.3 @backstage/plugin-kubernetes-common 0.7.5 @backstage/plugin-notifications-backend 0.2.0 @backstage/plugin-notifications-common 0.0.3 @backstage/plugin-notifications-node 0.1.3 @backstage/plugin-notifications 0.2.0 @backstage/plugin-org 0.6.24 @backstage/plugin-permission-backend 0.5.41 @backstage/plugin-permission-common 0.7.13 @backstage/plugin-permission-node 0.7.28 @backstage/plugin-permission-react 0.4.22 @backstage/plugin-proxy-backend 0.4.15 @backstage/plugin-scaffolder-backend-module-azure 0.1.9 @backstage/plugin-scaffolder-backend-module-bitbucket-cloud 0.1.7 @backstage/plugin-scaffolder-backend-module-bitbucket-server 0.1.7 @backstage/plugin-scaffolder-backend-module-bitbucket 0.2.7 @backstage/plugin-scaffolder-backend-module-confluence-to-markdown 0.2.18 @backstage/plugin-scaffolder-backend-module-cookiecutter 0.2.41 @backstage/plugin-scaffolder-backend-module-gerrit 0.1.9 @backstage/plugin-scaffolder-backend-module-gitea 0.1.7 @backstage/plugin-scaffolder-backend-module-github 0.2.7 @backstage/plugin-scaffolder-backend-module-gitlab 0.3.3 @backstage/plugin-scaffolder-backend 1.22.4 @backstage/plugin-scaffolder-common 1.5.1 @backstage/plugin-scaffolder-node-test-utils 0.1.3 @backstage/plugin-scaffolder-node 0.4.3 @backstage/plugin-scaffolder-react 1.8.4 @backstage/plugin-scaffolder 1.19.3 @backstage/plugin-search-backend-module-catalog 0.1.23 @backstage/plugin-search-backend-module-elasticsearch 1.4.0 @backstage/plugin-search-backend-module-techdocs 0.1.22 @backstage/plugin-search-backend-node 1.2.21 @backstage/plugin-search-backend 1.5.7 @backstage/plugin-search-common 1.2.11 @backstage/plugin-search-react 1.7.10 @backstage/plugin-search 1.4.10 @backstage/plugin-signals-backend 0.1.3 @backstage/plugin-signals-node 0.1.3 @backstage/plugin-signals-react 0.0.3 @backstage/plugin-signals 0.0.5 @backstage/plugin-techdocs-backend 1.10.4 @backstage/plugin-techdocs-node 1.12.3 @backstage/plugin-techdocs-react 1.2.3 @backstage/plugin-techdocs 1.10.4 @backstage/plugin-user-settings-backend 0.2.16 @backstage/plugin-user-settings 0.8.5 @backstage/release-manifests 0.0.11 @backstage/repo-tools 0.8.0 @backstage/test-utils 1.5.4 @backstage/theme 0.5.3 @backstage/types 1.1.1 @backstage/version-bridge 1.0.8

👀 Have you spent some time to check if this bug has been raised before?

🏢 Have you read the Code of Conduct?

Are you willing to submit PR?

None

drodil commented 2 weeks ago

Might be related: #20698 and #20700

freben commented 2 weeks ago

There's a couple of obvious close listeners being added:

When creating event streams: https://github.com/backstage/backstage/blob/32cc4e64681bea7cbc27f141bae8dd714031c5e7/plugins/techdocs-backend/src/service/router.ts#L346

In the cache middleware: https://github.com/backstage/backstage/blob/32cc4e64681bea7cbc27f141bae8dd714031c5e7/plugins/techdocs-backend/src/cache/cacheMiddleware.ts#L70

Just noting quickly, haven't dug deep. In the traceback, the first one seems to be the culprit. I guess it gets called several times for a given socket?

drodil commented 2 weeks ago

@freben it seems to be related to the buildLogTransport someway, described very well in the #20968. We forked the techdocs backend plugin completely to be able to override this (until we have support for #24587 or some other proper fix) with the following:

 const buildLogTransport = {
          pipe: () => {},
          log: () => {},
          on: () => {},
          emit: () => {},
          write: () => {},
          once: () => {},
          _writableState: {
            objectMode: true,
          },
        };

After adding that to the router options, the error is now gone and memory usage seems to be stable. Probably this needs a bit closer look why is this adding so many listeners to the socket..

Additionally this is related to new backend system update we did quite lately and we had this workaround already in use in the old backend.

freben commented 2 weeks ago

I'm a bit surprised then - in the current iteration of the code after #20700, no fallback is made if buildLogTransport is not passed in. But the stream transport IS, no matter what. Does this keep happening if you do NOT specify a buildLogTransport at all?

drodil commented 2 weeks ago

Yeah it does. Passing this custom transport seems the only way to fix this