ECONNRESET when running "many" tests

[schw4rzlicht]

Hey there,

we run into random ECONNRESET problems for weeks now and tried many things to mitigate the problem. We use supertest for e2e testing a GraphQL API built with Nest.js.

We run it like so:

GraphQLQuery.ts:

export class GraphQlQuery {
    private static _bindings: Map<string, supertest.SuperTest<supertest.Test>>;

    static queryGraphQl(app: INestApplication, query: GraphQlQuery, token?: string): supertest.Test {
        if (!GraphQlQuery._bindings) {
            GraphQlQuery._bindings = new Map();
        }

        let binding = GraphQlQuery._bindings.get(app.getHttpServer().testHash);
        if (!binding) {
            binding = request(app.getHttpServer());
            GraphQlQuery._bindings.set(app.getHttpServer().testHash, binding);
        }

        const req = binding.post("/graphql");

        if (token) {
            req.set({ Authorization: `Bearer ${token}` });
        }

        req.send({ operationName: null, query: query.build() })
            .expect((response) => {
                if (!response.ok) {
                    console.error(JSON.stringify(response, null, 2));
                }
            })
            .expect(200);

        return req;
    }

    // ...
}

Somewhere in the tests, we can then do

GraphQlQuery.queryGraphQl(app, query, jwt).then((response) => {
    expect(response).toBe(something);
});

The reason we do all the GraphQLQuery._bindings-stuff is because we are aware that those bindings should be reused. We create the Nest.js app in our beforeAll()-hooks, where we also add the testHash to the HTTP Server. It's basically a UUID to ensure that we can map bindings to test-suites. We also run Jest with --runInBand on the CI, so nothing should run concurrently. We have around 80 tests now which all rely on supertest and the more tests we build, the worse it gets. Up until to a point where we need to run the whole CI-step multiple times until ECONNRESET doesn't happen any more. Sometimes up to 10 times or so, with each pipeline needing ~5 minutes.

Unfortunately I didn't find much information about our problems and since Nest.js recommends supertest (and I think I can say that on behalf of the team, we really like the approach), I am wondering what we're missing.

Any help appreciated!

[jordancalder]

Any update here? I seem to be getting a similar error:

Error: ECONNRESET: Connection reset by peer
      at Test.assert (./node_modules/supertest/lib/test.js:193:18)
      at localAssert (./node_modules/supertest/lib/test.js:159:12)
      at ./node_modules/supertest/lib/test.js:156:5

[Dezzpil]

I wrote test to reproduce problem

describe('a lot of calls', function () {
  this.timeout(10000)

  it('should handle many requests', (done) => {
    const app = express();
    app.get('/', (req, res) => {
      res.send('hi');
    })

    const agent = request(app)

    const reqFn = (i) => {
      return new Promise((resolve, reject) => {
        const st = setTimeout(() => {
          clearTimeout(st)
          const req = agent.get('/')
          req.end((err, res) => {
            if (err) {
              console.log(i, err.message);
              reject(err)
            } else {
              res.text.should.be.eql('hi');
              resolve()
            }
          });
        }, 100)
      })
    }
    const promises = []
    for (let i = 0; i < 200; i++) {
      promises.push(reqFn(i))
    }
    Promise.all(promises)
      .then(() => done())
      .catch(err => done(err))
  });
});

There are may be different output in terminal after executions:

...
160 ECONNREFUSED: Connection refused
159 ECONNREFUSED: Connection refused

or

...
133 ECONNREFUSED: Connection refused
132 ECONNREFUSED: Connection refused

etc.

But i have no idea how to fix this :(

[schw4rzlicht]

@Dezzpil thanks a lot! But I'm not quite sure if that directly relates to our problem, since we only see ECONNRESET and never ECONNREFUSED :/

[lucasraziel]

same error with me. ECONNRESET is showing when running many tests, but It does not show when running the same test individually. In my case, I am not using Nest

[cmwd]

It seems that when calling supertest many times in a concurrent manner this code https://github.com/visionmedia/supertest/blob/28116f9060c1904a433e7d69f1c14db32cf60279/lib/test.js#L61 will spin up multiple HTTP servers or start listening on multiple ports (?).

My solution is to call app.getHttpServer().listen(0); after test app initialization in the beforeAll hook. This way the HTTP server can warm up and all supertest calls use the same address.

So far worked fine for me. Hope that helps in your case as well.

[niftylettuce]

Another issue that @shadowgate15 @spence-s and I found was that port 5000 is being used by AirPlay receiver on macOS Monterey via https://developer.apple.com/forums/thread/682332. I believe that @shadowgate15 is changing tests to use get-port instead to launch port for ZUUL / server.listen.

[simplenotezy]

Good find @niftylettuce - that might be it.

Perhaps we should find a solution to avoid using port 5000, if that is the case?

[simplenotezy]

@niftylettuce I did try to disable AirPlay Receiver, but still getting random "Connection reset by peer"

[shadowgate15]

We need to fix the tests to use beforeEach and afterEach to make sure the app is started on a random port and closes the app after the test.

This issue should be reopened @niftylettuce.

[shadowgate15]

And assign it to me I'll fix it later on this week.

[simplenotezy]

@shadowgate15 I believe the ticket was never reopened; however, still a relevant issue to tackle! 😊

[titanism]

PR welcome

[frbuceta]

This is happening to me with Node v20 only.

Look at the evidence of this PR https://github.com/loopbackio/loopback-next/pull/9751.

What solutions can I adopt?

UPDATE:

It seems that removing the for and using es6 syntax works...

-> FOR TEST v20 https://github.com/loopbackio/loopback-next/actions/runs/5745322969/job/15573163172?pr=9751 -> ES6 TEST v20 https://github.com/loopbackio/loopback-next/actions/runs/5745513122/job/15573621754?pr=9751

[guli-bn]

Also started experiencing this in tests while bumping our app to node v20 with many requests with array.map that returns many request-handlers awaited by a Promise.all(). Worked fine in node 18.

[yelworc]

FWIW, I also noticed this issue with Node 20 earlier. The test that ran into it worked fine with Node 18 so far, up to v18.17.1; as of v18.18.0, it's also breaking there. Not sure if this helps at all, but maybe it's useful for narrowing down the cause? 😄

[gjthompson1]

Same node 20 (20.7.0) causing this difference in behaviour.

[andreainnocenti]

same for us, everything worked fine until node.js v18.17.1, v18.18.0 broke something.

[aercolino-cognizant-cepsa]

Same here. My GitHub action stopped working when Node version passed from 18.17.1 to 18.18.0, always failing in my supertest tests, each with the error read ECONNRESET.

[soccermax]

same problem here with node 18.18.0 and parallel requests.

[Drew-Kimberly]

Same here - bumping node from 18.17.x to 18.18.0

[andreainnocenti]

it looks fixed with node.js v18.18.2

[Drew-Kimberly]

This looks like it was introduced in Node with the libuv@>=1.45.0 upgrade. It was introduced in Node 20.3.0 and Node 18.18.0.

It was reverted in Node 18.18.1 but never in 20.x 😢

[luiof]

We are experiencing the same issue with the latest node.js v20.9.0 version. Same for the 20.8.1 version. The libuv seems to be the 1.46.0. So, for now are there any workaround? Without considering an older version of node.

[eddyw]

What worked for us is:

-const request = supertest(server);
+const request = supertest(`http://127.0.0.1:${server.address().port}`);

I couldn't figure out the exact reason but I suspected it was attempting to start another server so providing a URI instead somehow just fixed it.

Hopefully it's useful for at least one person going through this difficult and painful time. 😅

[aercolino-cognizant-cepsa]

It was reverted in Node 18.18.1 but never in 20.x 😢

Really upsetting.

To run my tests in Node.js 20.11.0, I had to replace

server = await supertest(app)
response = await server.get(path)

with

server = await app.listen()
port = server.address().port
response = await fetch(`http://localhost:${port}${path}`)

[vbuzivskoy]

This looks like it was introduced in Node with the libuv@>=1.45.0 upgrade. It was introduced in Node 20.3.0 and Node 18.18.0.

It was reverted in Node 18.18.1 but never in 20.x 😢

Hey) Did you have a chance to find out the root cause of the problem that is in the libuv?

[zyf0330]

This looks like it was introduced in Node with the libuv@>=1.45.0 upgrade. It was introduced in Node 20.3.0 and Node 18.18.0. It was reverted in Node 18.18.1 but never in 20.x 😢

Hey) Did you have a chance to find out the root cause of the problem that is in the libuv?

Node 18.18.1 reverts libuv change which is about importing io_uring and doesn't add it back again, but Node 20.x doesn't. Using Axios to do parallel http requests has no this problem. I think it shouldn't be duty of Node or libuv. They are chaning, but Supertest doesn't.

[Marcosaurios]

We also experienced the same issue, although we found out the ECONNRESET happened to us when upgrading to Node20, in combination of calling supertest in a Promise.all array of promises.

Untangling the Promise.all call and awaiting individually, solved the problem for us.

[nodify-at]

I encountered similar issues and resolved them by modifying the code as follows (node version: v20.14.0):

Before:

let app: ReturnType<typeof express>
let test: TestAgent

beforeEach(() => {
    app = express()
    // Middleware setup, etc.
    test = supertest(app)
})

After:

let app: ReturnType<typeof express>
let test: TestAgent
let server: ReturnType<typeof app.listen>

beforeEach(() => {
    app = express()
    // Middleware setup, etc.
    server = app.listen(0)
    test = supertest(server)
})

afterEach(() => {
    server.close()
})

It appears that passing the server to supertest instead of the app itself resolves the issue.