EPIPE write, ECONNRESET issues when switch to nodejs 20

[roman-kupriyanov]

Hello,

After the migration to nodejs 20.16.0 from 18.20.4 some of our tests that use Mockttp server started to fail with EPIPE write or ECONNRESET read or ECONNRESET socket hang up errors. We have around 200 tests in general and only some of them are affected. A weird part is that if to run failed tests individually they are passing, which makes me think that it may be related to some parallel running issues or race conditions. We use Jest as a test runner.

Any help would be appreciated. Thanks!

[pimterry]

Oooh, that sounds very interesting! I haven't seen this before myself, and Mockttp's own tests seem to run fine in 20.16.0 so I'm not sure exactly what would cause this.

Is there any pattern to which tests fail? Is there anything notable about those tests?

If you try using other Node versions (older Node v20 versions, or Node v22) do those fail in the same way? If not, it would be really interesting to try a few different versions and see if you can work out exactly which version is breaking this for you.

[roman-kupriyanov]

Hi @pimterry,

Indeed a very interesting issue it is 🙂 Regarding your questions:

If you try using other Node versions (older Node v20 versions, or Node v22) do those fail in the same way? If not, it would be really interesting to try a few different versions and see if you can work out exactly which version is breaking this for you.

I tried version 20.0.0 and it is not working right away, so seems that the issue is appearing from very beginning of the node 20 version. I also tried 22.6.0 and it fails the tests in the same way. And again the latest 18.x version fixes everything immediately.

I also use the latest to date mockttp version: 3.15.1.

Is there any pattern to which tests fail? Is there anything notable about those tests?

There are two patterns usually:

• When we expect some specific response from API, but instead getting the socket errors:

  expect(received).toEqual(expected) // deep equality
  
  - Expected  - 3
  + Received  + 7
  
    Object {
      "details": Object {
  -     "messages": Array [],
  -     "returnCode": 0,
  +     "connectionCode": "ECONNRESET",
  +     "httpStatusCode": undefined,
  +     "messages": Array [
  +       "read ECONNRESET",
  +     ],
      },
  -   "status": "OK",
  +   "status": "ERROR",
  +   "type": "CONNECTION_ERROR",
    }
  
    10244 |       )(packageName)(packageOptions);
    10245 |       // assert
  > 10246 |       expect(createPackageResult).toEqual({
          |                                   ^
    10247 |         status: ResponseStatus.OK,
    10248 |         details: {
    10249 |           returnCode: 0,

These tests usually use forAnyRequest().thenJson() mock:

      await mockServer.forAnyRequest().thenJson(
        200,
        {
          data: [],
          messages,
          reports: {},
          returnCode,
        },
        {
          version: '2.5',
          'content-type': 'application/json',
        }
      );

• And the second case is during the check if API was even called:

    expect(received).toBe(expected) // Object.is equality

    Expected: true
    Received: false

      6726 |         const seenRequests = await endevorEndpoint.getSeenRequests();
      6727 |         const calledOnce = seenRequests.length === 1;
    > 6728 |         expect(calledOnce).toBe(true);
           |                            ^
      6729 |
      6730 |         expect(isErrorEndevorResponse(generateResult)).toBe(false);
      6731 |       });

This one happens usually with the forPut() or forGet() mock.

[pimterry]

Ok, that's very useful thanks. I think it's likely that this is a Node bug (or intentional breaking change that's causing issues), but unfortunately I'm also a Node maintainer, so I can't really pass the buck here :laughing:, and whether the fix is in Node or Mockttp it'd be useful to know exactly what's going on.

Can you share more detail about how exactly are you're sending the requests, and any kinds of client configuration there? One thing that has changed a few times in Node during major version bumps is a slow tightening of which kinds of subtly invalid connections or unusual requests are accepted, so it might be that there's something unusual there triggering this.

If you test with some v19 versions, does this fail the same way there too? If we can pin this down to a specific version where it breaks then that would reduce the set of possibly related changes in Node significantly, which would be very helpful to understand the issue.

Is there any chance you can put together a reproduction of the issue, or share enough code that I can run the tests myself? That would make it very dramatically easier to investigate and fix.