TypeError: fetch failed

[kyrylkov]

Bug Description

Fetch throws 'TypeError: fetch failed' when posting

Reproducible By

Request with private production credentials to ERP

Expected Behavior

Successful response (ok). node-fetch works just fine.

Logs & Screenshots

 TypeError: fetch failed
    at Object.processResponse (C:\Users\kyryl\dev\entriwise\server\node_modules\undici\lib\fetch\index.js:183:23)
    at Fetch.fetchFinale (C:\Users\kyryl\dev\entriwise\server\node_modules\undici\lib\fetch\index.js:937:17)
    at Fetch.mainFetch (C:\Users\kyryl\dev\entriwise\server\node_modules\undici\lib\fetch\index.js:744:17)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

Environment

Linux and Windows, v17.5 v17.6

Additional context

Request takes up to several minutes until response

[mcollina]

Thanks for reporting, would you like to include a code to reproduce your problem? We'd also need the server counterpart.

[kyrylkov]

@mcollina We cannot share production environment credentials, but we could give you sandbox credentials via remote desktop. Is there any other way to give you more info for debugging?

[mcollina]

I think it would be better if you produced a server that returned that kind of response. We do not provide support for private code.

[kyrylkov]

Matteo, the server is NetSuite ERP, so we cannot provide it. Also note we've replaced undici fetch with node-fetch and it's working fine. So this error is not the server response.

We can provide you with isolated fetch request, no code required. But we cannot share the credentials to run this request.

[mcollina]

There is something in that server response that it is making undici.fetch fail. What I'm asking is a way to reproduce this without touching your systems. Only you can write it.

[kyrylkov]

Right but I don't understand how? The only way we can reproduce it ourselves is by executing that request to that account with those credentials.

[Linkgoron]

There's supposed to be an additional cause property on that error, but I don't see it in the log. Does it exist?

[kyrylkov]

@Linkgoron No. The code is like this. But 2nd line is never reached.

const fetchRes = await fetch(webservicesUrl, options);
if (!fetchRes.ok) throw new Error(`HTTP error! Status: ${fetchRes.status}`);

[ronag]

Sorry but with this little information it's impossible for us to do anything.

[kyrylkov]

@ronag So I'm trying to get you more information. Can we add some debugging code in undici to help you diagnose it?

[janat08]

I also get some error, it's not the same:

"TypeError: fetch failed\n    at Object.processResponse (C:\\Users\\jeyan\\Documents\\boughtwithslowsearch\\node_modules\\undici\\lib\\fetch\\index.js:183:23)\n    at Fetch.fetchFinale (C:\\Users\\jeyan\\Documents\\boughtwithslowsearch\\node_modules\\undici\\lib\\fetch\\index.js:937:17)\n    at Fetch.mainFetch (C:\\Users\\jeyan\\Documents\\boughtwithslowsearch\\node_modules\\undici\\lib\\fetch\\index.js:744:17)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)"

[akauppi]

Maybe this can help?

I am experiencing the same with a Firebase Emulators host.

$ git clone https://github.com/akauppi/firebase-jest-testing.git
$ cd firebase-jest-testing

$ git checkout 0d2a5f43022453a34a3de9510f9e436b4d890078    # reproducible in this commit

$ npm install

In one terminal:

$ npm run start

In another:

$ npm run test:fns:greet
...
(node:37466) ExperimentalWarning: VM Modules is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
 FAIL  sample/test-fns/greet.test.js
  Cloud Function callables
    ✕ returns a greeting (15 ms)

  ● Cloud Function callables › returns a greeting

    TypeError: fetch failed

      at Object.processResponse (../../node_modules/undici/lib/fetch/index.js:183:23)

Test Suites: 1 failed, 1 total
Tests:       1 failed, 1 total
Snapshots:   0 total
Time:        0.324 s, estimated 1 s
Ran all test suites matching /greet.test.js/i.

---

Added: Once npm run start is running, you can exercise the server by:

$ curl -v -X POST -d '{"data":"abc"}' -H "Content-type: application/json" http://localhost:5002/demo-1/mars-cent
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying ::1:5002...
* connect to ::1 port 5002 failed: Connection refused
*   Trying 127.0.0.1:5002...
* Connected to localhost (127.0.0.1) port 5002 (#0)
> POST /demo-1/mars-central2/greet HTTP/1.1
> Host: localhost:5002
> User-Agent: curl/7.77.0
> Accept: */*
> Content-type: application/json
> Content-Length: 14
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< x-powered-by: Express
< vary: Origin
< content-type: application/json; charset=utf-8
< content-length: 28
< etag: W/"1c-90pTE8vnJrTHpo49hh2KVRcGoxw"
< date: Sun, 27 Feb 2022 12:33:58 GMT
< connection: close
< 
* Closing connection 0
{"result":"Greetings, abc."} 

---

Trying with 127.0.0.1 instead of localhost puts the problem at rest:

$ EMUL_HOST=127.0.0.1 npm run test:fns:greet

> test:fns:greet
> NODE_OPTIONS=--experimental-vm-modules jest --config sample/test-fns/jest.config.js -f greet.test.js --verbose --all

(node:38224) ExperimentalWarning: VM Modules is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
(node:38224) ExperimentalWarning: buffer.Blob is an experimental feature. This feature could change at any time
 PASS  sample/test-fns/greet.test.js
  Cloud Function callables
    ✓ returns a greeting (33 ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        0.32 s, estimated 1 s
Ran all test suites matching /greet.test.js/i.

[akauppi]

As a continuation to my above report, from a CI run.

It looks like Undici (4.14.1) fetch may have a general problem with host names. Here, emul is supposed to point to a Docker container under Google Cloud Build. The run fails to find it.

Step #2: > ci:test
Step #2: > wait-for emul:6767 && wait-for emul:5002 && npm run -s _ci_warm-up && npm run -s _test1
Step #2: 
Step #2: Unexpected error: TypeError: fetch failed
Step #2:     at Object.processResponse (/workspace/node_modules/undici/lib/fetch/index.js:183:23)
Step #2:     at Fetch.fetchFinale (/workspace/node_modules/undici/lib/fetch/index.js:937:17)
Step #2:     at Fetch.mainFetch (/workspace/node_modules/undici/lib/fetch/index.js:744:17)
Step #2:     at processTicksAndRejections (node:internal/process/task_queues:96:5) {
Step #2:   cause: Error: getaddrinfo ENOTFOUND emul
Step #2:       at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26) {
Step #2:     errno: -3008,
Step #2:     code: 'ENOTFOUND',
Step #2:     syscall: 'getaddrinfo',
Step #2:     hostname: 'emul'
Step #2:   }

[ronag]

I don't think this has anything to do with undici per se. Does it work with the normal http client in node?

[akauppi]

I don't think this has anything to do with undici per se. Does it work with the normal http client in node?

It works with curl - I am not very versed in using the normal http client, so.. don't know.

Got some more light to why localhost isn't recognized by fetch.

My /etc/hosts (macOS 12.2.1 Intel):

##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1       localhost
255.255.255.255 broadcasthost
::1             localhost
# Added by Docker Desktop
# To allow the same kube context to work on the host and the container:
127.0.0.1 kubernetes.docker.internal
# End of section

If I comment out the line ::1 localhost, things work as normal.

[ronag]

@akauppi: Sorry but I don't think this is an issue we can resolve in undici. Sounds more like a node core dns thing. I suggest you open an issue in the node repo.

[ronag]

@akauppi: As i wrote above undici is probably not the place for the fix. @kyrylkov: If you want help with private code you can mail me.

[kyrylkov]

@ronag emailed you

[kyrylkov]

@ronag I'm still having these errors. Cannot use undici fetch at all. Here is updated trace from Node v18:

TypeError: fetch failed
    at Object.processResponse (/home/ubuntu/server/node_modules/undici/lib/fetch/index.js:200:23)
    at /home/ubuntu/server/node_modules/undici/lib/fetch/index.js:941:38
    at node:internal/process/task_queues:140:7
    at AsyncResource.runInAsyncScope (node:async_hooks:202:9)
    at AsyncResource.runMicrotask (node:internal/process/task_queues:137:8)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    ... 2 lines matching cause stack trace ...
    at process.processTimers (node:internal/timers:507:7) {
  cause: Error: [object Object]
      at makeNetworkError (/home/ubuntu/server/node_modules/undici/lib/fetch/response.js:366:11)
      at httpNetworkFetch (/home/ubuntu/server/node_modules/undici/lib/fetch/index.js:1751:12)
      at processTicksAndRejections (node:internal/process/task_queues:95:5)
      at runNextTicks (node:internal/process/task_queues:64:3)
      at listOnTimeout (node:internal/timers:533:9)
      at process.processTimers (node:internal/timers:507:7)
      at async httpNetworkOrCacheFetch (/home/ubuntu/server/node_modules/undici/lib/fetch/index.js:1440:29)
      at async httpFetch (/home/ubuntu/server/node_modules/undici/lib/fetch/index.js:1028:33)
      at async schemeFetch (/home/ubuntu/server/node_modules/undici/lib/fetch/index.js:888:14)
      at async /home/ubuntu/server/node_modules/undici/lib/fetch/index.js:569:16
      at async mainFetch (/home/ubuntu/server/node_modules/undici/lib/fetch/index.js:553:16) {
    [cause]: undefined
  }
}

@ronag Could you re-open or reply to my emails?

[s-bilic]

Had the same issue, had to switch from node v18 to v16 as a temp workaround.

[ItzaMi]

Had the same issue, had to switch from node v18 to v16 as a temp workaround.

This worked for me. Also was having the same issue without any resolution and downgrading to 16 ended up doing the trick!

[kyrylkov]

We also keep getting this error from time to time in Node.js 18.7.0

[mcollina]

I think I know what the problem is.

In node v17 we switched the default ordering of dns entries to follow what the OS is providing us vs reordering to put ipv4 always first.

I think we should try multiple addresses when dealing with localhost when establishing a new connection.

Wdyt @ronag?

[ronag]

I think I know what the problem is.

In node v17 we switched the default ordering of dns entries to follow what the OS is providing us vs reordering to put ipv4 always first.

I think we should try multiple addresses when dealing with localhost when establishing a new connection.

Wdyt @ronag?

SGTM!

[mcollina]

Closing in favor of https://github.com/nodejs/undici/issues/1602, it's clearer.

[aegonwolf]

Hello, I have been told I should post my example here as well since I might have the same error: https://stackoverflow.com/questions/73599889/fastapi-refuses-connection-on-a-parameterized-endpoint-if-called-sequentially-wi?noredirect=1#comment129970573_73599889 I am using svelte and fastapi and it seems that I can not fetch two endpoints together but only one of them each.

[zvictor]

As a temporary solution, please try this:

// Workaround from https://stackoverflow.com/a/72416352/599991
const dns = require('node:dns')
dns.setDefaultResultOrder('ipv4first')

[bnb]

FWIW that doesn't solve the bug for me @zvictor

[kyrylkov]

@zvictor It doesn't look like DNS issue because it happens after a whole bunch of previous similar requests complete successfuly. And upon retry the failed request also completes successfully.

[mcollina]

@bnb if that doesn't solve the problem... I would definitely need a client/server combo that reproduces the problem.

[bnb]

FWIW after a bit more tweaking it seems to have solved it (or at least, it's working now)

[mcollina]

what was the problem?

[bnb]

I believe the above code sample worked after I'd completely reverted all the changes I'd made to my tests in attempts to get this functionality to work. You can find the working code here.

[sugoidesune]

So this just started happening out of the blue. Working with firebase functions. From one minute to the next. I didn't make any changes to my environment nor did i update or install packages between the time it worked and when it stopped. Restarted Firebase functions and it works again...

TypeError: fetch failed
>      at Object.fetch (node:internal/deps/undici/undici:14152:11)
>      at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
>    cause: ConnectTimeoutError: Connect Timeout Error
>        at onConnectTimeout (node:internal/deps/undici/undici:7967:28)
>        at node:internal/deps/undici/undici:7925:50
>        at Immediate._onImmediate (node:internal/deps/undici/undici:7954:37)
>        at process.processImmediate (node:internal/timers:476:21) {
>      code: 'UND_ERR_CONNECT_TIMEOUT'
>    }
>  }
> Node.js v18.14.0

[jpcmf]

I'm facing the same issue with Next.js and Strapi. In localhost, the Next.js app can't fetch the API from Strapi. I recently updated Node.js to version 18.15.0.

error - TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11413:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async fetchQuery (webpack-internal:///./src/utils/fetch.ts:18:22)
    at async Promise.all (index 0)
    at async getServerSideProps (webpack-internal:///./src/pages/index.tsx:47:20) {
  cause: Error: connect ECONNREFUSED ::1:1337
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16) {
    errno: -4078,
    code: 'ECONNREFUSED',
    syscall: 'connect',
    address: '::1',
    port: 1337
  },
  page: '/'
}

[gazhay]

Just hit this same bug. No real changes to code, other fetches, curl, etc work fine

    at Object.fetch (node:internal/deps/undici/undici:11118:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  cause: InvalidArgumentError: invalid connection header

[kyrylkov]

@mcollina Could you please re-open as this started happening again in Node.js 20 with latest fetch and timeout of 5 minutes.

[hholst80]

Closing in favor of #1602, it's clearer.

@kyrylkov

[kyrylkov]

@hholst80 They are not related https://github.com/nodejs/undici/issues/1602#issuecomment-1216654581

No localhost involved in our case.

[fyapy]

+1

[kyrylkov]

We didn't solve it. It pops up once in a while. We're looking at migrating away from undici or Node.js altogether.

[stormsson]

Same issue here, fetching a remote url from nextJS ; localhost is not involved...

[jayco]

Not sure if this a Node, Next or Undici thing, but I was getting the same issue and it was pretty confusing.

I did some debugging, I downgraded from latest 20 to LTS, and in my case, the difference looks to be in the way the request error is being handled (or maybe not being handled in 20).

Same code for each below, just different node versions.

LTS 18.6.0

Error: 401:Unauthorized - invalid_token: The access token is invalid or has expired
    at list (webpack-internal:///(sc_server)/./lib/helpscout/client/list.ts:36:23)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async GET (webpack-internal:///(sc_server)/./app/api/hello/route.ts:18:18)
    at async eval (webpack-internal:///(sc_server)/./node_modules/next/dist/server/future/route-modules/app-route/module.js:240:37)

Node 20.1.0

TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11522:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  cause: Error [ERR_SOCKET_CONNECTION_TIMEOUT]: Socket connection timeout
      at new NodeError (node:internal/errors:399:5)
      at internalConnectMultiple (node:net:1099:20)
      at Timeout.internalConnectMultipleTimeout (node:net:1638:3)
      at listOnTimeout (node:internal/timers:575:11)
      at process.processTimers (node:internal/timers:514:7) {
    code: 'ERR_SOCKET_CONNECTION_TIMEOUT'
  }
}

[OhadRon]

+1 here. Happens to me when trying to upload files to Supabase Storage. Node v20, out of the blue with no code changes or package changes.

node:internal/deps/undici/undici:11522
    Error.captureStackTrace(err, this);
          ^

TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11522:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  cause: Error
      at makeNetworkError (node:internal/deps/undici/undici:6844:35)
      at httpRedirectFetch (node:internal/deps/undici/undici:11105:16)
      at httpFetch (node:internal/deps/undici/undici:11071:28)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async schemeFetch (node:internal/deps/undici/undici:10979:18)
      at async node:internal/deps/undici/undici:10855:20
      at async mainFetch (node:internal/deps/undici/undici:10851:20)
}

[marco-digennaro]

+1 node v20, simple fetch request, suddenly, no code changes

TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11522:11) { cause: SocketError: other side closed at Socket.onSocketEnd (node:internal/deps/undici/undici:9740:26) at Socket.emit (node:events:523:35) at endReadableNT (node:internal/streams/readable:1367:12) at process.processTicksAndRejections (node:internal/process/task_queues:82:21) { code: 'UND_ERR_SOCKET',

[offminded]

I was having the same issue with node server in my dev env and fixed it using NODE_TLS_REJECT_UNAUTHORIZED=0 flag running the node server. My GraphQL server is using a self-signed certificate so the issue was related to nodejs issue with the SSL.

[kyrylkov]

@offminded None of our issues related to self-signed certificates as they happen in production against well known URLs and certificates.

[BSN4]

same here try :

    const bug = await fetch('https://fapi.binance.com/fapi/v1/exchangeInfo')
    console.log(bug.json())

[BSN4]

I don't want to open new bug issue , because we already have one with all things you need @mcollina @RafaelGSS @ronag can you please reopen this issue... i've decided to downgrade my node version from 20.2.0 to 18.16.0 because of this

[kyrylkov]

@BSN4 your example works just fine for me, returning

{
  code: 0,
  msg: "Service unavailable from a restricted location according to 'b. Eligibility' in https://www.binance.com/en/terms. Please contact customer service if you believe you received this message in error."
}