Tasks sometimes fail on start

[olizilla]

3/31/2023, 3:47:29 PM | node:internal/deps/undici/undici:11279 | backup
3/31/2023, 3:47:29 PM | fetchParams.controller.controller.error(new TypeError("terminated", { | backup
3/31/2023, 3:47:29 PM | ^ | backup
3/31/2023, 3:47:29 PM | TypeError: terminated | backup
3/31/2023, 3:47:29 PM | at Fetch.onAborted (node:internal/deps/undici/undici:11279:53) | backup
3/31/2023, 3:47:29 PM | at Fetch.emit (node:events:525:35) | backup
3/31/2023, 3:47:29 PM | at Fetch.terminate (node:internal/deps/undici/undici:10534:14) | backup
3/31/2023, 3:47:29 PM | at Object.onError (node:internal/deps/undici/undici:11374:36) | backup
3/31/2023, 3:47:29 PM | at Request.onError (node:internal/deps/undici/undici:8168:31) | backup
3/31/2023, 3:47:29 PM | at errorRequest (node:internal/deps/undici/undici:10220:17) | backup
3/31/2023, 3:47:29 PM | at TLSSocket.onSocketClose (node:internal/deps/undici/undici:9668:9) | backup
3/31/2023, 3:47:29 PM | at TLSSocket.emit (node:events:525:35) | backup
3/31/2023, 3:47:29 PM | at node:net:322:12 | backup
3/31/2023, 3:47:29 PM | at TCP.done (node:_tls_wrap:588:7) { | backup
3/31/2023, 3:47:29 PM | [cause]: SocketError: other side closed | backup
3/31/2023, 3:47:29 PM | at TLSSocket.onSocketEnd (node:internal/deps/undici/undici:9647:26) | backup
3/31/2023, 3:47:29 PM | at TLSSocket.emit (node:events:525:35) | backup
3/31/2023, 3:47:29 PM | at endReadableNT (node:internal/streams/readable:1359:12) | backup
3/31/2023, 3:47:29 PM | at process.processTicksAndRejections (node:internal/process/task_queues:82:21) { | backup
3/31/2023, 3:47:29 PM | code: 'UND_ERR_SOCKET', | backup
3/31/2023, 3:47:29 PM | socket: { | backup
3/31/2023, 3:47:29 PM | localAddress: '10.5.4.173', | backup
3/31/2023, 3:47:29 PM | localPort: 54664, | backup
3/31/2023, 3:47:29 PM | remoteAddress: '104.18.23.52', | backup
3/31/2023, 3:47:29 PM | remotePort: 443, | backup
3/31/2023, 3:47:29 PM | remoteFamily: 'IPv4', | backup
3/31/2023, 3:47:29 PM | timeout: undefined, | backup
3/31/2023, 3:47:29 PM | bytesWritten: 244, | backup
3/31/2023, 3:47:29 PM | bytesRead: 1672024 | backup
3/31/2023, 3:47:29 PM | } | backup
3/31/2023, 3:47:29 PM | } | backup
3/31/2023, 3:47:29 PM | } | backup
3/31/2023, 3:47:29 PM | Node.js v18.15.0

[olizilla]

p-retry does not retry on TypeErrors! in node it will only retry a type error if it's message matches https://github.com/sindresorhus/p-retry/blob/285fdc3f890a6f3d199b3d4abb0bad7c172c2233/index.js#L8

but this one is a Terminated. We really do want "other side hung up" to be a thing we retry tho!

[olizilla]

A lead! remoteAddress: '104.18.23.52', shown in the error is the ip for w3s.link

❯ dig w3s.link +short
104.18.22.52
104.18.23.52

so this is failing on pulling the input file from w3s!

[olizilla]

wtf is this url format! both subdomain and path?? and why are we getting redirected to dweb.link https://github.com/web3-storage/backup-infra/blob/5807912f4128d8a9be9dd3ede0b7451a0f7632cf/terraspace/app/stacks/ecs-cluster/tfvars/base.tfvars#L17-L35

[olizilla]

switching to dweb.link gets us further into the file, but still fails with "other side hung up"

[olizilla]

we need to use the local ipfs node to get the input file.