search

TooTallNate / proxy-agents

Node.js HTTP Proxy Agents Monorepo
https://proxy-agents.n8.io
872 stars 229 forks source link

[BUG]: EPIPE socket "error" event after "end" event #267

Open mxschmitt opened 6 months ago

mxschmitt commented 6 months ago

Hey! Thanks for your great library, I wasn't able to quickly figure out what the root cause might be, one idea which I had was putting target?.unpipe(socket) into socket.on('end', function onclientend() { which seems to work.

See the EPIPE error below at the end which gets thrown / logged - maybe since it only gets logged in the debug logs and we internally add a custom error event handler where it gets surfaced then, this does not get considered a bug in this library? Instead we could probably also just ignore this kind of error.

Downstream issue: https://github.com/microsoft/playwright/issues/28701

Logs:

  proxy ← ← ← CONNECT location.services.mozilla.com:443 HTTP/1.1  +0ms
  proxy ↑ ↑ ↑ connecting to proxy target { host: 'location.services.mozilla.com', port: 443, localAddress: undefined } +0ms
  proxy ↓ ↓ ↓ proxy target location.services.mozilla.com:443 "connect" event +0ms
  proxy → → → HTTP/1.1 200 Connection established +0ms
  proxy ← ← ← CONNECT contile.services.mozilla.com:443 HTTP/1.1  +385ms
  proxy ↑ ↑ ↑ connecting to proxy target { host: 'contile.services.mozilla.com', port: 443, localAddress: undefined } +385ms
  proxy ← ← ← CONNECT spocs.getpocket.com:443 HTTP/1.1  +8ms
  proxy ↑ ↑ ↑ connecting to proxy target { host: 'spocs.getpocket.com', port: 443, localAddress: undefined } +8ms
  proxy ← ← ← GET http://localhost:8907/target-false-169.254.3.4.html HTTP/1.1  +5ms
  proxy ← ← ← Request Header: [ 'Host', '169.254.3.4:8907' ] +1ms
  proxy ← ← ← Request Header: [ 'User-Agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:120.0) Gecko/20100101 Firefox/120.0' ] +0ms
  proxy ← ← ← Request Header: [ 'Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8' ] +0ms
  proxy ← ← ← Request Header: [ 'Accept-Language', 'en-US' ] +0ms
  proxy ← ← ← Request Header: [ 'Accept-Encoding', 'gzip, deflate' ] +0ms
  proxy ← ← ← Request Header: [ 'Connection', 'keep-alive' ] +0ms
  proxy ↑ ↑ ↑ ignoring hop-by-hop header "Connection" +7ms
  proxy ← ← ← Request Header: [ 'Upgrade-Insecure-Requests', '1' ] +1ms
  proxy ↑ ↑ ↑ adding new "X-Forwarded-For" header: "::ffff:127.0.0.1" +0ms
  proxy ↑ ↑ ↑ adding new "Via" header: "1.1 Maxs-Laptop.fritz.box (proxy/2.1.1)" +0ms
  proxy ↑ ↑ ↑ GET /target-false-169.254.3.4.html HTTP/1.1  +1ms
  proxy ↓ ↓ ↓ HTTP/1.1 200 +137ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Date: Tue, 19 Dec 2023 21:59:33 GMT" +0ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Connection: keep-alive" +0ms
  proxy → → → ignoring hop-by-hop header "Connection" +137ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Keep-Alive: timeout=5" +0ms
  proxy → → → ignoring hop-by-hop header "Keep-Alive" +1ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Content-Length: 47" +1ms
  proxy → → → HTTP/1.1 200 +0ms
  proxy → → → "finish" event +0ms
  proxy → → → cleanup +0ms
  proxy ← ← ← GET http://localhost:8907/favicon.ico HTTP/1.1  +105ms
  proxy ← ← ← Request Header: [ 'Host', '169.254.3.4:8907' ] +0ms
  proxy ← ← ← Request Header: [ 'User-Agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:120.0) Gecko/20100101 Firefox/120.0' ] +0ms
  proxy ← ← ← Request Header: [ 'Accept', 'image/avif,image/webp,*/*' ] +0ms
  proxy ← ← ← Request Header: [ 'Accept-Language', 'en-US' ] +0ms
  proxy ← ← ← Request Header: [ 'Accept-Encoding', 'gzip, deflate' ] +0ms
  proxy ← ← ← Request Header: [ 'Connection', 'keep-alive' ] +0ms
  proxy ↑ ↑ ↑ ignoring hop-by-hop header "Connection" +104ms
  proxy ← ← ← Request Header: [ 'Referer', 'http://169.254.3.4:8907/target-false-169.254.3.4.html' ] +0ms
  proxy ↑ ↑ ↑ adding new "X-Forwarded-For" header: "::ffff:127.0.0.1" +0ms
  proxy ↑ ↑ ↑ adding new "Via" header: "1.1 Maxs-Laptop.fritz.box (proxy/2.1.1)" +0ms
  proxy ↑ ↑ ↑ GET /favicon.ico HTTP/1.1  +1ms
  proxy ↓ ↓ ↓ HTTP/1.1 404 +103ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Cache-Control: no-cache, no-store" +0ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Content-Type: text/plain" +0ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Date: Tue, 19 Dec 2023 21:59:33 GMT" +0ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Connection: keep-alive" +0ms
  proxy → → → ignoring hop-by-hop header "Connection" +103ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Keep-Alive: timeout=5" +0ms
  proxy → → → ignoring hop-by-hop header "Keep-Alive" +0ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Content-Length: 79" +0ms
  proxy → → → HTTP/1.1 404 +0ms
  proxy → → → "finish" event +0ms
  proxy → → → cleanup +0ms
  proxy ← ← ← GET http://localhost:8907/foo.html HTTP/1.1  +24ms
  proxy ← ← ← Request Header: [ 'Host', '1.non.existent.domain.for.the.test' ] +0ms
  proxy ← ← ← Request Header: [ 'User-Agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:120.0) Gecko/20100101 Firefox/120.0' ] +1ms
  proxy ← ← ← Request Header: [ 'Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8' ] +0ms
  proxy ← ← ← Request Header: [ 'Accept-Language', 'en-US' ] +0ms
  proxy ← ← ← Request Header: [ 'Accept-Encoding', 'gzip, deflate' ] +0ms
  proxy ← ← ← Request Header: [ 'Connection', 'keep-alive' ] +0ms
  proxy ↑ ↑ ↑ ignoring hop-by-hop header "Connection" +24ms
  proxy ← ← ← Request Header: [ 'Upgrade-Insecure-Requests', '1' ] +0ms
  proxy ↑ ↑ ↑ adding new "X-Forwarded-For" header: "::ffff:127.0.0.1" +0ms
  proxy ↑ ↑ ↑ adding new "Via" header: "1.1 Maxs-Laptop.fritz.box (proxy/2.1.1)" +0ms
  proxy ↑ ↑ ↑ GET /foo.html HTTP/1.1  +0ms
  proxy ↓ ↓ ↓ HTTP/1.1 404 +24ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Cache-Control: no-cache, no-store" +0ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Content-Type: text/plain" +0ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Date: Tue, 19 Dec 2023 21:59:33 GMT" +0ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Connection: keep-alive" +0ms
  proxy → → → ignoring hop-by-hop header "Connection" +24ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Keep-Alive: timeout=5" +0ms
  proxy → → → ignoring hop-by-hop header "Keep-Alive" +0ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Content-Length: 76" +0ms
  proxy → → → HTTP/1.1 404 +0ms
  proxy → → → "finish" event +0ms
  proxy → → → cleanup +0ms
  proxy ↓ ↓ ↓ proxy target contile.services.mozilla.com:443 "connect" event +20ms
  proxy → → → HTTP/1.1 200 Connection established +20ms
  proxy ← ← ← GET http://localhost:8907/favicon.ico HTTP/1.1  +24ms
  proxy ← ← ← Request Header: [ 'Host', '1.non.existent.domain.for.the.test' ] +1ms
  proxy ← ← ← Request Header: [ 'User-Agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:120.0) Gecko/20100101 Firefox/120.0' ] +0ms
  proxy ← ← ← Request Header: [ 'Accept', 'image/avif,image/webp,*/*' ] +0ms
  proxy ← ← ← Request Header: [ 'Accept-Language', 'en-US' ] +0ms
  proxy ← ← ← Request Header: [ 'Accept-Encoding', 'gzip, deflate' ] +0ms
  proxy ← ← ← Request Header: [ 'Connection', 'keep-alive' ] +0ms
  proxy ↑ ↑ ↑ ignoring hop-by-hop header "Connection" +25ms
  proxy ← ← ← Request Header: [ 'Referer', 'http://1.non.existent.domain.for.the.test/foo.html' ] +0ms
  proxy ↑ ↑ ↑ adding new "X-Forwarded-For" header: "::ffff:127.0.0.1" +0ms
  proxy ↑ ↑ ↑ adding new "Via" header: "1.1 Maxs-Laptop.fritz.box (proxy/2.1.1)" +0ms
  proxy ↑ ↑ ↑ GET /favicon.ico HTTP/1.1  +0ms
  proxy ↓ ↓ ↓ HTTP/1.1 404 +4ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Cache-Control: no-cache, no-store" +0ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Content-Type: text/plain" +0ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Date: Tue, 19 Dec 2023 21:59:33 GMT" +0ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Connection: keep-alive" +0ms
  proxy → → → ignoring hop-by-hop header "Connection" +4ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Keep-Alive: timeout=5" +0ms
  proxy → → → ignoring hop-by-hop header "Keep-Alive" +0ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Content-Length: 79" +0ms
  proxy → → → HTTP/1.1 404 +0ms
  proxy → → → "finish" event +1ms
  proxy → → → cleanup +0ms
  proxy ← ← ← POST http://localhost:8907/ HTTP/1.1  +2ms
  proxy ← ← ← Request Header: [ 'Host', 'ocsp.digicert.com' ] +1ms
  proxy ← ← ← Request Header: [ 'User-Agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:120.0) Gecko/20100101 Firefox/120.0' ] +0ms
  proxy ← ← ← Request Header: [ 'Accept', '*/*' ] +0ms
  proxy ← ← ← Request Header: [ 'Accept-Language', 'en-US,en;q=0.5' ] +0ms
  proxy ← ← ← Request Header: [ 'Accept-Encoding', 'gzip, deflate' ] +0ms
  proxy ← ← ← Request Header: [ 'Content-Type', 'application/ocsp-request' ] +0ms
  proxy ← ← ← Request Header: [ 'Content-Length', '83' ] +0ms
  proxy ← ← ← Request Header: [ 'Connection', 'keep-alive' ] +0ms
  proxy ↑ ↑ ↑ ignoring hop-by-hop header "Connection" +3ms
  proxy ← ← ← Request Header: [ 'Pragma', 'no-cache' ] +0ms
  proxy ← ← ← Request Header: [ 'Cache-Control', 'no-cache' ] +0ms
  proxy ↑ ↑ ↑ adding new "X-Forwarded-For" header: "::ffff:127.0.0.1" +0ms
  proxy ↑ ↑ ↑ adding new "Via" header: "1.1 Maxs-Laptop.fritz.box (proxy/2.1.1)" +0ms
  proxy ↑ ↑ ↑ POST / HTTP/1.1  +0ms
  proxy ↓ ↓ ↓ HTTP/1.1 404 +4ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Cache-Control: no-cache, no-store" +0ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Content-Type: text/plain" +0ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Date: Tue, 19 Dec 2023 21:59:33 GMT" +0ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Connection: keep-alive" +0ms
  proxy → → → ignoring hop-by-hop header "Connection" +3ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Keep-Alive: timeout=5" +0ms
  proxy → → → ignoring hop-by-hop header "Keep-Alive" +0ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Content-Length: 78" +0ms
  proxy → → → HTTP/1.1 404 +0ms
  proxy → → → "finish" event +0ms
  proxy → → → cleanup +0ms
  proxy ← ← ← POST http://localhost:8907/ HTTP/1.1  +43ms
  proxy ← ← ← Request Header: [ 'Host', 'r3.o.lencr.org' ] +1ms
  proxy ← ← ← Request Header: [ 'User-Agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:120.0) Gecko/20100101 Firefox/120.0' ] +0ms
  proxy ← ← ← Request Header: [ 'Accept', '*/*' ] +0ms
  proxy ← ← ← Request Header: [ 'Accept-Language', 'en-US,en;q=0.5' ] +0ms
  proxy ← ← ← Request Header: [ 'Accept-Encoding', 'gzip, deflate' ] +0ms
  proxy ← ← ← Request Header: [ 'Content-Type', 'application/ocsp-request' ] +0ms
  proxy ← ← ← Request Header: [ 'Content-Length', '85' ] +0ms
  proxy ← ← ← Request Header: [ 'Connection', 'keep-alive' ] +0ms
  proxy ↑ ↑ ↑ ignoring hop-by-hop header "Connection" +44ms
  proxy ← ← ← Request Header: [ 'Pragma', 'no-cache' ] +0ms
  proxy ← ← ← Request Header: [ 'Cache-Control', 'no-cache' ] +0ms
  proxy ↑ ↑ ↑ adding new "X-Forwarded-For" header: "::ffff:127.0.0.1" +0ms
  proxy ↑ ↑ ↑ adding new "Via" header: "1.1 Maxs-Laptop.fritz.box (proxy/2.1.1)" +0ms
  proxy ↑ ↑ ↑ POST / HTTP/1.1  +0ms
  proxy ↓ ↓ ↓ HTTP/1.1 404 +44ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Cache-Control: no-cache, no-store" +0ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Content-Type: text/plain" +0ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Date: Tue, 19 Dec 2023 21:59:33 GMT" +0ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Connection: keep-alive" +0ms
  proxy → → → ignoring hop-by-hop header "Connection" +44ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Keep-Alive: timeout=5" +0ms
  proxy → → → ignoring hop-by-hop header "Keep-Alive" +0ms
  proxy ↓ ↓ ↓ Proxy Response Header: "Content-Length: 78" +0ms
  proxy → → → HTTP/1.1 404 +0ms
  proxy → → → "finish" event +0ms
  proxy → → → cleanup +0ms
  proxy ↓ ↓ ↓ proxy target spocs.getpocket.com:443 "connect" event +156ms
  proxy → → → HTTP/1.1 200 Connection established +156ms
  proxy ← ← ← HTTP request spocs.getpocket.com:443 socket "end" event +290ms
  proxy ← ← ← HTTP request contile.services.mozilla.com:443 socket "end" event +1ms
  proxy ← ← ← HTTP request location.services.mozilla.com:443 socket "end" event +0ms
  proxy ↓ ↓ ↓ proxy target contile.services.mozilla.com:443 "end" event +164ms
  proxy ← ← ← HTTP request contile.services.mozilla.com:443 socket "close" event +30ms
  proxy ↓ ↓ ↓ proxy target contile.services.mozilla.com:443 "close" event +0ms
  proxy ↓ ↓ ↓ proxy target spocs.getpocket.com:443 "end" event +151ms
  proxy ← ← ← HTTP request spocs.getpocket.com:443 socket "close" event +151ms
  proxy ↓ ↓ ↓ proxy target spocs.getpocket.com:443 "close" event +0ms
  proxy ← ← ← HTTP request location.services.mozilla.com:443 socket "error" event:
  proxy ← ← ← Error: write EPIPE
    at afterWriteDispatched (node:internal/stream_base_commons:160:15)
    at writeGeneric (node:internal/stream_base_commons:151:3)
    at Socket._writeGeneric (node:net:952:11)
    at Socket._write (node:net:964:8)
    at writeOrBuffer (node:internal/streams/writable:447:12)
    at _write (node:internal/streams/writable:389:10)
    at Socket.Writable.write (node:internal/streams/writable:393:10)
    at Socket.ondata (node:internal/streams/readable:817:22)
    at Socket.emit (node:events:514:28)
    at addChunk (node:internal/streams/readable:376:12)
    at readableAddChunk (node:internal/streams/readable:349:9)
    at Socket.Readable.push (node:internal/streams/readable:286:10)
    at TCP.onStreamRead (node:internal/stream_base_commons:190:23) +31ms
  proxy ← ← ← HTTP request location.services.mozilla.com:443 socket "close" event +0ms
  proxy ↓ ↓ ↓ proxy target location.services.mozilla.com:443 "end" event +31ms
  proxy ↓ ↓ ↓ proxy target location.services.mozilla.com:443 "close" event +0ms