ulixee / hero

The web browser built for scraping
MIT License
649 stars 32 forks source link

HPE_INVALID_CONSTANT - Expected HTTP/ when using proxy with mitm #210

Closed fueryon closed 1 year ago

fueryon commented 1 year ago

I'm trying to run Hero with http proxy that requires authorization. Running Miner with ULX_DISABLE_MITM=0 always causes following error:

2023-01-10T16:45:34.207Z ERROR [hero-core/connections/ConnectionToHeroClient] ConnectionToClient.HandleRequestError { context: {}, sessionId: undefined, sessionName: undefined } Error: Parse Error: Expected HTTP/
    at Socket.socketOnData (_http_client.js:515:22)
    at Socket.emit (events.js:400:28)
    at addChunk (internal/streams/readable.js:293:12)
    at readableAddChunk (internal/streams/readable.js:267:9)
    at Socket.Readable.push (internal/streams/readable.js:206:10)
    at Pipe.onStreamRead (internal/stream_base_commons.js:188:23) {
  bytesParsed: 0,
  code: 'HPE_INVALID_CONSTANT',
  reason: 'Expected HTTP/',
  rawPacket: <Buffer 00 00 18 04 00 00 00 00 00 00 05 00 10 00 00 00 03 00 00 00 fa 00 06 00 10 01 40 00 04 00 10 00 00>
}

What I've tried: various proxies - with, and without an authorization; commercial and local; http and socks5 running miner from cli and using docker image various uri's (both http and https) different node versions (14 and 18) different machines and different networks

only possibility to use proxy is to disable mitm in .env, but then I can't use proxies that require authorization (and i understand that it's a browser limitation)

my environment: Ubuntu 20.04.5 LTS

failing code:

import Hero from '@ulixee/hero';

(async () => {
  const hero = new Hero({
      connectionToCore: { host: 'ws://localhost:8080' },
      upstreamProxyUrl: 'socks5://192.168.200.2:10864'
  });
  await hero.goto('https://icanhazip.com/');
  const title = await hero.document.title;
  console.log(title)
  const intro = await hero.document.querySelector('pre').textContent;
  console.log(intro)
  await hero.close();
})();

What am I missing?

blakebyrnes commented 1 year ago

Hi, sorry, I just moved a duplicate of this ticket over to Unblocked. See answer here: https://github.com/ulixee/unblocked/issues/29#issuecomment-1376318887

blakebyrnes commented 1 year ago

Update: this is due to not supporting http2 correctly with these services. Will be included in next release later this week or early next.