ulixee / hero

The web browser built for scraping
MIT License
652 stars 32 forks source link

Tor Proxy #124

Closed ctaity closed 1 year ago

ctaity commented 2 years ago

Using socks 5 tor proxy, when try to navigate to: http://lockbitapt6vx57t3eeqjofwgcglmutr3a35nygvokja5uuccip4ykyd.onion

we got this error: net::ERR_HTTP_RESPONSE_CODE_FAILURE

async navigate(@Res() res: Response) {
    const hero = new Hero({
      showChrome: true,
      showChromeAlive: false,
      showChromeInteractions: false,
      upstreamProxyUrl: process.env.TOR_PROXY,
    });

    try {
      const uri = buildUrl('http://lockbitapt6vx57t3eeqjofwgcglmutr3a35nygvokja5uuccip4ykyd.onion');
      this.logger.log(`Navigating to:${uri}`);
      await hero.goto(uri, { timeoutMs: 30 * 1000 });
      this.logger.log('Waiting for main page to load');
      // await hero.activeTab.waitForLoad(LocationStatus.AllContentLoaded);
      await hero.document.querySelector('a[href*=\\/conditions]').$waitForExists({ timeoutMs: 40 * 1000 });
      const profile = await hero.exportUserProfile();
      this.logger.debug('returning cookies');
      res.status(HttpStatus.OK).send(profile.cookies);
    } catch (e) {
      this.logger.error('Problem loading page', e);
      res.writeHead(HttpStatus.INTERNAL_SERVER_ERROR, `Error with Hero -> ${e}`).end(`Error with Hero -> ${e}`);
    } finally {
      await hero.close();
    }
  }
blakebyrnes commented 2 years ago

@ctaity per discussions on Discord, I can't really see anything odd going on in logs. Does it take a really long time to connect over the proxy? I think it times out after 10 seconds if the socket doesn't start responding.

blakebyrnes commented 1 year ago

I'm going to close this as we have another user using socks5 tor proxies fine.