search

apify / got-scraping

HTTP client made for scraping based on got.
422 stars 32 forks source link

Passing SSL_OP_LEGACY_SERVER_CONNECT down to Node #83

Open corford opened 1 year ago

corford commented 1 year ago

Node v18 removed the OpenSSL option to accept legacy servers. This causes Got to throw the following error when scraping servers that don't support RFC 5746 secure renegotiation:

RequestError: C0B70F932E7F0000:error:0A000152:SSL routines:final_renegotiate:unsafe legacy renegotiation disabled:../deps/openssl/openssl/ssl/statem/extensions.c:908:

Looking at the got-scraping source and also this issue https://github.com/apify/got-scraping/issues/75, it seems it should be possible to pass SSL_OP_LEGACY_SERVER_CONNECT down to Got via _unixOptions.secureOptions. But I can't seem to get it working.

This is what I tried (in my case via a Cheerio preNavigationHook(), since I'm trying to scrape this server using the Crawlee SDK, not got-scraping directly):

  import crypto from 'crypto';
  gotOptions._unixOptions = {
    secureOptions: crypto.constants.SSL_OP_LEGACY_SERVER_CONNECT,
  };
  gotOptions.https = {
    ciphers: [
      // Chrome v92
      'TLS_AES_128_GCM_SHA256',
      'TLS_AES_256_GCM_SHA384',
      'TLS_CHACHA20_POLY1305_SHA256',
      'ECDHE-ECDSA-AES128-GCM-SHA256',
      'ECDHE-RSA-AES128-GCM-SHA256',
      'ECDHE-ECDSA-AES256-GCM-SHA384',
      'ECDHE-RSA-AES256-GCM-SHA384',
      'ECDHE-ECDSA-CHACHA20-POLY1305',
      'ECDHE-RSA-CHACHA20-POLY1305',
      // Legacy:
      'ECDHE-RSA-AES128-SHA',
      'ECDHE-RSA-AES256-SHA',
      'AES128-GCM-SHA256',
      'AES256-GCM-SHA384',
      'AES128-SHA',
      'AES256-SHA',
    ].join(':'),
  };

Here are some SO links related to this OpenSSL issue for additional context: https://stackoverflow.com/questions/71603314/ssl-error-unsafe-legacy-renegotiation-disabled https://stackoverflow.com/questions/74324019/allow-legacy-renegotiation-for-nodejs

teammakdi commented 9 months ago

Facing the same issue @barjin @B4nan

Example url: https://www.hagerty.com/ads.txt

@corford let me know if any solutions

teammakdi commented 9 months ago

Right now I've added this ssl config as workaround

openssl.cnf

nodejs_conf = openssl_init

[openssl_init]
ssl_conf = ssl_sect

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
Options = UnsafeLegacyRenegotiation