search

nodejs / node

Node.js JavaScript runtime ✨🐢🚀✨
https://nodejs.org
Other
105.08k stars 28.46k forks source link

HTTP 449 error sending request from node version 17 and above #52449

Open wkuc opened 2 months ago

wkuc commented 2 months ago

Version

20.12.1

Platform

Darwin wkuc-na 23.1.0 Darwin Kernel Version 23.1.0: Mon Oct 9 21:28:12 PDT 2023; root:xnu-10002.41.9~6/RELEASE_ARM64_T8103 arm64

Subsystem

No response

What steps will reproduce the bug?

Just run code in node version >= 17:

const https = require('https');

https.get('https://autokult.pl', (res) => {
      console.log(process.versions.node, res.statusCode);
    let rawData = '';
    res.on('data', (chunk) => {
        rawData += chunk;
    });
    res.on('end', () => {
        //console.log(rawData)
    });
}).on('error', (e) => {
    console.error(`Got error: ${e.message}`);
});

Response code is 449.

How often does it reproduce? Is there a required condition?

Always in node environment >= 17

What is the expected behavior? Why is that the expected behavior?

Response should return status 200

What do you see instead?

Response code is 449.

Additional information

No response

targos commented 2 months ago

@nodejs/http

ShogunPanda commented 2 months ago

I tried on both node 20 and 21 without "luck": I always receive 200. Are you behind a proxy?

targos commented 2 months ago

I can reproduce with Node.js 21 from my network (no proxy):

> node -e "fetch('https://autokult.pl').then(console.log)"
Response {
  status: 449,
  statusText: '',
  headers: Headers {
    server: 'nginx',
    date: 'Wed, 10 Apr 2024 13:28:25 GMT',
    'content-length': '0',
    connection: 'keep-alive'
  },
  body: ReadableStream { locked: false, state: 'readable', supportsBYOB: true },
  bodyUsed: false,
  ok: false,
  redirected: false,
  type: 'basic',
  url: 'https://autokult.pl/'
}

With cURL, it returns 200.

ShogunPanda commented 2 months ago

Yeah, I observed the same from my SSH server in Finland. Will dig into it.

ShogunPanda commented 2 months ago

I tried with something simpler:

import {connect} from 'tls'

const client = connect({host: 'autokult.pl', port: 443})
client.setEncoding('utf-8')
client.on('data', console.log)
client.write('GET / HTTP/1.1\r\nHost: autokult.pl\r\n\r\n')

and I received 200. I guess we're sending something the server doesn't like. Still digging.

climba03003 commented 2 months ago

I am able to reproduce the issue with the author script.

Using Windows and WSL to test.

climba03003 commented 2 months ago

I suspect the 449 is due to the cipher list. Maybe it is the bug on openssl.

When you have the list TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA256:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!SRP:!CAMELLIA which is the default one will fail with 449. When ever remove the last one !CAMELLIA will return 200.

Note that the same cipher list on node@16 will works.

I though the problem is about OpenSSL 3 which is the default starting with node@17

You may check with the below command

# success
node --tls-cipher-list='TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA256:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!SRP' -e "fetch('https://autokult.pl').then(console.log)"
# failed with default
node --tls-cipher-list='TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA256:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!SRP:!CAMELLIA' -e "fetch('https://autokult.pl').then(console.log)"
climba03003 commented 2 months ago

After digging it deeper, the problem has two symptom.

  1. Cipher List which related to OpenSSL 3.

Removing :!CAMELLIA in the list will works in all case

  1. Failed whenever it provide servername in tls.connect

Removing servername will work for default cipher list.

ShogunPanda commented 2 months ago

I wonder if it is a server misconfiguration or a bug on our own. Give no other reports for a 449 happened before I lean on the former.

climba03003 commented 2 months ago

I wonder if it is a server misconfiguration or a bug on our own.

I am not good at C, but what I see in the code. When you got the servername or not, the code path for https handshake will be different. That would be the root cause of why sometimes works but sometimes do not.

wkuc commented 2 months ago

I also wonder where the misconfiguration is....

Here are websites with the same problem.

All websites are among the largest in Poland. It is worth adding that they belong to the same owner, which may mean the same server configuration.

Therefore, I expect that all modern browsers have no problem downloading these pages (they receive HTTP 200).

What could it mean that the implementation in Node is non-standard?

ShogunPanda commented 2 months ago

I think the opposite. 449 is a Microsoft specific HTTP response code. So it seems like an internal server (since they have Nginx on front) is messing up.

wkuc commented 2 months ago

I'm trying to play the role of an ordinary user who knows a little bit of coding and wants to download HTML from the above urls (it is possible that there are also other websites with a similar configuration).

Such a user can run the simplest commands to download html.

The problem is that very simple out-of-the-box solutions (which are easy to find on the Internet) work in:

but node >= 17 no longer works (including the simplest example: fetch("https://www.wp.pl") from node 21).

From the point of view of a non-advanced user, the image in your mind is that only the node has some problems (it's probably broken), something is wrong with it. It is possible that he will not dig deeper to discover an interesting secret.

ShogunPanda commented 2 months ago

I totally see your point and I also agree, but also think about the opposite case: the POV of a maintainer. So far only this owner leads to this behavior. What if we fix this and break somebody else?

Anyway, I'm not saying we're not going to intervene on this. We just want to make sure we do the right thing.

ShogunPanda commented 2 months ago

Ok, I found new leads. Given this snippet:

require('node:https').request('https://autokult.pl', r => {
  console.log(r.statusCode)
  console.log(r.headers)

  r.resume()
}).end()

Here's my results

Command Result
node 1.js fail
node --tls-min-v1.2 1.js fail
node --tls-min-v1.3 1.js pass
node --tls-max-v1.2 1.js pass
node --tls-max-v1.3 1.js fail

Which means that TLS selection is messing up.

@nodejs/crypto I have no knowledge in TLS. Is this ringing any bell?