http2.connect() doesn't support self-signed cert in `ca` field

[satyarohith]

deno 1.42.4 (release, aarch64-apple-darwin)
v8 12.3.219.9
typescript 5.4.3

Bug

➜  http2 deno run -A client.mjs 
Error: invalid peer certificate: Other(CaUsedAsEndEntity)
    at async node:http2:279:11
error: Uncaught (in promise) Error: invalid peer certificate: Other(CaUsedAsEndEntity)
    at async node:http2:569:24

To reproduce the bug, you can follow the instructions from https://nodejs.org/api/http2.html#server-side-example. I'm pasting code converted to ES modules for convenience:

1. generate certs using the following command:

   openssl req -x509 -newkey rsa:2048 -nodes -sha256 -subj '/CN=localhost' \
   -keyout localhost-privkey.pem -out localhost-cert.pem

2. create server.mjs

   
   import { createSecureServer } from 'node:http2';
   import { readFileSync } from 'node:fs';

const server = createSecureServer({ key: readFileSync('localhost-privkey.pem'), cert: readFileSync('localhost-cert.pem'), }); server.on('error', (err) => console.error(err));

server.on('stream', (stream, headers) => { // stream is a Duplex stream.respond({ 'content-type': 'text/html; charset=utf-8', ':status': 200, }); stream.end('

Hello World

'); });

server.listen(8443);

3. create client.js
```js
import { connect } from 'node:http2';
import { readFileSync } from 'node:fs';

const client = connect('https://localhost:8443', {
  ca: readFileSync('localhost-cert.pem'),
});
client.on('error', (err) => console.error(err));

const req = client.request({ ':path': '/' });

req.on('response', (headers, flags) => {
  for (const name in headers) {
    console.log(`${name}: ${headers[name]}`);
  }
});

req.setEncoding('utf8');
let data = '';
req.on('data', (chunk) => { data += chunk; });
req.on('end', () => {
  console.log(`\n${data}`);
  client.close();
});
req.end();

Run the below command to test nodejs

➜  http2 node server.mjs & node client.mjs
[1] 5576
:status: 200
content-type: text/html; charset=utf-8
date: Mon, 22 Apr 2024 01:36:06 GMT

<h1>Hello World</h1>

Then run the below command to test deno (assuming that you didn't kill the server)

➜  http2 deno run -A client.mjs 
Error: invalid peer certificate: Other(CaUsedAsEndEntity)
    at async node:http2:279:11
error: Uncaught (in promise) Error: invalid peer certificate: Other(CaUsedAsEndEntity)
    at async node:http2:569:24

[lucacasonato]

To be honest, this is just correct behaviour from rustls. Node's behaviour is abysmal and we should not implement it - it directly contradicts the spec - CA certs must never also be end entity certs. You must always have a different CA cert and end entity cert certs.