How to modify the certificate size limit when deploying a TLS server

[shilohshi]

Hi, I have noticed that when deploying a server using botan or botan-tls-testserver, there seems to be a direct or indirect limit on certificate size in the code. Specifically, when the server certificate exceeds a certain size (e.g., over around 120KB), the client receives an ERR_SSL_PROTOCOL_ERROR upon subsequent access. I would greatly appreciate any guidance on how to manually modify this size limit. Thank you!

[randombit]

Thanks for the report. I’m not sure offhand why this would occur and it certainly seems like a bug.

Am I understanding correctly that the situation is that if the server is provisioned with a large cert, then upon connecting, the client receives the alert?

Which version(s) have you tried?

Also if you force the client to connect using TLS v1.2 vs v1.3 do you see any difference in behavior?

[randombit]

Was able to replicate this with master. Only affects TLS 1.3, errors with "No sufficient server certificate available"

@reneme any ideas

Util for creating a big cert

#include <botan/rsa.h>
#include <botan/x509self.h>
#include <botan/system_rng.h>
#include <botan/pkcs8.h>
#include <iostream>

int main() {
   using namespace Botan;

   auto& rng = system_rng();

   auto rsa = RSA_PrivateKey(rng, 4096);

   X509_Cert_Options opts;
   opts.common_name = "CN";

   for(size_t i = 0; i != 10000; ++i) {
      opts.more_dns.push_back("dns" + std::to_string(i) + ".domain.com");
   }

   auto cert = X509::create_self_signed_cert(opts, rsa, "SHA-256", rng);

   std::cout << PKCS8::PEM_encode(rsa) << "\n";
   std::cout << cert.PEM_encode() << "\n";
}

[randombit]

Interesting with OpenSSL cli I instead get

805B7CF4EC770000:error:0A000098:SSL routines:read_state_machine:excessive message size:ssl/statem/statem.c:647:

both TLS 1.2 or TLS 1.3

[reneme]

For the record: There is size_t TLS_Policy::maximum_certificate_chain_size() which allows configuring the maximum size of the transferred certificate chain in bytes. Though, this is set to 0 (unlimited) by default.

[reneme]

Interesting with OpenSSL cli I instead get

805B7CF4EC770000:error:0A000098:SSL routines:read_state_machine:excessive message size:ssl/statem/statem.c:647:

OpenSSL seems to restrict the CERTIFICATE message to 100kB by default: https://github.com/openssl/openssl/blob/e1886edf4bffef7663eb5bb4bee43d0232d8b68d/include/openssl/ssl.h.in#L663-L664

Was able to replicate this with master. Only affects TLS 1.3, errors with "No sufficient server certificate available"

It works for me, actually! The error message you're seeing (@randombit) indicates that the server didn't find a suitable certificate in the Credentials_Manager. I adapted your certificate creation program slightly (CN=localhost and more_dns.push_back("localhost")) and it worked for me, using ./botan tls_server ... and ./botan tls_client localhost. Both for TLS 1.3 and 1.2.

For the record: the resulting certificate is about 200kB in DER format.

[reneme]

@shilohshi could it be that your client relies on OpenSSL internally? Then you might actually be running into a client-side limit rather than on the server.

Please try this against your Botan-based server with the large certificate configured:

openssl s_client -connect your.domain:443 -servername your.domain -debug

If this prints 100+kB of hex barf, you are receiving your certificate from the server but OpenSSL aborts the connection because of the limitation mentioned in my previous post. If so, try again without the -debug and you should see what Jack saw, namely:

> openssl s_client -connect your.domain:443 -servername your.domain
40679B56567F0000:error:0A000098:SSL routines:read_state_machine:excessive message size:../ssl/statem/statem.c:610

[shilohshi]

Thank you very much for your prompt response! @randombit @reneme

I am using Botan version 3.3.0, and I deployed a website using the following command:

botan tls_http_server large_cert.crt large_cert.key

with the certificate (172KB) and key provided below: large_cert_and_key.zip

When I access the URL of my deployed website in Chrome, I receive the ERR_SSL_PROTOCOL_ERROR message. At this point, the error message displayed by the Botan server is as follows:

Session: illegal_parameter [Botan TLS Alert:47]

However, if I switch to the following certificate (78KB) and key, then the above error message will not be received: small_cert_and_key.zip

For the record: There is size_t TLS_Policy::maximum_certificate_chain_size() which allows configuring the maximum size of the transferred certificate chain in bytes. Though, this is set to 0 (unlimited) by default.

If I understand correctly, does this mean that Botan does not impose any restrictions on the size of certificates or TLS records by default? If that is the case, does it imply that the access issues are actually originating from the browser rather than the server deployed with Botan?

Thank you very much for your assistance!

[reneme]

If I understand correctly, does this mean that Botan does not impose any restrictions on the size of certificates or TLS records by default? If that is the case, does it imply that the access issues are actually originating from the browser rather than the server deployed with Botan?

Yes, Botan does not impose such restrictions by default. And, yes, I strongly believe that it is the browser that is rejecting your large certificate.

When I access the URL of my deployed website in Chrome, I receive the ERR_SSL_PROTOCOL_ERROR message. At this point, the error message displayed by the Botan server is as follows: Session: illegal_parameter [Botan TLS Alert:47]

My theory: Chrome rejects your large certificate (printing the protocol error) and aborts the handshake by sending a TLS alert "illegal_parameter", that's what your Botan-based server receives and prints. I saw the same behaviour when trying to connect using OpenSSL (as stated above).

[randombit]

BoringSSL seems to have the same 100 kb limit on the chain, presumably inherited from OpenSSL, so Chrome likely rejects any chains larger.

[shilohshi]

Thank you very much for your explanation; I understand now.

[reneme]

Closing as 'not a bug'.