Performance and certificate limits

[aleksy-zalenski]

Hello!

We are planning to run this solution for an nginx server that will be used to serve multiple sites. Therefore I wanted to ask the following question:

1. Is there any limit in regards to the number of certificates this solution can handle? To give you an idea, we plan to have 100+ domains on that server.
2. In regards to the performance, can we expect any downgrade on performance when using this solution? Is the private key fetched from ACM for every request or is there some caching mechanism implemented that will store the key locally to improve performance?

Thanks in advance for your help! :)

[alcioa]

@aleksy-zalenski Hi.

1. The enclave currently supports one server key and its optional associated certificate chain per token. There are currently maximum 8 tokens and 1024 sessions per token supported. This could be increased in the future if demand arrives. Meanwhile, please see the Using multiple certificates chapter in the docs on how to configure this. Could you provide more info on how you wish to integrate this? i.e. Nginx config file with one stanza file per domain for each private key?

2. This solution handles the provisioning and renewal of your ACM certificates (and the private key) on your EC2 instance. After you start the service the private key is provisioned in the enclave and NGINX uses the pkcs#11 engine over AF_VSOCK for local requests between your EC2 instance and the enclave. For example, when a signing operation is requested, the data is sent to the enclave, it gets signed by the local stored private key and the output is provided back. The private key is isolated and never leaves the enclave once provisioned.

You can setup your solution today and do an NGINX benchmark to see your performance areas of interest for up to 8 domains.

[aleksy-zalenski]

Hello @alcioa,

First of all, thank you very much for your quick response on this one. I really appreciate it.

In regards to your answers:

1. Oh I see, so basically the limit is only 8 certificates per instance, right? In regards to how I plan to integrate it -- basically like you mentioned I was planning to have a separate nginx stanza file for each domain (certificate and private key). Unfortunately the limit of 8 certs is way too low for us and kind of makes this solution unusable. May I ask you what's the reason of only up to 8 certs being supported and if it is possible to somehow increase it?
2. Got ya, so basically everything is done on the instance level. That's cool. So the singing is basically done in the enclave while NGINX is running on the instance itself and request is being handled there, right?

In case you need any additional input from my end, don't hesitate to let me know.

[alcioa]

@aleksy-zalenski currently investigating supporting either 128 tokens or more key-cert pairs per token. Meanwhile you can run some ab to check some benchmarks if curious (keep in mind that x86_64 enclave needs at least 2 vCPUs from your EC2 instance). Are you interested exclusively in NGINX? Apache HTTPD?

[aleksy-zalenski]

Hello @alcioa,

Thanks for the information :)

Do you have any rough estimate on when do you think your investigation regarding supporting more certificates will finish and can be implemented? I assume that supporting 128 tokens means we will be able to support 128 AWS ACM certificates, right?

In regards to the ab testing I've already run some tests last week and the results look promising. Below are the details:

[ec2-user@ip-10-100-1-119 ~]$ ab -n 10000 -c 20 https://test.oneeach.dev/
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking test.oneeach.dev (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests

Server Software:        nginx/1.20.0
Server Hostname:        test.oneeach.dev
Server Port:            443
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
Server Temp Key:        ECDH P-256 256 bits
TLS Server Name:        test.oneeach.dev

Document Path:          /
Document Length:        3520 bytes

Concurrency Level:      20
Time taken for tests:   43.153 seconds
Complete requests:      10000
Failed requests:        0
Total transferred:      37540000 bytes
HTML transferred:       35200000 bytes
Requests per second:    231.73 [#/sec] (mean)
Time per request:       86.306 [ms] (mean)
Time per request:       4.315 [ms] (mean, across all concurrent requests)
Transfer rate:          849.54 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        9   69  11.6     67    1071
Processing:     1   17   2.9     17      45
Waiting:        1   17   2.9     17      44
Total:         10   86  12.0     84    1088

Percentage of the requests served within a certain time (ms)
  50%     84
  66%     86
  75%     87
  80%     88
  90%     96
  95%    102
  98%    105
  99%    108
 100%   1088 (longest request)

It was run with only a single certificate installed, but I don't think more certificates will change anything in terms of performance since the requests are anyway handled independently.

In regards to the web server, to be honest for our use case it doesn't matter that much and we can use either Nginx or Apache. I've used Nginx since it was officially supported, but we can use whatever supports more AWS ACM certificates ;)

[alcioa]

Hi, PR #61 is up. Will be included in the next 1.2.0 RPM release which should be around mid April.

[alcioa]

@aleksy-zalenski anymore info needed here?

[azalenski-branch]

Hello @alcioa,

Thank you very much for implementing the fix, I really appreciate it.

Just for the future, in case we will need more than 128 ACM certs, do you think it will be possible to further increase the limit, say to 256?

[alcioa]

Hi, yes it can be done as there is no pkcs#11 spec restriction for that, but some more extensive changes might be needed in order to scale better as that number grows from I/O and memory perspective.

[alcioa]

@aleksy-zalenski the latest v1.1.2 RPM has been released with support of up to 128 ACM certificates. Will close this.

[aleksy-zalenski]

Hello @alcioa,

We are really close to hitting the 128 ACM certificates limit. As discussed previously, can we have this limit increased to 256 certificates per instance? If needed, I can create a separate Github issue.

Thanks in advance!