Dynamic SSL example?

[s3rj1k]

Is there a possibility to set path to SSL cert dynamically using TLS_SNI headers and some additional string mangling using JS?

If possible can we have an example for this?

[xeioex]

Hi @s3rj1k,

Yes it is possible. See this for the details.

$ssl_server_name corresponds to TLS_SNI. Using js_set you can implement arbitrary transformations.

js_include http.js;
js_set $js_cert_path js_cert_path;

...
ssl_certificate     $js_cert_path;
ssl_certificate_key  ...;

http.js:
function js_cert_path(r) {
    return r.variables.ssl_server_name.toUpperCase();
}

[s3rj1k]

@xeioex Hi, thanks for quick response, Can I'll be bold enough to ask additional info on example on how to check for file existence using JS.

That should cover all my needs.

The idea bind this is to use dynamic SSL with wildcard certs.

Thanks.

[xeioex]

@s3rj1k

how to check for file existence using JS.

The is no special method to check for file existence. For example, in node.js fs.exists()API is deprecated because of possible race conditions. The recommended way is to read/write file directly and handle the results.

[s3rj1k]

@xeioex no fs.stat() or fs.access() equivalent in njs?

[xeioex]

@s3rj1k

No, they are not implemented yet.

[s3rj1k]

@xeioex ability to somehow check if existence with fs.access() would be a great addition.

Is there any ETA on this?

[xeioex]

@s3rj1k We plan to extend fs module in the middle term future (BTW, patches are also welcome).

[s3rj1k]

@xeioex any updates on this?

because this code is super slow

var fs = require('fs');
var PREFIX = '/etc/nginx/ssl/';
var KEY_SUFFIX = '.key';
var CERTIFICATE_SUFFIX = '.crt';

function js_cert(r) {
    return read_cert_or_key(PREFIX, r.variables.ssl_server_name.toLowerCase(), CERTIFICATE_SUFFIX);
}

function js_key(r) {
    return read_cert_or_key(PREFIX, r.variables.ssl_server_name.toLowerCase(), KEY_SUFFIX);
}

function read_cert_or_key(prefix, domain, suffix) {
    var none_wildcard_path = String.prototype.concat(prefix, domain, suffix);
    var wildcard_path = String.prototype.concat(prefix, domain.replace(/.*?\./, '*.'), suffix);
    var data = '';

    try {
        data = fs.readFileSync(none_wildcard_path);
    } catch (e) {
        try {
            data = fs.readFileSync(wildcard_path);
        } catch (e) {
            data = '';
        }
    }

    return data;
}

[xeioex]

@s3rj1k

because this code is super slow

can you share your performance results? Please also note that loading certificates dynamically is pricy (especially for encrypted certificates, slowdown may be x8 and more (handshakes per second)):

http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_certificate

Note that using variables implies that a certificate will be loaded for each SSL handshake, and this may have a negative impact on performance.

[s3rj1k]

I'll do another bunch of tests to confirm, I think something went wrong during initial test.

Observing strange results...

[s3rj1k]

@xeioex Ok, so I am seeing x7 slowdown when using njs with above code if comparing with direct variable usage (meaning ssl_certificate /etc/nginx/ssl/$ssl_server_name.crt).

also this exact magnitude of slowdown is observed then map

map $ssl_server_name $ssl_cert {
hostnames;
...
}

is used that maps filepaths to SNI hostname.

I assumed that map would be faster.

Moreover using map as storage for content of certificates does not help, same x7 slowdown.

I am assuming this is somehow related to memory copying inside nginx? What can be done to speed this up somehow?

I am also curious performance of ngx_http_keyval_module then used as certificate store, sadly I do not have Nginx Plus.

Provided greater performance of K/V + njs solution, could be a great selling point of Nginx Plus.

[xeioex]

@s3rj1k

Ok, so I am seeing x7 slowdown.

Most probably it means that you are using encrypted certificates which is not recommended for dynamic certs. Encrypted certificates are protected with hash-functions which are designed to be slow.

Not-encrypted certs would give you 2-3x slow down (handshake/second). Overall slowdown may be reduced using keep-alive connections. So, dynamic certs is inherently pricy and you should only do this if you have no other options (you have more than 10k sites per nginx instance which are constantly changing). Otherwise the choice is not justified.

[xeioex]

@s3rj1k

Ok, so I am seeing x7 slowdown when using njs with above code if comparing with direct variable usage (meaning ssl_certificate /etc/nginx/ssl/$ssl_server_name.crt).

I doubt the results because of this: http://hg.nginx.org/nginx/file/tip/src/http/modules/ngx_http_ssl_module.c#l865

Any variable in ssl_certificate or ssl_certificate_key makes certificates dynamic.

[s3rj1k]

@xeioex I my use case dynamic certs are needed, so I am hoping to achieve similar performance to direct use of variables in ssl_certificate.

I am using Lets Encrypt certs issued for domain.net, *.domain.net.

I am not convinced that certificate encryption are to blame for a slowdown, same certs work faster then used without njs, using variable $ssl_server_name in ssl_certificate.

How can I check and confirm that this certs are encrypted, just to make sure?

I am testing with vegeta, below are results for njs and for variable in ssl_certificate.

Variable:

echo "GET https://sub.domain.net/" | vegeta attack -duration=30s -max-workers=250 -rate=15000 -timeout 300s | tee nginx_variable.bin | vegeta report
Requests      [total, rate, throughput]  450004, 15000.17, 15000.11
Duration      [total, attack, wait]      30.000036967s, 29.999917402s, 119.565µs
Latencies     [mean, 50, 95, 99, max]    1.89612ms, 267.038µs, 4.172452ms, 9.222129ms, 1.793154975s
Bytes In      [total, mean]              0, 0.00
Bytes Out     [total, mean]              0, 0.00
Success       [ratio]                    100.00%
Status Codes  [code:count]               201:450004
Error Set:

njs:

echo "GET https://sub.domain.net/" | vegeta attack -duration=30s -max-workers=250 -rate=15000 -timeout 300s | tee njs.bin | vegeta report
Requests      [total, rate, throughput]  450004, 15000.15, 6514.58
Duration      [total, attack, wait]      1m9.076482659s, 29.999964519s, 39.07651814s
Latencies     [mean, 50, 95, 99, max]    7.452839ms, 6.71063ms, 14.517587ms, 21.371967ms, 1m8.383661226s
Bytes In      [total, mean]              0, 0.00
Bytes Out     [total, mean]              0, 0.00
Success       [ratio]                    100.00%
Status Codes  [code:count]               201:450004
Error Set:

Looking at mean and 50 latencies I see x7 slowdown for njs vs variable

Can you do tests from your side to double check results, maybe I am doing something wrong?

[xeioex]

@s3rj1k

please, share your nginx.conf and js file.

[s3rj1k]

@xeioex

js:

var fs = require('fs');
var PREFIX = '/etc/nginx/ssl/';
var KEY_SUFFIX = '.key';
var CERTIFICATE_SUFFIX = '.crt';

function js_cert(r) {
    return read_cert_or_key(PREFIX, r.variables.ssl_server_name.toLowerCase(), CERTIFICATE_SUFFIX);
}

function js_key(r) {
    return read_cert_or_key(PREFIX, r.variables.ssl_server_name.toLowerCase(), KEY_SUFFIX);
}

function read_cert_or_key(prefix, domain, suffix) {
    var none_wildcard_path = String.prototype.concat(prefix, domain, suffix);
    var wildcard_path = String.prototype.concat(prefix, domain.replace(/.*?\./, '*.'), suffix);
    var data = '';

    try {
        data = fs.readFileSync(none_wildcard_path);
    } catch (e) {
        try {
            data = fs.readFileSync(wildcard_path);
        } catch (e) {
            data = '';
        }
    }

    return data;
}

nginx_vhost.conf:

js_include /etc/nginx/conf.d/cert.js;
js_set $js_cert js_cert;
js_set $js_key js_key;

server {
    listen 443 ssl;

    server_name _;

    ssl_certificate data:$js_cert;
    ssl_certificate_key data:$js_key;

    ssl_protocols TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_dhparam /etc/nginx/ssl/dhparams.pem;
    ssl_ciphers EECDH+AESGCM:EDH+AESGCM;
    ssl_ecdh_curve secp384r1;

    ssl_session_cache shared:SSL:100m;
    ssl_session_timeout 10m;

    location / {
        return 201;
    }
}

[xeioex]

@s3rj1k

Can you do tests from your side to double check results, maybe I am doing something wrong?

1) What you need to measure is handshakes per second, you are measuring something different. Keep-alive option should be disabled: -keepalive=false.

echo "GET https://dev.xeioex.pro:8443/" | vegeta attack -duration=5s -max-workers=250 -rate=15000 -keepalive=false -timeout 300s | vegeta report
Requests      [total, rate, throughput]  1341, 268.01, 164.41
Duration      [total, attack, wait]      8.156297728s, 5.003607868s, 3.15268986s
Latencies     [mean, 50, 95, 99, max]    1.158997452s, 617.806913ms, 7.790309768s, 8.109414508s, 8.14398642s
Bytes In      [total, mean]              0, 0.00
Bytes Out     [total, mean]              0, 0.00
Success       [ratio]                    100.00%
Status Codes  [code:count]               201:1341  
Error Set:

The heaviest operation for TLS handshake is RSA sign by the server. You can measure it for a single CPU using openssl speed rsa2048. sign/s - is the upmost value to expect for 1 worker.

openssl speed rsa2048
Doing 2048 bit private rsa's for 10s: 18542 2048 bit private RSA's in 10.00s
Doing 2048 bit public rsa's for 10s: 413764 2048 bit public RSA's in 10.00s
...
                  sign    verify    sign/s verify/s
rsa 2048 bits 0.000539s 0.000024s   1854.2  41376.4

2) while benchmarking enable 1 worker in nginx to get comparable results worker_processes 1.

[s3rj1k]

@xeioex How about setting worker_processes statically to nproc value, would it skew results?

I did a bunch of other tests with response body greater than zero, the bigger the response body size the lesser the difference between njs to variable.

So I can assume that your suggestion it pretty solid and confirms my latest tests.

I am just curious about worker_processes static distribution.

keepalive is a valid point, I thought I disabled it at some point in nginx.conf, seems that that was not the case.

[xeioex]

@s3rj1k

How about setting worker_processes statically to nproc value, would it skew results?

My guess, the result will differ no more than several percents.

[s3rj1k]

@xeioex Thanks, settings this Issue as solved.