SSL handshake fails because IP SANs are missing

[soupdiver]

I've used the command from the README for generating a SSL certificate. I use the same cert on the server and on the forwarder. When starting the forwarder I get the following error:

Failed to tls handshake with x.x.x.x x509: cannot validate certificate for x.x.x.x because it doesn't contain any IP SANs

So I see it is a problem with the cert and not the forwarder itself but maybe you can update the README.

[alphazero-es]

@soupdiver Thanks for the heads-up. The readme needs an update, but what we really need is to add a flag to skip verification. We'll use this issue to push that through.

[soupdiver]

:+1:

[alphazero-es]

@soupdiver: Felix, pull branch ISSUE-221 and give it a whirl & ping me back on your end w/ ack|nack. It works for me. /Danke

[soupdiver]

@alphazero-es ack Thanks!

[alphazero-es]

@soupdiver great. @jordansissel Jordan, what's your take on this? This can merge clean to master. README is updated so it is buyer beware.

[jordansissel]

(edited this comment to remove incorrect information)

Basically, in order to use tls in Go, now you MUST specify a CN in the subject field. Wildcards are supported, but '*' does not match '.'

If you're using self signed certs or have a closed ecosystem of SSL certs in your environment, it's possible that CN=* or CN=*.example.com might just be fine for you.

master branch currently sets the ServerName (which must match the cert CN field) to the address of the server being talked to, so be aware of this.

I'm open to exposing a setting you can choose for the cert CN setting (what Go calls ServerName), if that would be useful.

[PavelPolyakov]

@jordansissel I've compiled my logstash-forwarder using go version go1.3 linux/amd64 and created my certs using your solution - however I still have this:

Have I've missed something? Could you give an example how could I create the certificate specifying the IP SAN ?

@alphazero-es Am I understand right that your pull request hasn't yet been merged and we're not able to use the

    "ssl strict verify": false,

yet ?

ALL: Am I understand right that if I would specify the domain name in the logstash.json, then it would work without updating the certificate?

Regards,

[alphazero-es]

@PavelPolyakov The consensus on our end is that it would violate expectation of security for users. @jordansissel has explored and found a viable and practical workaround.

In the interim, if using master, use Go1.2 or satisfy Go1.3's requirements per crypto/tls.

[PavelPolyakov]

@alphazero-es thanks for the answer,

do you mean the workaround about creating the certificate using CN=* ?

openssl req -subj '/CN=*/' -x509 -batch -nodes -newkey rsa:2048 -keyout lumberjack.key -out lumberjack.crt

The issue is, that even if I've created the certificate using the command provided, and updated it both on the receiver host and on the logstash-forwarder host - I still see that message.

[alphazero-es]

@PavelPolyakov Yes, it requires a minor change to the publisher code as well. If you can wait until Friday, hopefully we'll push it through then. In the interim, please use Go1.2, or the 221 branch (fully noting that it is NOT acceptable to us.

[driskell]

@PavelPolyakov Using CN=* will only work if you connect via hostname. If you connect via IP then the CN is completely ignored and the certificate verification requires that an IP Subject Alternate Name is added to the certificate. If you connect via hostname the CN=* will work, or even CN=hostname.you.connect.with will work.

I'm in process of writing a script to do all this in Log Courier. Currently I have a "make selfsigned" which works - but it needs manual editing of openssl.cnf to add IP SAN. I will be writing a script to do all this soon. It at least means people who just want quick security can get up and running fast. I'll post here when I'm done but @jordansissel or @alphazero-es are probably already doing something similar.

[PavelPolyakov]

@driskell When I connect via hostname it states the next:

where the blank space is my domain

@alphazerp-es I've just recompiled it using go 1.2.2 but still have the same:

my logstash-config.json:

Tomorrow I would try to do that again :) Anyway, thanks for your help.

[alphazero-es]

@PavelPolyakov don't understand the 1.2.x issue. I can't at this moment switch focus to this -- I would have to reinstall 1.2 and all that -- so this will have to wait until Friday (or possibly @jordansissel can provide some insight). if this is absolutely a blocker for you, then I suggest you just build from the prior commit and all should be OK.

Thank you for your patience.

/R

[driskell]

@PavelPolyakov this might be due to "*" matches "localhost" and "example" but not "example.com" according to my interpretation of it.

Try using the same hostname in certificate as you do in the config to connect.

"" is then not a solution as you'd need ".*" presumedly for "example.com"

[jordansissel]

@driskell I think you'er right, I didn't test . (I tested * against 'localhost'). Testing now.

[jordansissel]

What I can confirm:

• Without Subject (or SAN, aka Subject Alternative Name), connecting to a hostname gets you: Failed to tls handshake with 127.0.0.1 x509: certificate is valid for , not localhost
• Same as above, but connecting with "servers": [ "127.0.0.1:5043" ] (ip, not hostname): Failed to tls handshake with 127.0.0.1 x509: cannot validate certificate for 127.0.0.1 because it doesn't contain any IP SANs

IPs and Hostnames are configured differently, or appears to be so.

To permit IPs as the server name, your ssl cert must include IP:a.b.c.d as a subjectAltName field, for example, from a sample ssl config:

[v3_ca]
subjectAltName = IP:127.0.0.1

Generating a cert with this allows lsf to connect to 127.0.0.1 (by 'name') and it's succesful:

% openssl req -config ssl.conf -x509  -batch -nodes -newkey rsa:2048 -keyout lumberjack.key -out lumberjack.crt
# ... run logstash with this key/cert
% 2014/07/08 19:47:38.215155 Setting trusted CA from file: ./lumberjack.crt
2014/07/08 19:47:38.216058 Connecting to [127.0.0.1]:5043 (127.0.0.1)
2014/07/08 19:47:38.221611 Connected to 127.0.0.1

In this respect, DNS seems much easier to work with, but I won't burden you with requiring dns vs ip server names.

[jordansissel]

Also confirmed:

• * does not match '.' so you can't just do a simple wildcard.
• *.* does match foo.com
• *.*.pants.domain matches super.fancy.pants.domain

[jordansissel]

@PavelPolyakov Your SSL cert, if you are pointing atn an IP address for the server name, must include a subjectAlternativeName field that has IP:1.2.3.4 or whatever your IP is.

[PavelPolyakov]

ok guys, sorry, but I still don't have it working :(

1. I've tried the next command:

openssl req -subj '/CN=subdomain.domain.com/' -x509 -batch -nodes -newkey rsa:2048 -keyout logstash-forwarder.key -out logstash-forwarder.crt

The in config I had the next:

  "servers": [
            "subdomain.domain.com:12345"
        ],

The output was the next:

2014/07/09 08:37:36.341373 Connecting to [XX.XX.XX.XX]:12345 (subdomain.mydomain.com) 
2014/07/09 08:37:36.348066 Failed to tls handshake with XX.XX.XX.XX x509: certificate is valid for *, not subdomain.mydomain.com

So... looks like it still doesn't want to connect.

The certificate looks the next way:

1. @jordansissel

Then I've tried your variant with IP. I've created the ssl.conf with the next content:

[v3_ca]
subjectAltName = IP:XX.XX.XX.XX

Then I'm trying to generate the keys, and here is the output:

Pavels-Mac-mini:new_cert pavelp$ openssl req -config ./ssl.conf -x509  -batch -nodes -newkey rsa:2048 -keyout logstash-forwarder.key -out logstash-forwarder.crt
Generating a 2048 bit RSA private key
...............................+++
..........+++
writing new private key to 'logstash-forwarder.key'
-----
unable to find 'distinguished_name' in config
problems making Certificate Request
140735222621024:error:0E06D06C:configuration file routines:NCONF_get_string:no value:conf_lib.c:335:group=req name=distinguished_name

So I'm not able even to create the crt file (while .key is created).

I've tried to launch everything with the binary compiled using the go-1.3 .

However, currently I'm ok having the logstash-forwarder built using the go-1.2 . And I wait patiently for Friday, when the official solution would be exposed, hope it would work for me :)

Much thanks for the help.

Regards,

[ponny]

Same issue. Followed this guide and it broke: https://www.digitalocean.com/community/tutorials/how-to-use-logstash-and-kibana-to-centralize-and-visualize-logs-on-ubuntu-14-04

[ponny]

Tried with my ip: sudo openssl req -subj '/CN=xx.xx.xx.xxx/' -x509 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

No luck. Same error.

[ponny]

Added "ssl strict verify": false

No IP SANs errors any more. No data either ಠ_ಠ

[jordansissel]

@PavelPolyakov Your certificate error says certificate is valid for *, not subdomain.mydomain.com which indicates your certificate is still for "*" not for "subdomain.mydomain.com"

@ponny Based on what I have tested, you cannot add IP addresses in the CN field, you MUST add a subjectAltName as described in this comment: https://github.com/elasticsearch/logstash-forwarder/issues/221#issuecomment-48390920

Allowing folks to disable ssl cert verification disables the very security that SSL provides.

We CAN solve this in a way that doesn't continuously confuse users. However, for NOW you must use the instructions I gave in the comment here: https://github.com/elasticsearch/logstash-forwarder/issues/221#issuecomment-48390920 - generate a correct certificate and it works.

[ponny]

Where will I find this ssl config on Ubuntu 14.04? Do I need to do this on the logstash server or the forwarder nodes?

[jordansissel]

@ponny what ssl config? I'm confused now. You mean an ssl config file? You can copy whatever your distro ships with (probably in /etc/ssl/openssl.cnf?) and add the subjectAltName lines.

[ponny]

@jordansissel That did it. Thanks.

[ponny]

The reason it wasn't sending data before was my clumsy fingers typed "falSE" which threw the config in to a spin.

[jordansissel]

ouch! Typos like that are annoying ;)

[PavelPolyakov]

@jordansissel However, I've shoen the certificate image, it said that it's issued for 'subdomain.domain.com', so why it's still considered as '*' ?

And the other question - any thoughts why I have an error during the certificates generation with the ssl.conf param?

[driskell]

@PavelPolyakov Check that your LSF config points to the correct certificate, and that you restarted it. If it says the certificate it loaded was for "*" it probably is. So either its loading the wrong file or hasn't reloaded it.

[ArkeologeN]

Hey,

I'm following this link to deploy ELK stack: https://www.digitalocean.com/community/tutorials/how-to-use-logstash-and-kibana-to-centralize-and-visualize-logs-on-ubuntu-14-04

I've successfully exported certificates, keys, compiled code and installed logstash-forwader on my client machine(s).

The logstash server machine works fine and delivering logs to elasticsearch while the logstash-forwarder throws the following error in logs:

Jul 12 17:46:46 ip-172-31-42-13 logstash-forwarder[11532]: 2014/07/12 17:46:46.430561 Failed to tls handshake with [STATIC_IP]:5000 x509: cannot validate certificate for [STATIC_IP]:5000 because it doesn't contain any IP SANs

Any quick through is highly appreciated. I tried exporting certificates by following line:

sudo openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

for save case, I also tried

openssl req -config ./ssl.conf -x509  -batch -nodes -newkey rsa:2048 -keyout logstash-forwarder.key -out logstash-forwarder.crt

Which throws:

Generating a 2048 bit RSA private key
...............................+++
..........+++
writing new private key to 'logstash-forwarder.key'
-----
unable to find 'distinguished_name' in config
problems making Certificate Request
140735222621024:error:0E06D06C:configuration file routines:NCONF_get_string:no value:conf_lib.c:335:group=req name=distinguished_name

Any quick clue?

[zSprawl]

I've messed with it for a few hours and can not seem to get a valid certificate. Does anyone have a step by step handy?

[ArkeologeN]

@zSprawl I had been following this: https://www.digitalocean.com/community/tutorials/how-to-use-logstash-and-kibana-to-centralize-and-visualize-logs-on-ubuntu-14-04 but seems its also not working.. hours already so far and no luck to make logstash-forwarder working..

[zSprawl]

@ArkeologeN I'm following the exact same document and stuck the exact same place. I think we posted on both threads too, hehe. Hopefully someone can help. I think I've tried all the solutions above, but I might not have done it right.

[ArkeologeN]

@zSprawl true story.. no luck! neither certificates works nor any other stuff.. Well, I'm hoping to see project leader to solve the mystery.. hehe!

[driskell]

Hi

Try downloading this file and building it: https://github.com/driskell/log-courier/blob/develop/src/lc-tlscert/lc-tlscert.go

I think go build lc-tlscert.go should work.

The KEY thing, mentioned here several times, is that IF you connect using an IP address then your certificate MUST contain a matching IP SAN to pass validation with Go 1.3, which nearly all certificate guides online (including ELK ones) do not currently cover.

The above program will ask for additional DNS names and IP addresses. Enter the IP you connect with there. It will add an IP SAN to the resulting certificate and it should work.

The program is based on Go's own certificate generator it uses to test TLS so should be good.

[driskell]

updated comment to elaborate a bit more

[zSprawl]

Script worked perfectly. Thank you!

[ArkeologeN]

@driskell you rock! It works! However, now I faced timeout i/o exception after few moments of working happiness! :D Might be something wrong with message format..

[driskell]

@arkeologen the timeouts are likely due to logstash pipeline full. This causes timeouts on the connections. You need a faster logstash or a broker to buffer logs.

180 fixes it - try that. Log Courier in my github has completely rewritten protocol but as such needs extra steps to get the logstash bit working. If you can get #180 built tho things will stop timing out completely or almost completely.

[PavelPolyakov]

@driskell

Hi,

I've tried your go script to generate the certs. But even that hadn't helped me :(

I've stated my domain in the certificate as CA as well as as DNS, I even stated the IP SAN. I've uploaded the new certificates to the logstash server and reloaded it.

I feel myself cursed making to work go-1.3 , logstash-forwarder and the logstash.

Waiting for the changes in the official documentation, hope that would help.

Any thoughts why it doesn't work for me? I've tried everything so far - starting from the generation the cert using the command line, finishing with the custom tool...

When do we expect the official update regarding the issue?

Thanks everyone for the help.

Regards,

[driskell]

Hi @PavelPolyakov

Please make sure you are using the latest version of Logstash Forwarder. The error you state only occurs in the older versions and was recently fixed in the master branch.

"either ServerName or InsecureSkipVerify must be specified in the tls."

Jason

[PavelPolyakov]

@driskell

Thanks, the update it work finally!

[jordansissel]

We're going to try and provide a script to help generate certs for users. Please stand by.

[iainlbc]

For our AWS VPC based environment we ended up generating the CSR with CN of *.us-west-1.compute.internal and using the internal dns names for the logstash endpoints. Cheers

[sandstrom]

Here is a snippet that works. Perhaps in the future Logstash Forwarder will make TLS optional, for those who've used other methods to secure the network (e.g. ipsec) or are running on an internal network.

# use this for config https://gist.github.com/sandstrom/a92a9d384999c659d96a
openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout insecure.key -out insecure.crt -config insecure.cnf -days 3650

[driskell]

@sandstorm Loving the openssl command lol. But I would advise others to steer well clear of it!

Using my script I posted earlier is the best option for now until there's an official LSF one.

If you absolutely want rid of TLS tho; Log Courier, which is where the lc-tlscert script I posted comes from, allows TCP only.

[jordansissel]

I repeat this often, but now again: If you don't need security, I highly recommend using any other transport tool.

I'm still open to discussion about the future features of this project, but right now, we are not removing TLS.

[jordansissel]

I made an attempt to get an openssl config file that would prompt you for things to make a self-signed cert, but I can't figure out yet how to make it prompt you for ip/hostname/etc.

I'll try a different approach tomorrow.

[devryan]

Wow, @sandstrom's post above actually worked for me after nothing else did (I understand it's completely insecure but hey, it works). I'd had Go 1.2.2, and latest as of today logstash-forwarder codebase.