SSL handshake fails because IP SANs are missing

[soupdiver]

I've used the command from the README for generating a SSL certificate. I use the same cert on the server and on the forwarder. When starting the forwarder I get the following error:

Failed to tls handshake with x.x.x.x x509: cannot validate certificate for x.x.x.x because it doesn't contain any IP SANs

So I see it is a problem with the cert and not the forwarder itself but maybe you can update the README.

[DanielRedOak]

On the same track here, I am trying to use a SAN on my cert with instances behind a load balancer but it doesn't seem to be taking those into consideration. I am receiving errors like: Failed to tls handshake with xxx.xxx.xxx.xxx x509: certificate is valid for fqdn hostname, not load balanced dns name

I verified the cert has the fqdn in the CN and the load balanced dns name in the SAN but no go.

[jordansissel]

@DanielRedOak please put in a gist your logstash-forwarder config with the "ssl ca" file contents as well as the SSL certificate you provide to your logstash servers.

[DanielRedOak]

I can do that, although I should add that I am re-using certs provided by puppet on these machines. They all use the ca provided by puppet, and their own certs signed against that ca which is all done when the machine first checks in. It worked up until I built the latest to get the persistence and rotation fixes :D

[jordansissel]

Well that's why it fails, probably. Puppet generates certs for each server (the fqdn). Therefore, you pointing lsf at a dns name that is not the fqdn of the server, it doesn't match and is failing to do so correctly.

If you have a dns name "logstash.example.com" that resolves to 10.0.0.1, and 10.0.0.1's fqdn is "fancypants.example.com", then logstash-forwarder will expect the SSL cert to verify as "logstash.example.com" but it receives one for "fancypants.example.com" and fails to match. You must have, for now, your "fancypants.example.com" (the fqdn of a server) use an ssl cert that matches what the logstash-forwarder expects.

[DanielRedOak]

Fortunately that's where the subject alternative names (SAN) are supposed to come in AFAIK. Since all of these logstash servers are sitting behind a load balancer, they have their own FQDN on the cert but also have the dns name for the load balancer as a subject alternative name.

[jordansissel]

I'd have to see the cert to know where to look for a bug. Sorry you're still having issues.

[DanielRedOak]

Gist here: https://gist.github.com/DanielRedOak/6bfb6b500e949043fb5e with output from openssl for viewing

[TinLe]

FYI. I was adding logstash-forwarder to new servers today and run into exact same TLS error. It is not Go v1.3. Something in the latest logstash-forwarder commit broke TLS.

I am using Go v1.2.1. The binary I built with it works fine (commit 66c29b5cb208c1be7a675707939a2335c2b66605 from 4/29/2014). But I pulled trunk today (commit 8bb400188c919ffc0f7c21be3aada51e5ce7c5c7) and the new binary built with Go v1.2.1 is broken. I was getting exact same errors being reported here. Replacing that binary with the one built earlier and everything is working again.

[driskell]

@tinle the latest commit fixed TLS. Previously it was not validating the server name at all. Your problem is likely that your certificate is not matching the server IP or name you use to connect. So now the validation is fixed it's stopped working.

[TinLe]

@driskell Ah, that's possible. I have not had time to test that today, but will tomorrow. I might as well whip up a script to gen certs to go with this.

[TinLe]

@driskell Just wanted to close the loop. I generated a new cert, with right data in openssl.cnf file and this time it works fine with latest binary built from trunk. Still using go v1.2.1 though.

All is well in the world.

[pascalrobert]

I updated my clone of the repo (master branch) and built logstash-forwarder with Go 1.3. I had the TLS errors, and saw this issue by doing a Google search. Downgraded to Go 1.2.2, and still having the same problem. logstash-forwarder is connecting to Logstash with a hostname (DNS).

If I create a new certificate with * as the subject, I'm getting:

Connecting to [192.168.1.108]:6790 (logstash.xxx) Failed to tls handshake with 192.168.1.108 x509: certificate is valid for *, not logstash.xxx

If I recreate the certificate with 'logstash.xxx' as the subject on server, but the certificate on the "client" is still with '*' as the subject, I get:

Failed to tls handshake with 192.168.1.108 x509: certificate signed by unknown authority

If I copy the certificate where the subject is 'logstash.xxx' on the client, everything works.

[jordansissel]

@pascalrobert - Sorry for the confusion. Downgrading to Go 1.2 is not an option anymore because we made the code work with Go 1.3. You can remove the 'ServerName' setting in publisher1.go if you wish, and it should work in Go 1.2, but this may diminish security, I don't know.

It sounds like you got it working and even despite the minefield that TLS presents. I am happy for that.

[rekuplex]

Hi, tried a lot of solutions from this thread, but I'm still stuck with the same error message:

"Failed to tls handshake with x.x.x.x x509: certificate is valid for , not serverB" Here I'm using this command to create the certs:

"sudo openssl req -subj '/CN=server.domain./' -x509 -nodes -days 3650 -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt"

Tried it with the subjectAltName and used IP's instead of hostnames but the same error appeared.

Worth mentioning that my shipper server (serverB) and central log server (serverA) both have 2 interfaces on different subnets, but on the DNS entries I only registered one of them for each one.

Any ideas on how to troubleshoot this? , been stuck here for a week now.

Thanks

[larryj0nes]

@rekuplex I spent several hours tonight trying a lot of the solutions in this thread. Then I realized my own mistake was not restarting logstash on the host server after regenerating the certificate and private key. I was only reloading the logstash-forwarder service on the client. What ended up working for me was using these parameters in my openssl config:

[ v3_ca ] subjectAltName = IP.1:xxx.xxx.xxx.xxx,DNS.1:server.mydomain,DNS.2:,DNS.3:.mydomain

[rekuplex]

Finally solved it after building it from the last commit and just adding the -subject -subj '/CN=server.domain/ to the command.

[gdubicki]

@driskell Thanks for the lc-tlscert script solution. It's the only one that worked for me. Please just add it to the README at https://github.com/elasticsearch/logstash-forwarder#generating-an-ssl-certificate and then maybe we can at last consider this issue closed...?

[molszanski]

So much time wasted...

In a nutshell, here is what worked for me:

1) Install go compiler on a "monkey" instance. 2) Downloaded file: https://github.com/driskell/log-courier/blob/develop/src/lc-tlscert/lc-tlscert.go 3) Compile it to "make-shiny-certs" 4) ./make-shiny-certs 5) got 2 files: .cert .key 6) Copy that files to a Logstash server instance. Update configs! 7) Copy files to a logstash-forwarder instance. Update configts! 8) Restart logstash service and logstash forwarder

[FlorinAndrei]

@driskell - Jason, lc-tlscert.go is great but one issue with it is that it hardcodes the client IPs or hostnames. This is not feasible in some cloud environments, where your instances may or may not come up on well-known addresses or hostnames. Also, what happens if you have a large but variable number of clients?

Basically, it would work in an old-school bare-metal environment where things are not changing too much.

What's needed is a solution that works in a fluid environment, where the client fleet may change quite a bit.

[jordansissel]

@FlorinAndrei You can achieve this with your own SSL certificate authority to sign certificates per client, etc. It really depends on your environment's trust needs. I won't say it's easy, but we could make it easier. Puppet, Chef, and Kerberos all do trust management pretty well, for example.

[jviney]

Apart from creating your own CA, is there any solution for cloud based infrastructure where nodes can come and go at any time?

[jordansissel]

@jviney I don't know how you prefer to define trust and distribute credentials among your infrastructure, so without more details, I can't offer you any suggestions. OpenSSL provides tools to create and manage certificate authorities, certificates, and keys. Logstash-forwarder only supports certificate-based authentication at this time.

[driskell]

@florinandrei It only hard codes the server IP or hostname? Not the client?

In this case you generally have a fixed hostname hitting an ELB (use CNAME to make it friendlier). You could share the same private key amongst any logstash auto scaling group to ease. If you need different then I'd call it more complex and you'd need your own certificate request / response APIA for when servers are booting.

[torrancew]

@jviney We're a puppet shop, and I've found it pretty simple to re-use the certs Puppet provisions its agents with for LSF. Since all of our clode nodes check into puppet, they always have a cert signed by a single CA, and this solution has been pretty elegant for us. If you have a CM solution with a similar architecture, you may be able to re-use those.

[jviney]

@torrancew Thanks for the idea. We're using chef, so something similar would probably work for us.

[ram-devsecops]

Hi @torrancew @driskell @jordansissel I am trying to use the puppet certs to setup Logstash forwarder, but I am getting below error, any ideas on how to troubleshoot this?

2014/11/04 13:23:49.882056 Connecting to [x.x.x.x]:5043 (x.x.x.x) 2014/11/04 13:23:49.893056 Failed to tls handshake with x.x.x.x EOF

below are my conf files for logstash-forwarder on windows and logstash on linux...

https://gist.github.com/ramreddy26/3e1d3dfe636675bc3e77

https://gist.github.com/ramreddy26/641e27b55ed88ae9e031

[driskell]

@ramreddy26 You should be using private key not public on logstash side.

Also there's not much point in the key and certificate on forwarded side - server doesn't currently check any of it so you could omit those.

Then it should work. EOF looks suspicious though so check firewalls and any load balancers too.

[ram-devsecops]

Thankyou @driskell its working now... I used the private key on logstash and also instead of IP, I used the fqdn name on forwarder.

[ram-devsecops]

Hi @driskell @jordansissel forwarder is running fine but I am having issue with my indexer, when I run it manually its working(/opt/logstash/bin/logstash agent -f /opt/logstash/indexer.conf). But when I run it as service its not picking logs from forwarder, below is the error from log file. Indexer was working as a service before we switched our shippers to forwarder.

{:timestamp=>"2014-11-05T11:04:41.829000-0500", :message=>"Using milestone 1 input plugin 'lumberjack'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin. For more information on plugin milestones, see http://logstash.net/docs/1.4.2/plugin-milestones", :level=>:warn} {:timestamp=>"2014-11-05T11:04:41.868000-0500", :message=>"Invalid setting for lumberjack input plugin:\n\n input {\n lumberjack {\n # This setting must be a path\n # File does not exist or cannot be opened /etc/puppetlabs/puppet/ssl/private_keys/xxxx.pem\n ssl_key => \"/etc/puppetlabs/puppet/ssl/private_keys/xxxx.pem\"\n ...\n }\n }", :level=>:error}
{:timestamp=>"2014-11-05T11:04:41.881000-0500", :message=>"Error: Something is wrong with your configuration."} {:timestamp=>"2014-11-05T11:04:41.884000-0500", :message=>"You may be interested in the '--configtest' flag which you can\nuse to validate logstash's configuration before you choose\nto restart a running system."}

[torrancew]

@ramreddy26 the logstash user does not have the ability to read those files (specifically, folders above those files). I suggest you create a copy of the certificate, put it somewhere like /etc/logstash, and set permissions on it so that the logstash user has exclusive rights to the copy. In our org, we captured this logic in a define that allows us to create user-specific copies of any or all of (local) public/private key, CA cert, and CRL from the master.

[ram-devsecops]

thank you @torrancew that worked.

[Bradox]

@zSprawl and @ArkeologeN, I was able to solve your problem adding some lines to the ssl conf file

[req] distinguished_name = test

[test] countryName = country name two letters countryName_default = country name two letters countryName_min = 2 countryName_max = 2

localityName = city organizationalUnitName = your company or section

commonName = your name commonName_max = 64

emailAddress = your email emailAddress_max = 40

[v3_ca] subjectAltName = IP:127.0.0.1

[Priyantha]

I am getting the following when I try to compile the lc-tlscert.go script:

priyantha@logging:~$ go build lc-tlscert.go can't load package: lc-tlscert.go:5:1: expected 'package', found '<' lc-tlscert.go:8:19: illegal rune literal

[driskell]

How did you download it? Make sure you pull down the RAW page:

https://raw.githubusercontent.com/driskell/log-courier/develop/src/lc-tlscert/lc-tlscert.go

[kiranos]

Hm so if I understand this correclty, I have to generate a new cert each time I install logstash-forwarder on a new machine? (I dont know hostname nor ip before hand.. )

[driskell]

@kiranos You only need generate a certificate for the server - that's Logstash. So no, not each time you install logstash-forwarder. Only each time you add a new logstash node. But you can always use the same certificate on all the logstash machines too if that is allowed within your security policies.

[kiranos]

@driskell hm I tried that, added the correct "common name" that my logstash server has, the fqdn but it did not work. I got:

x509: certificate is valid for , not "fqdn"

Then I tried your "lc-tlscert" it works but I have to add this:

"The next step is to add any additional DNS names and IP addresses that clients may use to connect to the server"

Which implies I have to redo this proceedure each time I have to add a new client?

[driskell]

@kiranos

"The next step is to add any additional DNS names and IP addresses that clients may use to connect to the server"

You may have misunderstood, sorry. All of your logstash-forwarder will be connecting to the same server, using the same address, usually, and its only that address you need here.

[kiranos]

oh ok thanks I'll see if I can redo from scratch, I want to get this stable and as hassle free as possible.

[kiranos]

@driskell it seems to work, what magic are you using? :)

openssl req -x509 -nodes -days 365 -newkey rsa:4096 -keyout logstash-forwarder.key -out logstash-forwarder.crt

and specifiying the logstash servers fqdn as "common name" did not work, but specifiying it in our lc-tlscert did.

[driskell]

Haha no magic. Are you sure you didn't specify an IP address in the "additional DNS names and IP addresses" section? That's usually the bit where the openssl command falls over, as you cannot specify "IP SAN" on the command line. When connecting via IP it never uses common name to verify, it uses these "IP SAN" instead.

[kiranos]

what I get to specify is:

Country Name (2 letter code) [AU]: State or Province Name (full name) [Some-State]: Locality Name (eg, city) []: Organization Name (eg, company) [Internet Widgits Pty Ltd]: Organizational Unit Name (eg, section) []: Common Name (e.g. server FQDN or YOUR name) []:logstash.mydomain.com Email Address []:

I set all above including "common name"

and In logstash-forwarder I used the logstash.mydomain.com:5000 to connect to logstash. Still I got "x509: certificate is valid for , not logstash.mydomain.com"

[driskell]

That looks right so I cannot see why it didn't work - possibly logstash hadn't loaded the new certificate properly? Very odd!

[kiranos]

ok I'll redo again and see if it fixes it. might have had an old cert configued :(

[kiranos]

Got it to work now. If others are having issues. Here is a few steps to get it to work:

• You should use a domain to connect to your logstash server from the logstash-forwarder (to validate the ssl-cert) for example: logstash.domain.com
• The logstash server should reply with this domain if you use "hostname -f" if not fix it in /etc/hosts /etc/hostname (OPTIONAL)
• Create a pointer for this domain so "dig logstash.domain.com" replies with your logstash servers ip.

create the SSL-cert

openssl req -x509 -nodes -days 365 -newkey rsa:4096 -keyout selfsigned.key -out selfsigned.crt

Generating a 4096 bit RSA private key
..........................................................................................++
................................++
writing new private key to 'selfsigned.key'
-----

Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:logstash.domain.com  <-- VERY IMPORTANT TO USE YOUR DOMAIN YOU SETUP HERE
Email Address []:

When this is done use the generetad files and all should be good to go.

With logstash-forwarder use:

    "servers": [ "logstash.domain.com:5000" ],

[driskell]

Hi @kiranos - your second step should be unnecessary.

But if anyone does need to connect via IP address (no DNS available) - lc-tlscert will allow setting the IP SAN required to make validation work of IP instead of DNS

[kiranos]

yes I just think its good practice to set a correct fqdn, about your lc-tlscert it worked great! just wanted to supply a howto for using the standard tools.

[josh-padnick]

First, special thanks to @driskell for the great script. Worked like a charm.

Second, I also ran into the error message: x509: certificate is valid for , not logstash.mydomain.com, but I was able to fix it once I updated not only the .crt file for logstash-forwarder, but also the one for the logstash server in /etc/pki/tls/certs/logstash-forwarder.crt.

Once I updated that cert, it works great.

[swapnil-jaiswal]

Hi. I am new to Logstash-forwarder, i have downloaded the script lc-tlscert. Compiled the script and logstash-forwarder via the latest version of go ( /usr/bin/go version go version go1.4.1 linux/amd64)

Version of other softwares in stack : OpenSSL 1.0.1e-fips 11 Feb 2013

lumberjack code - via git hub v0.3.1.

I generated the cert using the ip sans / dns hostnames and common name as the fqdn of the server
Logstash Server( 1.4.2-modified) is also running on the same machine as log forwarder.

The new certificate was copied to the appropriate locations in lumberjack folder as well as /etc/pki/tls/certs/logstash-forwarder.crt (for logstash server)

Still I am not able to authenticate the lumjerjack.

On starting lumberjack i am getting following error.

2015/01/28 12:34:24.292164 Connecting to [172.31.22.235]:5043 (logstash.compute.internal) 2015/01/28 12:34:24.312255 Failed to tls handshake with 172.31.22.235 x509: certificate signed by unknown authority 2015/01/28 12:34:25.312763 Connecting to [127.0.0.1]:5043 (logstash.compute.internal) 2015/01/28 12:34:25.317503 Failed to tls handshake with 127.0.0.1 x509: certificate signed by unknown authority 2015/01/28 12:34:26.317956 Connecting to [172.31.22.235]:5043 (logstash.compute.internal) 2015/01/28 12:34:26.323748 Failed to tls handshake with 172.31.22.235 x509: certificate signed by unknown authority 2015/01/28 12:34:27.324190 Connecting to [127.0.0.1]:5043 (logstash.compute.internal) 2015/01/28 12:34:27.329292 Failed to tls handshake with 127.0.0.1 x509: certificate signed by unknown authority ...

My certificate details : openssl x509 -text -in logstash-forwarder.crt Certificate: Data: Version: 3 (0x2) Serial Number: b0:57:40:af:b6:14:93:04:8c:11:24:53:cb:24:0e:09 Signature Algorithm: sha256WithRSAEncryption Issuer: O=Log Courier, CN=logstash.compute.internal Validity Not Before: Jan 28 12:24:59 2015 GMT Not After : Jan 26 12:24:59 2025 GMT Subject: O=Log Courier, CN=logstash.compute.internal Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: bb:4f:5d:55:b4:1f:d1:7b:88:b0:fc:8e:df:1e:08: 24:f3 ... Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment, Certificate Sign X509v3 Extended Key Usage: TLS Web Server Authentication X509v3 Basic Constraints: critical CA:TRUE X509v3 Subject Alternative Name: DNS:logstash.compute.internal, DNS:ip-172-31-22-235.ap-southeast-1.compute.internal, DNS:ip-172-31-22-235, DNS:logstash.mydomain.com, DNS:, DNS:., DNS:.., DNS:..., DNS:...., DNS:....., DNS:......, DNS:localhost, IP Address:172.31.22.235, IP Address:127.0.0.1 Signature Algorithm: sha256WithRSAEncryption 8b:10:ac:2d:7c:1a:ae:67:9d:cb:b1:62:61:f9:cc:ac:3b:af: fe:28:bd:07:6f:a4:35:06:6b:ae:f9:5a:e3:de:0d:f3:00:a2: 41:cb:8e:01 ... -----BEGIN CERTIFICATE----- .. -----END CERTIFICATE-----

I event appended the above certificate to the file /etc/ssl/certs/ca-bundle.trust.crt

But the same error persists. Am I missing something obvious ?

Thanks in advance!

[swapnil-jaiswal]

I appended the certificate to the wrong file openssl x509 -text -in logstash-forwarder.crt >> ca-bundle.crt

did the trick :)