Xpack communication between logstash and es cluster fails with "PKIX path building failed

[aklira]

Xpack.monitoring and Xpack.management communication between logstash and es cluster fails with "PKIX path building failed.

The error sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target is raised by Logstash when it is not able to trust the certificate exposed by the server while the certificate is a perfectly working SAN certificate.

From the curl verbose output the CN is myapplication.mydomain.com, while logstash connects to a host named myapplication-myservice1.mydomain.com. They are different and Logstash performs a full hostname validation, which fails.

According to rfc5280#section-4.2.1.6 The subject alternative name extension allows identities to be bound to the subject of the certificate. These identities may be included in addition to or in place of the identity in the subject field of the certificate. [...] Whenever such identities are to be bound into a certificate, the subject alternative name (or issuer alternative name) extension MUST be used;

• Version: Logstash 6.7.0
• Operating System: Red Hat Enterprise Linux Server 7.2 (Maipo)

[xakraz]

Same issue on my side with the elasticsearch output plugin ...

Logstash logs

[WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, 
but got an error. {:url=>"https://elastic:xxxxxx@quickstart-es-http.default.svc.cluster.local:9200/", 
:error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, 
:error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@quickstart-es-http.default.svc.cluster.local:9200/]
[Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}

Config:

output {
      elasticsearch {
        hosts    => [ "${ELASTIC_HOST}" ]
        user     => "${ELASTIC_USERNAME}"
        password => "${ELASTIC_PASSWORD}"
        ssl      => true
        ssl_certificate_verification => true
        cacert   => '/ca.crt'

        index    => "logs-%{[@metadata][index_prefix]}-%{[@metadata][index_name]}-%{+YYYY.MM.dd}"
        id       => "output-elasticsearch"
      }
}

$ curl --cacert /ca.crt "https://${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}@quickstart-es-http.default.svc.cluster.local:9200/"
{
  "name" : "quickstart-es-default-0",
  "cluster_name" : "quickstart",
  "cluster_uuid" : "-jsLCLX4TeKWHEnRlrpo0g",
  "version" : {
    "number" : "7.4.2",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "2f90bbf7b93631e52bafb59b3b049cb44ec25e96",
    "build_date" : "2019-10-28T20:40:44.881551Z",
    "build_snapshot" : false,
    "lucene_version" : "8.2.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

$ openssl x509 -in tls.crt -text -nocert

...
X509v3 Subject Alternative Name: 
                DNS:quickstart-es-http.default.es.local, 
                DNS:quickstart-es-http, 
                DNS:quickstart-es-http.default,
                DNS:quickstart-es-http.default.svc,
                DNS:quickstart-es-http.default.svc.cluster.local

$ logstash --version

logstash 7.4.2

The official Elastic docker image

Ref:

• https://discuss.elastic.co/t/issue-with-output-elasticsearch-running-on-eck-hostunreachableerror/223109