SSL - The requested feature is not available when logging to OpenDistro for Elasticsearch

[AlexanderRasmussen]

Bug Report

Describe the bug When deploying Fluent Bit into a Kubernetes Cluster, the pods can't log to our Opendistro for Elasticsearch Cluster. When I try to log to the cluster with "http_user", "http_passwd", "tls on" I get the following error: SSL - The requested feature is not available.

Output from logs:

[2020/09/24 09:30:24] [debug] [io_tls] /lib/mbedtls-2.16.5/library/ssl_tls.c 3626: handshake message: msglen = 13947, type = 13, hslen = 17429
[2020/09/24 09:30:24] [debug] [io_tls] /lib/mbedtls-2.16.5/library/ssl_tls.c 3698: TLS handshake fragmentation not supported
[2020/09/24 09:30:24] [debug] [io_tls] /lib/mbedtls-2.16.5/library/ssl_tls.c 4369: mbedtls_ssl_handle_message_type() returned -28800 (-0x7080)
[2020/09/24 09:30:24] [debug] [io_tls] /lib/mbedtls-2.16.5/library/ssl_cli.c 2708: mbedtls_ssl_read_record() returned -28800 (-0x7080)
[2020/09/24 09:30:24] [debug] [io_tls] /lib/mbedtls-2.16.5/library/ssl_tls.c 8094: <= handshake
[2020/09/24 09:30:24] [error] [io_tls] flb_io_tls.c:356 SSL - The requested feature is not available
[2020/09/24 09:30:24] [debug] [io_tls] /lib/mbedtls-2.16.5/library/ssl_tls.c 8725: => write close notify
[2020/09/24 09:30:24] [debug] [io_tls] /lib/mbedtls-2.16.5/library/ssl_tls.c 8741: <= write close notify
[2020/09/24 09:30:24] [debug] [io_tls] /lib/mbedtls-2.16.5/library/ssl_tls.c 8934: => free
[2020/09/24 09:30:24] [debug] [io_tls] /lib/mbedtls-2.16.5/library/ssl_tls.c 8999: <= free
[2020/09/24 09:30:24] [debug] [upstream] connection #49 failed to OurElasticsearchCluster.domain.net:9200
[2020/09/24 09:30:24] [debug] [retry] re-using retry for task_id=1 attemps=3
[2020/09/24 09:30:24] [ warn] [engine] failed to flush chunk '1-1600939771.849501446.flb', retry in 698 seconds: task_id=1, input=tail.0 > output=es.0

To Reproduce

• Create a Opendistro for Elasticsearch Cluster. With security enabled and SSL/TLS on the Elasticsearch REST endpoint
• Create an Internal Elasticsearch user.
• Setup FluentBit according to the following guide: Fluent-Bit Kubernetes
• Add "index", "logstash_prefix", "http_user", "http_passwd" to the output-elasticsearch.conf file following the guide lines here Fluent-Bit Elasticsearch
• Add the following tls settings "tls on", "tls.verify off", "tls.ca_file /path/to/ca" to the output-elasticsearch.conf file following the guide lines here Fluent-Bit TLS
• Deploy Fluent-Bit

Expected behavior

Fluent-Bit to log to the OpenDistro for Elasticsearch Cluster

Your Environment

• Version used: fluent/fluent-bit:1.5.6
• Environment name and version (e.g. Kubernetes? What version?): On-Prem v1.16.7
• OpenDistro for Elasticsearch version: 1.7.0 (Elasticsearh 7.6.1)
• Server type and version: Virtual Machine
• Operating System and version: RHEL 7.8
• Configuration: See below
• Filters and plugins: See below

  
  [SERVICE]
  Flush         1
  Log_Level     trace
  Daemon        off
  Parsers_File  parsers.conf
  HTTP_Server   On
  HTTP_Listen   0.0.0.0
  HTTP_Port     2020

[INPUT] Name tail Tag kube. Path /var/log/containers/.log Parser docker DB /var/log/flb_kube.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10

[FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc:443 Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token Kube_Tag_Prefix kube.var.log.containers. Merge_Log On Merge_Log_Key log_processed K8S-Logging.Parser On K8S-Logging.Exclude Off

[OUTPUT] Name es Match * Host OurElasticsearchCluster.domain.net:9200 Port 9200 Logstash_Format On Replace_Dots On Retry_Limit False index project-code logstash_prefix project-code http_user user http_passwd password tls on tls.debug 4 tls.verify off tls.ca_file /secure/ca-cert.crt

[PARSER] Name apache Format regex Regex ^(?[^ ]) [^ ] (?[^ ]) [(?[^]])] "(?\S+)(?: +(?[^\"]?)(?: +\S)?)?" (?[^ ]) (?[^ ])(?: "(?[^\"])" "(?[^\"])")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z

[PARSER] Name apache2 Format regex Regex ^(?[^ ]) [^ ] (?[^ ]) [(?[^]])] "(?\S+)(?: +(?[^ ]) +\S)?" (?[^ ]) (?[^ ])(?: "(?[^\"])" "(?[^\"])")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z

[PARSER] Name apache_error Format regex Regex ^[[^ ] (?[^]])] [(?[^]])](?: [pid (?[^]])])?( [client (?[^]])])? (?.)$

[PARSER] Name nginx Format regex Regex ^(?[^ ]) (?[^ ]) (?[^ ]) [(?[^]])] "(?\S+)(?: +(?[^\"]?)(?: +\S)?)?" (?[^ ]) (?[^ ])(?: "(?[^\"])" "(?[^\"])")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z

[PARSER] Name json Format json Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z

[PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On

[PARSER] Name syslog Format regex Regex ^\<(?[0-9]+)>(?[^ ] {1,2}[^ ] [^ ]) (?[^ ]) (?[a-zA-Z0-9_\/.-])(?:[(?[0-9]+)])?(?:[^\:]\:)? (?.)$ Time_Key time Time_Format %b %d %H:%M:%S

**Additional context**

Currently we have a fluent-bit setup that logs to an unsecure Elasticsearch cluster, and that's woking just fine, but we need to move to a new setup that has proper access rights.

I have also tried to configure Fluentd to see if i could get that to work, and it seems to work fine. 

I would guess that if there isn't a solution to make Fluent-Bit log directly to the Opendistro For Elasticsearch Cluster the solution is either Fluend as a daemonset or Fluent-Bit --> Fluentd --> Elasticsearch.

[edsiper]

pls share the complete Fluent Bit log that contains all tls debug messages

[dvp3010]

looks like related to https://github.com/ARMmbed/mbedtls/issues/1840.

I am also facing the same issue. when using splunk HEC output plugin.

[OUTPUT]
    Name splunk
    Match *
    Host <splunk-hec host>
    Port <splunk-hec port>
    tls On
    tls.Verify Off
    tls.debug 4
    Splunk_Token <splunk token>
    Splunk_Send_Raw On  

[2020/10/01 19:42:51] [debug] [io_tls] \lib\mbedtls-2.16.5\library\ssl_tls.c 3626: handshake message: msglen = 16384, type = 11, hslen = 17641
[2020/10/01 19:42:51] [debug] [io_tls] \lib\mbedtls-2.16.5\library\ssl_tls.c 3698: TLS handshake fragmentation not supported
[2020/10/01 19:42:51] [debug] [io_tls] \lib\mbedtls-2.16.5\library\ssl_tls.c 4369: mbedtls_ssl_handle_message_type() returned -28800 (-0x7080)
[2020/10/01 19:42:51] [debug] [io_tls] \lib\mbedtls-2.16.5\library\ssl_tls.c 5699: mbedtls_ssl_read_record() returned -28800 (-0x7080)
[2020/10/01 19:42:51] [debug] [io_tls] \lib\mbedtls-2.16.5\library\ssl_tls.c 8094: <= `handshake`
[2020/10/01 19:42:51] [error] [io_tls] flb_io_tls.c:364 SSL - The requested feature is not available

[grep4error]

FYI, we are running fluent-bit 1.5.7 with opendistro-for-es 1.9.0, tls is on, and it works just fine. Here's a sample of the output config that we use

    [OUTPUT]
      Name            es
      Alias           es-nginx-ingress
      Match           kube.nginx-ingress.*
      Host            logging-client-service.logs.svc.cluster.local
      Port            9200
      Logstash_Format On
      Replace_Dots    On
      Retry_Limit     1
      HTTP_User       logstash
      HTTP_Passwd     youknowit
      tls             on
      tls.verify      off
      Generate_ID     On
      Logstash_Prefix    nginx-ingress
      Type            _doc

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.