kafka ssl no longer works with fluentd v1.16.4 and above

[sschroed99]

We have been utilizing the rdkafka2 output plugin with v1.16.2-debian-1.0. We attempted to upgrade to v1.16.5-debian-1.0, but we are now receiving the below error. It looks like it started with v1.16.4-debian-1.0, as it still works with v1.16.3-debian-1.0.

Utilizing kafka2 output plugin does work with v1.16.5-debian-1.0. Looks like as of v1.16.4-debian-1.0 they upgraded from debian 11 to 12 and openssl went from OpenSSL 1.1.1w 11 Sep 2023 to OpenSSL 3.0.11 19 Sep 2023 (Library: OpenSSL 3.0.11 19 Sep 2023).

2024-04-25 16:05:27 +0000 [warn]: #0 Send exception occurred: Unsupported value "ssl" for configuration property "security.protocol": OpenSSL not available at build time at /usr/local/bundle/gems/rdkafka-0.15.1/lib/rdkafka/config.rb:263:in `block (2 levels) in native_config'
2024-04-25 16:05:27 +0000 [warn]: #0 failed to flush the buffer. retry_times=2 next_retry_time=2024-04-25 16:05:28 +0000 chunk="616edf45a85eec5bb3505a80859b7b7f" error_class=Rdkafka::Config::ConfigError error="Unsupported value \"ssl\" for configuration property \"security.protocol\": OpenSSL not available at build time"
  2024-04-25 16:05:27 +0000 [warn]: #0 /usr/local/bundle/gems/rdkafka-0.15.1/lib/rdkafka/config.rb:263:in `block (2 levels) in native_config'
  2024-04-25 16:05:27 +0000 [warn]: #0 /usr/local/bundle/gems/rdkafka-0.15.1/lib/rdkafka/config.rb:253:in `each'
  2024-04-25 16:05:27 +0000 [warn]: #0 /usr/local/bundle/gems/rdkafka-0.15.1/lib/rdkafka/config.rb:253:in `block in native_config'
  2024-04-25 16:05:27 +0000 [warn]: #0 <internal:kernel>:90:in `tap'
  2024-04-25 16:05:27 +0000 [warn]: #0 /usr/local/bundle/gems/rdkafka-0.15.1/lib/rdkafka/config.rb:251:in `native_config'
  2024-04-25 16:05:27 +0000 [warn]: #0 /usr/local/bundle/gems/rdkafka-0.15.1/lib/rdkafka/config.rb:201:in `producer'
  2024-04-25 16:05:27 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.19.2/lib/fluent/plugin/out_rdkafka2.rb:387:in `block in get_producer'
  2024-04-25 16:05:27 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.19.2/lib/fluent/plugin/out_rdkafka2.rb:384:in `synchronize'
  2024-04-25 16:05:27 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.19.2/lib/fluent/plugin/out_rdkafka2.rb:384:in `get_producer'
  2024-04-25 16:05:27 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.19.2/lib/fluent/plugin/out_rdkafka2.rb:421:in `write'
  2024-04-25 16:05:27 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.16.4/lib/fluent/plugin/output.rb:1225:in `try_flush'
  2024-04-25 16:05:27 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.16.4/lib/fluent/plugin/output.rb:1538:in `flush_thread_run'
  2024-04-25 16:05:27 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.16.4/lib/fluent/plugin/output.rb:510:in `block (2 levels) in start'
  2024-04-25 16:05:27 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.16.4/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

[mensfeld]

Can you provide me with a docker file that I can use to reproduce it? Otherwise I will not be able to look into this.

[sschroed99]

Below is the dockerfile I am using.

FROM fluent/fluentd:v1.16-debian-2
LABEL maintainer=""
USER root
RUN fluent-gem install fluent-plugin-rewrite-tag-filter
RUN apt -y update
RUN apt -y install build-essential
RUN fluent-gem install fluent-plugin-kafka
RUN fluent-gem install fluentd-plugins-pulsar
RUN fluent-gem install fluent-plugin-beats
RUN fluent-gem install fluent-plugin-forest
RUN fluent-gem install fluent-plugin-elasticsearch
RUN fluent-gem install fluent-plugin-record-reformer
RUN fluent-gem install rdkafka
RUN fluent-gem install fluent-plugin-prometheus --version=2.1.0
RUN fluent-gem install fluent-plugin-grafana-loki
RUN mkdir -p /tmp/gems
COPY ffi-1.13.1.gem /tmp/gems
COPY mini_portile2-2.5.0.gem /tmp/gems
WORKDIR /tmp/gems
RUN fluent-gem install --local ffi-1.13.1.gem
RUN fluent-gem install --local mini_portile2-2.5.0.gem

[mensfeld]

Thanks. I will look at it in around a month as it is not my top priority at the moment.

[sschroed99]

Understood, thank you for your time!

[sschroed99]

Hi @mensfeld, just wanted to check in to see if you were able to get some free time to review this? Thanks!

[mensfeld]

@sschroed99 no If you want to get this higher in my priorities, I do have a paid support if you rely on my software and can't wait for it: https://karafka.io/docs/Pro-Support/#prioritized-bug-fixes

[mensfeld]

@sschroed99 can you try with the most recent rdkafka version? The one released today.

[sschroed99]

I will give it a shot tomorrow.On Oct 1, 2024, at 12:42 PM, Maciej Mensfeld @.***> wrote: @sschroed99 can you try with the most recent rdkafka version? The one released today.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

[sschroed99]

I just attempted the latest version of rdkafka with fluentd 1.17.1, but looks like the same results.

024-10-11 19:21:21 +0000 [warn]: #0 Send exception occurred: Unsupported value "ssl" for configuration property "security.protocol": OpenSSL not available at build time at /usr/local/bundle/gems/rdkafka-0.19.0/lib/rdkafka/config.rb:316:in `block (2 levels) in native_config'
2024-10-11 19:21:21 +0000 [error]: #0 Hit limit for retries. dropping all chunks in the buffer queue. retry_times=5 records=86 error_class=Rdkafka::Config::ConfigError error="Unsupported value \"ssl\" for configuration property \"security.protocol\": OpenSSL not available at build time"
  2024-10-11 19:21:21 +0000 [error]: #0 /usr/local/bundle/gems/rdkafka-0.19.0/lib/rdkafka/config.rb:316:in `block (2 levels) in native_config'
  2024-10-11 19:21:21 +0000 [error]: #0 /usr/local/bundle/gems/rdkafka-0.19.0/lib/rdkafka/config.rb:306:in `each'
  2024-10-11 19:21:21 +0000 [error]: #0 /usr/local/bundle/gems/rdkafka-0.19.0/lib/rdkafka/config.rb:306:in `block in native_config'
  2024-10-11 19:21:21 +0000 [error]: #0 <internal:kernel>:90:in `tap'
  2024-10-11 19:21:21 +0000 [error]: #0 /usr/local/bundle/gems/rdkafka-0.19.0/lib/rdkafka/config.rb:304:in `native_config'
  2024-10-11 19:21:21 +0000 [error]: #0 /usr/local/bundle/gems/rdkafka-0.19.0/lib/rdkafka/config.rb:244:in `producer'
  2024-10-11 19:21:21 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.19.3/lib/fluent/plugin/out_rdkafka2.rb:362:in `block in get_producer'
  2024-10-11 19:21:21 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.19.3/lib/fluent/plugin/out_rdkafka2.rb:359:in `synchronize'
  2024-10-11 19:21:21 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.19.3/lib/fluent/plugin/out_rdkafka2.rb:359:in `get_producer'
  2024-10-11 19:21:21 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.19.3/lib/fluent/plugin/out_rdkafka2.rb:396:in `write'
  2024-10-11 19:21:21 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.17.1/lib/fluent/plugin/output.rb:1225:in `try_flush'
  2024-10-11 19:21:21 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.17.1/lib/fluent/plugin/output.rb:1538:in `flush_thread_run'
  2024-10-11 19:21:21 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.17.1/lib/fluent/plugin/output.rb:510:in `block (2 levels) in start'
  2024-10-11 19:21:21 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.17.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2024-10-11 19:21:23 +0000 [warn]: #0 Send exception occurred: Unsupported value "ssl" for configuration property "security.protocol": OpenSSL not available at build time at /usr/local/bundle/gems/rdkafka-0.19.0/lib/rdkafka/config.rb:316:in `block (2 levels) in native_config'
2024-10-11 19:21:23 +0000 [warn]: #0 failed to flush the buffer. retry_times=0 next_retry_time=2024-10-11 19:21:24 +0000 chunk="624386457c83a2834c0c079ea281ccae" error_class=Rdkafka::Config::ConfigError error="Unsupported value \"ssl\" for configuration property \"security.protocol\": OpenSSL not available at build time"
  2024-10-11 19:21:23 +0000 [warn]: #0 /usr/local/bundle/gems/rdkafka-0.19.0/lib/rdkafka/config.rb:316:in `block (2 levels) in native_config'
  2024-10-11 19:21:23 +0000 [warn]: #0 /usr/local/bundle/gems/rdkafka-0.19.0/lib/rdkafka/config.rb:306:in `each'
  2024-10-11 19:21:23 +0000 [warn]: #0 /usr/local/bundle/gems/rdkafka-0.19.0/lib/rdkafka/config.rb:306:in `block in native_config'
  2024-10-11 19:21:23 +0000 [warn]: #0 <internal:kernel>:90:in `tap'
  2024-10-11 19:21:23 +0000 [warn]: #0 /usr/local/bundle/gems/rdkafka-0.19.0/lib/rdkafka/config.rb:304:in `native_config'
  2024-10-11 19:21:23 +0000 [warn]: #0 /usr/local/bundle/gems/rdkafka-0.19.0/lib/rdkafka/config.rb:244:in `producer'
  2024-10-11 19:21:23 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.19.3/lib/fluent/plugin/out_rdkafka2.rb:362:in `block in get_producer'
  2024-10-11 19:21:23 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.19.3/lib/fluent/plugin/out_rdkafka2.rb:359:in `synchronize'
  2024-10-11 19:21:23 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.19.3/lib/fluent/plugin/out_rdkafka2.rb:359:in `get_producer'
  2024-10-11 19:21:23 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.19.3/lib/fluent/plugin/out_rdkafka2.rb:396:in `write'
  2024-10-11 19:21:23 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.17.1/lib/fluent/plugin/output.rb:1225:in `try_flush'
  2024-10-11 19:21:23 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.17.1/lib/fluent/plugin/output.rb:1538:in `flush_thread_run'
  2024-10-11 19:21:23 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.17.1/lib/fluent/plugin/output.rb:510:in `block (2 levels) in start'
  2024-10-11 19:21:23 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.17.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

[mensfeld]

Yeah still not seeing ssl

[mensfeld]

Too bad. It will have to wait as it is low on my priorities list.