Kafka outputs on Windows

arasic commented 2 months ago

Bug Report

Describe the bug After configuring kafka outputs for Windows 11 Home, when log events were written to kafka outputs, the error: [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: fluent-bit#producer-1: 192.168.2.5:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 0ms in state APIVERSION_QUERY) was being displayed.

To Reproduce

Rubular link if applicable:

Example log message if applicable:

[2024/08/14 10:55:12] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: fluent-bit#producer-1: 192.168.2.5:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 0ms in state APIVERSION_QUERY)
[2024/08/14 10:55:12] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: fluent-bit#producer-1: 192.168.2.5:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed)
[2024/08/14 10:55:12] [debug] in produce_message

Steps to reproduce the problem: I used the following output config if the configurations:

[OUTPUT]
Name kafka
Match *
Brokers <address>
Topics <topic name>
rdkafka.enable.ssl.certificate.verification true
rdkafka.request.required.acks 1
rdkafka.security.protocol SASL_SSL
rdkafka.sasl.mechanism SCRAM-SHA-256
rdkafka.compression.codec zstd
rdkafka.sasl.username <sasl username>
rdkafka.sasl.password <sasl password>
rdkafka.ssl.ca.location cacert.crt

Expected behavior Event messages being successfully written to kafka messaging queue

Your Environment

Version used: v3.1.5

Configuration: I used the following output config if the configurations:

[OUTPUT]
Name kafka
Match *
Brokers <address>
Topics <topic name>
rdkafka.enable.ssl.certificate.verification true
rdkafka.request.required.acks 1
rdkafka.security.protocol SASL_SSL
rdkafka.sasl.mechanism SCRAM-SHA-256
rdkafka.compression.codec zstd
rdkafka.sasl.username <sasl username>
rdkafka.sasl.password <sasl password>
rdkafka.ssl.ca.location cacert.crt

Environment name and version (e.g. Kubernetes? What version?):
Server type and version:
Operating System and version: Windows 11 Home
Filters and plugins: kafka outputs

Additional context

arasic commented 2 months ago

my question is essentially if kafka outputs for Windows supports SASL_SSL as protocol

arasic commented 2 months ago

In another instance, I tried with ssl as protocol.

Configuration used

[OUTPUT]
    Name kafka
    Match *
    Brokers <broker-address>
    Topics <topic-name>
    rdkafka.enable.ssl.certificate.verification false
    rdkafka.security.protocol ssl
    rdkafka.ssl.ca.location kafka-ca-cert.crt
    rdkafka.ssl.certificate.location kafka.crt
    rdkafka.ssl.key.location kafka.key
    rdkafka.api.version.request false
    rdkafka.debug All

Here is the extended log error:

[2024/08/15 12:01:09] [debug] [kafka:kafka.0] created event channels: read=964 write=996
[2024/08/15 12:01:09] [error] [flb_kafka] cannot configure 'enable.ssl.certificate.verification' property
[2024/08/15 12:01:09] [error] [flb_kafka] cannot configure 'security.protocol' property
[2024/08/15 12:01:09] [error] [flb_kafka] cannot configure 'ssl.ca.location' property
[2024/08/15 12:01:09] [error] [flb_kafka] cannot configure 'ssl.certificate.location' property
[2024/08/15 12:01:09] [error] [flb_kafka] cannot configure 'ssl.key.location' property
[2024/08/15 12:01:09] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: 192.168.2.5:9092/bootstrap: Added new broker with NodeId -1
[2024/08/15 12:01:09] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: 192.168.2.5:9092/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))
[2024/08/15 12:01:09] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: librdkafka v2.4.0 (0x20400ff) fluent-bit#producer-1 initialized (builtin.features sasl,regex,lz4,sasl_gssapi,sasl_plain,plugins, SSL ZLIB SNAPPY ZSTD CURL SASL_SCRAM SASL_OAUTHBEARER PLUGINS HDRHISTOGRAM, debug 0xfffff)
[2024/08/15 12:01:09] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd::0/internal]: :0/internal: Enter main broker thread
[2024/08/15 12:01:09] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: Client configuration:
[2024/08/15 12:01:09] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:192.168.2.5:9092/bootstrap]: 192.168.2.5:9092/bootstrap: Enter main broker thread
[2024/08/15 12:01:09] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:192.168.2.5:9092/bootstrap]: 192.168.2.5:9092/bootstrap: Received CONNECT op
[2024/08/15 12:01:09] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:192.168.2.5:9092/bootstrap]: 192.168.2.5:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT
[2024/08/15 12:01:09] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]:   client.id = fluent-bit
[2024/08/15 12:01:09] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:192.168.2.5:9092/bootstrap]: Broadcasting state change
[2024/08/15 12:01:09] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:192.168.2.5:9092/bootstrap]: 192.168.2.5:9092/bootstrap: broker in state TRY_CONNECT connecting
[2024/08/15 12:01:09] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:192.168.2.5:9092/bootstrap]: 192.168.2.5:9092/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
[2024/08/15 12:01:09] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]:   client.software.version = 2.4.0
[2024/08/15 12:01:09] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:192.168.2.5:9092/bootstrap]: Broadcasting state change
[2024/08/15 12:01:09] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]:   metadata.broker.list = 192.168.2.5:9092
[2024/08/15 12:01:09] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:192.168.2.5:9092/bootstrap]: 192.168.2.5:9092/bootstrap: Connecting to ipv4#192.168.2.5:9092 (plaintext) with socket 1196
[2024/08/15 12:01:09] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]:   debug = generic,broker,topic,metadata,feature,queue,msg,protocol,cgrp,security,fetch,interceptor,plugin,consumer,admin,eos,mock,assignor,conf,all
[2024/08/15 12:01:09] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]:   log_cb = 00007FF7C6D5F5B0
[2024/08/15 12:01:09] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]:   opaque = 000001C00EF8C2C0
[2024/08/15 12:01:09] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]:   api.version.request = true
[2024/08/15 12:01:09] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]:   dr_msg_cb = 00007FF7C6D5F830
[2024/08/15 12:01:09] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: New local topic: <topic-name>
[2024/08/15 12:01:09] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: NEW <topic-name> [-1] 000001C00EF990F0 refcnt 000001C00EF99180 (at rd_kafka_topic_new0:488)
[2024/08/15 12:01:09] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: leader query
[2024/08/15 12:01:09] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: Hinted cache of 1/1 topic(s) being queried
[2024/08/15 12:01:09] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: Skipping metadata refresh of 1 topic(s): leader query: no usable brokers
[2024/08/15 12:01:09] [ info] [output:kafka:kafka.0] brokers='192.168.2.5:9092' topics='<topic-name>'
[2024/08/15 12:01:09] [ info] [sp] stream processor started
[2024/08/15 12:01:09] [debug] [input:tail:zeek-all] inode=1688849860690590 file=C:\Users\<username>\Documents\NetworkLogs\conn\20240715.log promote to TAIL_EVENT

The line:

[2024/08/15 12:01:09] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:192.168.2.5:9092/bootstrap]: 192.168.2.5:9092/bootstrap: Connecting to ipv4#192.168.2.5:9092 (plaintext) with socket 1196

indicates that it is connecting without using ssl/tls connectivity even if it was configured.

While testing fluent-bit in Linux, I was able to connect and ship messages.

arasic commented 2 months ago

I am closing this issue because its related with how fluent-bit was built from source code. Using exe/msi Win64 OpenSSL v3.3.1 from https://slproweb.com/products/Win32OpenSSL.html does not work. You must use vcpkg to install openssl

Lets consider that VS is already installed (https://visualstudio.microsoft.com/downloads/) and make sure that Desktop development with C++ is selected in installation. After installation is complete, launch Developer Command Prompt for Visual Studio

Lets suppose you install vcpkg in C:\

cd C:\
git clone https://github.com/microsoft/vcpkg.git
cd vcpkg && bootstrap-vcpkg.bat
.\vcpkg.exe integrate install

Now lets install openssl: from C:\vcpkg, type:

.\vcpkg.exe --triplet x64-windows-static install openssl

Set in the env variables of Windows: OPENSSL_ROOT_DIR=C:\vcpkg\installed\x64-windows-static

Now to to fluent-bit\build and run:

cmake .. -G "Visual Studio 17 2022" -A x64
cmake --build . --config Release

fluent / fluent-bit

Kafka outputs on Windows #9218

Bug Report