search

fluent / fluent-plugin-kafka

Kafka input and output plugin for Fluentd
Other
303 stars 178 forks source link

fluent-input-kafka use ssl error: SSL_CTX_use_certificate: ca md too weak #440

Closed ly123-liu closed 2 years ago

ly123-liu commented 2 years ago

Describe the bug

we use fluent to consume kafka messages with ssl cert , fluent start with error below:

2022-01-12 15:00:45 +0800 [error]: #0 unexpected error error_class=OpenSSL::SSL::SSLError error="SSL_CTX_use_certificate: ca md too weak" 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/ssl_socket_with_timeout.rb:59:in initialize' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/ssl_socket_with_timeout.rb:59:innew' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/ssl_socket_with_timeout.rb:59:in initialize' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/connection.rb:130:innew' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/connection.rb:130:in open' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/connection.rb:101:inblock in send_request' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/instrumenter.rb:23:in instrument' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/connection.rb:100:insend_request' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/broker.rb:200:in send_request' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/broker.rb:44:infetch_metadata' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:427:in block in fetch_cluster_info' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:422:ineach' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:422:in fetch_cluster_info' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:402:incluster_info' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:102:in refresh_metadata!' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:106:inrefresh_metadata_if_necessary!' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:452:in random_broker' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:382:inlist_topics' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/consumer.rb:634:in cluster_topics' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/consumer.rb:614:insubscribe_to_regex' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/consumer.rb:606:in block in scan_for_subscribing' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/consumer.rb:601:ineach' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/consumer.rb:601:in scan_for_subscribing' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/consumer.rb:118:insubscribe' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.16.0/lib/fluent/plugin/in_kafka_group.rb:229:in block in setup_consumer' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.16.0/lib/fluent/plugin/in_kafka_group.rb:221:ineach' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.16.0/lib/fluent/plugin/in_kafka_group.rb:221:in setup_consumer' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.16.0/lib/fluent/plugin/in_kafka_group.rb:202:instart' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/compat/call_super_mixin.rb:42:in start' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/root_agent.rb:200:inblock in start' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/root_agent.rb:189:in block (2 levels) in lifecycle' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/root_agent.rb:188:ineach' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/root_agent.rb:188:in block in lifecycle' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/root_agent.rb:175:ineach' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/root_agent.rb:175:in lifecycle' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/root_agent.rb:199:instart' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/engine.rb:248:in start' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/engine.rb:147:inrun' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/supervisor.rb:590:in block in run_worker' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/supervisor.rb:825:inmain_process' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/supervisor.rb:584:in run_worker' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/command/fluentd.rb:338:in<top (required)>' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in require' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:inrequire' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/bin/fluentd:8:in <top (required)>' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/bin/fluentd:23:inload' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/bin/fluentd:23:in `

' 2022-01-12 15:00:45 +0800 [error]: #0 unexpected error error_class=OpenSSL::SSL::SSLError error="SSL_CTX_use_certificate: ca md too weak" 2022-01-12 15:00:45 +0800 [error]: #0 suppressed same stacktrace

To Reproduce

since we generate kafka cert without set message digest, default message digest seems sha1WithRSAEncryption , is there fluent-kafka-input plugin can check tls with sha1 and continue to work ?

    Signature Algorithm: sha1WithRSAEncryption
    Issuer: CN = Yingmi Infra Kafka RootCA
    Validity
        Not Before: Mar  8 02:15:36 2021 GMT
        Not After : Feb 12 02:15:36 2121 GMT
    Subject Public Key Info:
        Public Key Algorithm: rsaEncryption

Expected behavior

1

Your Environment

no special

Your Configuration

@type kafka_group

brokers kafka.infra:9092 consumer_group log-fluentd-test topics /log-prod.*/ format text message_key <key (Optional, for text format only, default is message)>

kafka_message_key <key (Optional, If specified, set kafka's message key to this key)>

add_headers <If true, add kafka's message headers to record>

add_prefix <tag prefix (Optional)>

add_suffix <tag suffix (Optional)>

retry_emit_limit 3 time_source record time_format <string (Optional when use_record_time is used)>

ruby-kafka consumer options

max_bytes 1048576

max_wait_time (integer) :default => nil (Use default of ruby-kafka)

min_bytes (integer) :default => nil (Use default of ruby-kafka)

offset_commit_interval (integer) :default => nil (Use default of ruby-kafka)

offset_commit_threshold (integer) :default => nil (Use default of ruby-kafka)

fetcher_max_queue_size (integer) :default => nil (Use default of ruby-kafka)

start_from_beginning true

get_kafka_client_log true connect_timeout 10 socket_timeout 30 ssl_verify_hostname false ssl_ca_cert /ca.crt ssl_client_cert /log.crt ssl_client_cert_key /log.key ssl_client_cert_chain PEM

Your Error Log

2022-01-12 15:00:45 +0800 [error]: #0 unexpected error error_class=OpenSSL::SSL::SSLError error="SSL_CTX_use_certificate: ca md too weak"
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/ssl_socket_with_timeout.rb:59:in `initialize'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/ssl_socket_with_timeout.rb:59:in `new'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/ssl_socket_with_timeout.rb:59:in `initialize'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/connection.rb:130:in `new'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/connection.rb:130:in `open'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/connection.rb:101:in `block in send_request'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/instrumenter.rb:23:in `instrument'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/connection.rb:100:in `send_request'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/broker.rb:200:in `send_request'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/broker.rb:44:in `fetch_metadata'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:427:in `block in fetch_cluster_info'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:422:in `each'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:422:in `fetch_cluster_info'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:402:in `cluster_info'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:102:in `refresh_metadata!'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:106:in `refresh_metadata_if_necessary!'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:452:in `random_broker'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:382:in `list_topics'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/consumer.rb:634:in `cluster_topics'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/consumer.rb:614:in `subscribe_to_regex'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/consumer.rb:606:in `block in scan_for_subscribing'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/consumer.rb:601:in `each'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/consumer.rb:601:in `scan_for_subscribing'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/consumer.rb:118:in `subscribe'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.16.0/lib/fluent/plugin/in_kafka_group.rb:229:in `block in setup_consumer'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.16.0/lib/fluent/plugin/in_kafka_group.rb:221:in `each'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.16.0/lib/fluent/plugin/in_kafka_group.rb:221:in `setup_consumer'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.16.0/lib/fluent/plugin/in_kafka_group.rb:202:in `start'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/compat/call_super_mixin.rb:42:in `start'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/root_agent.rb:200:in `block in start'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/root_agent.rb:189:in `block (2 levels) in lifecycle'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/root_agent.rb:188:in `each'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/root_agent.rb:188:in `block in lifecycle'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/root_agent.rb:175:in `each'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/root_agent.rb:175:in `lifecycle'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/root_agent.rb:199:in `start'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/engine.rb:248:in `start'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/engine.rb:147:in `run'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/supervisor.rb:590:in `block in run_worker'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/supervisor.rb:825:in `main_process'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/supervisor.rb:584:in `run_worker'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/command/fluentd.rb:338:in `<top (required)>'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/bin/fluentd:8:in `<top (required)>'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/bin/fluentd:23:in `load'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/bin/fluentd:23:in `<main>'
2022-01-12 15:00:45 +0800 [error]: #0 unexpected error error_class=OpenSSL::SSL::SSLError error="SSL_CTX_use_certificate: ca md too weak"
  2022-01-12 15:00:45 +0800 [error]: #0 suppressed same stacktrace

Additional context

No response

github-actions[bot] commented 2 years ago

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

github-actions[bot] commented 2 years ago

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

github-actions[bot] commented 2 years ago

This issue was automatically closed because of stale in 30 days