search

confluentinc / confluent-kafka-dotnet

Confluent's Apache Kafka .NET client
https://github.com/confluentinc/confluent-kafka-dotnet/wiki
Apache License 2.0
91 stars 869 forks source link

ClientConfig.SecurityProtocol not compatible with official config values. #1743

Open randomravings opened 2 years ago

randomravings commented 2 years ago

When instantiating a client config, eg:

var config = new ProducerConfig(IDictionary<string, string> config)

using the string configs from the Apache Kafka documentation for security.protocol then the ClientConfig throws and ArgumentException because it cannot convert it to a value in the ClientConfig.SecurityProtocol.

This is the line that breaks: https://github.com/confluentinc/confluent-kafka-dotnet/blob/df4fb025d3dfab26179d3e9d4aa6cf954f0acbc5/src/Confluent.Kafka/Config_gen.cs#L623

As an example, if I were to use the following config from a file:

{
  "bootstrap.servers": "localhost:9092",
  "group.id": "some-group-id",
  "auto.offset.reset": "earliest",
  "enable.auto.commit": "true",
  "auto.commit.interval.ms": 5000,
  "max.poll.interval.ms": 300000,
  "security.protocol": "SASL_SSL",
  "sasl.mechanism": "OAUTHBEARER"
}

For "sasl.mechanism" it is solved in the ClientConfig.SaslMechanism

I would suggest a similar approach be taken wrt. ClientConfig.SecurityProtocol.

mhowlett commented 2 years ago

it seems librdkafka config expects these constants as lowercase (and expects sasl.mechanism as uppercase). i think making librdkafka case insensitive to these values would be the ideal fix.

edenhill commented 2 years ago

I think @randomravings is right, there should be strictly typed enums in the dotnet client for things like SecurityProtocol.

But Matt is also right, everyone is right!, librdkafka should be more forgiving. I'll fix the latter.

edenhill commented 2 years ago

Scratch that, librdkafka's enums are already case insensitive.

mhowlett commented 2 years ago

right, looks like an issue on the .NET side.

huaxing-yuan commented 1 year ago

I encountered the same error following get started document: https://developer.confluent.io/get-started/dotnet/#configuration

If in producer.properties, we specify SASL_SSL as specified in the documentation, SecurityProtocol throws argument exception. but if we try to use enum value SaslSsl, SecurityProtocol does not throw exception, but when building Producer, another exception throws: Invalid value "SaslSsl" for configuration property "security.protocol"

   at Confluent.Kafka.Impl.SafeConfigHandle.Set(String name, String value)
   at Confluent.Kafka.Producer`2.<>c__DisplayClass56_0.<.ctor>b__5(KeyValuePair`2 kvp)
   at System.Collections.Generic.List`1.ForEach(Action`1 action)
   at Confluent.Kafka.Producer`2..ctor(ProducerBuilder`2 builder)
   at Confluent.Kafka.ProducerBuilder`2.Build()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

I would suggest the same implementation SaslMechanism as @randomravings has mentioned.

huaxing-yuan commented 1 year ago

Digging in the code, I think maybe the pb is here:

https://github.com/confluentinc/confluent-kafka-dotnet/blob/77a4f81b79b2320493f1dee237f5e45769b24a37/src/Confluent.Kafka/Config.cs#L159

before searching in EnumNameToConfigValueSubstitutes, we didn't transform result in lower case, nor counting by ignoring cases. so the value SASL_SSL couldn't be found.

The Stacktrace matches this hypothese System.ArgumentException: 'Requested value 'SASL_SSL' was not found.' image