How to use environment variable for brokerList in kubernetes deployment

[tjsiron]

I'm running into an issue where I'm trying to deploy a C# azure function and deploying it to kubernetes.

The KafkaTriggerAttribute.brokerList attribute seems to be what is used when building the metadata for the ScaledObject.

The only way I can get the ScaledObject kubernetes resource to generate correctly is by using the brokerlist directly in the function rather than providing any kind of app settings variable. The app settings variable does seem to work for the actual runtime, but not the KEDA trigger. The below function produces the correct ScaledObject resource

 [FunctionName("Cars")]
        public static async void Run(
           [KafkaTrigger("my-confluent-oss-cp-kafka-headless:9092", "cars", ConsumerGroup = "cars")] byte[][] kafkaEvents)
        {
            foreach (var kafkaEvent in kafkaEvents)
            {
                var car = await deserializer.DeserializeAsync(kafkaEvent, false, SerializationContext.Empty);
                Console.WriteLine($"Custom deserialised user from batch: {JsonConvert.SerializeObject(car)}");
            }
        }

  triggers:
  - type: kafka
    metadata:
      type: kafkaTrigger
      consumerGroup: cars
      topic: cars
      brokerList: my-confluent-oss-cp-kafka-headless:9092
      authenticationMode: notSet
      protocol: notSet
      name: kafkaEvents

But I don't want to have to provide my brokerList at runtime. Altering it to LocalBroker makes the runtime work and pulling from an app settings variable work, but the ScaledObject reference breaks and it produces something the KEDA operator canot use.

        [FunctionName("Cars")]
        public static async void Run(
           [KafkaTrigger("LocalBroker", "cars", ConsumerGroup = "cars")] byte[][] kafkaEvents)
        {
            foreach (var kafkaEvent in kafkaEvents)
            {
                var car = await deserializer.DeserializeAsync(kafkaEvent, false, SerializationContext.Empty);
                Console.WriteLine($"Custom deserialised user from batch: {JsonConvert.SerializeObject(car)}");
            }
        }

 triggers:
  - type: kafka
    metadata:
      type: kafkaTrigger
      consumerGroup: cars-20
      topic: cars
      brokerList: LocalBroker
      authenticationMode: notSet
      protocol: notSet
      name: kafkaEvents

Long story short, I want to be able to use an environment variable in some way, but if I do, the ScaledObject resource is wrong. Am I missing something obvious?

[TsuyoshiUshio]

Hi @tjsiron Thank you for this issue. I realize that the issue. In this case, Kafka Trigger on C# is fine. However, The KEDA scaler configuration might be the issue. Under the hood, Kafka Trigger on this repo, substitute the LocalBroker as Environment variables. So that there is no connection between KEDA and this Kafka Trigger. For enabling the feature, could you create an issue on KEDA repo? https://github.com/kedacore/keda/issues For example, EventHubs trigger already implement it. https://keda.sh/docs/2.0/scalers/azure-event-hub/

[tjsiron]

Thank you! I will do that. I wasn't sure which repo it fell under.

[ryancrawcour]

@tjsiron did @TsuyoshiUshio's suggestions resolve the issue for you?

[derekrprice]

I ran into this too with JS functions. Most of the other plugins substitute strings like %VARIABLE% in bindings with environment variable values. This one apparently does not. https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-expressions-patterns