search

Azure / azure-functions-kafka-extension

Kafka extension for Azure Functions
MIT License
114 stars 82 forks source link

KafkaTrigger fails to load SslCertificateLocation and SslKeyLocation #321

Closed Irate-Walrus closed 2 years ago

Irate-Walrus commented 2 years ago
shrohilla commented 2 years ago

Please share your configuration with functions.json & other related details

Irate-Walrus commented 2 years ago

📝 I did have to change some values for confidentiality, but nothing meaningful.

function.json

{
"generatedBy": "Microsoft.NET.Sdk.Functions.Generator-4.1.0",
"configurationSource": "attributes",
"bindings": [
{
"type": "kafkaTrigger",
"sslCaLocation": "server.crt",
"sslCertificateLocation": "client.crt",
"sslKeyLocation": "client.key",
"protocol": "ssl",
"consumerGroup": "consumerGroup",
"topic": "%Topic%",
"brokerList": "%BrokerList%",
"authenticationMode": "notSet",
"lagThreshold": 1000,
"name": "events"
}
],
"disabled": false,
"scriptFile": "../bin/KafkaConnector.dll",
"entryPoint": "KafkaSolution.KafkaController.Subscriber"
}

host.json

{
"extensions": {
},
"logging": {
"logLevel": {
"default": "Warning",
"Function": "Debug",
"Host.Triggers.Kafka": "Warning"
},
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request"
}
}
},
"version": "2.0"
}

Working function.json

{
"generatedBy": "Microsoft.NET.Sdk.Functions.Generator-4.1.0",
"configurationSource": "attributes",
"bindings": [
{
"type": "kafkaTrigger",
"sslCaLocation": "server.crt",
"sslCertificateLocation": "%HOME%\\site\\wwwroot\\client.crt",
"sslKeyLocation": "%HOME%\\site\\wwwroot\\client.key",
"protocol": "ssl",
"consumerGroup": "consumerGroup",
"topic": "%Topic%",
"brokerList": "%BrokerList%",
"authenticationMode": "notSet",
"lagThreshold": 1000,
"name": "events"
}
],
"disabled": false,
"scriptFile": "../bin/KafkaConnector.dll",
"entryPoint": "KafkaSolution.KafkaController.Subscriber"
}
shrohilla commented 2 years ago

Can you share the details of below two points:-

  1. Can you add the complete relative path of the sslCertificateLocation & sslCaLocation and retry
  2. Can you check "Copy to Output Directory" configuration value, I am wondering if certificate is copied in bin directory
Irate-Walrus commented 2 years ago

Project Configuration for CopyToOutputDirectory

<ItemGroup>
    <None Update="server_ca.crt">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
    <None Update="client.crt">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
    <None Update="client.key">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
    <None Update="host.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
    <None Update="local.settings.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <CopyToPublishDirectory>Never</CopyToPublishDirectory>
    </None>
</ItemGroup>

wwwroot

image

Irate-Walrus commented 2 years ago
Some paths I've tried: Path Azure Local Debug (self-explanatory)
"%HOME%\\site\\wwwroot\\server.crt" ✔️ ❌
"D:\\home\\site\\wwwroot\\server.crt" ✔️ ❌
".\\server_ca.crt" ❌ ✔️
"server_ca.crt" ❌ ✔️
MichalLechowski commented 2 years ago

For WebJob (same nuget) I just use the same kinda approach, I have the certs in the root of my app and I use the same nuget version.

SslCaLocation = "rootcert.pem",
SslCertificateLocation = "clientcert.pem",
SslKeyLocation = "privatekey.pem",

This just works all right. What also works is when I set it up like this e.g.:

SslCaLocation = "pathInRootDir//rootcert.pem"

But it doesn't work with only single /

I use Docker to build and deploy to AKS on Linux nodes. But it also works locally since I don't see any certificates issues when running in Docker Desktop or just locally as ..NET 5 Console App (WebJob) and when I set a wrong file or a path, it throws an error. I have also had some kafka subs running as ACI in Azure, it works, too.

One difference I see is that I have AuthMode set to Plain but it shouldn't matter.

Irate-Walrus commented 2 years ago

Hi @MichalLechowski, Thank you for the information. I do believe the issue is platform dependent as the filename works correctly locally. The function is running .NET 6 on a Windows Premium Function. Deployment is done via a local build and upload as a package.

MichalLechowski commented 2 years ago

Okay.

Can you:

  1. Share your function code, the method with Input Kafka trigger? You can skip the body, just the whole declaration with attributes and such. If you have some passwords there, obviously remove them.
  2. Can you go to your Azure function then go to Advanced Tools then click Ok to open Kudu and then you have access to your function file system and check if actually the cert file is where you think it is? You need to open debug console in Kudu (CMD or PowerShell, doesn't matter) and there you have access to file system over simple GUI. Then you'll see your typical sites/wwwroot and such.
Irate-Walrus commented 2 years ago

@MichalLechowski ,

  1. Share your function code, the method with Input Kafka trigger? This is the broken configuration of the trigger, just a reflection of the function.json in above comment image

  2. Can you go to your Azure function then go to Advanced Tools then click Ok to open Kudu etc. Can confirm they are in D:\home\site\wwwroot as previously shown in above comment

MichalLechowski commented 2 years ago

Okay, I've setup a basic azf (.net 6 isolated) with kafka trigger and deployed to Azure Function App running on Windows. It looks like the problem is that you think that execution context is the same directory as dlls of your built project (basically wwwroot in this case) but it's probably not and you always need to set it up with a proper path, that's why "%HOME%\site\wwwroot\server.crt" works just fine for you and that is why plain filename works when running locally since locally it looks for the file in bin/Release/ directory (for Release environment) where all your dlls and certificates are.

You could play with it and somehow dynamically build the path but I don't it's worth it. Just use environment variables in Function App and in local.settings.json and set proper paths for diffrent environments. When running locally it'll use local.settings.json and when running in Azure, it'll use the ones setup in Configuration part of Function App in Azure.

That is not a problem with the kafka extension, just how Functions work in Azure. Maybe it'd be different for Azure Function running on Linux, I don't know but you can try if you have time.

That is also why there is ExecutionContext class available you can add in your function method signature to get the directory during runtime but you cannot use it in an attribute value, obviously.

Irate-Walrus commented 2 years ago

Thanks for the sanity check, it's good to hear you've experienced the same issue.

You could play with it and somehow dynamically build the path but I don't it's worth it.

Just regarding this, I thought the package already did in the AzureFunctionsFileHelper. AzureFunctionsFileHelper is called in KafkaProducerFactory. I believe this is the reason why I don't have this issue when using the Kafka output binding.

// KafkaProducerFactory.cs
public ProducerConfig GetProducerConfig(KafkaProducerEntity entity)
{
    if (!AzureFunctionsFileHelper.TryGetValidFilePath(entity.Attribute.SslCertificateLocation, out var resolvedSslCertificationLocation))
    {
        resolvedSslCertificationLocation = entity.Attribute.SslCertificateLocation;
    }

However, looking into this further, the KafkaTriggerAttributeBindingProvider doesn't actually use the helper, and therefore breaks without the file path.

// KafkaTriggerAttributeBindingProvider.cs
// CreateConsumerConfiguration
consumerConfig.SaslPassword = this.config.ResolveSecureSetting(nameResolver, attribute.Password);
consumerConfig.SaslUsername = this.config.ResolveSecureSetting(nameResolver, attribute.Username);
consumerConfig.SslKeyLocation = this.config.ResolveSecureSetting(nameResolver, attribute.SslKeyLocation);
consumerConfig.SslKeyPassword = this.config.ResolveSecureSetting(nameResolver, attribute.SslKeyPassword);
consumerConfig.SslCertificateLocation = this.config.ResolveSecureSetting(nameResolver, attribute.SslCertificateLocation);
consumerConfig.SslCaLocation = this.config.ResolveSecureSetting(nameResolver, attribute.SslCaLocation);

What I do still find confusing is why the server_ca.crt is found as seen in my initial comment and why the file loading behavior is different between the Kafka Output and the Kafka Trigger:

2022-04-19T03:09:32.209 [Debug] Librdkafka initialization: loading librdkafka from C:\home\site\wwwroot\bin\runtimes\win-x86\native\librdkafka.dll
2022-04-19T03:09:32.286 [Debug] Found SslCaLocation in C:\home\site\wwwroot\server_ca.crt
2022-04-19T03:09:32.306 [Debug] Libkafka: [thrd:app]: librdkafka built with OpenSSL version 0x1000211f
2022-04-19T03:09:32.306 [Debug] Libkafka: [thrd:app]: Loading CA certificate(s) from file C:\home\site\wwwroot\server_ca.crt
2022-04-19T03:09:32.306 [Debug] Libkafka: [thrd:app]: Loading public key from file client.crt
2022-04-19T03:09:32.307 [Error] Libkafka: [thrd:app]: .\crypto\bio\bss_file.c:406: error:02001002:system library:fopen:No such file or directory: fopen('client.crt','rb')
shrohilla commented 2 years ago

This seems a bug, we will look into this

MichalLechowski commented 2 years ago

That's weird, indeed. One thing you can check is what happens (just as an experiment) when you set it up manually instead of using an attribute. Configured manually and triggered with TimeTrigger, for instance, based on e.g. this: https://github.com/Azure/azure-functions-kafka-extension/blob/dev/samples/dotnet/ConsoleConsumer/Program.cs

I have been wondering if that might have anything to do with permissions. Maybe the file exists but the app has no permissions to access it. Just a thought.

shrohilla commented 2 years ago

@Irate-Walrus Thanks alot for reporting this bug, we had fixed this from 3.6.0 release