Figure out how to process event hub messages using an Azure Functions custom handler

[zmoog]

Requirements

• brew upgrade azure/functions/azure-functions-core-tools@4

Refs

• https://learn.microsoft.com/en-us/azure/azure-functions/functions-custom-handlers
• https://learn.microsoft.com/en-us/azure/azure-functions/create-first-function-vs-code-other?tabs=go,macos

[zmoog]

I need to install the azure-functions-core-tools:

$ brew search azure-functions-core-tools
==> Formulae
azure/functions/azure-functions-core-tools                       azure/functions/azure-functions-core-tools@2                     azure/functions/azure-functions-core-tools@4 ✔
azure/functions/azure-functions-core-tools-v3-preview            azure/functions/azure-functions-core-tools@3

I already have azure/functions/azure-functions-core-tools@4 from previous work on Azure Functions, so I am probably good.

Update the local install to the latest version available:

$ brew upgrade azure/functions/azure-functions-core-tools@4

[zmoog]

Create a new Function App with a custom worker runtime:

$ func init azure-streamer --worker-runtime custom

Writing .gitignore
Writing host.json
Writing local.settings.json
Writing /Users/zmoog/code/projects/zmoog/azure-streamer/.vscode/extensions.json

[zmoog]

Now, we have a Function App, so it's time to create a function.

Listing the function templates:

$ func templates list 
C# Templates:
  <redacted>

Custom Templates:
  Azure Blob Storage trigger
  Azure Cosmos DB trigger
  Azure Event Grid trigger
  Azure Event Hub trigger
  HTTP trigger
  IoT Hub (Event Hub)
  Kafka trigger
  Azure Queue Storage trigger
  RabbitMQ trigger
  SendGrid
  Azure Service Bus Queue trigger
  Azure Service Bus Topic trigger
  SignalR negotiate HTTP trigger
  Timer trigger

JavaScript Templates:
  <redacted>

PowerShell Templates:
  <redacted>

Python Templates:
  <redacted>

TypeScript Templates:
  <redacted>

Creating a new function using the "Azure Event Hub trigger" template:

$ func new --name receive  --template "Azure Event Hub trigger"                                 
Select a number for template:Azure Event Hub trigger
Function name: [EventHubTrigger] Writing /Users/zmoog/code/projects/zmoog/azure-streamer/receive/function.json
The function "receive" was created successfully from the "Azure Event Hub trigger" template.

$ tree .                                                                                        
.
├── host.json
├── local.settings.json
└── receive
    └── function.json

1 directory, 3 files
$ cat receive/function.json                                                                     
{
  "bindings": [
    {
      "type": "eventHubTrigger",
      "name": "eventHubMessages",
      "direction": "in",
      "eventHubName": "samples-workitems",
      "connection": "",
      "cardinality": "many",
      "consumerGroup": "$Default"
    }
  ]
}

[zmoog]

Now it's time to create a few resources in Azure.

Set environment variables

As suggested by the tutorial, I will use the following environment variables in all commands:

export RESOURCE_GROUP=mbranca-azure-streamer
export EVENT_HUB_NAMESPACE=mbrancaazurestreamer
export EVENT_HUB_NAME=activitylogs
export EVENT_HUB_AUTHORIZATION_RULE=ListenOnly
export STORAGE_ACCOUNT=mbrancaazurestreamer
export FUNCTION_APP=mbranca-azure-streamer
export LOCATION=eastus2

Create a resource group

az group create \
    --name $RESOURCE_GROUP \
    --location $LOCATION

Create an event hub

az eventhubs namespace create \
    --resource-group $RESOURCE_GROUP \
    --sku Basic \
    --name $EVENT_HUB_NAMESPACE

az eventhubs eventhub create \
    --resource-group $RESOURCE_GROUP \
    --name $EVENT_HUB_NAME \
    --namespace-name $EVENT_HUB_NAMESPACE \
    --retention-time-in-hours 24 \
    --cleanup-policy delete

az eventhubs eventhub authorization-rule create \
    --resource-group $RESOURCE_GROUP \
    --name $EVENT_HUB_AUTHORIZATION_RULE \
    --eventhub-name $EVENT_HUB_NAME \
    --namespace-name $EVENT_HUB_NAMESPACE \
    --rights Listen

Create a storage account and function app

az storage account create \
    --resource-group $RESOURCE_GROUP \
    --name $STORAGE_ACCOUNT \
    --sku Standard_LRS

az functionapp create \
    --resource-group $RESOURCE_GROUP \
    --name $FUNCTION_APP \
    --storage-account $STORAGE_ACCOUNT \
    --consumption-plan-location $LOCATION \
    --os-type linux \
    --runtime custom \
    --functions-version 4

Note that:

When the az functionapp create command creates your function app, it also creates an Application Insights resource with the same name. The function app is automatically configured with a setting named APPINSIGHTS_INSTRUMENTATIONKEY that connects it to Application Insights. You can view app telemetry after you deploy your functions to Azure, as described later in this tutorial.

The function app seems a sort of container that can host multiple functions.

[zmoog]

Configure your function app

Retrieve resource connection strings

AZURE_WEB_JOBS_STORAGE=$( \
    az storage account show-connection-string \
        --name $STORAGE_ACCOUNT \
        --query connectionString \
        --output tsv)
echo $AZURE_WEB_JOBS_STORAGE

EVENT_HUB_CONNECTION_STRING=$( \
    az eventhubs eventhub authorization-rule keys list \
        --resource-group $RESOURCE_GROUP \
        --name $EVENT_HUB_AUTHORIZATION_RULE \
        --eventhub-name $EVENT_HUB_NAME \
        --namespace-name $EVENT_HUB_NAMESPACE \
        --query primaryConnectionString \
        --output tsv)
echo $EVENT_HUB_CONNECTION_STRING

Update your function app settings

Next, use the following command to transfer the connection string values to app settings in your Azure Functions account:

az functionapp config appsettings set \
    --resource-group $RESOURCE_GROUP \
    --name $FUNCTION_APP \
    --settings \
        AzureWebJobsStorage=$AZURE_WEB_JOBS_STORAGE \
        EventHubConnectionString=$EVENT_HUB_CONNECTION_STRING \
        ELASTICSEARCH_ENDPOINTS=$EVENT_HUB_CONNECTION_STRING \
        ELASTICSEARCH_API_KEY=$EVENT_HUB_CONNECTION_STRING 

Check the settings values with:

az functionapp config appsettings list --resource-group $RESOURCE_GROUP --name $FUNCTION_APP