Trigger the Azure Function from Event Hub

[zmoog]

Relates: #16

Resources

• https://learn.microsoft.com/en-us/azure/azure-functions/functions-event-hub-cosmos-db?tabs=bash

[zmoog]

I will first try to use the tutorial named "Create a function in Java with an Event Hub trigger and an Azure Cosmos DB output binding" making the necessary adjustments.

[zmoog]

Set environment variables

As suggested by the tutorial, I will use the following environment variables in all commands:

export RESOURCE_GROUP=mbranca-esf
export EVENT_HUB_NAMESPACE=mbrancaesfns
export EVENT_HUB_NAME=azurelogs
export EVENT_HUB_AUTHORIZATION_RULE=ListenOnly
export STORAGE_ACCOUNT=mbrancaesfsa
export FUNCTION_APP=mbranca-esf
export LOCATION=eastus

[zmoog]

Create a resource group

az group create \
    --name $RESOURCE_GROUP \
    --location $LOCATION

[zmoog]

Create an event hub

az eventhubs namespace create \
    --resource-group $RESOURCE_GROUP \
    --sku Basic \
    --name $EVENT_HUB_NAMESPACE

az eventhubs eventhub create \
    --resource-group $RESOURCE_GROUP \
    --name $EVENT_HUB_NAME \
    --namespace-name $EVENT_HUB_NAMESPACE \
    --message-retention 1

az eventhubs eventhub authorization-rule create \
    --resource-group $RESOURCE_GROUP \
    --name $EVENT_HUB_AUTHORIZATION_RULE \
    --eventhub-name $EVENT_HUB_NAME \
    --namespace-name $EVENT_HUB_NAMESPACE \
    --rights Listen

[zmoog]

Create a storage account and function app

az storage account create \
    --resource-group $RESOURCE_GROUP \
    --name $STORAGE_ACCOUNT \
    --sku Standard_LRS

az functionapp create \
    --resource-group $RESOURCE_GROUP \
    --name $FUNCTION_APP \
    --storage-account $STORAGE_ACCOUNT \
    --consumption-plan-location $LOCATION \
    --os-type linux \
    --runtime python \
    --runtime-version 3.10 \
    --functions-version 4

Note that:

When the az functionapp create command creates your function app, it also creates an Application Insights resource with the same name. The function app is automatically configured with a setting named APPINSIGHTS_INSTRUMENTATIONKEY that connects it to Application Insights. You can view app telemetry after you deploy your functions to Azure, as described later in this tutorial.

The function app seems a sort of container that can host multiple functions.

[zmoog]

Configure your function app

Retrieve resource connection strings

AZURE_WEB_JOBS_STORAGE=$( \
    az storage account show-connection-string \
        --name $STORAGE_ACCOUNT \
        --query connectionString \
        --output tsv)
echo $AZURE_WEB_JOBS_STORAGE

EVENT_HUB_CONNECTION_STRING=$( \
    az eventhubs eventhub authorization-rule keys list \
        --resource-group $RESOURCE_GROUP \
        --name $EVENT_HUB_AUTHORIZATION_RULE \
        --eventhub-name $EVENT_HUB_NAME \
        --namespace-name $EVENT_HUB_NAMESPACE \
        --query primaryConnectionString \
        --output tsv)
echo $EVENT_HUB_CONNECTION_STRING

Update your function app settings

Next, use the following command to transfer the connection string values to app settings in your Azure Functions account:

az functionapp config appsettings set \
    --resource-group $RESOURCE_GROUP \
    --name $FUNCTION_APP \
    --settings \
        AzureWebJobsStorage=$AZURE_WEB_JOBS_STORAGE \
        EventHubConnectionString=$EVENT_HUB_CONNECTION_STRING

[zmoog]

Next step is write an actual function that's triggered by the event hub.

Collecting resources for Python:

• https://medium.com/analytics-vidhya/azure-functions-event-hub-2603e8804f73
• https://github.com/yokawasa/azure-functions-python-samples/blob/master/v1functions/eventhub-trigger-table-out-bindings/README.md

[zmoog]

Bootstrapping a new function app

10027  func init forwarder --python
10028  cd forwarder/
10029  ls -ltr
10030  python -m venv venv
10031  . venv/bin/activate
10032  pip install -r requirements.txt
10033  python --version
10034  pip install -r requirements.txt
10035  pip install --upgrade pip
10036  asdf local python 3.10.10
10037  ls -ltr
10038  ls -la
10039  python --version
10040  func
10041  func templates list -l python

func new --name process_eventhub --template "Azure Event Hub trigger"

[zmoog]

I created a new repo https://github.com/zmoog/forwarder/ ho host the PoC, and added a couple of commits:

• https://github.com/zmoog/forwarder/commit/2d3e65befe2eea14594dda6bc19b0158a15814d7
• https://github.com/zmoog/forwarder/commit/922c695709ad82c020475ac31eb175a3fe8a67ab

This requires a local file to work:

{
  "IsEncrypted": false,
  "Values": {
    "FUNCTIONS_WORKER_RUNTIME": "python",
    "AzureWebJobsStorage": "DefaultEndpointsProtocol= ... windows.net/",
    "receiverConnectionString": "Endpoint=sb://...0=;EntityPath=azurelogs"
  }
}

And by executing:

$ func start
Found Python version 3.8.10 (python3).

Azure Functions Core Tools
Core Tools Version:       4.0.4895 Commit hash: N/A  (64-bit)
Function Runtime Version: 4.13.0.19486

Functions:

    process_eventhub: eventHubTrigger

For detailed output, run func with --verbose flag.
[2023-03-09T07:53:22.277Z] Worker process started and initialized.
[2023-03-09T07:53:26.733Z] Host lock lease acquired by instance ID '000000000000000000000000000C2005'.
[2023-03-09T07:53:29.230Z] Executing 'Functions.process_eventhub' (Reason='(null)', Id=be909a66-a800-4d32-812e-3b27392953e5)
[2023-03-09T07:53:29.233Z] Trigger Details: PartionId: 1, Offset: 496904-496904, EnqueueTimeUtc: 2023-03-09T07:53:28.8820000+00:00-2023-03-09T07:53:28.8820000+00:00, SequenceNumber: 52-52, Count: 1
[2023-03-09T07:53:29.530Z] Python EventHub trigger processed an event: {"records": [{"RoleLocation": "Germany West Central", "ReleaseVersion": "6.2023.9.2+1c09383.release_2023w09", "time": "2023-03-09T07:45:28.6741316Z", "resourceId": "/SUBSCRIPTIONS/12C..... 719adb"}]}
[2023-03-09T07:53:29.591Z] Executed 'Functions.process_eventhub' (Succeeded, Id=be909a66-a800-4d32-812e-3b27392953e5, Duration=480ms)

It works! 🎉