Container App Job fails to use User Assigned Managed Identity

[willvelida]

This issue is a: (mark with an x)

• [ x] bug report -> please search issues before submitting
• [ ] documentation issue or request
• [ ] regression (a behavior that used to work and stopped in a new release)

Issue description

I have a Container App Cron job that uses a user-assigned managed identity. The managed identity has the AcrPull role, but the Container App job fails, and Log Analytics logs the following error:

MountVolume.SetUp failed for volume "identity-config" : configmap "biotrackr-auth-identity" not found

Steps to reproduce

Review the following Terraform template for creating the job:

data "azurerm_resource_group" "rg" {
  name = var.resource_group_name
}

data "azurerm_container_app_environment" "env" {
  name                = var.env_name
  resource_group_name = data.azurerm_resource_group.rg.name
}

data "azurerm_user_assigned_identity" "msi" {
  name                = var.uai_name
  resource_group_name = data.azurerm_resource_group.rg.name
}

data "azurerm_container_registry" "acr" {
  name                = var.acr_name
  resource_group_name = data.azurerm_resource_group.rg.name
}

resource "azurerm_container_app_job" "job" {
  name                         = var.aca_app_name
  location                     = data.azurerm_resource_group.rg.location
  resource_group_name          = data.azurerm_resource_group.rg.name
  tags                         = var.tags
  container_app_environment_id = data.azurerm_container_app_environment.env.id
  template {
    container {
      name   = var.aca_app_name
      image  = var.image_name
      cpu    = 0.25
      memory = "0.5Gi"
    }
  }
  replica_timeout_in_seconds = 60
  replica_retry_limit        = 3
  schedule_trigger_config {
    cron_expression          = "*/15 * * * *"
    parallelism              = 1
    replica_completion_count = 1
  }

  registry {
    server   = data.azurerm_container_registry.acr.login_server
    identity = data.azurerm_user_assigned_identity.msi.id
  }

  identity {
    type         = "UserAssigned"
    identity_ids = [data.azurerm_user_assigned_identity.msi.id]
  }
}

This is the terraform for implementing the ACR, MSI, and role assignment:

module "usi" {
  source   = "../modules/user-assigned-identity"
  name     = var.user_assigned_identity_name
  location = var.location
  rg_name  = module.resource_group.name
  tags     = var.tags
}

module "acr" {
  source                    = "../modules/container-registry"
  name                      = var.acr_name
  location                  = var.location
  rg_name                   = module.resource_group.name
  sku                       = var.acr_sku
  admin_enabled             = var.acr_admin_enabled
  user_assigned_identity_id = module.usi.user_assinged_identity_id
  tags                      = var.tags
}

module "acr_pull_role" {
  source       = "../modules/role-assignment"
  role_name    = var.acr_pull_role_name
  principal_id = module.usi.user_assinged_identity_principal_id
  scope_id     = module.acr.acr_id
}

This is the logic that the job is doing. Very basic .NET worker:

using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace Biotrackr.Auth
{
    public class AuthWorker : BackgroundService
    {
        private readonly ILogger<AuthWorker> _logger;
        private readonly IHostApplicationLifetime _appLifetime;

        public AuthWorker(ILogger<AuthWorker> logger, IHostApplicationLifetime appLifetime)
        {
            _logger = logger;
            _appLifetime = appLifetime;
        }

        protected override async Task<int> ExecuteAsync(CancellationToken stoppingToken)
        {
            try
            {
                _logger.LogInformation($"Hello there! Biotrackr.Auth running at {DateTime.Now}");
                return 0;
            }
            catch (Exception ex)
            {
                _logger.LogError($"Exception thrown: {ex.Message}");
                throw;
            }
        }
    }
}

Implemented in a Program.cs file like so:

using Biotrackr.Auth;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

IHost host = Host.CreateDefaultBuilder(args)
    .ConfigureServices(services =>
    {
        services.AddHostedService<AuthWorker>();
    })
    .Build();

host.Run();

The full code (with all Terraform modules and C# code) can be found here: https://github.com/willvelida/biotrackr

Expected behavior [What you expected to happen.]

For Log Analytics to log out the message, and for the container app job to be successful

Actual behavior [What actually happened.]

Container App job fails due to the following error:

MountVolume.SetUp failed for volume "identity-config" : configmap "biotrackr-auth-identity" not found

[anthonychu]

@willvelida can you please share your job's resource id with us at acasupport@ microsoft.com and reference this issue? Thanks! 🙂

[willvelida]

Thanks @anthonychu, just emailed you now :)

[willvelida]

@anthonychu we can mark this as resolved - Thanks to the team for helping me with this offline!

For those wondering what the solution is, I had a Container App that I was running as a Dapr cron job called biotrackr-auth. I used the same name for a normal job, and while the Terraform deployment destroyed the old Container App and created a new job, using the same name caused a transient issue as they were trying to execute at the same time.

I resolved it by using a new name for the job.