`aws.batch.JobDefinition` doesn't contain tags when creating a new revision

[tlinhart]

Describe what happened

We observe this strange behavior with aws.batch.JobDefinition tags:

• When the job definition resource is created it contains tags, be it auto-tags (we use pulumi-aws-tags library) or tags explicitly defined via tags argument.
• When the job definition resource is modified and the update leads to creating a new revision (e.g. changing the command), the new revision doesn't have the tags. However, the previous (now inactive) revision gets tagged during the update.
• When the job definition resource is modified and the update doesn't lead to creating a new revision (e.g. adding or changing the tags argument), the job definition resource receives the tags.

Sample program

import json

import pulumi
import pulumi_aws as aws
from pulumi_aws_tags import register_auto_tags

# Automatically inject tags to created AWS resources.
register_auto_tags(
    {"user:Project": pulumi.get_project(), "user:Stack": pulumi.get_stack()}
)

# Use resources managed by infra stack.
infra_stack = pulumi.StackReference("infra-prod")

vpc_id = infra_stack.get_output("vpc_id")
private_subnet_ids = infra_stack.get_output("private_subnet_ids")

# Create an AWS Batch compute environment.
batch_security_group = aws.ec2.SecurityGroup(
    "x-tmp-batch-tagging",
    vpc_id=vpc_id,
    egress=[
        aws.ec2.SecurityGroupEgressArgs(
            from_port=0, to_port=0, protocol="-1", cidr_blocks=["0.0.0.0/0"]
        )
    ],
)

batch_service_role = aws.iam.Role(
    "x-tmp-batch-tagging-service-role",
    assume_role_policy=json.dumps(
        {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Action": "sts:AssumeRole",
                    "Principal": {"Service": "batch.amazonaws.com"},
                }
            ],
        }
    ),
    managed_policy_arns=[
        "arn:aws:iam::aws:policy/service-role/AWSBatchServiceRole"
    ],
)

compute_environment = aws.batch.ComputeEnvironment(
    "x-tmp-batch-tagging",
    compute_environment_name_prefix="x-tmp-batch-tagging",
    type="MANAGED",
    service_role=batch_service_role.arn,
    compute_resources=aws.batch.ComputeEnvironmentComputeResourcesArgs(
        max_vcpus=1,
        security_group_ids=[batch_security_group.id],
        subnets=private_subnet_ids,
        type="FARGATE",
    ),
)

aws.ecs.Tag(
    "user-project-tag",
    resource_arn=compute_environment.ecs_cluster_arn,
    key="user:Project",
    value=pulumi.get_project(),
)

aws.ecs.Tag(
    "user-stack-tag",
    resource_arn=compute_environment.ecs_cluster_arn,
    key="user:Stack",
    value=pulumi.get_stack(),
)

# Create an AWS Batch job queue mapped to the compute environment.
job_queue = aws.batch.JobQueue(
    "x-tmp-batch-tagging",
    state="ENABLED",
    priority=10,
    compute_environments=[compute_environment.arn],
)

# Create a log group for the job.
job_log_group = aws.cloudwatch.LogGroup(
    "x-tmp-batch-tagging", retention_in_days=30
)

# Create an AWS Batch job definition.
job_execution_role = aws.iam.Role(
    "x-tmp-batch-tagging-execution-role",
    assume_role_policy=json.dumps(
        {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Action": "sts:AssumeRole",
                    "Principal": {"Service": "ecs-tasks.amazonaws.com"},
                }
            ],
        }
    ),
    managed_policy_arns=[
        "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
    ],
)

job_role = aws.iam.Role(
    "x-tmp-batch-tagging-job-role",
    assume_role_policy=json.dumps(
        {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Action": "sts:AssumeRole",
                    "Principal": {"Service": "ecs-tasks.amazonaws.com"},
                }
            ],
        }
    ),
)

job_definition = aws.batch.JobDefinition(
    "x-tmp-batch-tagging",
    type="container",
    platform_capabilities=["FARGATE"],
    container_properties=pulumi.Output.all(
        execution_role=job_execution_role.arn,
        job_role=job_role.arn,
        log_group=job_log_group.name,
    ).apply(
        lambda args: json.dumps(
            {
                "command": ["sleep", "15"],
                "image": "busybox",
                "logConfiguration": {
                    "logDriver": "awslogs",
                    "options": {"awslogs-group": args["log_group"]},
                },
                "runtimePlatform": {
                    "cpuArchitecture": "ARM64",
                    "operatingSystemFamily": "LINUX",
                },
                "resourceRequirements": [
                    {"type": "VCPU", "value": "0.25"},
                    {"type": "MEMORY", "value": "512"},
                ],
                "executionRoleArn": args["execution_role"],
                "jobRoleArn": args["job_role"],
            }
        )
    ),
    tags={"user:Foo": "bar"},
    propagate_tags=True,
)

# Export stack outputs.
pulumi.export("ecs_cluster", compute_environment.ecs_cluster_arn)
pulumi.export("job_definition", job_definition.name)
pulumi.export("job_queue", job_queue.name)

Log output

Here's the output of pulumi up after changing e.g. command in container properties which leads to a new job definition revision:

Updating (x-tmp-batch-tagging-dev):
     Type                        Name                                         Status              Info
     pulumi:pulumi:Stack         x-tmp-batch-tagging-x-tmp-batch-tagging-dev                      2 messages
 ~   └─ aws:batch:JobDefinition  x-tmp-batch-tagging                          updated (0.59s)     [diff: +tags,tagsAll~containerProperties,tags,tagsAll]

Outputs:
    ecs_cluster   : "xxx"
    job_definition: "xxx"
    job_queue     : "xxx"

Resources:
    ~ 1 updated
    10 unchanged

Duration: 11s

Here's the output after the same change when adding a --diff option:

Updating (x-tmp-batch-tagging-dev):
  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::pulumi:pulumi:Stack::x-tmp-batch-tagging-x-tmp-batch-tagging-dev]
    > pulumi:pulumi:StackReference: (read)
        [id=infra-prod]
        [urn=urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::pulumi:pulumi:StackReference::infra-prod]
        name: "infra-prod"
    ~ aws:batch/jobDefinition:JobDefinition: (update)
        [id=arn:aws:batch:eu-central-1:xxx:job-definition/x-tmp-batch-tagging-23bc289:1]
        [urn=urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::aws:batch/jobDefinition:JobDefinition::x-tmp-batch-tagging]
        [provider=urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::pulumi:providers:aws::default_6_51_0::b5382c19-dda7-41fd-86a6-43955c2b3f9d]
      ~ containerProperties: (json) {
          ~ command                     : [
                [0]: "sleep"
              ~ [1]: "15" => "10"
            ]
          - environment                 : []
            executionRoleArn            : "arn:aws:iam::xxx:role/x-tmp-batch-tagging-execution-role-bd1dc64"
          - fargatePlatformConfiguration: {
              - platformVersion: "LATEST"
            }
            image                       : "busybox"
            jobRoleArn                  : "arn:aws:iam::xxx:role/x-tmp-batch-tagging-job-role-d48642d"
          ~ logConfiguration            : {
                logDriver    : "awslogs"
                options      : {
                    awslogs-group: "x-tmp-batch-tagging-0eb1d94"
                }
              - secretOptions: []
            }
          - mountPoints                 : []
            resourceRequirements        : [
                [0]: {
                    type : "VCPU"
                    value: "0.25"
                }
                [1]: {
                    type : "MEMORY"
                    value: "512"
                }
            ]
            runtimePlatform             : {
                cpuArchitecture      : "ARM64"
                operatingSystemFamily: "LINUX"
            }
          - secrets                     : []
          - ulimits                     : []
          - volumes                     : []
        }
        --outputs:--
      ~ arn                    : "arn:aws:batch:eu-central-1:xxx:job-definition/x-tmp-batch-tagging-23bc289:1" => "arn:aws:batch:eu-central-1:xxx:job-definition/x-tmp-batch-tagging-23bc289:2"
      ~ containerProperties    : (json) {
          ~ command                     : [
                [0]: "sleep"
              ~ [1]: "15" => "10"
            ]
            executionRoleArn            : "arn:aws:iam::xxx:role/x-tmp-batch-tagging-execution-role-bd1dc64"
            fargatePlatformConfiguration: {
                platformVersion: "LATEST"
            }
            image                       : "busybox"
            jobRoleArn                  : "arn:aws:iam::xxx:role/x-tmp-batch-tagging-job-role-d48642d"
            logConfiguration            : {
                logDriver    : "awslogs"
                options      : {
                    awslogs-group: "x-tmp-batch-tagging-0eb1d94"
                }
                secretOptions: []
            }
            resourceRequirements        : [
                [0]: {
                    type : "VCPU"
                    value: "0.25"
                }
                [1]: {
                    type : "MEMORY"
                    value: "512"
                }
            ]
            runtimePlatform             : {
                cpuArchitecture      : "ARM64"
                operatingSystemFamily: "LINUX"
            }
        }
      ~ id                     : "arn:aws:batch:eu-central-1:xxx:job-definition/x-tmp-batch-tagging-23bc289:1" => "arn:aws:batch:eu-central-1:xxx:job-definition/x-tmp-batch-tagging-23bc289:2"
      ~ revision               : 1 => 2
      - tags                   : {
          - user:Foo    : "bar"
          - user:Project: "x-tmp-batch-tagging"
          - user:Stack  : "x-tmp-batch-tagging-dev"
        }
      - tagsAll                : {
          - user:Foo    : "bar"
          - user:Project: "x-tmp-batch-tagging"
          - user:Stack  : "x-tmp-batch-tagging-dev"
        }

    --outputs:--
    ecs_cluster   : "xxx"
    job_definition: "xxx"
    job_queue     : "xxx"
Resources:
    ~ 1 updated
    10 unchanged

Duration: 13s

Affected Resource(s)

aws.batch.JobDefinition

Output of pulumi about

CLI          
Version      3.136.1
Go Version   go1.23.2
Go Compiler  gc

Plugins
KIND      NAME             VERSION
resource  aws              6.51.0
resource  pulumi_aws_tags  1.1.0
language  python           unknown

Host     
OS       ubuntu
Version  24.04
Arch     x86_64

This project is written in python: executable='/home/xxx/x-tmp-batch-tagging/infra/venv/bin/python' version='3.12.3'

Current Stack: x-tmp-batch-tagging-dev

TYPE                                             URN
pulumi:pulumi:Stack                              urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::pulumi:pulumi:Stack::x-tmp-batch-tagging-x-tmp-batch-tagging-dev
pulumi:providers:pulumi                          urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::pulumi:providers:pulumi::default
pulumi:pulumi:StackReference                     urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::pulumi:pulumi:StackReference::infra-prod
pulumi:providers:aws                             urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::pulumi:providers:aws::default_6_51_0
aws:iam/role:Role                                urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::aws:iam/role:Role::x-tmp-batch-tagging-job-role
aws:iam/role:Role                                urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::aws:iam/role:Role::x-tmp-batch-tagging-service-role
aws:cloudwatch/logGroup:LogGroup                 urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::aws:cloudwatch/logGroup:LogGroup::x-tmp-batch-tagging
aws:iam/role:Role                                urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::aws:iam/role:Role::x-tmp-batch-tagging-execution-role
aws:batch/jobDefinition:JobDefinition            urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::aws:batch/jobDefinition:JobDefinition::x-tmp-batch-tagging
aws:ec2/securityGroup:SecurityGroup              urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::aws:ec2/securityGroup:SecurityGroup::x-tmp-batch-tagging
aws:batch/computeEnvironment:ComputeEnvironment  urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::aws:batch/computeEnvironment:ComputeEnvironment::x-tmp-batch-tagging
aws:ecs/tag:Tag                                  urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::aws:ecs/tag:Tag::user-project-tag
aws:ecs/tag:Tag                                  urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::aws:ecs/tag:Tag::user-stack-tag
aws:batch/jobQueue:JobQueue                      urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::aws:batch/jobQueue:JobQueue::x-tmp-batch-tagging

Found no pending operations associated with x-tmp-batch-tagging-dev

Backend        
Name           xxx
URL            s3://xxx
User           xxx
Organizations  
Token type     personal

Dependencies:
NAME             VERSION
pip              24.2
pulumi_aws_tags  1.1.0
setuptools       75.2.0
wheel            0.44.0

Pulumi locates its logs in /tmp by default

Additional context

We use tags propagation to propagate tags from the job and job definition to the ECS task. We found out in AWS Cost Explorer that the tags of ECS tasks corresponding to Batch jobs started to be missing on May 30 this year. This might relate to the Pulumi AWS package update from 6.34.1 to 6.37.1 that we performed on May 27.

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[t0yv0]

Thank you for reporting this strange behavior. The bug reproduces.

Starting with this:

import json
import pulumi as pulumi
import pulumi_aws as aws

test = aws.batch.JobDefinition(
    "test",
    name="my_test_batch_job_definition",
    type="container",
    container_properties=json.dumps(
        {
            "command": [
                "ls",
            ],
            "image": "busybox",
            "resourceRequirements": [
                {
                    "type": "VCPU",
                    "value": "1",
                },
                {
                    "type": "MEMORY",
                    "value": "512",
                },
            ]
        }
    ),
    tags={"env": "prod"}
)

pulumi.export("name", test.name)
pulumi.export("revision", test.revision)

Checking tagging with this snippet:

aws batch describe-job-definitions --job-definition-name my_test_batch_job_definition |
    jq '.jobDefinitions[]|{revision: .revision, tags: .tags}'

After a pulumi up, we get:

{
  "revision": 1,
  "tags": {
    "env": "prod"
  }
}

After changing the command and doing another pulumi up:

"command": [
    "ls", "--help"
],

We get this:

{
  "revision": 1,
  "tags": {
    "env": "prod"
  }
}
{
  "revision": 2,
  "tags": {}
}

[t0yv0]

I am finding the same behavior in Terraform proper, so I think this is something that needs fixing ideally in the upstream provider. I will file a bug there and link here.

[t0yv0]

I have filed this in https://github.com/hashicorp/terraform-provider-aws/issues/39795 - it might help to upvote it to get the maintainers attention! Once the fix is made available the Pulumi provider will likely inherit it and get fixed as well in a regular release.

[tlinhart]

Thanks! I upvoted the issue.