AWS::Glue::Job should allow tags to be updated

chirillc-aws commented 4 years ago

1. Title

2. Scope of request

Glue jobs created via CloudFormation ignore Tags values update in consecutive stack update runs. Stack update does not fail, but Glue job tags don't get updated.

3.Expected behavior

In Update, it should detect that tag value have been changed and update Glue Job tags correspondingly.

4. Test case recommendation (optional)

Create any Glue Job including tags via CloudFormation. Example below.

  GlueJob:
    Type: AWS::Glue::Job
    Properties:
      Role: ...
      DefaultArguments: 
        ....
      MaxRetries: "0"
      GlueVersion: "1.0"
      Description: "Glue Job"
      Command:
        Name: "pythonshell"
        ScriptLocation: ...
        PythonVersion: 3
      ExecutionProperty:
        MaxConcurrentRuns: 1
      Name: GlueJob
      Tags:
        CostCenter:
          Ref: TagCostCenter
        Department:
          Ref: TagDepartment
        Environment:
          Ref: TagEnvironment
        IO:
          Ref: TagIO
        Project:
          Ref: ProjectName

Then try updating any tag value and perform stack update. Observed behavior is that stack update does not fail, although tags update is ignored.

5. Links to existing API doc (optional)

6. Category

Analytics (Glue)

tdjuren commented 4 years ago

This can be extended to AWS::Glue::Crawler which also behaves in the same way.

The tagging structure is also different in Glue resources than it is in almost every other type of cloud formation template. Glue resources take in a JSON string, where all other resources take a list of tags.

Ricapar commented 4 years ago

I wasted so much time on this yesterday until I figured out what was actually happening. As it behaves right now, you update the tags in your CloudFormation, it tells you the resource is "Update in Progress", then "Update Complete"... but nothing has changed.

I had to change stuff to force a resource recreation for the tags to actually go in. I ended up deleting the stacks and recreating them just to get everything tagged properly after the fact.

I'll second @tdjuren's comment from above. While the Key: Value format is so much easier to type... it's entirely inconsistent with how tagging is done on nearly every single other CloudFormation resource. What happened here?

mgmarino commented 3 years ago

Is there any update on this? This issue is over a year old and I've seen no movement of this to the roadmap.

benMain commented 3 years ago

It's 2021 guys. Seriously, I hate having to change job properties just to force an update to Tags. Our IT management uses a growing series of tags to track costs (of which Glue is a significant percentage). I love your CF just force the Glue team to fix their SDK!

kimberlyamandalu commented 3 years ago

I am having similar issues. First, Glue seems to only be inheriting the tag key "Name" from CloudFormation stack. It ignores the rest of the tags. I have added tags to the Glue resource in my CFN template and those don't seem to get picked up either.

Is this a bug? Expected? Pls let us all know or kindly update the documentation. Thanks!

mgmarino commented 3 years ago

I am having similar issues. First, Glue seems to only be inheriting the tag key "Name" from CloudFormation stack. It ignores the rest of the tags.

My understanding is that the reason stack tags aren't propagated is because the formatting of the tag property on the Glue Resources does not correspond to the "standard" CF tag, which is what some posters here have (rightfully, IMHO) complained about.

I have added tags to the Glue resource in my CFN template and those don't seem to get picked up either.

This is the issue here. The workaround is to trigger a new resource generation (e.g. change the Name property).

kimberlyamandalu commented 3 years ago

@mgmarino Thanks for the explanation. That was helpful.

So it seems like there are 2 issues:

CF tags are not propagated because of formatting differences with Glue resource tags
Glue resource tags cannot be updated via cfn template unless resource is recreated

Do you know if AWS team is actively looking into this? thanks!

asyschikov commented 3 years ago

My colleague and I created a reusable Custom Resource (https://github.com/aws-samples/aws-glue-tagger-sar/) that addresses points raised in this issue and some more:

Automatic propagation of stack tags to AWS Glue resources.
Updating tags.
Having tagging syntax consistent with other AWS CloudFormation resources.

Usage example:

GlueTaggerTestStack:
  Type: 'Custom::GlueTaggerTestStack'
  DependsOn:
    - 'MyJob1'
    - 'AWSGlueTagger'
  Properties:
    ServiceToken: !ImportValue GlueTaggerFunction
    ResourceArn:
      - !Sub 'arn:aws:glue:${AWS::Region}:${AWS::AccountId}:job/${MyJob1}'
    Tags:
      - Key: "cost_id"
        Value: "0000"
      - Key: "owner"
        Value: "myorg"
      - Key: "project"
        Value: "myproject"

@kimberlyamandalu @benMain @luiseduardocolon @Ricapar @mgmarino

aws-cloudformation / cloudformation-coverage-roadmap