aws - ebs - InvalidVolume.NotFound when calling the CreateTags operation

[nitrocode]

Describe the bug

The following exception is thrown

An error occurred (InvalidVolume.NotFound) when calling the CreateTags operation

To Reproduce Steps to reproduce the behavior:

Expected behavior A clear and concise description of what you expected to happen.

Background (please complete the following information):

• OS: lambda
• Python Version: 3.8.1
• Custodian Version: 0.9.4.0
• Tool Version: N/A
• Cloud Provider: aws
• Policy:

  
  vars:
  mode: &mode
  type: periodic
  schedule: "rate(7 days)"
  role: arn:aws:iam::{account_id}:role/cloud-custodian
  tags:
    application: cloud-custodian
    team: sre
    env: production
  execution-options:
    output_dir: s3://output-bucket/cloud-custodian/output

policies:

• name: ebs-mark-unattached-deletion resource: ebs description: | Mark any unattached EBS volumes for deletion in 30 days. Volumes set to not delete on instance termination do have valid use cases as data drives, but 99% of the time they appear to be just garbage creation. mode: <<: *mode filters:
  • "tag:c7n_ignore": absent
  • "tag:c7n_status": absent
  • Attachments: []
  • type: mark-for-op tag: c7n_status msg: "Unused. If this resource should stay, add the tag c7n_ignore with a value of why it should exist." op: delete days: 30

  • Traceback:

[ERROR] ClientError: An error occurred (InvalidVolume.NotFound) when calling the CreateTags operation: The volume 'vol-05eeb39ae0ab38aa5' does not exist.
Traceback (most recent call last):
  File "/var/task/custodian_policy.py", line 4, in run
    return handler.dispatch_event(event, context)
  File "/var/task/c7n/handler.py", line 176, in dispatch_event
    p.push(event, context)
  File "/var/task/c7n/policy.py", line 1135, in push
    return mode.run(event, lambda_ctx)
  File "/var/task/c7n/policy.py", line 532, in run
    return PullMode.run(self)
  File "/var/task/c7n/policy.py", line 323, in run
    results = a.process(resources)
  File "/var/task/c7n/tags.py", line 682, in process
    _common_tag_processer(
  File "/var/task/c7n/tags.py", line 133, in _common_tag_processer
    raise error
  File "/var/lang/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/var/task/c7n/tags.py", line 688, in process_resource_set
    tagger.process_resource_set(client, resource_set, tags)
  File "/var/task/c7n/tags.py", line 430, in process_resource_set
    self.manager.retry(
  File "/var/task/c7n/utils.py", line 373, in _retry
    return func(*args, **kw)
  File "/var/runtime/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 626, in _make_api_call
    raise error_class(parsed_response, operation_name)

• custodian version --debug output

Additional context The workaround for now has been to pin down cloudcustodian to 0.9.1.0.

https://github.com/cloud-custodian/cloud-custodian/issues/5375 https://github.com/cloud-custodian/cloud-custodian/issues/3531

[kapilt]

do you have the full traceback?

[nitrocode]

Yes. I modified the ticket to add it.

[ajkerrigan]

This came up in Gitter today also, courtesy of @cpitstick-argo :

We run a lot of spot instances that wink in and out constantly. So sometimes, Cloud Custodian will attempt to tag a resource that doesn't exist, because in between the time it discovered the resource and tried to tag it, the tagging operation fails, so we get a lot of these types of errors: An error occurred (InvalidVolume.NotFound) when calling the CreateTags operation: The volume 'vol-XXXXXXXXXXXXX' does not exist.

With a similar sample policy:

- name: ebs-tag-unattached-volume
  resource: ebs
  description: |
    EBS volume is unattached
  filters:
    - Attachments: []
  actions:
    - type: tag
      key: "cloud-custodian-flagged"
      value: "ebs-volume-is-unattached"
    - type: mark-for-op
      tag: cloud-custodian-action
      op: delete
      hours: 0

[nitrocode]

I spoke with @ajkerrigan and he said it may be related to the last 2 lines in this block

https://github.com/cloud-custodian/cloud-custodian/blob/30bbbaca6f1ececb2a3aee8d695f8bf793d51a29/c7n/tags.py#L101-L119

https://github.com/cloud-custodian/cloud-custodian/blob/30bbbaca6f1ececb2a3aee8d695f8bf793d51a29/c7n/tags.py#L118-L119