Resource does not support custom properties

[YuriGal]

I've seen in nuke logs multiple times errors like

time="2024-07-29T15:48:21Z" level=warning component=libnuke error="*resources.CloudWatchDashboard does not support custom properties" handler=Filter type=CloudWatchDashboard

time="2024-07-29T15:48:21Z" level=warning component=libnuke error="*resources.CloudWatchEventsBus does not support custom properties" handler=Filter type=CloudWatchEventsBuses

time="2024-07-29T15:48:21Z" level=warning component=libnuke error="*resources.FirehoseDeliveryStream does not support custom properties" handler=Filte type=FirehoseDeliveryStream

time="2024-07-29T15:48:21Z" level=warning component=libnuke error="*resources.SESIdentity does not support custom properties" handler=Filter type=SESIdentity

but other times

CoudWatchDashboard - would remove
...

Are those different type of resources? Is it skipping some, unable to get properties? Thanks!

[YuriGal]

Not sure if related or not, noticed another error:

time="2024-07-29T16:50:43Z" level=error msg="Listing CloudWatchLogsLogGroup failed:\n    ThrottlingException: Rate exceeded" error="ThrottlingException: Rate exceeded"

[YuriGal]

This particular account has A LOT of stale resources - including cloudwatch dashboards and log groups. Looks like the nuke is missing/skipping many of them.

[YuriGal]

Seeing more rate exceeded errors

time="2024-07-29T19:15:21Z" level=error msg="Listing CloudWatchAlarm failed:\n    Throttling: Rate exceeded\n    \tstatus code: 400, request id: 68ec456e-b6a3-4aed-9ea2-f41440c8145b" error="Throttling: Rate exceeded\n\tstatus code: 400, request id: 68ec456e-b6a3-4aed-9ea2-f41440c8145b"

[ekristen]

You've got a lot of things happening in this issue here, I'll try to address them in one response.

What does your configuration look like?

does not support custom properties

This simply means that the resource does not have any properties defined, the one way you can filter it is by the default legacy string name. You can't filter by tag: or by any other property as they don't have properties.

This particular account has A LOT of stale resources - including cloudwatch dashboards and log groups. Looks like the nuke is missing/skipping many of them.

I assume you mean it's not deleting them? Either they are matching a filter OR it's possible that pagination support isn't in place. I'd need to know how many of each resource and your configuration.

Rate limiting

This means you are just hitting the API too much with the same account. It could be that you just have a lot of resource are a lot more endpoints are being called or you are re-running nuke to frequently and you have a lot of resources. For testing and tailoring of your config, I would advise you use --include on the cli to limit the resources you are targeting. You can also do that in the config if you'd prefer.

[YuriGal]

Apologies, my bad, I should've probably opened separate issues, but I just noted the errors as I encountered them. My config is pretty straightforward: nuke-config.yml.zip, I filter lot of files by tags, and some resources were excluded by type (some of them because they currently cannot be excluded by type).

This simply means that the resource does not have any properties

What was confusing to me is some messages said type=CloudWatchDashboard does not support custom properties, while the other said "CloudWatchDashboard would be removed." Is it to be expected?

This means you are just hitting the API too much with the same account

What is strange is the only place where I encountered rate limiting is when nuke tries to list particular resources, e.g. "Listing CloudWatchLogsLogGroup failed" no matter what other resources are included in the config.

[ekristen]

Since everything is nukeable by default, filters are used to exclude resources.

Since you are using __global__ with filters like property: "tag:github-branch-name" it attempts to lookup the property on every single resource of every single type, in this case CloudWatchDashboard doesn't have properties defined, it's a warning, but lack of properties does not mean it gets "filtered", although another request has been made for this feature.

So if you want a CloudWatchDashboard to be filtered (i.e. not removed) then you'd need to define the following, (this is not complete yaml, just a subsection of a config)

filters:
  CloudWatchDashboard:
    - NameOfDashboard
    - type: regex
      value: "NameOfDashboard.*"  

All properties have a "Legacy" string name, usually the human name of a resource, it varies by resource unfortunately, sometimes it's an ARN, it's something I'm working on standardizing.

Rate limiting ... If it's only happening on CloudWatchLogsLogGroup, can you find out how many are in our account? It might be that we are making too many calls. If you run with --include CloudWatchLogsLogGroup does it still hit rate limiting?

[YuriGal]

This account has 13317 log groups. And yes, the "ThrottlingException: Rate exceeded" consistently happen during "Listing CloudWatchLogsLogGroup" even if I target log groups exclusively.

[ekristen]

Yeah 13,000 is a ton.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/cloudwatch_limits_cwl.html

So 10/second but you can request an increase. It's possible that we might be able to add a throttling control to the resource but I couldn't do that for another couple of days/weeks but that would need to be tested pretty extensively.

[YuriGal]

Yeah, that account used to be a playground, and a lot of garbage accumulated there. We're ok for now, I can do a manual-scripted cleanup, but it would be great if the nuke could natively support this. Also curious: it looks like it gets rate-limited at just describing log groups - which is, if I am not mistaken, a single request? or something else is going on?

[ekristen]

It has pagination support. Can only request 50 at a time. So it's roughly 260 calls. I'll see about adding in limits if possible. You are definitely an edge case though.

[ekristen]

I'm going to close this. I opened a new issue around the rate limiting, but the tool is behaving as currently expected with log messages around supported properties.