Bouncer not deleting ipsets

Hello everyone! I'm running a Crowdsec installation for 3 services supposedly fine (I get IP bans in the correct scenarios) until I received an error in one of the bouncer logs stating that it couldn't create more new AWS WAF IPSets. I realized I had 100 existing IPSets and that was a current limit that I'd need to increase.

I have 3 EC2 instances. Each instance runs a different service via docker-compose stack. And in each stack there's a crowdsec and crowdsec-awf-waf-bouncer service running.

All three services share the same AWS WAF ACL (crowdsec-) and each service writes a new Group Rule. Here's the example configuration for the bouncer of the service "myservice":

api_key: redacted-api-key
api_url: "http://127.0.0.1:8080/"
update_frequency: 10s
waf_config:
  - web_acl_name: crowdsec-staging
    fallback_action: ban
    rule_group_name: crowdsec-waf-bouncer-ip-set-myservice
    scope: REGIONAL
    capacity: 300
    region: us-east-1
    ipset_prefix: myservice-crowdsec-ipset-a

From https://docs.crowdsec.net/u/bouncers/aws_waf/ for the ipset_prefix parameter it states: "All ipsets are deleted on shutdown."

And I noticed this is not happening. Everytime the docker-compose stack is restarted new IPSets are created and the old ones remain.

I have RTFM and STFW without results. I have no suspicious information from the logs of crowdsec and crowdsec-awf-waf-bouncer that I can use.

I have tried setting IAM AdministratorAccess policy to the EC2's IAM role in case it was lacking an IAM permissions but it seems not to be the case.

Has anyone detected this issue before? What could I be doing wrong?

Thanks in advance for reading.

Versions

Crowdsec: crowdsecurity/crowdsec:v1.6.2 Bouncer: crowdsecurity/aws-waf-bouncer:v0.1.7

Hello,

We are using this bouncer internally, and this is the first time we have seen this error. I've just tried to stop the bouncer, and the resources are properly deleted from the account:

time="2024-10-31T13:53:25Z" level=info msg="terminating bouncer process"
time="2024-10-31T13:53:25Z" level=info msg="Cleaning up resources" acl=cti_public_api_searchbar-rule region=us-east-1 scope=CLOUDFRONT
time="2024-10-31T13:53:26Z" level=info msg="deleting set crowdsec-ipset--IPV4-ban-33664642-5547-420d-b7fb-5ae02f5e714d" acl=cti_public_api_searchbar-rule component=ipset_manager region=us-east-1 scope=CLOUDFRONT
time="2024-10-31T13:53:26Z" level=info msg="Deleting IPSet crowdsec-ipset--IPV4-ban-33664642-5547-420d-b7fb-5ae02f5e714d" set=crowdsec-ipset--IPV4-ban-33664642-5547-420d-b7fb-5ae02f5e714d
time="2024-10-31T13:53:27Z" level=info msg="deleting set crowdsec-ipset--IPV4-ban-0c57b313-b6d7-44de-bbaf-f717dba6dc63" acl=cti_public_api_searchbar-rule component=ipset_manager region=us-east-1 scope=CLOUDFRONT
time="2024-10-31T13:53:27Z" level=info msg="Deleting IPSet crowdsec-ipset--IPV4-ban-0c57b313-b6d7-44de-bbaf-f717dba6dc63" set=crowdsec-ipset--IPV4-ban-0c57b313-b6d7-44de-bbaf-f717dba6dc63
time="2024-10-31T13:53:30Z" level=info msg="deleting set crowdsec-ipset--IPV4-ban-f5477bf8-d287-40b5-afed-67565b64c0d3" acl=cti_public_api_searchbar-rule component=ipset_manager region=us-east-1 scope=CLOUDFRONT
time="2024-10-31T13:53:30Z" level=info msg="Deleting IPSet crowdsec-ipset--IPV4-ban-f5477bf8-d287-40b5-afed-67565b64c0d3" set=crowdsec-ipset--IPV4-ban-f5477bf8-d287-40b5-afed-67565b64c0d3
time="2024-10-31T13:53:31Z" level=info msg="deleting set crowdsec-ipset--IPV6-ban-07416271-3ddb-4bf1-9764-796ed47c34c4" acl=cti_public_api_searchbar-rule component=ipset_manager region=us-east-1 scope=CLOUDFRONT
time="2024-10-31T13:53:31Z" level=info msg="Deleting IPSet crowdsec-ipset--IPV6-ban-07416271-3ddb-4bf1-9764-796ed47c34c4" set=crowdsec-ipset--IPV6-ban-07416271-3ddb-4bf1-9764-796ed47c34c4
time="2024-10-31T13:53:32Z" level=info msg="Cleaning up resources" acl=hub-cdn-acl region=us-east-1 scope=CLOUDFRONT
time="2024-10-31T13:53:33Z" level=info msg="deleting set hub-cdn-ipset--IPV4-ban-aa688add-0a1b-47dc-b471-85228c28aba8" acl=hub-cdn-acl component=ipset_manager region=us-east-1 scope=CLOUDFRONT
time="2024-10-31T13:53:33Z" level=info msg="Deleting IPSet hub-cdn-ipset--IPV4-ban-aa688add-0a1b-47dc-b471-85228c28aba8" set=hub-cdn-ipset--IPV4-ban-aa688add-0a1b-47dc-b471-85228c28aba8
time="2024-10-31T13:53:35Z" level=info msg="deleting set hub-cdn-ipset--IPV4-ban-02c3b350-0ab2-4d22-9694-67c29e5d3138" acl=hub-cdn-acl component=ipset_manager region=us-east-1 scope=CLOUDFRONT
time="2024-10-31T13:53:35Z" level=info msg="Deleting IPSet hub-cdn-ipset--IPV4-ban-02c3b350-0ab2-4d22-9694-67c29e5d3138" set=hub-cdn-ipset--IPV4-ban-02c3b350-0ab2-4d22-9694-67c29e5d3138
time="2024-10-31T13:53:36Z" level=info msg="deleting set hub-cdn-ipset--IPV4-ban-e1ab12ec-f642-42e0-8803-39ceaf80d23d" acl=hub-cdn-acl component=ipset_manager region=us-east-1 scope=CLOUDFRONT
time="2024-10-31T13:53:36Z" level=info msg="Deleting IPSet hub-cdn-ipset--IPV4-ban-e1ab12ec-f642-42e0-8803-39ceaf80d23d" set=hub-cdn-ipset--IPV4-ban-e1ab12ec-f642-42e0-8803-39ceaf80d23d
time="2024-10-31T13:53:37Z" level=info msg="deleting set hub-cdn-ipset--IPV6-ban-5aebefc2-82cf-4c71-90cf-21be0907a35b" acl=hub-cdn-acl component=ipset_manager region=us-east-1 scope=CLOUDFRONT
time="2024-10-31T13:53:37Z" level=info msg="Deleting IPSet hub-cdn-ipset--IPV6-ban-5aebefc2-82cf-4c71-90cf-21be0907a35b" set=hub-cdn-ipset--IPV6-ban-5aebefc2-82cf-4c71-90cf-21be0907a35b

For reference, we have the following policy attached to the instance (which is admittedly a bit too large if you have other ACLs you don't want the bouncer to have access to):

 {
            "Action": [
                "wafv2:UpdateWebACL",
                "wafv2:UpdateRuleGroup",
                "wafv2:UpdateIPSet",
                "wafv2:TagResource",
                "wafv2:ListWebACLs",
                "wafv2:ListRuleGroups",
                "wafv2:ListIPSets",
                "wafv2:GetWebACL",
                "wafv2:GetRuleGroup",
                "wafv2:GetIPSet",
                "wafv2:DeleteRuleGroup",
                "wafv2:DeleteIPSet",
                "wafv2:CreateRuleGroup",
                "wafv2:CreateIPSet"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:wafv2:us-east-1:XXXX:global/webacl/XXXX/XXXXX",
                "arn:aws:wafv2:us-east-1:XXXX:global/webacl/*/*",
                "arn:aws:wafv2:us-east-1:XXXX:global/rulegroup/*/*",
                "arn:aws:wafv2:us-east-1:XXXX:global/managedruleset/*/*",
                "arn:aws:wafv2:us-east-1:XXXX:*/ipset/*/*"
            ],
            "Sid": "ManageWAF"
        },

Do you see the Deleting IPSet .... in your logs when stopping the bouncer ? How many ACLs is the bouncer configured to manage ? If you have a lot, there might be an issue with the rate limiting on AWS side.

crowdsecurity / cs-aws-waf-bouncer

Bouncer not deleting ipsets #72

Versions