Updating WAF stack with 2.2.0 fails

aws-solutions / aws-waf-security-automations

This solution automatically deploys a single web access control list (web ACL) with a set of AWS WAF rules designed to filter common web-based attacks.

https://aws.amazon.com/solutions/aws-waf-security-automations

Apache License 2.0

838 stars 360 forks source link

Updating WAF stack with 2.2.0 fails #68

Closed rectalogic closed 5 years ago

rectalogic commented 5 years ago

Updating WAF stack v2.1 with v2.2.0 fails and rolls back:

21 Dec 2018 19:34:54    play-ids-WAFStack-1B0FJLMDJIWAP UPDATE_ROLLBACK_IN_PROGRESS The following resource(s) failed to update: [CreateUniqueID].
21 Dec 2018 19:34:53    CreateUniqueID  UPDATE_FAILED   Modifying service token is not allowed.

hvital commented 5 years ago

Hi @rectalogic

In this case, I would recommend create a separated new stack and associate you web app endpoit to it. Just be aware to delete the existing Access Log bucket event if you plan to reuse the same S3 location for logs.

hvital commented 5 years ago

Did it work?

rectalogic commented 5 years ago

Yes, I tried this on a test stack. We use the WAF stack as a nested stack in our main stack, so I changed the nested stack name so that CF destroyed the old one and recreated a new one. I did not delete the S3 bucket.

Will this result in an outage for our users though? When the old WebACL is detached and destroyed and the new one is attached to our ALBs, will in-flight HTTP connections be disrupted?

hvital commented 5 years ago

Hi @rectalogic

You shouldn't have any problem to associate the new webACL (https://amzn.to/2GQZVxt). However, as you've mentioned that this might be a prod/critical endpoint, I recommend:

Use a blue-green deployment approach by creating a new stack and just switch your endpoint to use the new webACL. This way you'll have a easy way to rollback if needed.
Change the new webACL to test-mode for a while just to triple check if it is working as expected. More info: https://amzn.to/2s3nglj

Regards