search

Rungutan / sentry-fargate-cf-stack

AWS CloudFormation template to launch a highly-available Sentry 20 stack through ECS Fargate at the minimum cost possible
Apache License 2.0
60 stars 16 forks source link

Update to 1.11.1 fails with Vendor response doesn't contain KafkaBroker1..3 key in object #33

Closed nodomain closed 3 years ago

nodomain commented 3 years ago

On my staging environment everything went fine yesterday. On my production one it fails with these errors: The only difference besides the instance type is that I manually extended the EBS storage on production already.

Please help - I already found that it seems to have something to do with the result of the lambda because it will output the keys only on "Create" but I am stuck. https://github.com/Rungutan/sentry-fargate-cf-stack/blob/main/cloudformation-template.yaml#L1828

CustomResource attribute error: Vendor response doesn't contain KafkaBroker3 key in object arn:aws:cloudformation:eu-central-1:722716701248:stack/sentry-prod/6e810c90-5cb1-11eb-8bbc-0207ca607726|KafkaGetBrokersLambdaResult|6c61345e-58f6-4b10-b64d-28e849cd4cdd in S3 bucket cloudformation-custom-resource-storage-eucentral1
image

Thanks

nodomain commented 3 years ago

I tried to further analyze the root cause of this by renaming the Kafka resource in the CF template to trigger the creation of a new Kafka cluster:

ClusterName:
        Fn::Sub: ${AWS::StackName}-Kafka

Unfortunately this results in the same errors and the CF stack is rolled back.

The result indeed does not contain the requested keys.

image
mariusmitrofan commented 3 years ago

The https://github.com/Rungutan/sentry-fargate-cf-stack/releases/tag/1.11.2 release patch should fix this. I think it should work with your existing changed-outside-of-cloudformation Kafka cluster, but not 100% sure about it...

@nodomain

nodomain commented 3 years ago

Fixed it by temporarily renaming KafkaGetBrokersLambdaResult to KafkaGetEndpointsLambdaResult in the latest version and applying it. This seems to have triggered the needed changes that were prerequisite to a successful upgrade. After the first run, I re-applied the original CF template in the latest version.