search

Rungutan / sentry-fargate-cf-stack

AWS CloudFormation template to launch a highly-available Sentry 20 stack through ECS Fargate at the minimum cost possible
Apache License 2.0
60 stars 16 forks source link

SnubaBootstrap is not run when exchanging the Kafka cluster #36

Closed nodomain closed 3 years ago

nodomain commented 3 years ago

I wanted to get rid of my storage settings that I changed manually before https://github.com/Rungutan/sentry-fargate-cf-stack/issues/25 was implemented, see also https://github.com/Rungutan/sentry-fargate-cf-stack/issues/33#issuecomment-770457244). So I renamed the cluster in the template and updated my stack. It took a long time before I got suspicious and looked at the logs.

This pointed me to further investigate the issue:

snuba.utils.streams.backends.abstract.ConsumerError: KafkaError{code=UNKNOWN_TOPIC_OR_PART,val=3,str="Subscribed topic not available: event-replacements: Broker: Unknown topic or partition"}

I suspected that the "bootstrap" docker was not executed. I think the reason is this line: https://github.com/Rungutan/sentry-fargate-cf-stack/blob/fccdf5ce69470775f3511049fbbf222ec817584b/cloudformation-template.yaml#L5487

I started the task manually and the stack finally upgraded with no errors.

The logs show that the missing topics were created:

2021-01-31T22:37:45.586+01:00 | 2021-01-31 21:37:45,585 Topic errors-replacements created
-- | --
  | 2021-01-31T22:37:45.586+01:00 | 2021-01-31 21:37:45,585 Topic event-replacements created

For the next run, I am going to change this line to YES - perhaps this should also be changed in the template, even if it is an edge case.

WDYT?

nodomain commented 3 years ago

Just wanted to confirm that it worked with my proposed change.

mariusmitrofan commented 3 years ago

Fixed with PR https://github.com/Rungutan/sentry-fargate-cf-stack/pull/37