getsentry / self-hosted

Sentry, feature-complete and packaged up for low-volume deployments and proofs-of-concept
https://develop.sentry.dev/self-hosted/
Other
7.57k stars 1.72k forks source link

Performance Alert Not Running Even Condition Is Match #3180

Open Charlieevox opened 3 weeks ago

Charlieevox commented 3 weeks ago

Self-Hosted Version

2.46.0

Docker Version

25.0.3

Docker Compose Version

2.24.6

Steps to Reproduce

Helo, i try to create alert but alert always not triggered.

image

But my alert not triggered, even we lowering metric but still not trigger

image

image

We using internal webhook.

Expected Result

We need metric get triggered, we already try to update to latest update and on docker list, all services is running not stopped

But if check on docker logs there's an error on sentry-self-hosted-snuba-subscription-consumer-events-1 :

2024-07-03 04:34:54,279 Initializing Snuba... 2024-07-03 04:34:56,976 Snuba initialization took 2.697246802970767s 2024-07-03 04:34:57,401 Initializing Snuba... 2024-07-03 04:35:00,036 Snuba initialization took 2.635713377967477s {"module": "builtins", "event": "Checking Clickhouse connections", "severity": "info", "timestamp": "2024-07-03T04:35:00.054755Z"} {"module": "snuba.migrations.connect", "event": "Snuba has only been tested on Clickhouse versions up to 23.3.19.33 (clickhouse:9000 - 23.8.11.29). Higher versions might not be supported.", "severity": "warning", "timestamp": "2024-07-03T04:35:00.062221Z"} 2024-07-03 04:35:03,067 New partitions assigned: {Partition(topic=Topic(name='snuba-commit-log'), index=0): 80546569} 2024-07-03 04:35:04,484 Caught exception, shutting down... Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/arroyo/processing/processor.py", line 322, in run self._run_once() File "/usr/local/lib/python3.11/site-packages/arroyo/processing/processor.py", line 410, in _run_once self.processing_strategy.submit(message) File "/usr/src/snuba/snuba/subscriptions/scheduler_processing_strategy.py", line 252, in submit self.__next_step.submit(message) File "/usr/src/snuba/snuba/subscriptions/combined_scheduler_executor.py", line 275, in submit tasks.extend([task for task in entity_scheduler[tick.partition].find(tick)]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/snuba/snuba/subscriptions/combined_scheduler_executor.py", line 275, in tasks.extend([task for task in entity_scheduler[tick.partition].find(tick)]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/snuba/snuba/subscriptions/scheduler.py", line 402, in find subscriptions = self.get_subscriptions() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/snuba/snuba/subscriptions/scheduler.py", line 375, in __get_subscriptions for uuid, data in self.__store.all() ^^^^^^^^^^^^^^^^^^ File "/usr/src/snuba/snuba/subscriptions/store.py", line 68, in all return [ ^ File "/usr/src/snuba/snuba/subscriptions/store.py", line 69, in (UUID(key.decode("utf-8")), self.codec.decode(val)) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/snuba/snuba/subscriptions/codecs.py", line 35, in decode return SubscriptionData.from_dict(data, self.entity_key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/snuba/snuba/subscriptions/data.py", line 231, in from_dict query=data["query"],


KeyError: 'query'
2024-07-03 04:35:04,491 Closing <snuba.subscriptions.scheduler_consumer.CommitLogTickConsumer object at 0x7f283ba92f10>...

### Actual Result

-

### Event ID

_No response_
hubertdeng123 commented 2 weeks ago

Hmmm, I suspect that data may have been corrupted during an upgrade of some sort. What version of self-hosted Sentry did you upgrade from before getting to 24.6.0?

hubertdeng123 commented 2 weeks ago

Perhaps a similar issue to https://github.com/getsentry/self-hosted/issues/2067?