Snuba consumes high cpu

samir34ml commented 3 months ago

Self-Hosted Version

24.3.0

CPU Architecture

x86_x64

Docker Version

25.0.4

Docker Compose Version

2.24.7

Steps to Reproduce

I have installed sentry self hosted on an aws EC2. The only different that I have made is the following. In the process of creating the EC2, I attached a new HDD disk to the server and mounted it. I used HDD disk to mount 2 volumes sentry-kafka and sentry-postgres. I edited the following file install/create-docker-volumes.sh (attached to the issue). create-docker-volumes.txt So, now I can mount all the docker volumes data and storage of both sentry-kafka and sentry-postgres to the HDD disk.

The reason why I did this is to save some money and use huge disk of HDD instead of SSD because these volumes consume huge amount of disk space.

Expected Result

normal behavior and events receiving and issue monitoring.

Actual Result

no events no issues high cpu utilization even after uninstalling sentry completely and installing it again.

It worked for a while around day or day and half, then sentry stops to report. No new events or issues or transactions. I had to reset it and re-install it from the beginning but now even before adding any project cpu is very high above 90%.

I tried to add some logs from the docker compose logs command output. logs.txt

So any idea regrading what to do next?

Thanks,

Event ID

No response

hubertdeng123 commented 3 months ago

Thanks for writing in. Could you answer these questions before proceeding?

What is your even throughput into your instance?
How much CPU are you using?

samir34ml commented 3 months ago

Hello @hubertdeng123

for question 1, I hope the attached screenshot answers your question.

And for question 2, I am using t3a.xlarge as EC2 from aws, which has 4 vCPU and 16 GB of memory.

Thanks,

csvan commented 3 months ago

As somebody who also deploys Sentry on AWS, I would strongly advice against using t3a instances. Their baseline performance is too low, and the burst performance does not scale well with how Sentry operates.

Look into using m7i instances, at least, or m7i-flex if minimising costs is a concern.

chris-laack commented 3 months ago

we use r6a.large instances in the aws. They work very well for us.

levani commented 3 months ago

Having the same issue on digitalocean droplet after upgrading to 24.3.0. CPU is constantly at 100%

azaslavsky commented 3 months ago

Can you see if the solution suggested here helps in this case as well?

getsantry[bot] commented 2 months ago

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

rihards-simanovics commented 2 months ago

hey just updated to 24.4.1, my CPU is constantly pegged to 100%, snuba seems to be the cause.

azaslavsky commented 2 months ago

Can you try the advice posted here?

rihards-simanovics commented 2 months ago

Can you try the advice posted here?

hmm actually runs worse without the rust, can see an extra 20% on top for the CPU utilisation from both sentry and arrouns 5% for snuba. have run the docker compose logs -f and it does look like everything is running as expected other than periodical tracebacks from sentry-self-hosted-post-process-forwarder-errors-1, sentry-self-hosted-metrics-consumer-1 and sentry-self-hosted-generic-metrics-consumer-1, in case it helps here are the specific logs:

sentry-self-hosted-post-process-forwarder-errors-1 traces

``` sentry-self-hosted-post-process-forwarder-errors-1 | Traceback (most recent call last): sentry-self-hosted-post-process-forwarder-errors-1 | File "/usr/local/bin/sentry", line 8, in sentry-self-hosted-post-process-forwarder-errors-1 | sys.exit(main()) sentry-self-hosted-post-process-forwarder-errors-1 | File "/usr/local/lib/python3.8/site-packages/sentry/runner/__init__.py", line 193, in main sentry-self-hosted-post-process-forwarder-errors-1 | func(**kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__ sentry-self-hosted-post-process-forwarder-errors-1 | return self.main(*args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main sentry-self-hosted-post-process-forwarder-errors-1 | rv = self.invoke(ctx) sentry-self-hosted-post-process-forwarder-errors-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | return _process_result(sub_ctx.command.invoke(sub_ctx)) sentry-self-hosted-post-process-forwarder-errors-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | return _process_result(sub_ctx.command.invoke(sub_ctx)) sentry-self-hosted-post-process-forwarder-errors-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | return ctx.invoke(self.callback, **ctx.params) sentry-self-hosted-post-process-forwarder-errors-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | return __callback(*args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func sentry-self-hosted-post-process-forwarder-errors-1 | return f(get_current_context(), *args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 69, in inner sentry-self-hosted-post-process-forwarder-errors-1 | return ctx.invoke(f, *args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | return __callback(*args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func sentry-self-hosted-post-process-forwarder-errors-1 | return f(get_current_context(), *args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner sentry-self-hosted-post-process-forwarder-errors-1 | return ctx.invoke(f, *args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | return __callback(*args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/run.py", line 730, in basic_consumer sentry-self-hosted-post-process-forwarder-errors-1 | run_processor_with_signals(processor) sentry-self-hosted-post-process-forwarder-errors-1 | File "/usr/local/lib/python3.8/site-packages/sentry/utils/kafka.py", line 13, in run_processor_with_signals sentry-self-hosted-post-process-forwarder-errors-1 | processor.run() sentry-self-hosted-post-process-forwarder-errors-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 288, in run sentry-self-hosted-post-process-forwarder-errors-1 | self._run_once() sentry-self-hosted-post-process-forwarder-errors-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 355, in _run_once sentry-self-hosted-post-process-forwarder-errors-1 | self.__message = self.__consumer.poll(timeout=1.0) sentry-self-hosted-post-process-forwarder-errors-1 | File "/usr/local/lib/python3.8/site-packages/sentry/consumers/synchronized.py", line 245, in poll sentry-self-hosted-post-process-forwarder-errors-1 | message = self.__consumer.poll(timeout) sentry-self-hosted-post-process-forwarder-errors-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 414, in poll sentry-self-hosted-post-process-forwarder-errors-1 | raise OffsetOutOfRange(str(error)) sentry-self-hosted-post-process-forwarder-errors-1 | arroyo.errors.OffsetOutOfRange: KafkaError{code=_AUTO_OFFSET_RESET,val=-140,str="fetch failed due to requested offset not available on the broker: Broker: Offset out of range (broker 1001)"} ```

sentry-self-hosted-metrics-consumer-1 traces

```log sentry-self-hosted-metrics-consumer-1 | 07:11:58 [ERROR] arroyo.processing.processor: Caught exception, shutting down... sentry-self-hosted-metrics-consumer-1 | 07:11:58 [INFO] arroyo.processing.processor: Closing ... sentry-self-hosted-metrics-consumer-1 | 07:11:58 [INFO] arroyo.processing.processor: Processor terminated sentry-self-hosted-metrics-consumer-1 | Traceback (most recent call last): sentry-self-hosted-metrics-consumer-1 | File "/usr/local/bin/sentry", line 8, in sentry-self-hosted-metrics-consumer-1 | sys.exit(main()) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/sentry/runner/__init__.py", line 193, in main sentry-self-hosted-metrics-consumer-1 | func(**kwargs) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__ sentry-self-hosted-metrics-consumer-1 | return self.main(*args, **kwargs) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main sentry-self-hosted-metrics-consumer-1 | rv = self.invoke(ctx) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke sentry-self-hosted-metrics-consumer-1 | return _process_result(sub_ctx.command.invoke(sub_ctx)) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke sentry-self-hosted-metrics-consumer-1 | return _process_result(sub_ctx.command.invoke(sub_ctx)) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke sentry-self-hosted-metrics-consumer-1 | return ctx.invoke(self.callback, **ctx.params) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke sentry-self-hosted-metrics-consumer-1 | return __callback(*args, **kwargs) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func sentry-self-hosted-metrics-consumer-1 | return f(get_current_context(), *args, **kwargs) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 69, in inner sentry-self-hosted-metrics-consumer-1 | return ctx.invoke(f, *args, **kwargs) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke sentry-self-hosted-metrics-consumer-1 | return __callback(*args, **kwargs) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func sentry-self-hosted-metrics-consumer-1 | return f(get_current_context(), *args, **kwargs) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner sentry-self-hosted-metrics-consumer-1 | return ctx.invoke(f, *args, **kwargs) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke sentry-self-hosted-metrics-consumer-1 | return __callback(*args, **kwargs) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/run.py", line 730, in basic_consumer sentry-self-hosted-metrics-consumer-1 | run_processor_with_signals(processor) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/sentry/utils/kafka.py", line 13, in run_processor_with_signals sentry-self-hosted-metrics-consumer-1 | processor.run() sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 288, in run sentry-self-hosted-metrics-consumer-1 | self._run_once() sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 355, in _run_once sentry-self-hosted-metrics-consumer-1 | self.__message = self.__consumer.poll(timeout=1.0) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 416, in poll sentry-self-hosted-metrics-consumer-1 | raise ConsumerError(str(error)) sentry-self-hosted-metrics-consumer-1 | arroyo.errors.ConsumerError: KafkaError{code=UNKNOWN_TOPIC_OR_PART,val=3,str="Subscribed topic not available: ingest-metrics: Broker: Unknown topic or partition"} sentry-self-hosted-metrics-consumer-1 | Updating certificates in /etc/ssl/certs... sentry-self-hosted-metrics-consumer-1 | 0 added, 0 removed; done. sentry-self-hosted-metrics-consumer-1 | Running hooks in /etc/ca-certificates/update.d... sentry-self-hosted-metrics-consumer-1 | done. sentry-self-hosted-metrics-consumer-1 | Traceback (most recent call last): sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 288, in run sentry-self-hosted-metrics-consumer-1 | self._run_once() sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 355, in _run_once sentry-self-hosted-metrics-consumer-1 | self.__message = self.__consumer.poll(timeout=1.0) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 416, in poll sentry-self-hosted-metrics-consumer-1 | raise ConsumerError(str(error)) sentry-self-hosted-metrics-consumer-1 | arroyo.errors.ConsumerError: KafkaError{code=UNKNOWN_TOPIC_OR_PART,val=3,str="Subscribed topic not available: ingest-metrics: Broker: Unknown topic or partition"} sentry-self-hosted-metrics-consumer-1 | 07:12:36 [ERROR] arroyo.processing.processor: Caught exception, shutting down... sentry-self-hosted-metrics-consumer-1 | 07:12:36 [INFO] arroyo.processing.processor: Closing ... sentry-self-hosted-metrics-consumer-1 | 07:12:36 [INFO] arroyo.processing.processor: Processor terminated sentry-self-hosted-metrics-consumer-1 | Traceback (most recent call last): sentry-self-hosted-metrics-consumer-1 | File "/usr/local/bin/sentry", line 8, in sentry-self-hosted-metrics-consumer-1 | sys.exit(main()) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/sentry/runner/__init__.py", line 193, in main sentry-self-hosted-metrics-consumer-1 | func(**kwargs) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__ sentry-self-hosted-metrics-consumer-1 | return self.main(*args, **kwargs) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main sentry-self-hosted-metrics-consumer-1 | rv = self.invoke(ctx) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke sentry-self-hosted-metrics-consumer-1 | return _process_result(sub_ctx.command.invoke(sub_ctx)) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke sentry-self-hosted-metrics-consumer-1 | return _process_result(sub_ctx.command.invoke(sub_ctx)) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke sentry-self-hosted-metrics-consumer-1 | return ctx.invoke(self.callback, **ctx.params) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke sentry-self-hosted-metrics-consumer-1 | return __callback(*args, **kwargs) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func sentry-self-hosted-metrics-consumer-1 | return f(get_current_context(), *args, **kwargs) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 69, in inner sentry-self-hosted-metrics-consumer-1 | return ctx.invoke(f, *args, **kwargs) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke sentry-self-hosted-metrics-consumer-1 | return __callback(*args, **kwargs) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func sentry-self-hosted-metrics-consumer-1 | return f(get_current_context(), *args, **kwargs) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner sentry-self-hosted-metrics-consumer-1 | return ctx.invoke(f, *args, **kwargs) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke sentry-self-hosted-metrics-consumer-1 | return __callback(*args, **kwargs) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/run.py", line 730, in basic_consumer sentry-self-hosted-metrics-consumer-1 | run_processor_with_signals(processor) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/sentry/utils/kafka.py", line 13, in run_processor_with_signals sentry-self-hosted-metrics-consumer-1 | processor.run() sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 288, in run sentry-self-hosted-metrics-consumer-1 | self._run_once() sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 355, in _run_once sentry-self-hosted-metrics-consumer-1 | self.__message = self.__consumer.poll(timeout=1.0) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 416, in poll sentry-self-hosted-metrics-consumer-1 | raise ConsumerError(str(error)) sentry-self-hosted-metrics-consumer-1 | arroyo.errors.ConsumerError: KafkaError{code=UNKNOWN_TOPIC_OR_PART,val=3,str="Subscribed topic not available: ingest-metrics: Broker: Unknown topic or partition"} sentry-self-hosted-metrics-consumer-1 | Updating certificates in /etc/ssl/certs... sentry-self-hosted-metrics-consumer-1 | 0 added, 0 removed; done. sentry-self-hosted-metrics-consumer-1 | Running hooks in /etc/ca-certificates/update.d... sentry-self-hosted-metrics-consumer-1 | done. sentry-self-hosted-metrics-consumer-1 | Traceback (most recent call last): sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 288, in run sentry-self-hosted-metrics-consumer-1 | self._run_once() sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 355, in _run_once sentry-self-hosted-metrics-consumer-1 | self.__message = self.__consumer.poll(timeout=1.0) sentry-self-hosted-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 416, in poll sentry-self-hosted-metrics-consumer-1 | raise ConsumerError(str(error)) sentry-self-hosted-metrics-consumer-1 | arroyo.errors.ConsumerError: KafkaError{code=UNKNOWN_TOPIC_OR_PART,val=3,str="Subscribed topic not available: ingest-metrics: Broker: Unknown topic or partition"} sentry-self-hosted-metrics-consumer-1 | 07:13:14 [ERROR] arroyo.processing.processor: Caught exception, shutting down... ```

sentry-self-hosted-generic-metrics-consumer-1 traces

```log sentry-self-hosted-generic-metrics-consumer-1 | 07:59:24 [ERROR] arroyo.processing.processor: Caught exception, shutting down... sentry-self-hosted-generic-metrics-consumer-1 | 07:59:24 [INFO] arroyo.processing.processor: Closing ... sentry-self-hosted-generic-metrics-consumer-1 | 07:59:24 [INFO] arroyo.processing.processor: Processor terminated sentry-self-hosted-generic-metrics-consumer-1 | Traceback (most recent call last): sentry-self-hosted-generic-metrics-consumer-1 | File "/usr/local/bin/sentry", line 8, in sentry-self-hosted-generic-metrics-consumer-1 | sys.exit(main()) sentry-self-hosted-generic-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/sentry/runner/__init__.py", line 193, in main sentry-self-hosted-generic-metrics-consumer-1 | func(**kwargs) sentry-self-hosted-generic-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__ sentry-self-hosted-generic-metrics-consumer-1 | return self.main(*args, **kwargs) sentry-self-hosted-generic-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main sentry-self-hosted-generic-metrics-consumer-1 | rv = self.invoke(ctx) sentry-self-hosted-generic-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke sentry-self-hosted-generic-metrics-consumer-1 | return _process_result(sub_ctx.command.invoke(sub_ctx)) sentry-self-hosted-generic-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke sentry-self-hosted-generic-metrics-consumer-1 | return _process_result(sub_ctx.command.invoke(sub_ctx)) sentry-self-hosted-generic-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke sentry-self-hosted-generic-metrics-consumer-1 | return ctx.invoke(self.callback, **ctx.params) sentry-self-hosted-generic-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke sentry-self-hosted-generic-metrics-consumer-1 | return __callback(*args, **kwargs) sentry-self-hosted-generic-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func sentry-self-hosted-generic-metrics-consumer-1 | return f(get_current_context(), *args, **kwargs) sentry-self-hosted-generic-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 69, in inner sentry-self-hosted-generic-metrics-consumer-1 | return ctx.invoke(f, *args, **kwargs) sentry-self-hosted-generic-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke sentry-self-hosted-generic-metrics-consumer-1 | return __callback(*args, **kwargs) sentry-self-hosted-generic-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func sentry-self-hosted-generic-metrics-consumer-1 | return f(get_current_context(), *args, **kwargs) sentry-self-hosted-generic-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner sentry-self-hosted-generic-metrics-consumer-1 | return ctx.invoke(f, *args, **kwargs) sentry-self-hosted-generic-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke sentry-self-hosted-generic-metrics-consumer-1 | return __callback(*args, **kwargs) sentry-self-hosted-generic-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/run.py", line 730, in basic_consumer sentry-self-hosted-generic-metrics-consumer-1 | run_processor_with_signals(processor) sentry-self-hosted-generic-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/sentry/utils/kafka.py", line 13, in run_processor_with_signals sentry-self-hosted-generic-metrics-consumer-1 | processor.run() sentry-self-hosted-generic-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 288, in run sentry-self-hosted-generic-metrics-consumer-1 | self._run_once() sentry-self-hosted-generic-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 355, in _run_once sentry-self-hosted-generic-metrics-consumer-1 | self.__message = self.__consumer.poll(timeout=1.0) sentry-self-hosted-generic-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 416, in poll sentry-self-hosted-generic-metrics-consumer-1 | raise ConsumerError(str(error)) sentry-self-hosted-generic-metrics-consumer-1 | arroyo.errors.ConsumerError: KafkaError{code=UNKNOWN_TOPIC_OR_PART,val=3,str="Subscribed topic not available: ingest-performance-metrics: Broker: Unknown topic or partition"} sentry-self-hosted-generic-metrics-consumer-1 | Updating certificates in /etc/ssl/certs... sentry-self-hosted-generic-metrics-consumer-1 | 0 added, 0 removed; done. sentry-self-hosted-generic-metrics-consumer-1 | Running hooks in /etc/ca-certificates/update.d... sentry-self-hosted-generic-metrics-consumer-1 | done. sentry-self-hosted-generic-metrics-consumer-1 | Traceback (most recent call last): sentry-self-hosted-generic-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 288, in run sentry-self-hosted-generic-metrics-consumer-1 | self._run_once() sentry-self-hosted-generic-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 355, in _run_once sentry-self-hosted-generic-metrics-consumer-1 | self.__message = self.__consumer.poll(timeout=1.0) sentry-self-hosted-generic-metrics-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 416, in poll sentry-self-hosted-generic-metrics-consumer-1 | raise ConsumerError(str(error)) sentry-self-hosted-generic-metrics-consumer-1 | arroyo.errors.ConsumerError: KafkaError{code=UNKNOWN_TOPIC_OR_PART,val=3,str="Subscribed topic not available: ingest-performance-metrics: Broker: Unknown topic or partition"} sentry-self-hosted-generic-metrics-consumer-1 | 07:59:53 [ERROR] arroyo.processing.processor: Caught exception, shutting down... ```

azaslavsky commented 2 months ago

Moved this over to the snuba repo for more domain-specific triage.

FrankHeijden commented 2 months ago

getsentry / snuba