search

getsentry / self-hosted

Sentry, feature-complete and packaged up for low-volume deployments and proofs-of-concept
https://develop.sentry.dev/self-hosted/
Other
7.51k stars 1.71k forks source link

Sentry stop logging after a few hours until a new DSN is generated #3088

Open bintangkevin29 opened 1 month ago

bintangkevin29 commented 1 month ago

Self-Hosted Version

23.3.1

CPU Architecture

x86_64

Docker Version

23.0.2

Docker Compose Version

2.17.2

Steps to Reproduce

Assume a succesfull installation of self-hosted sentry, and event logging works well. Within an hour later, it stopped logging. But whenever I generate and use a new DSN, it works again, but only for it to stop working also within an hour later. If I try using previous DSN, it still would not log.

Expected Result

Logging event works indefinitely

Actual Result

I'm not sure what to attach to identify my problem but one thing I noticed is from docker compose logs, ingest consumer won't fire on when using broken DSN

When it works: ``` sentry-self-hosted-nginx-1 | 2024-05-29T03:49:06.604566923Z 10.0.38.160 - - [29/May/2024:03:49:06 +0000] "POST /api/1/envelope/?sentry_key=1b5ea24aaa79475ca0fc4935be2c9500&sentry_version=7 HTTP/1.0" 200 2 "http://localhost:3000/" "Mozilla/5.0 (iPhone; CPU iPhone OS 16_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Mobile/15E148 Safari/604.1" "-" sentry-self-hosted-nginx-1 | 2024-05-29T03:49:06.621564590Z 10.0.38.161 - - [29/May/2024:03:49:06 +0000] "POST /api/1/envelope/?sentry_key=1b5ea24aaa79475ca0fc4935be2c9500&sentry_version=7 HTTP/1.0" 200 2 "http://localhost:3000/" "Mozilla/5.0 (iPhone; CPU iPhone OS 16_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Mobile/15E148 Safari/604.1" "-" sentry-self-hosted-nginx-1 | 2024-05-29T03:49:06.640194703Z 2024/05/29 03:49:06 [warn] 23#23: *2425 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000463, client: 10.0.38.158, server: , request: "POST /api/1/store/?sentry_key=1b5ea24aaa79475ca0fc4935be2c9500&sentry_version=7 HTTP/1.0", host: "10.0.38.186:9000", referrer: "http://localhost:3000/" sentry-self-hosted-nginx-1 | 2024-05-29T03:49:06.645601628Z 10.0.38.158 - - [29/May/2024:03:49:06 +0000] "POST /api/1/store/?sentry_key=1b5ea24aaa79475ca0fc4935be2c9500&sentry_version=7 HTTP/1.0" 200 41 "http://localhost:3000/" "Mozilla/5.0 (iPhone; CPU iPhone OS 16_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Mobile/15E148 Safari/604.1" "-" sentry-self-hosted-nginx-1 | 2024-05-29T03:49:06.664153144Z 10.0.38.159 - - [29/May/2024:03:49:06 +0000] "POST /api/1/envelope/?sentry_key=1b5ea24aaa79475ca0fc4935be2c9500&sentry_version=7 HTTP/1.0" 200 2 "http://localhost:3000/" "Mozilla/5.0 (iPhone; CPU iPhone OS 16_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Mobile/15E148 Safari/604.1" "-" sentry-self-hosted-nginx-1 | 2024-05-29T03:49:06.688075136Z 2024/05/29 03:49:06 [warn] 23#23: *2429 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000464, client: 10.0.38.160, server: , request: "POST /api/1/store/?sentry_key=1b5ea24aaa79475ca0fc4935be2c9500&sentry_version=7 HTTP/1.0", host: "10.0.38.186:9000", referrer: "http://localhost:3000/" sentry-self-hosted-nginx-1 | 2024-05-29T03:49:06.692642337Z 10.0.38.160 - - [29/May/2024:03:49:06 +0000] "POST /api/1/store/?sentry_key=1b5ea24aaa79475ca0fc4935be2c9500&sentry_version=7 HTTP/1.0" 200 41 "http://localhost:3000/" "Mozilla/5.0 (iPhone; CPU iPhone OS 16_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Mobile/15E148 Safari/604.1" "-" sentry-self-hosted-ingest-consumer-1 | 2024-05-29T03:49:07.701018092Z 03:49:07 [INFO] sentry.batching-kafka-consumer: Flushing 2 items (from {('ingest-events', 0): [1763, 1764]}): forced:False size:False time:True sentry-self-hosted-ingest-consumer-1 | 2024-05-29T03:49:07.733312017Z 03:49:07 [INFO] sentry.batching-kafka-consumer: Worker flush took 32ms sentry-self-hosted-clickhouse-1 | 2024-05-29T03:49:07.824884556Z 2024.05.29 03:49:07.823321 [ 87 ] {d7efba4d-1231-4fce-9e86-dbcb5a895f01} executeQuery: Read 3 rows, 310.00 B in 0.150 sec., 20 rows/sec., 2.02 KiB/sec. sentry-self-hosted-clickhouse-1 | 2024-05-29T03:49:07.829129823Z 2024.05.29 03:49:07.827642 [ 87 ] {} HTTPHandler: Done processing query sentry-self-hosted-clickhouse-1 | 2024-05-29T03:49:08.940022034Z 2024.05.29 03:49:08.939827 [ 2467 ] {ae9010fb-bf61-4410-9a4f-1490e27dd2c8} executeQuery: Read 2 rows, 132.00 B in 0.070 sec., 28 rows/sec., 1.84 KiB/sec. sentry-self-hosted-clickhouse-1 | 2024-05-29T03:49:08.941065448Z 2024.05.29 03:49:08.940459 [ 2467 ] {} HTTPHandler: Done processing query sentry-self-hosted-clickhouse-1 | 2024-05-29T03:49:09.060478804Z 2024.05.29 03:49:09.059140 [ 2459 ] {7eb7dc77-b56a-4a73-bd5b-1fb17730aea2} executeQuery: Read 2 rows, 4.55 KiB in 0.188 sec., 10 rows/sec., 24.15 KiB/sec. sentry-self-hosted-clickhouse-1 | 2024-05-29T03:49:09.060503912Z 2024.05.29 03:49:09.059537 [ 2459 ] {} HTTPHandler: Done processing query sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:10.880165428Z 03:49:10 [INFO] arroyo.processing.processor: Partitions revoked: [Partition(topic=Topic(name='transactions'), index=0)] sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:10.880396785Z 03:49:10 [INFO] arroyo.processing.processor: Closing ... sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:10.880516728Z 03:49:10 [INFO] arroyo.processing.processor: Waiting for to exit... sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:10.880621604Z 03:49:10 [INFO] arroyo.processing.processor: exited successfully, releasing assignment. sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:10.891654137Z 03:49:10 [INFO] arroyo.processing.processor: New partitions assigned: {Partition(topic=Topic(name='events'), index=0): 1757} sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:10.891682227Z Traceback (most recent call last): sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:10.891686610Z File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 181, in run sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:10.891689956Z self._run_once() sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:10.891692798Z File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 212, in _run_once sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:10.891695813Z self.__message = self.__consumer.poll(timeout=1.0) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:10.891698726Z File "/usr/local/lib/python3.8/site-packages/sentry/eventstream/kafka/synchronized.py", line 248, in poll sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:10.891701870Z message = self.__consumer.poll(timeout) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:10.891708956Z File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 407, in poll sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:10.891716993Z raise OffsetOutOfRange(str(error)) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:10.891719829Z arroyo.errors.OffsetOutOfRange: KafkaError{code=_AUTO_OFFSET_RESET,val=-140,str="fetch failed due to requested offset not available on the broker: Broker: Offset out of range (broker 1001)"} sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:10.891723242Z 03:49:10 [ERROR] arroyo.processing.processor: Caught exception, shutting down... sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:10.891726402Z 03:49:10 [INFO] arroyo.processing.processor: Closing ... sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:10.891729775Z 03:49:10 [INFO] arroyo.processing.processor: Partitions revoked: [Partition(topic=Topic(name='events'), index=0)] sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:10.896626171Z 03:49:10 [INFO] arroyo.processing.processor: Processor terminated sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:10.896709806Z /usr/local/lib/python3.8/site-packages/memcache.py:1303: SyntaxWarning: "is" with a literal. Did you mean "=="? sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:10.896715984Z if key is '': sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:10.896719092Z /usr/local/lib/python3.8/site-packages/memcache.py:1304: SyntaxWarning: "is" with a literal. Did you mean "=="? sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:10.896722194Z if key_extra_len is 0: sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:11.151193207Z 03:49:11 [INFO] arroyo.processing.processor: New partitions assigned: {Partition(topic=Topic(name='transactions'), index=0): 13} sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192183231Z Traceback (most recent call last): sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192203320Z File "/usr/local/bin/sentry", line 8, in sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192213652Z sys.exit(main()) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192217089Z File "/usr/local/lib/python3.8/site-packages/sentry/runner/__init__.py", line 191, in main sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192220193Z func(**kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192222915Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__ sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192225820Z return self.main(*args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192228645Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192231641Z rv = self.invoke(ctx) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192234834Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192245085Z return _process_result(sub_ctx.command.invoke(sub_ctx)) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192248274Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192251315Z return _process_result(sub_ctx.command.invoke(sub_ctx)) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192254466Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192257558Z return ctx.invoke(self.callback, **ctx.params) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192260525Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192263573Z return __callback(*args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192266351Z File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192269306Z return f(get_current_context(), *args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192272035Z File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 69, in inner sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192275149Z return ctx.invoke(f, *args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192277857Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192280912Z return __callback(*args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192283828Z File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192286910Z return f(get_current_context(), *args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192293174Z File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192301334Z return ctx.invoke(f, *args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192304206Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192307032Z return __callback(*args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192309834Z File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/run.py", line 381, in post_process_forwarder sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192313170Z eventstream.run_post_process_forwarder( sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192316785Z File "/usr/local/lib/python3.8/site-packages/sentry/utils/services.py", line 127, in sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192320151Z context[key] = (lambda f: lambda *a, **k: getattr(self, f)(*a, **k))(key) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192323156Z File "/usr/local/lib/python3.8/site-packages/sentry/eventstream/kafka/backend.py", line 314, in run_post_process_forwarder sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192326286Z consumer.run() sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192328983Z File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 181, in run sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192331851Z self._run_once() sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192334547Z File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 212, in _run_once sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192337902Z self.__message = self.__consumer.poll(timeout=1.0) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192340840Z File "/usr/local/lib/python3.8/site-packages/sentry/eventstream/kafka/synchronized.py", line 248, in poll sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192344093Z message = self.__consumer.poll(timeout) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192346958Z File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 407, in poll sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192350055Z raise OffsetOutOfRange(str(error)) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:11.192352900Z arroyo.errors.OffsetOutOfRange: KafkaError{code=_AUTO_OFFSET_RESET,val=-140,str="fetch failed due to requested offset not available on the broker: Broker: Offset out of range (broker 1001)"} sentry-self-hosted-post-process-forwarder-errors-1 exited with code 0 sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:12.561325843Z Updating certificates in /etc/ssl/certs... sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:13.212049852Z 0 added, 0 removed; done. sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:13.212078500Z Running hooks in /etc/ca-certificates/update.d... sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:13.213043274Z done. sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:13.883270875Z 03:49:13 [INFO] arroyo.processing.processor: Partitions revoked: [Partition(topic=Topic(name='transactions'), index=0)] sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:13.883295369Z 03:49:13 [INFO] arroyo.processing.processor: Closing ... sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:13.883300000Z 03:49:13 [INFO] arroyo.processing.processor: Waiting for to exit... sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:13.883303597Z 03:49:13 [INFO] arroyo.processing.processor: exited successfully, releasing assignment. sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:14.165204054Z 03:49:14 [INFO] arroyo.processing.processor: New partitions assigned: {Partition(topic=Topic(name='transactions'), index=0): 13} sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:17.649546944Z 03:49:17 [INFO] sentry.plugins.github: apps-not-configured sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:22.882622949Z 03:49:22 [INFO] arroyo.processing.processor: Partitions revoked: [Partition(topic=Topic(name='transactions'), index=0)] sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:22.882800519Z 03:49:22 [INFO] arroyo.processing.processor: Closing ... sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:22.882901822Z 03:49:22 [INFO] arroyo.processing.processor: Waiting for to exit... sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:22.883053408Z 03:49:22 [INFO] arroyo.processing.processor: exited successfully, releasing assignment. sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:22.887008222Z 03:49:22 [INFO] arroyo.processing.processor: New partitions assigned: {Partition(topic=Topic(name='events'), index=0): 1757} sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:22.889667896Z Traceback (most recent call last): sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:22.889692880Z File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 181, in run sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:22.889698035Z self._run_once() sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:22.889701236Z File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 212, in _run_once sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:22.889704381Z self.__message = self.__consumer.poll(timeout=1.0) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:22.889715732Z File "/usr/local/lib/python3.8/site-packages/sentry/eventstream/kafka/synchronized.py", line 248, in poll sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:22.889718909Z message = self.__consumer.poll(timeout) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:22.889725497Z File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 407, in poll sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:22.889728953Z raise OffsetOutOfRange(str(error)) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:22.889731893Z arroyo.errors.OffsetOutOfRange: KafkaError{code=_AUTO_OFFSET_RESET,val=-140,str="fetch failed due to requested offset not available on the broker: Broker: Offset out of range (broker 1001)"} sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:22.889735116Z 03:49:22 [ERROR] arroyo.processing.processor: Caught exception, shutting down... sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:22.890155596Z 03:49:22 [INFO] arroyo.processing.processor: Closing ... sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:22.890657767Z 03:49:22 [INFO] arroyo.processing.processor: Partitions revoked: [Partition(topic=Topic(name='events'), index=0)] sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:22.892019768Z 03:49:22 [INFO] arroyo.processing.processor: Processor terminated sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:22.892166608Z /usr/local/lib/python3.8/site-packages/memcache.py:1303: SyntaxWarning: "is" with a literal. Did you mean "=="? sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:22.892176321Z if key is '': sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:22.892179506Z /usr/local/lib/python3.8/site-packages/memcache.py:1304: SyntaxWarning: "is" with a literal. Did you mean "=="? sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:22.892183003Z if key_extra_len is 0: sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:23.174404748Z 03:49:23 [INFO] arroyo.processing.processor: New partitions assigned: {Partition(topic=Topic(name='transactions'), index=0): 13} sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218534862Z Traceback (most recent call last): sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218559500Z File "/usr/local/bin/sentry", line 8, in sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218563843Z sys.exit(main()) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218567134Z File "/usr/local/lib/python3.8/site-packages/sentry/runner/__init__.py", line 191, in main sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218570143Z func(**kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218573049Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__ sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218576113Z return self.main(*args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218579226Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218582212Z rv = self.invoke(ctx) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218585038Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218588071Z return _process_result(sub_ctx.command.invoke(sub_ctx)) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218591024Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218593898Z return _process_result(sub_ctx.command.invoke(sub_ctx)) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218596685Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218599580Z return ctx.invoke(self.callback, **ctx.params) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218602666Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218611549Z return __callback(*args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218614450Z File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218617622Z return f(get_current_context(), *args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218620402Z File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 69, in inner sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218623375Z return ctx.invoke(f, *args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218626075Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218629049Z return __callback(*args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218632069Z File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218635178Z return f(get_current_context(), *args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218638042Z File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218641232Z return ctx.invoke(f, *args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218644037Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218646990Z return __callback(*args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218649689Z File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/run.py", line 381, in post_process_forwarder sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218652608Z eventstream.run_post_process_forwarder( sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218656673Z File "/usr/local/lib/python3.8/site-packages/sentry/utils/services.py", line 127, in sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218660109Z context[key] = (lambda f: lambda *a, **k: getattr(self, f)(*a, **k))(key) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218663119Z File "/usr/local/lib/python3.8/site-packages/sentry/eventstream/kafka/backend.py", line 314, in run_post_process_forwarder sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218666218Z consumer.run() sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218669080Z File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 181, in run sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218672061Z self._run_once() sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218674760Z File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 212, in _run_once sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218677626Z self.__message = self.__consumer.poll(timeout=1.0) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218680621Z File "/usr/local/lib/python3.8/site-packages/sentry/eventstream/kafka/synchronized.py", line 248, in poll sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218683815Z message = self.__consumer.poll(timeout) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218686721Z File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 407, in poll sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218689742Z raise OffsetOutOfRange(str(error)) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:23.218692548Z arroyo.errors.OffsetOutOfRange: KafkaError{code=_AUTO_OFFSET_RESET,val=-140,str="fetch failed due to requested offset not available on the broker: Broker: Offset out of range (broker 1001)"} sentry-self-hosted-post-process-forwarder-errors-1 exited with code 0 sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:24.566814433Z Updating certificates in /etc/ssl/certs... sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:25.293733010Z 0 added, 0 removed; done. sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:25.293753360Z Running hooks in /etc/ca-certificates/update.d... sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:25.294486200Z done. sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:25.886274549Z 03:49:25 [INFO] arroyo.processing.processor: Partitions revoked: [Partition(topic=Topic(name='transactions'), index=0)] sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:25.886306208Z 03:49:25 [INFO] arroyo.processing.processor: Closing ... sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:25.886315917Z 03:49:25 [INFO] arroyo.processing.processor: Waiting for to exit... sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:25.886319382Z 03:49:25 [INFO] arroyo.processing.processor: exited successfully, releasing assignment. sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:26.197246836Z 03:49:26 [INFO] arroyo.processing.processor: New partitions assigned: {Partition(topic=Topic(name='transactions'), index=0): 13} sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:29.665696244Z 03:49:29 [INFO] sentry.plugins.github: apps-not-configured sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:34.883278184Z 03:49:34 [INFO] arroyo.processing.processor: Partitions revoked: [Partition(topic=Topic(name='transactions'), index=0)] sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:34.883568504Z 03:49:34 [INFO] arroyo.processing.processor: Closing ... sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:34.884219611Z 03:49:34 [INFO] arroyo.processing.processor: Waiting for to exit... sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:34.884245195Z 03:49:34 [INFO] arroyo.processing.processor: exited successfully, releasing assignment. sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:34.888622334Z 03:49:34 [INFO] arroyo.processing.processor: New partitions assigned: {Partition(topic=Topic(name='events'), index=0): 1757} sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:34.890819685Z Traceback (most recent call last): sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:34.890842944Z File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 181, in run sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:34.890848875Z self._run_once() sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:34.890853544Z File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 212, in _run_once sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:34.890858391Z self.__message = self.__consumer.poll(timeout=1.0) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:34.890862949Z File "/usr/local/lib/python3.8/site-packages/sentry/eventstream/kafka/synchronized.py", line 248, in poll sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:34.890867707Z message = self.__consumer.poll(timeout) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:34.890872299Z File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 407, in poll sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:34.890876645Z raise OffsetOutOfRange(str(error)) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:34.890881345Z arroyo.errors.OffsetOutOfRange: KafkaError{code=_AUTO_OFFSET_RESET,val=-140,str="fetch failed due to requested offset not available on the broker: Broker: Offset out of range (broker 1001)"} sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:34.890886180Z 03:49:34 [ERROR] arroyo.processing.processor: Caught exception, shutting down... sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:34.891499079Z 03:49:34 [INFO] arroyo.processing.processor: Closing ... sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:34.892059748Z 03:49:34 [INFO] arroyo.processing.processor: Partitions revoked: [Partition(topic=Topic(name='events'), index=0)] sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:34.898385812Z 03:49:34 [INFO] arroyo.processing.processor: Processor terminated sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:34.898397282Z /usr/local/lib/python3.8/site-packages/memcache.py:1303: SyntaxWarning: "is" with a literal. Did you mean "=="? sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:34.898418958Z if key is '': sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:34.898423884Z /usr/local/lib/python3.8/site-packages/memcache.py:1304: SyntaxWarning: "is" with a literal. Did you mean "=="? sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:34.898428501Z if key_extra_len is 0: sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209055568Z Traceback (most recent call last): sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209095992Z File "/usr/local/bin/sentry", line 8, in sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209102374Z sys.exit(main()) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209105686Z File "/usr/local/lib/python3.8/site-packages/sentry/runner/__init__.py", line 191, in main sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209108865Z func(**kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209118380Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__ sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209121849Z return self.main(*args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209124603Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209127546Z rv = self.invoke(ctx) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209130220Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209133057Z return _process_result(sub_ctx.command.invoke(sub_ctx)) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209136056Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209142030Z return _process_result(sub_ctx.command.invoke(sub_ctx)) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209145052Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209148304Z return ctx.invoke(self.callback, **ctx.params) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209151185Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209154096Z return __callback(*args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209156807Z File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209159631Z return f(get_current_context(), *args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209162580Z File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 69, in inner sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209165728Z return ctx.invoke(f, *args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209168631Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209171709Z return __callback(*args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209174379Z File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209177249Z return f(get_current_context(), *args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209179996Z File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209182919Z return ctx.invoke(f, *args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209185846Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209188877Z return __callback(*args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209191728Z File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/run.py", line 381, in post_process_forwarder sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209194812Z eventstream.run_post_process_forwarder( sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209198474Z File "/usr/local/lib/python3.8/site-packages/sentry/utils/services.py", line 127, in sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209201520Z context[key] = (lambda f: lambda *a, **k: getattr(self, f)(*a, **k))(key) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209204344Z File "/usr/local/lib/python3.8/site-packages/sentry/eventstream/kafka/backend.py", line 314, in run_post_process_forwarder sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209209012Z consumer.run() sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209211976Z File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 181, in run sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209215043Z self._run_once() sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209217896Z File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 212, in _run_once sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209220928Z self.__message = self.__consumer.poll(timeout=1.0) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209223648Z File "/usr/local/lib/python3.8/site-packages/sentry/eventstream/kafka/synchronized.py", line 248, in poll sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209226546Z message = self.__consumer.poll(timeout) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209229221Z File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 407, in poll sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209232111Z raise OffsetOutOfRange(str(error)) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:35.209235126Z arroyo.errors.OffsetOutOfRange: KafkaError{code=_AUTO_OFFSET_RESET,val=-140,str="fetch failed due to requested offset not available on the broker: Broker: Offset out of range (broker 1001)"} sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:35.229387193Z 03:49:35 [INFO] arroyo.processing.processor: New partitions assigned: {Partition(topic=Topic(name='transactions'), index=0): 13} sentry-self-hosted-post-process-forwarder-errors-1 exited with code 0 sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:37.354587356Z 0 added, 0 removed; done. sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:37.354616664Z Running hooks in /etc/ca-certificates/update.d... sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:37.355606007Z done. sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:37.883349943Z 03:49:37 [INFO] arroyo.processing.processor: Partitions revoked: [Partition(topic=Topic(name='transactions'), index=0)] sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:37.883375934Z 03:49:37 [INFO] arroyo.processing.processor: Closing ... sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:37.884784233Z 03:49:37 [INFO] arroyo.processing.processor: Waiting for to exit... sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:37.884807411Z 03:49:37 [INFO] arroyo.processing.processor: exited successfully, releasing assignment. sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:38.246652696Z 03:49:38 [INFO] arroyo.processing.processor: New partitions assigned: {Partition(topic=Topic(name='transactions'), index=0): 13} sentry-self-hosted-relay-1 | 2024-05-29T03:49:41.496016433Z 2024-05-29T03:49:41Z [relay_server::actors::upstream] ERROR: authentication encountered error: upstream request returned error 403 Forbidden sentry-self-hosted-relay-1 | 2024-05-29T03:49:41.496050223Z caused by: no error details sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:41.697340764Z 03:49:41 [INFO] sentry.plugins.github: apps-not-configured sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:46.885611375Z 03:49:46 [INFO] arroyo.processing.processor: Partitions revoked: [Partition(topic=Topic(name='transactions'), index=0)] sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:46.885632652Z 03:49:46 [INFO] arroyo.processing.processor: Closing ... sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:46.885636956Z 03:49:46 [INFO] arroyo.processing.processor: Waiting for to exit... sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:46.885646168Z 03:49:46 [INFO] arroyo.processing.processor: exited successfully, releasing assignment. sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:46.893589433Z 03:49:46 [INFO] arroyo.processing.processor: New partitions assigned: {Partition(topic=Topic(name='events'), index=0): 1757} sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:46.895298306Z Traceback (most recent call last): sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:46.895315382Z File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 181, in run sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:46.895319001Z self._run_once() sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:46.895321976Z File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 212, in _run_once sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:46.895325134Z self.__message = self.__consumer.poll(timeout=1.0) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:46.895328059Z File "/usr/local/lib/python3.8/site-packages/sentry/eventstream/kafka/synchronized.py", line 248, in poll sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:46.895330987Z message = self.__consumer.poll(timeout) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:46.895333790Z File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 407, in poll sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:46.895337177Z raise OffsetOutOfRange(str(error)) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:46.895340059Z arroyo.errors.OffsetOutOfRange: KafkaError{code=_AUTO_OFFSET_RESET,val=-140,str="fetch failed due to requested offset not available on the broker: Broker: Offset out of range (broker 1001)"} sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:46.895343161Z 03:49:46 [ERROR] arroyo.processing.processor: Caught exception, shutting down... sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:46.895648130Z 03:49:46 [INFO] arroyo.processing.processor: Closing ... sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:46.895769693Z 03:49:46 [INFO] arroyo.processing.processor: Partitions revoked: [Partition(topic=Topic(name='events'), index=0)] sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:46.900817310Z 03:49:46 [INFO] arroyo.processing.processor: Processor terminated sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:46.900891560Z /usr/local/lib/python3.8/site-packages/memcache.py:1303: SyntaxWarning: "is" with a literal. Did you mean "=="? sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:46.900897313Z if key is '': sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:46.900900323Z /usr/local/lib/python3.8/site-packages/memcache.py:1304: SyntaxWarning: "is" with a literal. Did you mean "=="? sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:46.900903404Z if key_extra_len is 0: sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201279516Z Traceback (most recent call last): sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201302817Z File "/usr/local/bin/sentry", line 8, in sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201307421Z sys.exit(main()) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201310364Z File "/usr/local/lib/python3.8/site-packages/sentry/runner/__init__.py", line 191, in main sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201313425Z func(**kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201316468Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__ sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201319630Z return self.main(*args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201322566Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201325497Z rv = self.invoke(ctx) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201328313Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201331231Z return _process_result(sub_ctx.command.invoke(sub_ctx)) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201334009Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201336919Z return _process_result(sub_ctx.command.invoke(sub_ctx)) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201339831Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201342997Z return ctx.invoke(self.callback, **ctx.params) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201345904Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201348856Z return __callback(*args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201351557Z File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201354595Z return f(get_current_context(), *args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201357397Z File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 69, in inner sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201360326Z return ctx.invoke(f, *args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201363559Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201366692Z return __callback(*args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201369711Z File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201378181Z return f(get_current_context(), *args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201380949Z File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201384049Z return ctx.invoke(f, *args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201386905Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201389735Z return __callback(*args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201392670Z File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/run.py", line 381, in post_process_forwarder sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201395984Z eventstream.run_post_process_forwarder( sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201399752Z File "/usr/local/lib/python3.8/site-packages/sentry/utils/services.py", line 127, in sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201402985Z context[key] = (lambda f: lambda *a, **k: getattr(self, f)(*a, **k))(key) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201405827Z File "/usr/local/lib/python3.8/site-packages/sentry/eventstream/kafka/backend.py", line 314, in run_post_process_forwarder sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201408874Z consumer.run() sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201411676Z File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 181, in run sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201414644Z self._run_once() sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201417641Z File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 212, in _run_once sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201420664Z self.__message = self.__consumer.poll(timeout=1.0) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201423683Z File "/usr/local/lib/python3.8/site-packages/sentry/eventstream/kafka/synchronized.py", line 248, in poll sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201426758Z message = self.__consumer.poll(timeout) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201429460Z File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 407, in poll sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201432325Z raise OffsetOutOfRange(str(error)) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:47.201435151Z arroyo.errors.OffsetOutOfRange: KafkaError{code=_AUTO_OFFSET_RESET,val=-140,str="fetch failed due to requested offset not available on the broker: Broker: Offset out of range (broker 1001)"} sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:47.275841250Z 03:49:47 [INFO] arroyo.processing.processor: New partitions assigned: {Partition(topic=Topic(name='transactions'), index=0): 13} sentry-self-hosted-post-process-forwarder-errors-1 exited with code 0 sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:48.667685410Z Updating certificates in /etc/ssl/certs... sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:49.646181800Z 0 added, 0 removed; done. sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:49.646202607Z Running hooks in /etc/ca-certificates/update.d... sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:49.652161667Z done. sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:49.885162732Z 03:49:49 [INFO] arroyo.processing.processor: Partitions revoked: [Partition(topic=Topic(name='transactions'), index=0)] sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:49.885199745Z 03:49:49 [INFO] arroyo.processing.processor: Closing ... sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:49.885204627Z 03:49:49 [INFO] arroyo.processing.processor: Waiting for to exit... sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:49.885207882Z 03:49:49 [INFO] arroyo.processing.processor: exited successfully, releasing assignment. sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-05-29T03:49:50.301004364Z 03:49:50 [INFO] arroyo.processing.processor: New partitions assigned: {Partition(topic=Topic(name='transactions'), index=0): 13} sentry-self-hosted-nginx-1 | 2024-05-29T03:49:53.270402746Z 10.0.38.161 - - [29/May/2024:03:49:53 +0000] "POST /api/1/envelope/?sentry_key=1b5ea24aaa79475ca0fc4935be2c9500&sentry_version=7 HTTP/1.0" 200 2 "http://localhost:3000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36" "-" sentry-self-hosted-post-process-forwarder-errors-1 | 2024-05-29T03:49:54.127857183Z 03:49:54 [INFO] sentry.plugins.github: apps-not-configured sentry-self-hosted-clickhouse-1 | 2024-05-29T03:49:54.297578926Z 2024.05.29 03:49:54.297364 [ 87 ] {edaf719a-de24-48b7-8be2-b0162dea0ea7} executeQuery: Read 1 rows, 148.00 B in 0.018 sec., 55 rows/sec., 8.01 KiB/sec. sentry-self-hosted-clickhouse-1 | 2024-05-29T03:49:54.297807258Z 2024.05.29 03:49:54.297715 [ 87 ] {} HTTPHandler: Done processing query ```

When it's not working: Stops at nginx, then nothing image

Event ID

No response

hubertdeng123 commented 1 month ago

Just to clarify, when your instance is properly ingesting errors, are you getting a crashlooping post-process-forwarder? What is the volume of your errors you're sending? You're right, this should be happening and it is abnormal behavior here.

bintangkevin29 commented 1 month ago

Hi @hubertdeng123 thank you for responding. I must apologize in advance as I still new on using sentry and there's some terminology that you mentioned that I don't understand yet.

volume of your errors you're sending?

My instance is actually not live yet, I'm still testing it out so the volume is not big if that's what you're asking. I'm testing the sentry by sending dummy messages using Sentry.captureMessage().

are you getting a crashlooping post-process-forwarder

I'm not sure, I don't think so. But sentry-self-hosted-post-process-forwarder-errors keep logging these even when idling (when not consuming any errors/events). I think these logs also appeared in my report above.

Log ``` sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.398330320Z 04:29:55 [INFO] arroyo.processing.processor: New partitions assigned: {Partition(topic=Topic(name='events'), index=0): 1757} sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.398827544Z Traceback (most recent call last): sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.398835508Z File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 181, in run sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.398838931Z self._run_once() sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.398841882Z File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 212, in _run_once sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.398845072Z self.__message = self.__consumer.poll(timeout=1.0) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.398847841Z File "/usr/local/lib/python3.8/site-packages/sentry/eventstream/kafka/synchronized.py", line 248, in poll sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.398851075Z message = self.__consumer.poll(timeout) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.398854000Z File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 407, in poll sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.398857036Z raise OffsetOutOfRange(str(error)) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.398859783Z arroyo.errors.OffsetOutOfRange: KafkaError{code=_AUTO_OFFSET_RESET,val=-140,str="fetch failed due to requested offset not available on the broker: Broker: Offset out of range (broker 1001)"} sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.398863496Z 04:29:55 [ERROR] arroyo.processing.processor: Caught exception, shutting down... sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.399274066Z 04:29:55 [INFO] arroyo.processing.processor: Closing ... sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.399587395Z 04:29:55 [INFO] arroyo.processing.processor: Partitions revoked: [Partition(topic=Topic(name='events'), index=0)] sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.400455442Z 04:29:55 [INFO] arroyo.processing.processor: Processor terminated sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.400595728Z /usr/local/lib/python3.8/site-packages/memcache.py:1303: SyntaxWarning: "is" with a literal. Did you mean "=="? sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.400604407Z if key is '': sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.400607315Z /usr/local/lib/python3.8/site-packages/memcache.py:1304: SyntaxWarning: "is" with a literal. Did you mean "=="? sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.400610660Z if key_extra_len is 0: sentry-self-hosted-post-process-forwarder-transactions-1 | 2024-06-03T04:29:55.435121585Z 04:29:55 [INFO] arroyo.processing.processor: New partitions assigned: {Partition(topic=Topic(name='transactions'), index=0): 13} sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659755316Z Traceback (most recent call last): sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659773769Z File "/usr/local/bin/sentry", line 8, in sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659777527Z sys.exit(main()) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659785889Z File "/usr/local/lib/python3.8/site-packages/sentry/runner/__init__.py", line 191, in main sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659789432Z func(**kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659792287Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__ sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659795210Z return self.main(*args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659798109Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659800991Z rv = self.invoke(ctx) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659803992Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659806841Z return _process_result(sub_ctx.command.invoke(sub_ctx)) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659809674Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659812498Z return _process_result(sub_ctx.command.invoke(sub_ctx)) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659815259Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659818272Z return ctx.invoke(self.callback, **ctx.params) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659821092Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659823954Z return __callback(*args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659826613Z File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659829456Z return f(get_current_context(), *args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659832327Z File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 69, in inner sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659835253Z return ctx.invoke(f, *args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659838034Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659840881Z return __callback(*args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659843638Z File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659846476Z return f(get_current_context(), *args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659854121Z File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659857075Z return ctx.invoke(f, *args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659860095Z File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659862869Z return __callback(*args, **kwargs) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659865613Z File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/run.py", line 381, in post_process_forwarder sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659868518Z eventstream.run_post_process_forwarder( sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659871966Z File "/usr/local/lib/python3.8/site-packages/sentry/utils/services.py", line 127, in sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659874983Z context[key] = (lambda f: lambda *a, **k: getattr(self, f)(*a, **k))(key) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659877871Z File "/usr/local/lib/python3.8/site-packages/sentry/eventstream/kafka/backend.py", line 314, in run_post_process_forwarder sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659880869Z consumer.run() sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659883596Z File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 181, in run sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659886463Z self._run_once() sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659889156Z File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 212, in _run_once sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659892095Z self.__message = self.__consumer.poll(timeout=1.0) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659894863Z File "/usr/local/lib/python3.8/site-packages/sentry/eventstream/kafka/synchronized.py", line 248, in poll sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659897747Z message = self.__consumer.poll(timeout) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659900496Z File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 407, in poll sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659903293Z raise OffsetOutOfRange(str(error)) sentry-self-hosted-post-process-forwarder-errors-1 | 2024-06-03T04:29:55.659906017Z arroyo.errors.OffsetOutOfRange: KafkaError{code=_AUTO_OFFSET_RESET,val=-140,str="fetch failed due to requested offset not available on the broker: Broker: Offset out of range (broker 1001)"} ```

Please let me know if I need to provide something for the clarity of this issue

azaslavsky commented 1 month ago

This OffsetOutOfRange problem is a known issue, can you try the fixes suggested at https://github.com/getsentry/self-hosted/issues/2951?

getsantry[bot] commented 3 days ago

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀