search

getsentry / self-hosted

Sentry, feature-complete and packaged up for low-volume deployments and proofs-of-concept
https://develop.sentry.dev/self-hosted/
Other
7.77k stars 1.75k forks source link

[24.6.0] Some errors - Failed to get watermark offsets: Broker: Not leader for partition #3162

Closed motoronik closed 2 months ago

motoronik commented 3 months ago

Self-Hosted Version

24.6.0

CPU Architecture

x86_64

Docker Version

25.0.4

Docker Compose Version

2.24.7

Steps to Reproduce

Expected Result

No errors in docker-compose logs

Actual Result

I see some errors in docker-compose logs can you help me fix errors please? After reinstall or restart container this problem not fixed.

Socket is not connected, Stack trace (when copying this message, always include the lines below)

clickhouse-1 | clickhouse-1 | 0. Poco::Net::SocketImpl::error(int, String const&) @ 0x0000000015b3dbf2 in /usr/bin/clickhouse clickhouse-1 | 1. Poco::Net::SocketImpl::peerAddress() @ 0x0000000015b40376 in /usr/bin/clickhouse clickhouse-1 | 2. DB::HTTPServerRequest::HTTPServerRequest(std::shared_ptr, DB::HTTPServerResponse&, Poco::Net::HTTPServerSession&) @ 0x0000000013154417 in /usr/bin/clickhouse clickhouse-1 | 3. DB::HTTPServerConnection::run() @ 0x0000000013152ba4 in /usr/bin/clickhouse clickhouse-1 | 4. Poco::Net::TCPServerConnection::start() @ 0x0000000015b42834 in /usr/bin/clickhouse clickhouse-1 | 5. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b43a31 in /usr/bin/clickhouse clickhouse-1 | 6. Poco::PooledThread::run() @ 0x0000000015c7a667 in /usr/bin/clickhouse clickhouse-1 | 7. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c7893c in /usr/bin/clickhouse clickhouse-1 | 8. ? @ 0x00007f17040f5609 in ? clickhouse-1 | 9. ? @ 0x00007f170401a353 in ? clickhouse-1 | (version 23.8.11.29.altinitystable (altinity build)) clickhouse-1 | 2024.06.25 07:31:46.355569 [ 543 ] {} ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 107, Net Exception: Socket is not connected, Stack trace (when copying this message, always include the lines below):

Topic ID in memory: vTVx8hbaQ1GUpfJ3DYdWfg does not match the topic ID for partition monitors-clock-tick-0 received:

kafka-1 | ===> Launching kafka ... kafka-1 | [2024-06-25 07:26:01,006] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$) kafka-1 | [2024-06-25 07:26:02,505] INFO Starting the log cleaner (kafka.log.LogCleaner) kafka-1 | [2024-06-25 07:26:02,926] INFO Updated connection-accept-rate max connection creation rate to 2147483647 (kafka.network.ConnectionQuotas) kafka-1 | [2024-06-25 07:26:02,944] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1001] Created data-plane acceptor and processors for endpoint : ListenerName(PLAINTEXT) (kafka.network.SocketServer) kafka-1 | [2024-06-25 07:26:03,030] INFO Creating /brokers/ids/1001 (is it secure? false) (kafka.zk.KafkaZkClient) kafka-1 | [2024-06-25 07:26:03,049] INFO Stat of the created znode at /brokers/ids/1001 is: 791,791,1719300363042,1719300363042,1,0,0,72159068302999553,194,0,791 kafka-1 | (kafka.zk.KafkaZkClient) kafka-1 | [2024-06-25 07:26:03,050] INFO Registered broker 1001 at path /brokers/ids/1001 with addresses: PLAINTEXT://kafka:9092, czxid (broker epoch): 791 (kafka.zk.KafkaZkClient) kafka-1 | [2024-06-25 07:26:03,329] WARN [Controller id=1001, targetBrokerId=1001] Connection to node 1001 (kafka/192.168.144.7:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) kafka-1 | [2024-06-25 07:26:03,332] WARN [RequestSendThread controllerId=1001] Controller 1001's connection to broker kafka:9092 (id: 1001 rack: null) was unsuccessful (kafka.controller.RequestSendThread) kafka-1 | java.io.IOException: Connection to kafka:9092 (id: 1001 rack: null) failed. kafka-1 | at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:70) kafka-1 | at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:298) kafka-1 | at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:251) kafka-1 | at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:130) kafka-1 | [2024-06-25 07:26:03,333] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread) kafka-1 | [2024-06-25 07:26:03,427] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1001] Enabling request processing. (kafka.network.SocketServer) kafka-1 | [2024-06-25 07:26:03,430] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.DataPlaneAcceptor) kafka-1 | [2024-06-25 07:26:03,524] ERROR [Broker id=1001] Topic ID in memory: vTVx8hbaQ1GUpfJ3DYdWfg does not match the topic ID for partition monitors-clock-tick-0 received:

Failed to get watermark offsets: Broker: Not leader for partition

monitors-clock-tick-1 | Running hooks in /etc/ca-certificates/update.d... monitors-clock-tick-1 | done. monitors-clock-tick-1 | Traceback (most recent call last): monitors-clock-tick-1 | File "/.venv/lib/python3.11/site-packages/arroyo/processing/processor.py", line 322, in run monitors-clock-tick-1 | self._run_once() monitors-clock-tick-1 | File "/.venv/lib/python3.11/site-packages/arroyo/processing/processor.py", line 384, in _run_once monitors-clock-tick-1 | self.message = self.consumer.poll(timeout=1.0) monitors-clock-tick-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ monitors-clock-tick-1 | File "/.venv/lib/python3.11/site-packages/arroyo/backends/kafka/consumer.py", line 394, in poll monitors-clock-tick-1 | message: Optional[ConfluentMessage] = self.consumer.poll( monitors-clock-tick-1 | ^^^^^^^^^^^^^^^^^^^^^ monitors-clock-tick-1 | File "/.venv/lib/python3.11/site-packages/arroyo/backends/kafka/consumer.py", line 282, in assignment_callback monitors-clock-tick-1 | self.__resolve_partition_starting_offset(partition) monitors-clock-tick-1 | File "/.venv/lib/python3.11/site-packages/arroyo/backends/kafka/consumer.py", line 227, in resolve_partition_offset_earliest monitors-clock-tick-1 | low, high = self.consumer.get_watermark_offsets(partition) monitors-clock-tick-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ monitors-clock-tick-1 | cimpl.KafkaException: KafkaError{code=NOT_LEADER_FOR_PARTITION,val=6,str="Failed to get watermark offsets: Broker: Not leader for partition"} monitors-clock-tick-1 | 07:33:15 [ERROR] arroyo.processing.processor: Caught exception, shutting down... monitors-clock-tick-1 | 07:33:15 [INFO] arroyo.processing.processor: Closing <arroyo.backends.kafka.consumer.KafkaConsumer object at 0x7f4e867318d0>... monitors-clock-tick-1 | 07:33:15 [INFO] arroyo.processing.processor: Partitions to revoke: [Partition(topic=Topic(name='monitors-clock-tick'), index=0)] monitors-clock-tick-1 | 07:33:15 [INFO] arroyo.processing.processor: Partition revocation complete. monitors-clock-tick-1 | 07:33:15 [WARNING] arroyo.backends.kafka.consumer: failed to delete offset for unknown partition: Partition(topic=Topic(name='monitors-clock-tick'), index=0) monitors-clock-tick-1 | 07:33:15 [INFO] arroyo.processing.processor: Processor terminated monitors-clock-tick-1 | Traceback (most recent call last): monitors-clock-tick-1 | File "/.venv/bin/sentry", line 4, in monitors-clock-tick-1 | raise SystemExit(main()) monitors-clock-tick-1 | ^^^^^^ monitors-clock-tick-1 | File "/usr/src/sentry/src/sentry/runner/main.py", line 148, in main monitors-clock-tick-1 | func(kwargs) monitors-clock-tick-1 | File "/.venv/lib/python3.11/site-packages/click/core.py", line 1157, in call monitors-clock-tick-1 | return self.main(args, kwargs) monitors-clock-tick-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^ monitors-clock-tick-1 | File "/.venv/lib/python3.11/site-packages/click/core.py", line 1078, in main monitors-clock-tick-1 | rv = self.invoke(ctx) monitors-clock-tick-1 | ^^^^^^^^^^^^^^^^ monitors-clock-tick-1 | File "/.venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke monitors-clock-tick-1 | return _process_result(sub_ctx.command.invoke(sub_ctx)) monitors-clock-tick-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ monitors-clock-tick-1 | File "/.venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke monitors-clock-tick-1 | return _process_result(sub_ctx.command.invoke(sub_ctx)) monitors-clock-tick-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ monitors-clock-tick-1 | File "/.venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke monitors-clock-tick-1 | return ctx.invoke(self.callback, ctx.params) monitors-clock-tick-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ monitors-clock-tick-1 | File "/.venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke monitors-clock-tick-1 | return __callback(args, kwargs) monitors-clock-tick-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ monitors-clock-tick-1 | File "/.venv/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func monitors-clock-tick-1 | return f(get_current_context(), *args, *kwargs) monitors-clock-tick-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ monitors-clock-tick-1 | File "/usr/src/sentry/src/sentry/runner/decorators.py", line 83, in inner monitors-clock-tick-1 | return ctx.invoke(f, args, **kwargs) monitors-clock-tick-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ monitors-clock-tick-1 | File "/.venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke monitors-clock-tick-1 | return callback(*args, kwargs) monitors-clock-tick-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ monitors-clock-tick-1 | File "/.venv/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func monitors-clock-tick-1 | return f(get_current_context(), *args, *kwargs) monitors-clock-tick-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ monitors-clock-tick-1 | File "/usr/src/sentry/src/sentry/runner/decorators.py", line 35, in inner monitors-clock-tick-1 | return ctx.invoke(f, args, kwargs) monitors-clock-tick-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ monitors-clock-tick-1 | File "/.venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke monitors-clock-tick-1 | return callback(*args, **kwargs) monitors-clock-tick-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ monitors-clock-tick-1 | File "/usr/src/sentry/src/sentry/runner/commands/run.py", line 386, in basic_consumer monitors-clock-tick-1 | run_processor_with_signals(processor, consumer_name) monitors-clock-tick-1 | File "/usr/src/sentry/src/sentry/utils/kafka.py", line 46, in run_processor_with_signals monitors-clock-tick-1 | processor.run() monitors-clock-tick-1 | File "/.venv/lib/python3.11/site-packages/arroyo/processing/processor.py", line 322, in run monitors-clock-tick-1 | self._run_once() monitors-clock-tick-1 | File "/.venv/lib/python3.11/site-packages/arroyo/processing/processor.py", line 384, in _run_once monitors-clock-tick-1 | self.message = self.consumer.poll(timeout=1.0) monitors-clock-tick-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ monitors-clock-tick-1 | File "/.venv/lib/python3.11/site-packages/arroyo/backends/kafka/consumer.py", line 394, in poll monitors-clock-tick-1 | message: Optional[ConfluentMessage] = self.consumer.poll( monitors-clock-tick-1 | ^^^^^^^^^^^^^^^^^^^^^ monitors-clock-tick-1 | File "/.venv/lib/python3.11/site-packages/arroyo/backends/kafka/consumer.py", line 282, in assignment_callback monitors-clock-tick-1 | self.resolve_partition_starting_offset(partition) monitors-clock-tick-1 | File "/.venv/lib/python3.11/site-packages/arroyo/backends/kafka/consumer.py", line 227, in __resolve_partition_offset_earliest monitors-clock-tick-1 | low, high = self.consumer.get_watermark_offsets(partition) monitors-clock-tick-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ monitors-clock-tick-1 | cimpl.KafkaException: KafkaError{code=NOT_LEADER_FOR_PARTITION,val=6,str="Failed to get watermark offsets: Broker: Not leader for partition"}

Event ID

No response

motoronik commented 3 months ago

More logs

worker-1 | 07:48:07 [WARNING] py.warnings: /.venv/lib/python3.11/site-packages/celery/app/utils.py:203: CDeprecationWarning: worker-1 | The 'BROKER_URL' setting is deprecated and scheduled for removal in worker-1 | version 6.0.0. Use the broker_url instead worker-1 | worker-1 | deprecated.warn(description=f'The {setting!r} setting', worker-1 | worker-1 | 07:48:07 [WARNING] py.warnings: /.venv/lib/python3.11/site-packages/celery/app/utils.py:203: CDeprecationWarning: worker-1 | The 'CELERY_TASK_SERIALIZER' setting is deprecated and scheduled for removal in worker-1 | version 6.0.0. Use the task_serializer instead worker-1 | worker-1 | deprecated.warn(description=f'The {setting!r} setting', worker-1 | worker-1 | 07:48:07 [WARNING] py.warnings: /.venv/lib/python3.11/site-packages/celery/app/utils.py:203: CDeprecationWarning: worker-1 | The 'CELERY_DEFAULT_EXCHANGE_TYPE' setting is deprecated and scheduled for removal in worker-1 | version 6.0.0. Use the task_default_exchange_type instead worker-1 | worker-1 | deprecated.warn(description=f'The {setting!r} setting', worker-1 | worker-1 | 07:48:07 [WARNING] py.warnings: /.venv/lib/python3.11/site-packages/celery/app/utils.py:203: CDeprecationWarning: worker-1 | The 'CELERY_TASK_RESULT_EXPIRES' setting is deprecated and scheduled for removal in worker-1 | version 6.0.0. Use the result_expires instead worker-1 | worker-1 | deprecated.warn(description=f'The {setting!r} setting', worker-1 | worker-1 | 07:48:07 [WARNING] py.warnings: /.venv/lib/python3.11/site-packages/celery/app/utils.py:203: CDeprecationWarning: worker-1 | The 'CELERY_CREATE_MISSING_QUEUES' setting is deprecated and scheduled for removal in worker-1 | version 6.0.0. Use the task_create_missing_queues instead worker-1 | worker-1 | deprecated.warn(description=f'The {setting!r} setting', worker-1 | worker-1 | 07:48:07 [WARNING] py.warnings: /.venv/lib/python3.11/site-packages/celery/app/utils.py:203: CDeprecationWarning: worker-1 | The 'CELERY_QUEUES' setting is deprecated and scheduled for removal in worker-1 | version 6.0.0. Use the task_queues instead worker-1 | worker-1 | deprecated.warn(description=f'The {setting!r} setting', worker-1 | worker-1 | 07:48:07 [WARNING] py.warnings: /.venv/lib/python3.11/site-packages/celery/app/utils.py:203: CDeprecationWarning: worker-1 | The 'BROKER_TRANSPORT_OPTIONS' setting is deprecated and scheduled for removal in worker-1 | version 6.0.0. Use the broker_transport_options instead worker-1 | worker-1 | deprecated.warn(description=f'The {setting!r} setting', worker-1 | worker-1 | 07:48:07 [WARNING] py.warnings: /.venv/lib/python3.11/site-packages/celery/app/utils.py:203: CDeprecationWarning: worker-1 | The 'CELERY_RESULT_BACKEND' setting is deprecated and scheduled for removal in worker-1 | version 6.0.0. Use the result_backend instead worker-1 | worker-1 | deprecated.warn(description=f'The {setting!r} setting', worker-1 | worker-1 | 07:48:07 [WARNING] py.warnings: /.venv/lib/python3.11/site-packages/celery/app/utils.py:203: CDeprecationWarning: worker-1 | The 'CELERY_TASK_PROTOCOL' setting is deprecated and scheduled for removal in worker-1 | version 6.0.0. Use the task_protocol instead worker-1 | worker-1 | deprecated.warn(description=f'The {setting!r} setting', worker-1 | worker-1 | 07:48:07 [WARNING] py.warnings: /.venv/lib/python3.11/site-packages/celery/app/utils.py:203: CDeprecationWarning: worker-1 | The 'CELERY_DEFAULT_QUEUE' setting is deprecated and scheduled for removal in worker-1 | version 6.0.0. Use the task_default_queue instead worker-1 | worker-1 | deprecated.warn(description=f'The {setting!r} setting', worker-1 | worker-1 | 07:48:07 [WARNING] py.warnings: /.venv/lib/python3.11/site-packages/celery/app/utils.py:203: CDeprecationWarning: worker-1 | The 'CELERY_ACCEPT_CONTENT' setting is deprecated and scheduled for removal in worker-1 | version 6.0.0. Use the accept_content instead worker-1 | worker-1 | deprecated.warn(description=f'The {setting!r} setting', worker-1 | worker-1 | 07:48:07 [WARNING] py.warnings: /.venv/lib/python3.11/site-packages/celery/app/utils.py:203: CDeprecationWarning: worker-1 | The 'CELERY_DEFAULT_ROUTING_KEY' setting is deprecated and scheduled for removal in worker-1 | version 6.0.0. Use the task_default_routing_key instead worker-1 | worker-1 | deprecated.warn(description=f'The {setting!r} setting', worker-1 | worker-1 | 07:48:07 [WARNING] py.warnings: /.venv/lib/python3.11/site-packages/celery/app/utils.py:203: CDeprecationWarning: worker-1 | The 'CELERY_IMPORTS' setting is deprecated and scheduled for removal in worker-1 | version 6.0.0. Use the imports instead worker-1 | worker-1 | deprecated.warn(description=f'The {setting!r} setting', worker-1 | worker-1 | 07:48:07 [WARNING] py.warnings: /.venv/lib/python3.11/site-packages/celery/app/utils.py:203: CDeprecationWarning: worker-1 | The 'CELERYBEAT_SCHEDULE' setting is deprecated and scheduled for removal in worker-1 | version 6.0.0. Use the beat_schedule instead worker-1 | worker-1 | deprecated.warn(description=f'The {setting!r} setting', worker-1 | worker-1 | 07:48:07 [WARNING] py.warnings: /.venv/lib/python3.11/site-packages/celery/app/utils.py:203: CDeprecationWarning: worker-1 | The 'CELERY_SEND_EVENTS' setting is deprecated and scheduled for removal in worker-1 | version 6.0.0. Use the worker_send_task_events instead worker-1 | worker-1 | deprecated.warn(description=f'The {setting!r} setting', worker-1 | worker-1 | 07:48:07 [WARNING] py.warnings: /.venv/lib/python3.11/site-packages/celery/app/utils.py:203: CDeprecationWarning: worker-1 | The 'CELERY_DEFAULT_EXCHANGE' setting is deprecated and scheduled for removal in worker-1 | version 6.0.0. Use the task_default_exchange instead worker-1 | worker-1 | deprecated.warn(description=f'The {setting!r} setting', worker-1 | worker-1 | 07:48:07 [WARNING] py.warnings: /.venv/lib/python3.11/site-packages/celery/app/utils.py:203: CDeprecationWarning: worker-1 | The 'CELERY_RESULT_SERIALIZER' setting is deprecated and scheduled for removal in worker-1 | version 6.0.0. Use the result_serializer instead worker-1 | worker-1 | deprecated.warn(description=f'The {setting!r} setting', worker-1 | worker-1 | 07:48:07 [WARNING] py.warnings: /.venv/lib/python3.11/site-packages/celery/app/utils.py:203: CDeprecationWarning: worker-1 | The 'CELERYBEAT_SCHEDULE_FILENAME' setting is deprecated and scheduled for removal in worker-1 | version 6.0.0. Use the beat_schedule_filename instead worker-1 | worker-1 | deprecated.warn(description=f'The {setting!r} setting', worker-1 | worker-1 | 07:48:07 [WARNING] py.warnings: /.venv/lib/python3.11/site-packages/celery/app/utils.py:203: CDeprecationWarning: worker-1 | The 'CELERYD_HIJACK_ROOT_LOGGER' setting is deprecated and scheduled for removal in worker-1 | version 6.0.0. Use the worker_hijack_root_logger instead worker-1 | worker-1 | deprecated.warn(description=f'The {setting!r} setting', worker-1 | worker-1 | 07:48:07 [WARNING] py.warnings: /.venv/lib/python3.11/site-packages/celery/app/utils.py:203: CDeprecationWarning: worker-1 | The 'CELERY_REDIRECT_STDOUTS' setting is deprecated and scheduled for removal in worker-1 | version 6.0.0. Use the worker_redirect_stdouts instead worker-1 | worker-1 | deprecated.warn(description=f'The {setting!r} setting', worker-1 | worker-1 | 07:48:07 [WARNING] py.warnings: /.venv/lib/python3.11/site-packages/celery/app/utils.py:203: CDeprecationWarning: worker-1 | The 'CELERY_DISABLE_RATE_LIMITS' setting is deprecated and scheduled for removal in worker-1 | version 6.0.0. Use the worker_disable_rate_limits instead worker-1 | worker-1 | deprecated.warn(description=f'The {setting!r} setting', worker-1 | worker-1 | 07:48:07 [WARNING] py.warnings: /.venv/lib/python3.11/site-packages/celery/app/utils.py:203: CDeprecationWarning: worker-1 | The 'CELERY_IGNORE_RESULT' setting is deprecated and scheduled for removal in worker-1 | version 6.0.0. Use the task_ignore_result instead worker-1 | worker-1 | deprecated.warn(description=f'The {setting!r} setting', worker-1 | worker-1 | 07:48:07 [WARNING] py.warnings: /.venv/lib/python3.11/site-packages/celery/app/utils.py:203: CDeprecationWarning: worker-1 | The 'CELERY_EAGER_PROPAGATES_EXCEPTIONS' setting is deprecated and scheduled for removal in worker-1 | version 6.0.0. Use the task_eager_propagates instead worker-1 | worker-1 | deprecated.warn(description=f'The {setting!r} setting', worker-1 | worker-1 | 07:48:07 [WARNING] py.warnings: /.venv/lib/python3.11/site-packages/celery/app/utils.py:203: CDeprecationWarning: worker-1 | The 'CELERY_ALWAYS_EAGER' setting is deprecated and scheduled for removal in worker-1 | version 6.0.0. Use the task_always_eager instead worker-1 | worker-1 | deprecated.warn(description=f'The {setting!r} setting', worker-1 | worker-1 | 07:48:07 [WARNING] celery.apps.worker: Please run celery upgrade settings path/to/settings.py to avoid these warnings and to allow a smoother upgrade to Celery 6.0. metrics-consumer-1 | 07:48:07 [INFO] arroyo.processing.processor: New partitions assigned: {Partition(topic=Topic(name='ingest-metrics'), index=0): 87} worker-1 | 07:48:07 [WARNING] py.warnings: /.venv/lib/python3.11/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine worker-1 | whether broker connection retries are made during startup in Celery 6.0 and above. worker-1 | If you wish to retain the existing behavior for retrying connections on startup, worker-1 | you should set broker_connection_retry_on_startup to True.

hubertdeng123 commented 3 months ago

That's a super weird error. Is this a clean install? Have you tried completely removing your kafka/zookeeper volumes and remaking them to proceed with the installation?

motoronik commented 3 months ago

@hubertdeng123 Hi, thank you for answer ;)

This my story - previously, I had some version 23 installed, there were eternal problems with Clickhouse, as well as with Kafka, I reinstalled my version several times.

Reinstalled different paths:

Method 1:

Method 2:

Nothing helped. Later I stopped trying to fix the errors in the logs who related with clickhouse and kafka.

Recently I saw the release of version 24.6.0 and decided to reinstall everything again.

I did:

hubertdeng123 commented 3 months ago

I'd recommend you also perform a sudo docker volume prune for kafka/zookeeper/clickhouse if they are still giving you problems. This may result in data loss of some sort for unprocessed events, but given that your instance is not working properly anyways it may be the best move forwards.

getsantry[bot] commented 2 months ago

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀