search

getsentry / self-hosted

Sentry, feature-complete and packaged up for low-volume deployments and proofs-of-concept
https://develop.sentry.dev/self-hosted/
Other
7.75k stars 1.75k forks source link

All issue detail : Sorry, the events for this issue could not be found. #1005

Closed kaneruan closed 3 years ago

kaneruan commented 3 years ago

Version Information

Version: 21.6.1

Steps to Reproduce

./install.sh docker-compose up -d

Expected Result

Actual Result

image

Logs

clickhouse_1                                | 2021.06.25 08:47:06.694917 [ 98 ] {} <Information> HTTPHandler: Done processing query
cron_1                                      |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
cron_1                                      |     return _process_result(sub_ctx.command.invoke(sub_ctx))
cron_1                                      |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
cron_1                                      |     return ctx.invoke(self.callback, **ctx.params)
cron_1                                      |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
cron_1                                      |     return callback(*args, **kwargs)
cron_1                                      |   File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
cron_1                                      |     return f(get_current_context(), *args, **kwargs)
cron_1                                      |   File "/usr/local/lib/python3.6/site-packages/sentry/runner/decorators.py", line 69, in inner
cron_1                                      |     return ctx.invoke(f, *args, **kwargs)
cron_1                                      |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
cron_1                                      |     return callback(*args, **kwargs)
cron_1                                      |   File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
cron_1                                      |     return f(get_current_context(), *args, **kwargs)
cron_1                                      |   File "/usr/local/lib/python3.6/site-packages/sentry/runner/decorators.py", line 28, in inner
cron_1                                      |     configure()
cron_1                                      |   File "/usr/local/lib/python3.6/site-packages/sentry/runner/__init__.py", line 132, in configure
cron_1                                      |     configure(ctx, py, yaml, skip_service_validation)
cron_1                                      |   File "/usr/local/lib/python3.6/site-packages/sentry/runner/settings.py", line 161, in configure
cron_1                                      |     skip_service_validation=skip_service_validation,
cron_1                                      |   File "/usr/local/lib/python3.6/site-packages/sentry/runner/initializer.py", line 371, in initialize_app
cron_1                                      |     setup_services(validate=not skip_service_validation)
cron_1                                      |   File "/usr/local/lib/python3.6/site-packages/sentry/runner/initializer.py", line 415, in setup_services
cron_1                                      |     service.validate()
cron_1                                      |   File "/usr/local/lib/python3.6/site-packages/sentry/utils/services.py", line 102, in <lambda>
cron_1                                      |     context[key] = (lambda f: lambda *a, **k: getattr(self, f)(*a, **k))(key)
cron_1                                      |   File "/usr/local/lib/python3.6/site-packages/sentry/buffer/redis.py", line 66, in validate
cron_1                                      |     raise InvalidConfiguration(str(e))
cron_1                                      | sentry.exceptions.InvalidConfiguration: Error -2 connecting to redis:6379. Name or service not known.
cron_1                                      | 08:14:10 [INFO] sentry.plugins.github: apps-not-configured
geoipupdate_1                               | error loading configuration file /sentry/GeoIP.conf: error opening file: open /sentry/GeoIP.conf: no such file or directory
geoipupdate_1                               | error loading configuration file /sentry/GeoIP.conf: error opening file: open /sentry/GeoIP.conf: no such file or directory
geoipupdate_1                               | error loading configuration file /sentry/GeoIP.conf: error opening file: open /sentry/GeoIP.conf: no such file or directory
kafka_1                                     | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root
kafka_1                                     | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/
kafka_1                                     | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=236MB
kafka_1                                     | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=3531MB
kafka_1                                     | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=240MB
kafka_1                                     | [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zookeeper:2181 sessionTimeout=40000 watcher=io.confluent.admin.utils.ZookeeperConnectionWatcher@cc34f4d
kafka_1                                     | [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
kafka_1                                     | [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 4194304 Bytes
kafka_1                                     | [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
kafka_1                                     | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.64.3:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                                     | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /192.168.64.26:51814, server: zookeeper/192.168.64.3:2181
kafka_1                                     | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server zookeeper/192.168.64.3:2181, sessionid = 0x10816031e330000, negotiated timeout = 40000
kafka_1                                     | [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x10816031e330000 closed
kafka_1                                     | [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x10816031e330000
kafka_1                                     | ===> Launching ...
kafka_1                                     | ===> Launching kafka ...
kafka_1                                     | [2021-06-25 08:14:05,535] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
kafka_1                                     | [2021-06-25 08:14:06,684] WARN The package io.confluent.support.metrics.collectors.FullCollector for collecting the full set of support metrics could not be loaded, so we are reverting to anonymous, basic metric collection. If you are a Confluent customer, please refer to the Confluent Platform documentation, section Proactive Support, on how to activate full metrics collection. (io.confluent.support.metrics.KafkaSupportConfig)
kafka_1                                     | [2021-06-25 08:14:06,685] WARN The support metrics collection feature ("Metrics") of Proactive Support is disabled. (io.confluent.support.metrics.SupportedServerStartable)
kafka_1                                     | [2021-06-25 08:14:48,067] INFO Starting the log cleaner (kafka.log.LogCleaner)
kafka_1                                     | [2021-06-25 08:14:48,335] INFO [kafka-log-cleaner-thread-0]: Starting (kafka.log.LogCleaner)
kafka_1                                     | [2021-06-25 08:14:48,828] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
kafka_1                                     | [2021-06-25 08:14:48,886] INFO [SocketServer brokerId=1001] Created data-plane acceptor and processors for endpoint : EndPoint(0.0.0.0,9092,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.network.SocketServer)
kafka_1                                     | [2021-06-25 08:14:48,889] INFO [SocketServer brokerId=1001] Started 1 acceptor threads for data-plane (kafka.network.SocketServer)
kafka_1                                     | [2021-06-25 08:14:49,032] INFO Creating /brokers/ids/1001 (is it secure? false) (kafka.zk.KafkaZkClient)
kafka_1                                     | [2021-06-25 08:14:49,098] INFO Stat of the created znode at /brokers/ids/1001 is: 634,634,1624608889056,1624608889056,1,0,0,74333596498984961,180,0,634
kafka_1                                     |  (kafka.zk.KafkaZkClient)
kafka_1                                     | [2021-06-25 08:14:49,099] INFO Registered broker 1001 at path /brokers/ids/1001 with addresses: ArrayBuffer(EndPoint(kafka,9092,ListenerName(PLAINTEXT),PLAINTEXT)), czxid (broker epoch): 634 (kafka.zk.KafkaZkClient)
kafka_1                                     | [2021-06-25 08:14:49,742] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
kafka_1                                     | [2021-06-25 08:14:49,866] INFO [SocketServer brokerId=1001] Started data-plane processors for 1 acceptors (kafka.network.SocketServer)
ingest-consumer_1                           | 08:46:43 [INFO] batching-kafka-consumer: Worker flush took 528ms
ingest-consumer_1                           | 08:46:44 [INFO] batching-kafka-consumer: Flushing 27 items (from {('ingest-events', 0): [140733, 140759]}): forced:False size:False time:True
ingest-consumer_1                           | 08:46:44 [INFO] batching-kafka-consumer: Worker flush took 395ms
ingest-consumer_1                           | 08:46:45 [INFO] batching-kafka-consumer: Flushing 29 items (from {('ingest-events', 0): [140760, 140788]}): forced:False size:False time:True
ingest-consumer_1                           | 08:46:46 [INFO] batching-kafka-consumer: Worker flush took 531ms
ingest-consumer_1                           | 08:46:47 [INFO] batching-kafka-consumer: Flushing 29 items (from {('ingest-events', 0): [140789, 140817]}): forced:False size:False time:True
ingest-consumer_1                           | 08:46:47 [INFO] batching-kafka-consumer: Worker flush took 620ms
ingest-consumer_1                           | 08:46:49 [INFO] batching-kafka-consumer: Flushing 35 items (from {('ingest-events', 0): [140818, 140852]}): forced:False size:False time:True
ingest-consumer_1                           | 08:46:49 [INFO] batching-kafka-consumer: Worker flush took 709ms
ingest-consumer_1                           | 08:46:50 [INFO] batching-kafka-consumer: Flushing 35 items (from {('ingest-events', 0): [140853, 140887]}): forced:False size:False time:True
ingest-consumer_1                           | 08:46:51 [INFO] batching-kafka-consumer: Worker flush took 618ms
ingest-consumer_1                           | 08:46:52 [INFO] batching-kafka-consumer: Flushing 31 items (from {('ingest-events', 0): [140888, 140918]}): forced:False size:False time:True
ingest-consumer_1                           | 08:46:52 [INFO] batching-kafka-consumer: Worker flush took 501ms
ingest-consumer_1                           | 08:46:53 [INFO] batching-kafka-consumer: Flushing 31 items (from {('ingest-events', 0): [140919, 140949]}): forced:False size:False time:True
ingest-consumer_1                           | 08:46:54 [INFO] batching-kafka-consumer: Worker flush took 564ms
ingest-consumer_1                           | 08:46:55 [INFO] batching-kafka-consumer: Flushing 41 items (from {('ingest-events', 0): [140950, 140990]}): forced:False size:False time:True
ingest-consumer_1                           | 08:46:56 [INFO] batching-kafka-consumer: Worker flush took 721ms
ingest-consumer_1                           | 08:46:57 [INFO] batching-kafka-consumer: Flushing 34 items (from {('ingest-events', 0): [140991, 141024]}): forced:False size:False time:True
ingest-consumer_1                           | 08:46:57 [INFO] batching-kafka-consumer: Worker flush took 484ms
ingest-consumer_1                           | 08:46:58 [INFO] batching-kafka-consumer: Flushing 38 items (from {('ingest-events', 0): [141025, 141062]}): forced:False size:False time:True
ingest-consumer_1                           | 08:46:59 [INFO] batching-kafka-consumer: Worker flush took 644ms
ingest-consumer_1                           | 08:47:00 [INFO] batching-kafka-consumer: Flushing 30 items (from {('ingest-events', 0): [141063, 141092]}): forced:False size:False time:True
ingest-consumer_1                           | 08:47:01 [INFO] batching-kafka-consumer: Worker flush took 505ms
ingest-consumer_1                           | 08:47:02 [INFO] batching-kafka-consumer: Flushing 26 items (from {('ingest-events', 0): [141093, 141118]}): forced:False size:False time:True
ingest-consumer_1                           | 08:47:02 [INFO] batching-kafka-consumer: Worker flush took 528ms
ingest-consumer_1                           | 08:47:03 [INFO] batching-kafka-consumer: Flushing 25 items (from {('ingest-events', 0): [141119, 141143]}): forced:False size:False time:True
ingest-consumer_1                           | 08:47:04 [INFO] batching-kafka-consumer: Worker flush took 604ms
ingest-consumer_1                           | 08:47:05 [INFO] batching-kafka-consumer: Flushing 36 items (from {('ingest-events', 0): [141144, 141179]}): forced:False size:False time:True
ingest-consumer_1                           | 08:47:05 [INFO] batching-kafka-consumer: Worker flush took 535ms
ingest-consumer_1                           | 08:47:06 [INFO] batching-kafka-consumer: Flushing 37 items (from {('ingest-events', 0): [141180, 141216]}): forced:False size:False time:True
memcached_1                                 | Signal handled: Terminated.
memcached_1                                 | Signal handled: Terminated.
memcached_1                                 | Signal handled: Terminated.
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:05 +0000] "POST /api/33/store/?sentry_key=fc8678a0071145128f0c8314c07f8407&sentry_version=7 HTTP/1.0" 200 41 "https://www.example.com/?islogout" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36" "220.202.202.194"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:05 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 8.1.0; Redmi 5 Plus Build/OPM1.171019.019; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/75.0.3770.156 Mobile Safari/537.36 JsSdk/2 NewsArticle/8.2.5 NetType/wifi TTWebView/0751130023403" "223.115.22.70"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:05 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 8.1.0; Redmi 5 Plus Build/OPM1.171019.019; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/75.0.3770.156 Mobile Safari/537.36 JsSdk/2 NewsArticle/8.2.5 NetType/wifi TTWebView/0751130023403" "223.115.22.70"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:05 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 9; STF-AL00; HMSCore 5.3.0.312; GMSCore 19.6.29) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.93 HuaweiBrowser/11.1.1.300 Mobile Safari/537.36" "121.236.205.134"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:05 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 9; STF-AL00; HMSCore 5.3.0.312; GMSCore 19.6.29) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.93 HuaweiBrowser/11.1.1.300 Mobile Safari/537.36" "121.236.205.134"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:05 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 6.0.1; vivo Y55 Build/MMB29M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.84 Mobile Safari/537.36 VivoBrowser/7.1.0.1" "120.208.199.202"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:05 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 6.0.1; vivo Y55 Build/MMB29M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.84 Mobile Safari/537.36 VivoBrowser/7.1.0.1" "120.208.199.202"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:05 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 8.1.0; Redmi 5 Plus Build/OPM1.171019.019; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/75.0.3770.156 Mobile Safari/537.36 JsSdk/2 NewsArticle/8.2.5 NetType/wifi TTWebView/0751130023403" "223.115.22.70"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:05 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 6.0.1; vivo Y55 Build/MMB29M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.84 Mobile Safari/537.36 VivoBrowser/7.1.0.1" "120.208.199.202"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:05 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 8.1.0; Redmi 5 Plus Build/OPM1.171019.019; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/75.0.3770.156 Mobile Safari/537.36 JsSdk/2 NewsArticle/8.2.5 NetType/wifi TTWebView/0751130023403" "223.115.22.70"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:06 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 8.1.0; Redmi 5 Plus Build/OPM1.171019.019; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/75.0.3770.156 Mobile Safari/537.36 JsSdk/2 NewsArticle/8.2.5 NetType/wifi TTWebView/0751130023403" "223.115.22.70"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:06 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 6.0.1; vivo Y55 Build/MMB29M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.84 Mobile Safari/537.36 VivoBrowser/7.1.0.1" "120.208.199.202"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:06 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 8.1.0; Redmi 5 Plus Build/OPM1.171019.019; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/75.0.3770.156 Mobile Safari/537.36 JsSdk/2 NewsArticle/8.2.5 NetType/wifi TTWebView/0751130023403" "223.115.22.70"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:06 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 9; STF-AL00; HMSCore 5.3.0.312; GMSCore 19.6.29) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.93 HuaweiBrowser/11.1.1.300 Mobile Safari/537.36" "121.236.205.134"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:06 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 6.0.1; vivo Y55 Build/MMB29M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.84 Mobile Safari/537.36 VivoBrowser/7.1.0.1" "120.208.199.202"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:06 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 6.0.1; vivo Y55 Build/MMB29M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.84 Mobile Safari/537.36 VivoBrowser/7.1.0.1" "120.208.199.202"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:06 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 7.1.2; vivo X9Plus L Build/N2G47H; wv) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.84 Mobile Safari/537.36 VivoBrowser/9.5.14.0" "117.136.63.97"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:06 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 8.1.0; Redmi 5 Plus Build/OPM1.171019.019; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/75.0.3770.156 Mobile Safari/537.36 JsSdk/2 NewsArticle/8.2.5 NetType/wifi TTWebView/0751130023403" "223.115.22.70"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:06 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 9; STF-AL00; HMSCore 5.3.0.312; GMSCore 19.6.29) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.93 HuaweiBrowser/11.1.1.300 Mobile Safari/537.36" "121.236.205.134"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:06 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 8.1.0; Redmi 5 Plus Build/OPM1.171019.019; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/75.0.3770.156 Mobile Safari/537.36 JsSdk/2 NewsArticle/8.2.5 NetType/wifi TTWebView/0751130023403" "223.115.22.70"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:06 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 8.1.0; Redmi 5 Plus Build/OPM1.171019.019; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/75.0.3770.156 Mobile Safari/537.36 JsSdk/2 NewsArticle/8.2.5 NetType/wifi TTWebView/0751130023403" "223.115.22.70"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:06 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 9; STF-AL00; HMSCore 5.3.0.312; GMSCore 19.6.29) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.93 HuaweiBrowser/11.1.1.300 Mobile Safari/537.36" "121.236.205.134"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:06 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 8.1.0; Redmi 5 Plus Build/OPM1.171019.019; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/75.0.3770.156 Mobile Safari/537.36 JsSdk/2 NewsArticle/8.2.5 NetType/wifi TTWebView/0751130023403" "223.115.22.70"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:06 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 8.1.0; Redmi 5 Plus Build/OPM1.171019.019; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/75.0.3770.156 Mobile Safari/537.36 JsSdk/2 NewsArticle/8.2.5 NetType/wifi TTWebView/0751130023403" "223.115.22.70"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:06 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 8.1.0; Redmi 5 Plus Build/OPM1.171019.019; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/75.0.3770.156 Mobile Safari/537.36 JsSdk/2 NewsArticle/8.2.5 NetType/wifi TTWebView/0751130023403" "223.115.22.70"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:06 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 9; STF-AL00; HMSCore 5.3.0.312; GMSCore 19.6.29) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.93 HuaweiBrowser/11.1.1.300 Mobile Safari/537.36" "121.236.205.134"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:06 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 6.0.1; vivo Y55 Build/MMB29M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.84 Mobile Safari/537.36 VivoBrowser/7.1.0.1" "120.208.199.202"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:06 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 9; STF-AL00; HMSCore 5.3.0.312; GMSCore 19.6.29) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.93 HuaweiBrowser/11.1.1.300 Mobile Safari/537.36" "121.236.205.134"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:06 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 8.1.0; Redmi 5 Plus Build/OPM1.171019.019; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/75.0.3770.156 Mobile Safari/537.36 JsSdk/2 NewsArticle/8.2.5 NetType/wifi TTWebView/0751130023403" "223.115.22.70"
nginx_1                                     | 10.1.140.70 - - [25/Jun/2021:08:47:06 +0000] "POST /api/21/store/?sentry_key=6d6d36ab3bfe47ccb4801165e71ccb9e&sentry_version=7 HTTP/1.0" 200 41 "https://m3ws.example.com/" "Mozilla/5.0 (Linux; Android 8.1.0; Redmi 5 Plus Build/OPM1.171019.019; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/75.0.3770.156 Mobile Safari/537.36 JsSdk/2 NewsArticle/8.2.5 NetType/wifi TTWebView/0751130023403" "223.115.22.70"
postgres_1                                  | ERROR:  duplicate key value violates unique constraint "sentry_eventuser_project_id_1a96e3b719e55f9a_uniq"
postgres_1                                  | DETAIL:  Key (project_id, hash)=(43, 3c5592676c7519b6dc2fbd0e6c1983fb) already exists.
postgres_1                                  | STATEMENT:  INSERT INTO "sentry_eventuser" ("project_id", "hash", "ident", "email", "username", "name", "ip_address", "date_added") VALUES (43, '3c5592676c7519b6dc2fbd0e6c1983fb', '1381148272', NULL, NULL, NULL, NULL, '2021-06-25T08:44:01.652004+00:00'::timestamptz) RETURNING "sentry_eventuser"."id"
postgres_1                                  | ERROR:  duplicate key value violates unique constraint "sentry_eventuser_project_id_1a96e3b719e55f9a_uniq"
postgres_1                                  | DETAIL:  Key (project_id, hash)=(43, a5f14e790dc138177838c3083b5139be) already exists.
postgres_1                                  | STATEMENT:  INSERT INTO "sentry_eventuser" ("project_id", "hash", "ident", "email", "username", "name", "ip_address", "date_added") VALUES (43, 'a5f14e790dc138177838c3083b5139be', '1849351321', NULL, NULL, NULL, NULL, '2021-06-25T08:44:32.058722+00:00'::timestamptz) RETURNING "sentry_eventuser"."id"
postgres_1                                  | ERROR:  duplicate key value violates unique constraint "sentry_eventuser_project_id_1a96e3b719e55f9a_uniq"
postgres_1                                  | DETAIL:  Key (project_id, hash)=(43, 3fb6ca473237338bbebd27fca09aaf6e) already exists.
postgres_1                                  | STATEMENT:  INSERT INTO "sentry_eventuser" ("project_id", "hash", "ident", "email", "username", "name", "ip_address", "date_added") VALUES (43, '3fb6ca473237338bbebd27fca09aaf6e', '428621662', NULL, NULL, NULL, NULL, '2021-06-25T08:44:49.349440+00:00'::timestamptz) RETURNING "sentry_eventuser"."id"
postgres_1                                  | ERROR:  duplicate key value violates unique constraint "sentry_grouprelease_group_id_46ba6e430d088d04_uniq"
postgres_1                                  | DETAIL:  Key (group_id, release_id, environment)=(6951, 13, ) already exists.
postgres_1                                  | STATEMENT:  INSERT INTO "sentry_grouprelease" ("project_id", "group_id", "release_id", "environment", "first_seen", "last_seen") VALUES (3, 6951, 13, '', '2021-06-25T08:30:51.175980+00:00'::timestamptz, '2021-06-25T08:30:51.175980+00:00'::timestamptz) RETURNING "sentry_grouprelease"."id"
postgres_1                                  | ERROR:  duplicate key value violates unique constraint "sentry_eventuser_project_id_1a96e3b719e55f9a_uniq"
postgres_1                                  | DETAIL:  Key (project_id, hash)=(3, 3e4a36a1988f622f8b7f8359212515d9) already exists.
postgres_1                                  | STATEMENT:  INSERT INTO "sentry_eventuser" ("project_id", "hash", "ident", "email", "username", "name", "ip_address", "date_added") VALUES (3, '3e4a36a1988f622f8b7f8359212515d9', '1552084915', NULL, NULL, NULL, NULL, '2021-06-25T08:45:19.717856+00:00'::timestamptz) RETURNING "sentry_eventuser"."id"
postgres_1                                  | ERROR:  duplicate key value violates unique constraint "sentry_eventuser_project_id_1a96e3b719e55f9a_uniq"
postgres_1                                  | DETAIL:  Key (project_id, hash)=(2, 4d8eba46855e380d8758f0172117113e) already exists.
postgres_1                                  | STATEMENT:  INSERT INTO "sentry_eventuser" ("project_id", "hash", "ident", "email", "username", "name", "ip_address", "date_added") VALUES (2, '4d8eba46855e380d8758f0172117113e', '875184786', NULL, NULL, NULL, NULL, '2021-06-25T08:45:57.550843+00:00'::timestamptz) RETURNING "sentry_eventuser"."id"
postgres_1                                  | ERROR:  duplicate key value violates unique constraint "sentry_grouprelease_group_id_46ba6e430d088d04_uniq"
postgres_1                                  | DETAIL:  Key (group_id, release_id, environment)=(172767, 101, production) already exists.
postgres_1                                  | STATEMENT:  INSERT INTO "sentry_grouprelease" ("project_id", "group_id", "release_id", "environment", "first_seen", "last_seen") VALUES (23, 172767, 101, 'production', '2021-06-25T08:31:13.515000+00:00'::timestamptz, '2021-06-25T08:31:13.515000+00:00'::timestamptz) RETURNING "sentry_grouprelease"."id"
postgres_1                                  | ERROR:  duplicate key value violates unique constraint "sentry_eventuser_project_id_1a96e3b719e55f9a_uniq"
postgres_1                                  | DETAIL:  Key (project_id, hash)=(43, af5849f1a6ebfb90bc8dbf684e9988cf) already exists.
postgres_1                                  | STATEMENT:  INSERT INTO "sentry_eventuser" ("project_id", "hash", "ident", "email", "username", "name", "ip_address", "date_added") VALUES (43, 'af5849f1a6ebfb90bc8dbf684e9988cf', '1611660449', NULL, NULL, NULL, NULL, '2021-06-25T08:46:48.347380+00:00'::timestamptz) RETURNING "sentry_eventuser"."id"
postgres_1                                  | ERROR:  duplicate key value violates unique constraint "sentry_eventuser_project_id_1a96e3b719e55f9a_uniq"
postgres_1                                  | DETAIL:  Key (project_id, hash)=(2, 236f46bfb97bf42250243e001652a38c) already exists.
postgres_1                                  | STATEMENT:  INSERT INTO "sentry_eventuser" ("project_id", "hash", "ident", "email", "username", "name", "ip_address", "date_added") VALUES (2, '236f46bfb97bf42250243e001652a38c', '439600624', NULL, NULL, NULL, NULL, '2021-06-25T08:46:48.695441+00:00'::timestamptz) RETURNING "sentry_eventuser"."id"
postgres_1                                  | ERROR:  duplicate key value violates unique constraint "sentry_eventuser_project_id_1a96e3b719e55f9a_uniq"
postgres_1                                  | DETAIL:  Key (project_id, hash)=(2, 5b71d83c2a2d1082f36a7538a791e28c) already exists.
postgres_1                                  | STATEMENT:  INSERT INTO "sentry_eventuser" ("project_id", "hash", "ident", "email", "username", "name", "ip_address", "date_added") VALUES (2, '5b71d83c2a2d1082f36a7538a791e28c', '1766891017', NULL, NULL, NULL, NULL, '2021-06-25T08:46:49.504592+00:00'::timestamptz) RETURNING "sentry_eventuser"."id"
post-process-forwarder_1                    |     rv = self.invoke(ctx)
post-process-forwarder_1                    |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
post-process-forwarder_1                    |     return _process_result(sub_ctx.command.invoke(sub_ctx))
post-process-forwarder_1                    |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
post-process-forwarder_1                    |     return _process_result(sub_ctx.command.invoke(sub_ctx))
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
relay_1                                     | 2021-06-25T08:16:48Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_envelope
post-process-forwarder_1                    |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
post-process-forwarder_1                    |     return ctx.invoke(self.callback, **ctx.params)
post-process-forwarder_1                    |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
post-process-forwarder_1                    |     return callback(*args, **kwargs)
post-process-forwarder_1                    |   File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
post-process-forwarder_1                    |     return f(get_current_context(), *args, **kwargs)
post-process-forwarder_1                    |   File "/usr/local/lib/python3.6/site-packages/sentry/runner/decorators.py", line 69, in inner
post-process-forwarder_1                    |     return ctx.invoke(f, *args, **kwargs)
post-process-forwarder_1                    |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
post-process-forwarder_1                    |     return callback(*args, **kwargs)
post-process-forwarder_1                    |   File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
post-process-forwarder_1                    |     return f(get_current_context(), *args, **kwargs)
post-process-forwarder_1                    |   File "/usr/local/lib/python3.6/site-packages/sentry/runner/decorators.py", line 29, in inner
post-process-forwarder_1                    |     return ctx.invoke(f, *args, **kwargs)
post-process-forwarder_1                    |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
post-process-forwarder_1                    |     return callback(*args, **kwargs)
post-process-forwarder_1                    |   File "/usr/local/lib/python3.6/site-packages/sentry/runner/commands/run.py", line 334, in post_process_forwarder
post-process-forwarder_1                    |     initial_offset_reset=options["initial_offset_reset"],
post-process-forwarder_1                    |   File "/usr/local/lib/python3.6/site-packages/sentry/utils/services.py", line 102, in <lambda>
post-process-forwarder_1                    |     context[key] = (lambda f: lambda *a, **k: getattr(self, f)(*a, **k))(key)
post-process-forwarder_1                    |   File "/usr/local/lib/python3.6/site-packages/sentry/eventstream/kafka/backend.py", line 203, in run_post_process_forwarder
post-process-forwarder_1                    |     raise Exception(error)
post-process-forwarder_1                    | Exception: KafkaError{code=COORDINATOR_LOAD_IN_PROGRESS,val=14,str="JoinGroup failed: Broker: Coordinator load in progress"}
post-process-forwarder_1                    | 08:15:38 [INFO] sentry.plugins.github: apps-not-configured
post-process-forwarder_1                    | 08:15:42 [INFO] sentry.eventstream.kafka.backend: Received partition assignment: [TopicPartition{topic=events,partition=0,offset=-1001,error=None}]
smtp_1                                      | Authenticators: cram_md5 plaintext
smtp_1                                      | Routers: accept dnslookup ipliteral manualroute queryprogram redirect
smtp_1                                      | Transports: appendfile/maildir/mailstore autoreply lmtp pipe smtp
smtp_1                                      | Fixed never_users: 0
smtp_1                                      | Configure owner: 0:0
smtp_1                                      | Size of off_t: 8
smtp_1                                      |   994 delivering 1lwhTO-0000Fy-8g
smtp_1                                      |   994 R: dnslookup for zijieyang@example.net
smtp_1                                      |   995 T: remote_smtp for zijieyang@example.net
smtp_1                                      |   994 LOG: retry_defer MAIN
smtp_1                                      |   994   == zijieyang@example.net R=dnslookup T=remote_smtp defer (-53): retry time not reached for any host for 'example.net'
smtp_1                                      |   993 LOG: MAIN
smtp_1                                      |   993   <= root@localhost H=sentry_onpremise_worker_1.sentry_onpremise_default (b75481ef063d) [192.168.64.2] P=esmtp S=17401 id=20210625084601.80.94587@localhost
smtp_1                                      |   993 LOG: smtp_connection MAIN
smtp_1                                      |   993   SMTP connection from sentry_onpremise_worker_1.sentry_onpremise_default (b75481ef063d) [192.168.64.2] closed by QUIT
smtp_1                                      |   997 Exim version 4.92 uid=101 gid=101 pid=997 D=80001
smtp_1                                      | Berkeley DB: Berkeley DB 5.3.28: (September  9, 2013)
smtp_1                                      | Support for: crypteq iconv() IPv6 GnuTLS move_frozen_messages DANE DKIM DNSSEC Event OCSP PRDR SOCKS TCP_Fast_Open
smtp_1                                      | Lookups (built-in): lsearch wildlsearch nwildlsearch iplsearch cdb dbm dbmjz dbmnz dnsdb dsearch nis nis0 passwd
smtp_1                                      | Authenticators: cram_md5 plaintext
smtp_1                                      | Routers: accept dnslookup ipliteral manualroute queryprogram redirect
smtp_1                                      | Transports: appendfile/maildir/mailstore autoreply lmtp pipe smtp
smtp_1                                      | Fixed never_users: 0
smtp_1                                      | Configure owner: 0:0
smtp_1                                      | Size of off_t: 8
smtp_1                                      |   997 delivering 1lwhTP-0000G1-LE
smtp_1                                      |   997 R: dnslookup for jiajianhuang@example.net
smtp_1                                      |   998 T: remote_smtp for jiajianhuang@example.net
smtp_1                                      |   997 LOG: retry_defer MAIN
smtp_1                                      |   997   == jiajianhuang@example.net R=dnslookup T=remote_smtp defer (-53): retry time not reached for any host for 'example.net'
snuba-api_1                                 | worker 1 buried after 1 seconds
snuba-api_1                                 | goodbye to uWSGI.
snuba-api_1                                 | *** Starting uWSGI 2.0.18 (64bit) on [Fri Jun 25 08:13:41 2021] ***
snuba-api_1                                 | compiled with version: 8.3.0 on 17 June 2021 17:19:11
snuba-api_1                                 | os: Linux-3.10.0-1062.18.1.el7.x86_64 #1 SMP Tue Mar 17 23:49:17 UTC 2020
snuba-api_1                                 | nodename: 94ff184646ae
snuba-api_1                                 | machine: x86_64
snuba-api_1                                 | clock source: unix
snuba-api_1                                 | pcre jit disabled
snuba-api_1                                 | detected number of CPU cores: 6
snuba-api_1                                 | current working directory: /usr/src/snuba
snuba-api_1                                 | detected binary path: /usr/local/bin/uwsgi
snuba-api_1                                 | your memory page size is 4096 bytes
snuba-api_1                                 | detected max file descriptor number: 1048576
snuba-api_1                                 | lock engine: pthread robust mutexes
snuba-api_1                                 | thunder lock: enabled
snuba-api_1                                 | uwsgi socket 0 bound to TCP address 0.0.0.0:1218 fd 3
snuba-api_1                                 | Python version: 3.8.10 (default, May 12 2021, 15:56:47)  [GCC 8.3.0]
snuba-api_1                                 | Set PythonHome to /usr/local
snuba-api_1                                 | Python main interpreter initialized at 0x561476a78c00
snuba-api_1                                 | python threads support enabled
snuba-api_1                                 | your server socket listen backlog is limited to 100 connections
snuba-api_1                                 | your mercy for graceful operations on workers is 60 seconds
snuba-api_1                                 | mapped 145808 bytes (142 KB) for 1 cores
snuba-api_1                                 | *** Operational MODE: single process ***
snuba-api_1                                 | initialized 38 metrics
snuba-api_1                                 | spawned uWSGI master process (pid: 1)
snuba-api_1                                 | spawned uWSGI worker 1 (pid: 14, cores: 1)
snuba-api_1                                 | metrics collector thread started
snuba-api_1                                 | WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0x561476a78c00 pid: 14 (default app)
snuba-outcomes-consumer_1                   | 2021-06-25 08:46:32,006 Completed processing <Batch: 1 message, open for 1.01 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:46:33,260 Completed processing <Batch: 18 messages, open for 0.98 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:46:34,217 Completed processing <Batch: 14 messages, open for 0.96 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:46:35,635 Completed processing <Batch: 19 messages, open for 1.42 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:46:36,440 Completed processing <Batch: 20 messages, open for 0.80 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:46:37,501 Completed processing <Batch: 12 messages, open for 1.06 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:46:38,364 Completed processing <Batch: 9 messages, open for 0.86 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:46:41,216 Completed processing <Batch: 10 messages, open for 2.85 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:46:42,000 Completed processing <Batch: 24 messages, open for 0.78 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:46:42,796 Completed processing <Batch: 10 messages, open for 0.80 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:46:43,671 Completed processing <Batch: 11 messages, open for 0.87 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:46:46,007 Completed processing <Batch: 10 messages, open for 2.34 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:46:46,778 Completed processing <Batch: 32 messages, open for 0.77 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:46:47,698 Completed processing <Batch: 16 messages, open for 0.92 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:46:48,664 Completed processing <Batch: 12 messages, open for 0.97 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:46:51,200 Completed processing <Batch: 19 messages, open for 2.54 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:46:52,151 Completed processing <Batch: 25 messages, open for 0.95 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:46:52,940 Completed processing <Batch: 14 messages, open for 0.79 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:46:54,277 Completed processing <Batch: 15 messages, open for 1.34 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:46:55,200 Completed processing <Batch: 19 messages, open for 0.92 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:46:57,179 Completed processing <Batch: 13 messages, open for 1.98 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:46:58,001 Completed processing <Batch: 25 messages, open for 0.82 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:46:58,809 Completed processing <Batch: 9 messages, open for 0.81 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:47:00,032 Completed processing <Batch: 10 messages, open for 1.22 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:47:00,835 Completed processing <Batch: 6 messages, open for 0.80 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:47:01,686 Completed processing <Batch: 11 messages, open for 0.85 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:47:02,526 Completed processing <Batch: 9 messages, open for 0.84 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:47:03,849 Completed processing <Batch: 8 messages, open for 1.32 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:47:04,675 Completed processing <Batch: 14 messages, open for 0.83 seconds>.
snuba-outcomes-consumer_1                   | 2021-06-25 08:47:06,639 Completed processing <Batch: 10 messages, open for 1.96 seconds>.
snuba-sessions-consumer_1                   | %3|1624608865.191|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#192.168.64.26:9092 failed: Connection refused (after 0ms in state CONNECT, 30 identical error(s) suppressed)
snuba-sessions-consumer_1                   | %3|1624608865.199|FAIL|rdkafka#consumer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#192.168.64.26:9092 failed: Connection refused (after 1ms in state CONNECT, 30 identical error(s) suppressed)
snuba-sessions-consumer_1                   | 2021-06-25 08:14:51,272 Caught ConsumerError('KafkaError{code=COORDINATOR_LOAD_IN_PROGRESS,val=14,str="JoinGroup failed: Broker: Coordinator load in progress"}'), shutting down...
snuba-sessions-consumer_1                   | Traceback (most recent call last):
snuba-sessions-consumer_1                   |   File "/usr/local/bin/snuba", line 33, in <module>
snuba-sessions-consumer_1                   |     sys.exit(load_entry_point('snuba', 'console_scripts', 'snuba')())
snuba-sessions-consumer_1                   |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
snuba-sessions-consumer_1                   |     return self.main(*args, **kwargs)
snuba-sessions-consumer_1                   |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
snuba-sessions-consumer_1                   |     rv = self.invoke(ctx)
snuba-sessions-consumer_1                   |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
snuba-sessions-consumer_1                   |     return _process_result(sub_ctx.command.invoke(sub_ctx))
snuba-sessions-consumer_1                   |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
snuba-sessions-consumer_1                   |     return ctx.invoke(self.callback, **ctx.params)
snuba-sessions-consumer_1                   |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
snuba-sessions-consumer_1                   |     return callback(*args, **kwargs)
snuba-sessions-consumer_1                   |   File "/usr/src/snuba/snuba/cli/consumer.py", line 172, in consumer
snuba-sessions-consumer_1                   |     consumer.run()
snuba-sessions-consumer_1                   |   File "/usr/src/snuba/arroyo/processing/processor.py", line 108, in run
snuba-sessions-consumer_1                   |     self._run_once()
snuba-sessions-consumer_1                   |   File "/usr/src/snuba/arroyo/processing/processor.py", line 138, in _run_once
snuba-sessions-consumer_1                   |     self.__message = self.__consumer.poll(timeout=1.0)
snuba-sessions-consumer_1                   |   File "/usr/src/snuba/arroyo/backends/kafka/consumer.py", line 393, in poll
snuba-consumer_1                            | 2021-06-25 08:46:32,519 Completed processing <Batch: 11 messages, open for 2.05 seconds>.
snuba-consumer_1                            | 2021-06-25 08:46:33,317 Completed processing <Batch: 19 messages, open for 0.80 seconds>.
snuba-consumer_1                            | 2021-06-25 08:46:34,288 Completed processing <Batch: 13 messages, open for 0.97 seconds>.
snuba-consumer_1                            | 2021-06-25 08:46:35,662 Completed processing <Batch: 21 messages, open for 1.37 seconds>.
snuba-consumer_1                            | 2021-06-25 08:46:36,476 Completed processing <Batch: 18 messages, open for 0.81 seconds>.
snuba-consumer_1                            | 2021-06-25 08:46:37,508 Completed processing <Batch: 12 messages, open for 1.03 seconds>.
snuba-consumer_1                            | 2021-06-25 08:46:38,399 Completed processing <Batch: 11 messages, open for 0.89 seconds>.
snuba-consumer_1                            | 2021-06-25 08:46:39,481 Completed processing <Batch: 8 messages, open for 1.08 seconds>.
snuba-consumer_1                            | 2021-06-25 08:46:41,498 Completed processing <Batch: 5 messages, open for 2.02 seconds>.
snuba-consumer_1                            | 2021-06-25 08:46:42,376 Completed processing <Batch: 22 messages, open for 0.88 seconds>.
snuba-consumer_1                            | 2021-06-25 08:46:43,288 Completed processing <Batch: 12 messages, open for 0.91 seconds>.
snuba-consumer_1                            | 2021-06-25 08:46:44,127 Completed processing <Batch: 13 messages, open for 0.84 seconds>.
snuba-consumer_1                            | 2021-06-25 08:46:45,084 Completed processing <Batch: 4 messages, open for 0.96 seconds>.
snuba-consumer_1                            | 2021-06-25 08:46:46,063 Completed processing <Batch: 16 messages, open for 0.98 seconds>.
snuba-consumer_1                            | 2021-06-25 08:46:46,935 Completed processing <Batch: 16 messages, open for 0.87 seconds>.
snuba-consumer_1                            | 2021-06-25 08:46:47,760 Completed processing <Batch: 15 messages, open for 0.82 seconds>.
snuba-consumer_1                            | 2021-06-25 08:46:48,623 Completed processing <Batch: 15 messages, open for 0.86 seconds>.
snuba-consumer_1                            | 2021-06-25 08:46:51,233 Completed processing <Batch: 13 messages, open for 2.61 seconds>.
snuba-consumer_1                            | 2021-06-25 08:46:52,188 Completed processing <Batch: 27 messages, open for 0.95 seconds>.
snuba-consumer_1                            | 2021-06-25 08:46:53,076 Completed processing <Batch: 15 messages, open for 0.89 seconds>.
snuba-consumer_1                            | 2021-06-25 08:46:53,899 Completed processing <Batch: 16 messages, open for 0.82 seconds>.
snuba-consumer_1                            | 2021-06-25 08:46:54,947 Completed processing <Batch: 15 messages, open for 1.05 seconds>.
snuba-consumer_1                            | 2021-06-25 08:46:57,209 Completed processing <Batch: 13 messages, open for 2.26 seconds>.
snuba-consumer_1                            | 2021-06-25 08:46:58,781 Completed processing <Batch: 26 messages, open for 1.57 seconds>.
snuba-consumer_1                            | 2021-06-25 08:47:00,072 Completed processing <Batch: 19 messages, open for 1.29 seconds>.
snuba-consumer_1                            | 2021-06-25 08:47:00,938 Completed processing <Batch: 7 messages, open for 0.87 seconds>.
snuba-consumer_1                            | 2021-06-25 08:47:01,845 Completed processing <Batch: 12 messages, open for 0.91 seconds>.
snuba-cleanup_1                             | SHELL=/bin/bash
snuba-cleanup_1                             | BASH_ENV=/container.env
snuba-cleanup_1                             | */5 * * * * gosu snuba snuba cleanup --dry-run False > /proc/1/fd/1 2>/proc/1/fd/2
snuba-cleanup_1                             | 2021-06-25 07:25:26,792 Dropped 0 partitions on clickhouse:9000
snuba-cleanup_1                             | 2021-06-25 07:30:04,313 Dropped 0 partitions on clickhouse:9000
snuba-cleanup_1                             | 2021-06-25 07:35:03,718 Dropped 0 partitions on clickhouse:9000
snuba-cleanup_1                             | 2021-06-25 07:40:04,344 Dropped 0 partitions on clickhouse:9000
snuba-cleanup_1                             | 2021-06-25 07:45:03,998 Dropped 0 partitions on clickhouse:9000
snuba-cleanup_1                             | 2021-06-25 07:50:03,533 Dropped 0 partitions on clickhouse:9000
snuba-cleanup_1                             | 2021-06-25 07:55:03,976 Dropped 0 partitions on clickhouse:9000
snuba-cleanup_1                             | 2021-06-25 08:00:03,480 Dropped 0 partitions on clickhouse:9000
snuba-cleanup_1                             | 2021-06-25 08:05:04,055 Dropped 0 partitions on clickhouse:9000
snuba-cleanup_1                             | SHELL=/bin/bash
snuba-cleanup_1                             | BASH_ENV=/container.env
snuba-sessions-consumer_1                   |     raise ConsumerError(str(error))
snuba-sessions-consumer_1                   | arroyo.errors.ConsumerError: KafkaError{code=COORDINATOR_LOAD_IN_PROGRESS,val=14,str="JoinGroup failed: Broker: Coordinator load in progress"}
snuba-sessions-consumer_1                   | 2021-06-25 08:15:23,408 New partitions assigned: {Partition(topic=Topic(name='ingest-sessions'), index=0): 0}
snuba-sessions-consumer_1                   | 2021-06-25 08:15:29,411 Partitions revoked: [Partition(topic=Topic(name='ingest-sessions'), index=0)]
snuba-sessions-consumer_1                   | 2021-06-25 08:15:29,562 New partitions assigned: {Partition(topic=Topic(name='ingest-sessions'), index=0): 0}
snuba-sessions-consumer_1                   | 2021-06-25 08:15:32,563 Partitions revoked: [Partition(topic=Topic(name='ingest-sessions'), index=0)]
snuba-sessions-consumer_1                   | 2021-06-25 08:15:32,586 New partitions assigned: {Partition(topic=Topic(name='ingest-sessions'), index=0): 0}
snuba-consumer_1                            | 2021-06-25 08:47:02,991 Completed processing <Batch: 8 messages, open for 1.14 seconds>.
snuba-consumer_1                            | 2021-06-25 08:47:04,313 Completed processing <Batch: 6 messages, open for 1.32 seconds>.
snuba-consumer_1                            | 2021-06-25 08:47:06,702 Completed processing <Batch: 20 messages, open for 2.39 seconds>.
snuba-replacer_1                            | %3|1624608814.789|FAIL|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Failed to resolve 'kafka:9092': Name or service not known (after 67ms in state CONNECT)
snuba-replacer_1                            | %3|1624608816.788|FAIL|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Failed to resolve 'kafka:9092': Name or service not known (after 66ms in state CONNECT, 1 identical error(s) suppressed)
snuba-replacer_1                            | %3|1624608833.731|FAIL|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#192.168.64.26:9092 failed: No route to host (after 3008ms in state CONNECT)
snuba-replacer_1                            | %3|1624608834.733|FAIL|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#192.168.64.26:9092 failed: Connection refused (after 11ms in state CONNECT)
snuba-replacer_1                            | %3|1624608847.724|FAIL|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#192.168.64.26:9092 failed: Connection refused (after 0ms in state CONNECT, 13 identical error(s) suppressed)
snuba-replacer_1                            | %3|1624608877.728|FAIL|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#192.168.64.26:9092 failed: Connection refused (after 0ms in state CONNECT, 30 identical error(s) suppressed)
snuba-replacer_1                            | 2021-06-25 08:14:51,745 Caught ConsumerError('KafkaError{code=COORDINATOR_LOAD_IN_PROGRESS,val=14,str="JoinGroup failed: Broker: Coordinator load in progress"}'), shutting down...
snuba-replacer_1                            | Traceback (most recent call last):
snuba-replacer_1                            |   File "/usr/local/bin/snuba", line 33, in <module>
snuba-replacer_1                            |     sys.exit(load_entry_point('snuba', 'console_scripts', 'snuba')())
snuba-replacer_1                            |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
snuba-replacer_1                            |     return self.main(*args, **kwargs)
snuba-replacer_1                            |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
snuba-replacer_1                            |     rv = self.invoke(ctx)
snuba-replacer_1                            |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
snuba-replacer_1                            |     return _process_result(sub_ctx.command.invoke(sub_ctx))
snuba-replacer_1                            |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
snuba-replacer_1                            |     return ctx.invoke(self.callback, **ctx.params)
snuba-replacer_1                            |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
snuba-replacer_1                            |     return callback(*args, **kwargs)
snuba-replacer_1                            |   File "/usr/src/snuba/snuba/cli/replacer.py", line 131, in replacer
snuba-replacer_1                            |     replacer.run()
snuba-replacer_1                            |   File "/usr/src/snuba/arroyo/processing/processor.py", line 108, in run
snuba-replacer_1                            |     self._run_once()
snuba-replacer_1                            |   File "/usr/src/snuba/arroyo/processing/processor.py", line 138, in _run_once
snuba-replacer_1                            |     self.__message = self.__consumer.poll(timeout=1.0)
snuba-replacer_1                            |   File "/usr/src/snuba/arroyo/backends/kafka/consumer.py", line 393, in poll
snuba-replacer_1                            |     raise ConsumerError(str(error))
snuba-replacer_1                            | arroyo.errors.ConsumerError: KafkaError{code=COORDINATOR_LOAD_IN_PROGRESS,val=14,str="JoinGroup failed: Broker: Coordinator load in progress"}
snuba-replacer_1                            | 2021-06-25 08:15:35,192 New partitions assigned: {Partition(topic=Topic(name='event-replacements-legacy'), index=0): 0}
sentry-cleanup_1                            | SHELL=/bin/bash
sentry-cleanup_1                            | BASH_ENV=/container.env
sentry-cleanup_1                            | 0 0 * * * gosu sentry sentry cleanup --days 90 > /proc/1/fd/1 2>/proc/1/fd/2
sentry-cleanup_1                            | SHELL=/bin/bash
sentry-cleanup_1                            | BASH_ENV=/container.env
sentry-cleanup_1                            | 0 0 * * * gosu sentry sentry cleanup --days 90 > /proc/1/fd/1 2>/proc/1/fd/2
sentry-cleanup_1                            | SHELL=/bin/bash
sentry-cleanup_1                            | BASH_ENV=/container.env
sentry-cleanup_1                            | 0 0 * * * gosu sentry sentry cleanup --days 90 > /proc/1/fd/1 2>/proc/1/fd/2
snuba-subscription-consumer-events_1        | 2021-06-25 08:46:29,200 Flushing 29 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1379715, hi=1379744)})
snuba-subscription-consumer-events_1        | 2021-06-25 08:46:29,201 Worker flush took 0ms
snuba-subscription-consumer-events_1        | 2021-06-25 08:46:32,206 Flushing 54 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1379744, hi=1379798)})
snuba-subscription-consumer-events_1        | 2021-06-25 08:46:32,207 Worker flush took 0ms
snuba-subscription-consumer-events_1        | 2021-06-25 08:46:36,208 Flushing 30 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1379798, hi=1379828)})
snuba-subscription-consumer-events_1        | 2021-06-25 08:46:36,209 Worker flush took 0ms
snuba-subscription-consumer-events_1        | 2021-06-25 08:46:38,211 Flushing 44 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1379828, hi=1379872)})
snuba-subscription-consumer-events_1        | 2021-06-25 08:46:38,211 Worker flush took 0ms
snuba-subscription-consumer-events_1        | 2021-06-25 08:46:40,213 Flushing 33 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1379872, hi=1379905)})
snuba-subscription-consumer-events_1        | 2021-06-25 08:46:40,213 Worker flush took 0ms
snuba-subscription-consumer-events_1        | 2021-06-25 08:46:42,216 Flushing 6 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1379905, hi=1379911)})
snuba-subscription-consumer-events_1        | 2021-06-25 08:46:42,218 Worker flush took 0ms
snuba-subscription-consumer-events_1        | 2021-06-25 08:46:45,219 Flushing 27 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1379911, hi=1379938)})
snuba-subscription-consumer-events_1        | 2021-06-25 08:46:45,220 Worker flush took 0ms
snuba-subscription-consumer-events_1        | 2021-06-25 08:46:47,222 Flushing 54 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1379938, hi=1379992)})
snuba-subscription-consumer-events_1        | 2021-06-25 08:46:47,223 Worker flush took 0ms
snuba-subscription-consumer-events_1        | 2021-06-25 08:46:49,224 Flushing 29 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1379992, hi=1380021)})
snuba-subscription-consumer-events_1        | 2021-06-25 08:46:49,225 Worker flush took 0ms
snuba-subscription-consumer-events_1        | 2021-06-25 08:46:51,232 Flushing 8 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1380021, hi=1380029)})
snuba-subscription-consumer-events_1        | 2021-06-25 08:46:51,233 Worker flush took 0ms
snuba-subscription-consumer-events_1        | 2021-06-25 08:46:55,226 Flushing 55 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1380029, hi=1380084)})
snuba-subscription-consumer-events_1        | 2021-06-25 08:46:55,227 Worker flush took 0ms
snuba-subscription-consumer-events_1        | 2021-06-25 08:46:57,238 Flushing 31 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1380084, hi=1380115)})
snuba-subscription-consumer-events_1        | 2021-06-25 08:46:57,239 Worker flush took 0ms
snuba-subscription-consumer-events_1        | 2021-06-25 08:47:00,231 Flushing 18 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1380115, hi=1380133)})
snuba-subscription-consumer-events_1        | 2021-06-25 08:47:00,232 Worker flush took 0ms
snuba-subscription-consumer-events_1        | 2021-06-25 08:47:02,233 Flushing 48 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1380133, hi=1380181)})
snuba-subscription-consumer-events_1        | 2021-06-25 08:47:02,235 Worker flush took 1ms
snuba-subscription-consumer-events_1        | 2021-06-25 08:47:05,235 Flushing 19 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1380181, hi=1380200)})
snuba-subscription-consumer-events_1        | 2021-06-25 08:47:05,236 Worker flush took 0ms
snuba-cleanup_1                             | */5 * * * * gosu snuba snuba cleanup --dry-run False > /proc/1/fd/1 2>/proc/1/fd/2
snuba-cleanup_1                             | SHELL=/bin/bash
snuba-cleanup_1                             | BASH_ENV=/container.env
snuba-cleanup_1                             | */5 * * * * gosu snuba snuba cleanup --dry-run False > /proc/1/fd/1 2>/proc/1/fd/2
snuba-cleanup_1                             | 2021-06-25 08:15:04,133 Dropped 0 partitions on clickhouse:9000
snuba-cleanup_1                             | 2021-06-25 08:20:04,353 Dropped 0 partitions on clickhouse:9000
snuba-cleanup_1                             | 2021-06-25 08:25:04,946 Dropped 0 partitions on clickhouse:9000
snuba-cleanup_1                             | 2021-06-25 08:30:05,009 Dropped 0 partitions on clickhouse:9000
snuba-cleanup_1                             | 2021-06-25 08:35:05,029 Dropped 0 partitions on clickhouse:9000
snuba-cleanup_1                             | 2021-06-25 08:40:04,147 Dropped 0 partitions on clickhouse:9000
snuba-cleanup_1                             | 2021-06-25 08:45:04,776 Dropped 0 partitions on clickhouse:9000
redis_1                                     | 1:M 25 Jun 2021 08:41:22.053 * 10000 changes in 60 seconds. Saving...
redis_1                                     | 1:M 25 Jun 2021 08:41:22.061 * Background saving started by pid 35
redis_1                                     | 35:C 25 Jun 2021 08:41:26.779 * DB saved on disk
redis_1                                     | 35:C 25 Jun 2021 08:41:26.787 * RDB: 5 MB of memory used by copy-on-write
redis_1                                     | 1:M 25 Jun 2021 08:41:26.874 * Background saving terminated with success
redis_1                                     | 1:M 25 Jun 2021 08:42:27.066 * 10000 changes in 60 seconds. Saving...
redis_1                                     | 1:M 25 Jun 2021 08:42:27.074 * Background saving started by pid 36
redis_1                                     | 36:C 25 Jun 2021 08:42:34.935 * DB saved on disk
redis_1                                     | 36:C 25 Jun 2021 08:42:34.954 * RDB: 5 MB of memory used by copy-on-write
redis_1                                     | 1:M 25 Jun 2021 08:42:35.003 * Background saving terminated with success
redis_1                                     | 1:M 25 Jun 2021 08:43:36.016 * 10000 changes in 60 seconds. Saving...
redis_1                                     | 1:M 25 Jun 2021 08:43:36.026 * Background saving started by pid 37
redis_1                                     | 37:C 25 Jun 2021 08:43:43.156 * DB saved on disk
redis_1                                     | 37:C 25 Jun 2021 08:43:43.166 * RDB: 5 MB of memory used by copy-on-write
redis_1                                     | 1:M 25 Jun 2021 08:43:43.253 * Background saving terminated with success
redis_1                                     | 1:M 25 Jun 2021 08:44:44.019 * 10000 changes in 60 seconds. Saving...
redis_1                                     | 1:M 25 Jun 2021 08:44:44.037 * Background saving started by pid 38
redis_1                                     | 38:C 25 Jun 2021 08:44:49.317 * DB saved on disk
redis_1                                     | 38:C 25 Jun 2021 08:44:49.328 * RDB: 5 MB of memory used by copy-on-write
redis_1                                     | 1:M 25 Jun 2021 08:44:49.353 * Background saving terminated with success
redis_1                                     | 1:M 25 Jun 2021 08:45:50.045 * 10000 changes in 60 seconds. Saving...
redis_1                                     | 1:M 25 Jun 2021 08:45:50.056 * Background saving started by pid 39
redis_1                                     | 39:C 25 Jun 2021 08:45:54.911 * DB saved on disk
redis_1                                     | 39:C 25 Jun 2021 08:45:54.923 * RDB: 5 MB of memory used by copy-on-write
redis_1                                     | 1:M 25 Jun 2021 08:45:54.966 * Background saving terminated with success
redis_1                                     | 1:M 25 Jun 2021 08:46:55.011 * 10000 changes in 60 seconds. Saving...
redis_1                                     | 1:M 25 Jun 2021 08:46:55.021 * Background saving started by pid 40
redis_1                                     | 40:C 25 Jun 2021 08:47:00.027 * DB saved on disk
redis_1                                     | 40:C 25 Jun 2021 08:47:00.036 * RDB: 6 MB of memory used by copy-on-write
redis_1                                     | 1:M 25 Jun 2021 08:47:00.132 * Background saving terminated with success
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:46:34,907 Flushing 32 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1379807, hi=1379839)})
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:46:34,907 Worker flush took 0ms
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:46:37,909 Flushing 36 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1379839, hi=1379875)})
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:46:37,910 Worker flush took 0ms
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:46:40,910 Flushing 30 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1379875, hi=1379905)})
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:46:40,911 Worker flush took 0ms
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:46:42,912 Flushing 11 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1379905, hi=1379916)})
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:46:42,913 Worker flush took 0ms
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:46:44,915 Flushing 25 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1379916, hi=1379941)})
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:46:44,916 Worker flush took 0ms
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:46:46,918 Flushing 25 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1379941, hi=1379966)})
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:46:46,919 Worker flush took 0ms
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:46:48,920 Flushing 33 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1379966, hi=1379999)})
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:46:48,921 Worker flush took 0ms
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:46:50,923 Flushing 29 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1379999, hi=1380028)})
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:46:50,924 Worker flush took 0ms
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:46:52,925 Flushing 19 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1380028, hi=1380047)})
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:46:52,926 Worker flush took 0ms
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:46:54,928 Flushing 30 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1380047, hi=1380077)})
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:46:54,929 Worker flush took 0ms
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:46:56,930 Flushing 35 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1380077, hi=1380112)})
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:46:56,931 Worker flush took 0ms
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:46:58,933 Flushing 28 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1380112, hi=1380140)})
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:46:58,934 Worker flush took 0ms
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:47:00,937 Flushing 17 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1380140, hi=1380157)})
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:47:00,938 Worker flush took 0ms
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:47:02,940 Flushing 19 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1380157, hi=1380176)})
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:47:02,941 Worker flush took 0ms
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:47:04,944 Flushing 23 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1380176, hi=1380199)})
snuba-subscription-consumer-transactions_1  | 2021-06-25 08:47:04,945 Worker flush took 0ms
subscription-consumer-events_1              |     rv = self.invoke(ctx)
subscription-consumer-events_1              |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
subscription-consumer-events_1              |     return _process_result(sub_ctx.command.invoke(sub_ctx))
subscription-consumer-events_1              |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
subscription-consumer-events_1              |     return _process_result(sub_ctx.command.invoke(sub_ctx))
subscription-consumer-events_1              |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
subscription-consumer-events_1              |     return ctx.invoke(self.callback, **ctx.params)
subscription-consumer-events_1              |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
subscription-consumer-events_1              |     return callback(*args, **kwargs)
subscription-consumer-events_1              |   File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
subscription-consumer-events_1              |     return f(get_current_context(), *args, **kwargs)
subscription-consumer-events_1              |   File "/usr/local/lib/python3.6/site-packages/sentry/runner/decorators.py", line 69, in inner
subscription-consumer-events_1              |     return ctx.invoke(f, *args, **kwargs)
subscription-consumer-events_1              |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
subscription-consumer-events_1              |     return callback(*args, **kwargs)
subscription-consumer-events_1              |   File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
subscription-consumer-events_1              |     return f(get_current_context(), *args, **kwargs)
subscription-consumer-events_1              |   File "/usr/local/lib/python3.6/site-packages/sentry/runner/decorators.py", line 29, in inner
subscription-consumer-events_1              |     return ctx.invoke(f, *args, **kwargs)
subscription-consumer-events_1              |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
subscription-consumer-events_1              |     return callback(*args, **kwargs)
subscription-consumer-events_1              |   File "/usr/local/lib/python3.6/site-packages/sentry/runner/commands/run.py", line 388, in query_subscription_consumer
subscription-consumer-events_1              |     subscriber.run()
subscription-consumer-events_1              |   File "/usr/local/lib/python3.6/site-packages/sentry/snuba/query_subscription_consumer.py", line 160, in run
subscription-consumer-events_1              |     wait_for_topics(admin_client, [self.topic])
subscription-consumer-events_1              |   File "/usr/local/lib/python3.6/site-packages/sentry/utils/batching_kafka_consumer.py", line 37, in wait_for_topics
subscription-consumer-events_1              |     f"Timeout when waiting for Kafka topic '{topic}' to become available, last error: {last_error}"
subscription-consumer-events_1              | RuntimeError: Timeout when waiting for Kafka topic 'events-subscription-results' to become available, last error: KafkaError{code=LEADER_NOT_AVAILABLE,val=5,str="Broker: Leader not available"}
subscription-consumer-events_1              | 08:15:34 [INFO] sentry.plugins.github: apps-not-configured
subscription-consumer-events_1              | 08:15:37 [INFO] sentry.snuba.query_subscription_consumer: query-subscription-consumer.on_assign (offsets='{0: None}' partitions='[TopicPartition{topic=events-subscription-results,partition=0,offset=-1001,error=None}]')
snuba-transactions-consumer_1               | 2021-06-25 08:46:41,034 Completed processing <Batch: 10 messages, open for 0.78 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:46:41,789 Completed processing <Batch: 7 messages, open for 0.75 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:46:42,608 Completed processing <Batch: 8 messages, open for 0.82 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:46:43,519 Completed processing <Batch: 12 messages, open for 0.91 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:46:44,547 Completed processing <Batch: 13 messages, open for 1.03 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:46:45,323 Completed processing <Batch: 8 messages, open for 0.78 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:46:46,087 Completed processing <Batch: 14 messages, open for 0.76 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:46:46,882 Completed processing <Batch: 11 messages, open for 0.79 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:46:47,679 Completed processing <Batch: 14 messages, open for 0.80 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:46:48,451 Completed processing <Batch: 12 messages, open for 0.77 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:46:49,356 Completed processing <Batch: 16 messages, open for 0.90 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:46:50,416 Completed processing <Batch: 4 messages, open for 1.06 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:46:51,173 Completed processing <Batch: 11 messages, open for 0.75 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:46:51,945 Completed processing <Batch: 12 messages, open for 0.77 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:46:52,711 Completed processing <Batch: 9 messages, open for 0.76 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:46:53,562 Completed processing <Batch: 17 messages, open for 0.85 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:46:54,488 Completed processing <Batch: 15 messages, open for 0.93 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:46:55,380 Completed processing <Batch: 11 messages, open for 0.89 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:46:56,605 Completed processing <Batch: 12 messages, open for 1.22 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:46:57,494 Completed processing <Batch: 13 messages, open for 0.89 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:46:58,634 Completed processing <Batch: 12 messages, open for 1.14 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:46:59,419 Completed processing <Batch: 15 messages, open for 0.78 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:47:00,185 Completed processing <Batch: 2 messages, open for 0.77 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:47:01,248 Completed processing <Batch: 11 messages, open for 1.06 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:47:02,571 Completed processing <Batch: 12 messages, open for 1.32 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:47:03,359 Completed processing <Batch: 7 messages, open for 0.79 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:47:04,171 Completed processing <Batch: 4 messages, open for 0.81 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:47:05,006 Completed processing <Batch: 16 messages, open for 0.83 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:47:05,810 Completed processing <Batch: 4 messages, open for 0.80 seconds>.
snuba-transactions-consumer_1               | 2021-06-25 08:47:06,948 Completed processing <Batch: 11 messages, open for 1.14 seconds>.
subscription-consumer-transactions_1        |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
subscription-consumer-transactions_1        |     return _process_result(sub_ctx.command.invoke(sub_ctx))
subscription-consumer-transactions_1        |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
subscription-consumer-transactions_1        |     return ctx.invoke(self.callback, **ctx.params)
subscription-consumer-transactions_1        |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
subscription-consumer-transactions_1        |     return callback(*args, **kwargs)
subscription-consumer-transactions_1        |   File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
subscription-consumer-transactions_1        |     return f(get_current_context(), *args, **kwargs)
subscription-consumer-transactions_1        |   File "/usr/local/lib/python3.6/site-packages/sentry/runner/decorators.py", line 69, in inner
subscription-consumer-transactions_1        |     return ctx.invoke(f, *args, **kwargs)
subscription-consumer-transactions_1        |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
subscription-consumer-transactions_1        |     return callback(*args, **kwargs)
subscription-consumer-transactions_1        |   File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
subscription-consumer-transactions_1        |     return f(get_current_context(), *args, **kwargs)
subscription-consumer-transactions_1        |   File "/usr/local/lib/python3.6/site-packages/sentry/runner/decorators.py", line 29, in inner
subscription-consumer-transactions_1        |     return ctx.invoke(f, *args, **kwargs)
subscription-consumer-transactions_1        |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
subscription-consumer-transactions_1        |     return callback(*args, **kwargs)
subscription-consumer-transactions_1        |   File "/usr/local/lib/python3.6/site-packages/sentry/runner/commands/run.py", line 388, in query_subscription_consumer
subscription-consumer-transactions_1        |     subscriber.run()
subscription-consumer-transactions_1        |   File "/usr/local/lib/python3.6/site-packages/sentry/snuba/query_subscription_consumer.py", line 160, in run
subscription-consumer-transactions_1        |     wait_for_topics(admin_client, [self.topic])
subscription-consumer-transactions_1        |   File "/usr/local/lib/python3.6/site-packages/sentry/utils/batching_kafka_consumer.py", line 37, in wait_for_topics
subscription-consumer-transactions_1        |     f"Timeout when waiting for Kafka topic '{topic}' to become available, last error: {last_error}"
subscription-consumer-transactions_1        | RuntimeError: Timeout when waiting for Kafka topic 'transactions-subscription-results' to become available, last error: KafkaError{code=LEADER_NOT_AVAILABLE,val=5,str="Broker: Leader not available"}
subscription-consumer-transactions_1        | 08:15:28 [INFO] sentry.plugins.github: apps-not-configured
subscription-consumer-transactions_1        | 08:15:31 [INFO] sentry.snuba.query_subscription_consumer: query-subscription-consumer.on_assign (offsets='{0: None}' partitions='[TopicPartition{topic=transactions-subscription-results,partition=0,offset=-1001,error=None}]')
subscription-consumer-transactions_1        | 08:15:37 [INFO] sentry.snuba.query_subscription_consumer: query-subscription-consumer.commit_offsets (offsets='{0: None}' partitions='[0]')
subscription-consumer-transactions_1        | 08:15:37 [INFO] sentry.snuba.query_subscription_consumer: query-subscription-consumer.on_revoke (offsets='{}' partitions='[TopicPartition{topic=transactions-subscription-results,partition=0,offset=-1001,error=None}]')
subscription-consumer-transactions_1        | 08:15:37 [INFO] sentry.snuba.query_subscription_consumer: query-subscription-consumer.on_assign (offsets='{0: None}' partitions='[TopicPartition{topic=transactions-subscription-results,partition=0,offset=-1001,error=None}]')
symbolicator-cleanup_1                      | SHELL=/bin/bash
symbolicator-cleanup_1                      | BASH_ENV=/container.env
symbolicator-cleanup_1                      | 55 23 * * * gosu symbolicator symbolicator cleanup > /proc/1/fd/1 2>/proc/1/fd/2
symbolicator-cleanup_1                      | SHELL=/bin/bash
symbolicator-cleanup_1                      | BASH_ENV=/container.env
symbolicator-cleanup_1                      | 55 23 * * * gosu symbolicator symbolicator cleanup > /proc/1/fd/1 2>/proc/1/fd/2
symbolicator-cleanup_1                      | SHELL=/bin/bash
symbolicator-cleanup_1                      | BASH_ENV=/container.env
symbolicator-cleanup_1                      | 55 23 * * * gosu symbolicator symbolicator cleanup > /proc/1/fd/1 2>/proc/1/fd/2
web_1                                       | 08:17:29 [WARNING] django.request: Not Found: /api/0/issues/172815/events/latest/ (status_code=404 request=<WSGIRequest: GET '/api/0/issues/172815/events/latest/'>)
web_1                                       | 08:23:36 [WARNING] django.request: Not Found: /api/0/issues/172815/events/latest/ (status_code=404 request=<WSGIRequest: GET '/api/0/issues/172815/events/latest/'>)
web_1                                       | 08:37:00 [WARNING] django.request: Not Found: /api/0/issues/172650/events/latest/ (status_code=404 request=<WSGIRequest: GET '/api/0/issues/172650/events/latest/'>)
web_1                                       | 08:37:01 [WARNING] django.request: Not Found: /api/0/issues/172650/events/latest/ (status_code=404 request=<WSGIRequest: GET '/api/0/issues/172650/events/latest/'>)
web_1                                       | 08:41:23 [WARNING] django.request: Not Found: /api/0/issues/6596/events/latest/ (status_code=404 request=<WSGIRequest: GET '/api/0/issues/6596/events/latest/'>)
web_1                                       | 08:41:27 [WARNING] django.request: Not Found: /api/0/issues/6596/events/latest/ (status_code=404 request=<WSGIRequest: GET '/api/0/issues/6596/events/latest/'>)
web_1                                       | 08:41:30 [WARNING] django.request: Not Found: /api/0/issues/6596/events/latest/ (status_code=404 request=<WSGIRequest: GET '/api/0/issues/6596/events/latest/'>)
web_1                                       | 08:41:36 [WARNING] django.request: Not Found: /api/0/issues/120323/events/latest/ (status_code=404 request=<WSGIRequest: GET '/api/0/issues/120323/events/latest/'>)
web_1                                       | 08:41:36 [WARNING] django.request: Not Found: /api/0/issues/120323/events/latest/ (status_code=404 request=<WSGIRequest: GET '/api/0/issues/120323/events/latest/'>)
web_1                                       | 08:41:41 [WARNING] django.request: Not Found: /api/0/issues/120669/events/latest/ (status_code=404 request=<WSGIRequest: GET '/api/0/issues/120669/events/latest/'>)
web_1                                       | 08:41:41 [WARNING] django.request: Not Found: /api/0/issues/5508/events/latest/ (status_code=404 request=<WSGIRequest: GET '/api/0/issues/5508/events/latest/'>)
web_1                                       | 08:41:41 [WARNING] django.request: Not Found: /api/0/issues/147958/events/latest/ (status_code=404 request=<WSGIRequest: GET '/api/0/issues/147958/events/latest/'>)
web_1                                       | 08:41:41 [WARNING] django.request: Not Found: /api/0/issues/147958/events/latest/ (status_code=404 request=<WSGIRequest: GET '/api/0/issues/147958/events/latest/'>)
web_1                                       | 08:41:42 [WARNING] django.request: Not Found: /api/0/issues/147958/events/latest/ (status_code=404 request=<WSGIRequest: GET '/api/0/issues/147958/events/latest/'>)
web_1                                       | 08:41:43 [WARNING] django.request: Not Found: /api/0/issues/147958/events/latest/ (status_code=404 request=<WSGIRequest: GET '/api/0/issues/147958/events/latest/'>)
web_1                                       | 08:41:47 [WARNING] django.request: Not Found: /api/0/issues/120554/events/latest/ (status_code=404 request=<WSGIRequest: GET '/api/0/issues/120554/events/latest/'>)
web_1                                       | 08:41:48 [WARNING] django.request: Not Found: /api/0/issues/120554/events/latest/ (status_code=404 request=<WSGIRequest: GET '/api/0/issues/120554/events/latest/'>)
web_1                                       | 08:41:48 [WARNING] django.request: Not Found: /api/0/issues/120554/events/latest/ (status_code=404 request=<WSGIRequest: GET '/api/0/issues/120554/events/latest/'>)
web_1                                       | 08:41:49 [WARNING] django.request: Not Found: /api/0/issues/120554/events/latest/ (status_code=404 request=<WSGIRequest: GET '/api/0/issues/120554/events/latest/'>)
web_1                                       | 08:42:00 [WARNING] django.request: Not Found: /api/0/issues/120554/events/latest/ (status_code=404 request=<WSGIRequest: GET '/api/0/issues/120554/events/latest/'>)
web_1                                       | [uwsgi-http key: sentry.example.net client_addr: 192.168.64.28 client_port: 62089] hr_write(): Broken pipe [plugins/http/http.c line 565]
web_1                                       | [uwsgi-http key: sentry.example.net client_addr: 192.168.64.28 client_port: 138] hr_write(): Broken pipe [plugins/http/http.c line 565]
web_1                                       | 08:46:13 [WARNING] django.request: Not Found: /api/0/issues/6596/events/latest/ (status_code=404 request=<WSGIRequest: GET '/api/0/issues/6596/events/latest/'>)
web_1                                       | 08:46:14 [WARNING] django.request: Not Found: /api/0/issues/6596/events/latest/ (status_code=404 request=<WSGIRequest: GET '/api/0/issues/6596/events/latest/'>)
web_1                                       | 08:46:38 [WARNING] django.request: Not Found: /api/0/issues/6596/events/latest/ (status_code=404 request=<WSGIRequest: GET '/api/0/issues/6596/events/latest/'>)
web_1                                       | [uwsgi-http key: sentry.example.net client_addr: 192.168.64.28 client_port: 55017] hr_write(): Broken pipe [plugins/http/http.c line 565]
web_1                                       | 08:46:41 [WARNING] django.request: Not Found: /api/0/issues/6596/events/latest/ (status_code=404 request=<WSGIRequest: GET '/api/0/issues/6596/events/latest/'>)
web_1                                       | [uwsgi-http key: sentry.example.net client_addr: 192.168.64.28 client_port: 29930] hr_write(): Broken pipe [plugins/http/http.c line 565]
web_1                                       | [uwsgi-http key: sentry.example.net client_addr: 192.168.64.28 client_port: 36586] hr_write(): Broken pipe [plugins/http/http.c line 565]
web_1                                       | 08:46:42 [WARNING] django.request: Not Found: /api/0/issues/6596/events/latest/ (status_code=404 request=<WSGIRequest: GET '/api/0/issues/6596/events/latest/'>)
zookeeper_1                                 | CONFLUENT_PLATFORM_LABEL=
zookeeper_1                                 | CONFLUENT_SUPPORT_METRICS_ENABLE=false
zookeeper_1                                 | CONFLUENT_VERSION=5.5.0
zookeeper_1                                 | CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar
zookeeper_1                                 | HOME=/root
zookeeper_1                                 | HOSTNAME=cc72a30bf025
zookeeper_1                                 | KAFKA_VERSION=
zookeeper_1                                 | LANG=C.UTF-8
zookeeper_1                                 | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
zookeeper_1                                 | PWD=/
zookeeper_1                                 | PYTHON_PIP_VERSION=8.1.2
zookeeper_1                                 | PYTHON_VERSION=2.7.9-1
zookeeper_1                                 | SCALA_VERSION=2.12
zookeeper_1                                 | SHLVL=1
zookeeper_1                                 | ZOOKEEPER_CLIENT_PORT=2181
zookeeper_1                                 | ZOOKEEPER_LOG4J_ROOT_LOGLEVEL=WARN
zookeeper_1                                 | ZOOKEEPER_TOOLS_LOG4J_LOGLEVEL=WARN
zookeeper_1                                 | ZULU_OPENJDK_VERSION=8=8.38.0.13
zookeeper_1                                 | _=/usr/bin/env
zookeeper_1                                 | ===> User
zookeeper_1                                 | uid=0(root) gid=0(root) groups=0(root)
zookeeper_1                                 | ===> Configuring ...
zookeeper_1                                 | ===> Running preflight checks ...
zookeeper_1                                 | ===> Check if /var/lib/zookeeper/data is writable ...
zookeeper_1                                 | ===> Check if /var/lib/zookeeper/log is writable ...
zookeeper_1                                 | ===> Launching ...
zookeeper_1                                 | ===> Launching zookeeper ...
zookeeper_1                                 | [2021-06-25 08:13:41,614] WARN Either no config or no quorum defined in config, running  in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
zookeeper_1                                 | [2021-06-25 08:13:41,905] WARN o.e.j.s.ServletContextHandler@4d95d2a2{/,null,UNAVAILABLE} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler)
zookeeper_1                                 | [2021-06-25 08:13:41,905] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler)
worker_1                                    |     self, "Failed to establish a new connection: %s" % e)
worker_1                                    | urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7faf90057710>: Failed to establish a new connection: [Errno 101] Network is unreachable
worker_1                                    |
worker_1                                    | During handling of the above exception, another exception occurred:
worker_1                                    |
worker_1                                    | Traceback (most recent call last):
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/sentry_sdk/transport.py", line 326, in send_event_wrapper
worker_1                                    |     self._send_event(event)
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/sentry_sdk/transport.py", line 234, in _send_event
worker_1                                    |     headers={"Content-Type": "application/json", "Content-Encoding": "gzip"},
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/sentry_sdk/transport.py", line 180, in _send_request
worker_1                                    |     headers=headers,
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/urllib3/request.py", line 72, in request
worker_1                                    |     **urlopen_kw)
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/urllib3/request.py", line 150, in request_encode_body
worker_1                                    |     return self.urlopen(method, url, **extra_kw)
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/urllib3/poolmanager.py", line 324, in urlopen
worker_1                                    |     response = conn.urlopen(method, u.request_uri, **kw)
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 667, in urlopen
worker_1                                    |     **response_kw)
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 667, in urlopen
worker_1                                    |     **response_kw)
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 667, in urlopen
worker_1                                    |     **response_kw)
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
worker_1                                    |     _stacktrace=sys.exc_info()[2])
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment
worker_1                                    |     raise MaxRetryError(_pool, url, error or ResponseError(cause))
worker_1                                    | urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='sentry.example.com', port=443): Max retries exceeded with url: /api/1/store/ (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7faf90057710>: Failed to establish a new connection: [Errno 101] Network is unreachable',))
worker_1                                    | 08:46:58 [ERROR] sentry_sdk.errors: Internal error in sentry_sdk
kaneruan commented 3 years ago

any one help?

chadwhitacre commented 3 years ago

We're missing some repro steps after docker-compose up -d. You tried to send events to your Sentry install and they're not showing up? How did you try to send the events?

I am seeing some network connectivity issues in the logfile you shared, but without more info on the steps you're following I'm not sure what matters and what's a red herring.

BYK commented 3 years ago

It simply looks like your server is quite overloaded and keeps processing through a backlog. I advice you to leave it running for a while and try to minimize any new events coming in to see whether it stabilizes.

djbe commented 3 years ago

I have the same issue. My kafka & zookeeper services keep restarting.

Kafka logs (this endlessly repeats):

===> User uid=1000(appuser) gid=1000(appuser) groups=1000(appuser) ===> Configuring ... ===> Running preflight checks ... ===> Check if /var/lib/kafka/data is writable ... ===> Check if Zookeeper is healthy ... SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/share/java/cp-base-new/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/share/java/cp-base-new/slf4j-simple-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] log4j:WARN No appenders could be found for logger (io.confluent.admin.utils.cli.ZookeeperReadyCommand). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

Zookeeper logs (also endlessly repeats):

===> User uid=1000(appuser) gid=1000(appuser) groups=1000(appuser) ===> Configuring ... ===> Running preflight checks ... ===> Check if /var/lib/zookeeper/data is writable ... Command [/usr/local/bin/dub path /var/lib/zookeeper/data writable] FAILED !

This all started happening after pulling in the latest repo changes (coming from commit 0cfaa73b0586018c667b0c8cafa2c605a7bf29f5) and running the install script again as usual (have done this many times).

BYK commented 3 years ago

Pretty sure you ran out of space @djbe

djbe commented 3 years ago

Pretty sure you ran out of space @djbe

Hmmm no, disk use is at 45%

BYK commented 3 years ago

@djbe then I think your issue might be related to #1009. Can you try my suggestion there?

djbe commented 3 years ago

Can confirm that your suggestion there fixes this issue 🎉

BYK commented 3 years ago

Looks like the Kafka upgrade was not smooth. I'll dig into this.

BYK commented 3 years ago

Folding this into #1009 as they seem to be the same and #1009 is a clearer report.