Closed kargh closed 1 year ago
kargh
commented
2 years ago Just to be clear, since that but report is quite long:
piyushredpanda
commented
2 years ago @kargh : Thanks for the report, we are looking into it. To clarify though, we do not support a downgrade.
Also, presumably this is a 1-node cluster -- is that right?
kargh
commented
2 years ago It is a 10 node cluster.
I didn't have much of a choice as far as downgrading, I wanted to get the node back online.
On Wed, Nov 16, 2022, 5:35 PM piyushredpanda @.***> wrote:
@kargh https://github.com/kargh : Thanks for the report, we are looking into it. To clarify though, we do not support a downgrade.
Also, presumably this is a 1-node cluster -- is that right?
— Reply to this email directly, view it on GitHub https://github.com/redpanda-data/redpanda/issues/7343#issuecomment-1317821961, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB3DY7A2T6LXPRS4H7NY4E3WIVVT7ANCNFSM6AAAAAASCWSIZY . You are receiving this because you were mentioned.Message ID: @.***>
emaxerrno
commented
2 years ago @piyushredpanda - i don't think that matters because the cluster never actually upgraded, so the global barrier shouldn't come into play.
piyushredpanda
commented
2 years ago @emaxerrno, yes @kargh also mentioned they downgraded the cluster:
Downgraded from 22.3.2 to 22.3.1
which is what I was trying to clarify we do not support, for future reference.
andrwng
commented
2 years ago Just to clarify the signals here, @kargh do you know roughly when you started upgrading the first node? Looks like Nov 16 15:35:47 is when things started going bad with the upgrade, but looks like there's a bit more segfaulting that had been happening a few days ago.
How many topics and partitions are running on this cluster? How long was this node down before being upgraded? Do you see similar symptoms on other nodes?
Also, do you happen to have the logs starting from before the node started going bad?
travisdowns
commented
2 years ago @kargh - for the errors you show immediately after you write "From dmesg:": on what repdanda version did those occur?
kargh
commented
2 years ago @travisdowns Yeah, looking at the timestamps, some of those are from prior to the upgrade. That would have been on v22.2.7. The opcode errors would have been from v22.3.2.
kargh
commented
2 years ago @andrwng That error message happened right after I upgraded the node and tried to start it -- within 5 minutes of running the yum update command.
The original issue Alex was helping me with was:
Nov 16 15:17:39 srv-01-11-301.iad1.trmr.io rpk[55700]: WARN 2022-11-16 15:17:39,429 [shard 59] seastar - Exceptional future ignored: std::bad_alloc (std::bad_alloc), backtrace: 0x4d1d6c4 0x4a1298b 0x3dfb960 0x4ace59f 0x4ad2277 0x4b15745 0x4a6f10f /opt/redpanda/lib/libc.so.6+0x91016 /opt/redpanda/lib/libc.so.6+0x1166cf
Nov 16 15:17:39 srv-01-11-301.iad1.trmr.io rpk[55700]: --------
Nov 16 15:17:39 srv-01-11-301.iad1.trmr.io rpk[55700]: seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void> seastar::future<void>::handle_exception_type<auto ssx::spawn_with_gate_then<cluster::persisted_stm::make_snapshot_in_background()::$_3>(seastar::gate&, cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(seastar::broken_condition_variable const&)>(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(cluster::persisted_stm::make_snapshot_in_background()::$_3&&), seastar::futurize<cluster::persisted_stm::make_snapshot_in_background()::$_3>::type seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void> seastar::future<void>::handle_exception_type<auto ssx::spawn_with_gate_then<cluster::persisted_stm::make_snapshot_in_background()::$_3>(seastar::gate&, cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(seastar::broken_condition_variable const&)>(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)>(seastar::future<void> seastar::future<void>::handle_exception_type<auto ssx::spawn_with_gate_then<cluster::persisted_stm::make_snapshot_in_background()::$_3>(seastar::gate&, cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(seastar::broken_condition_variable const&)>(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::future<void> seastar::future<void>::handle_exception_type<auto ssx::spawn_with_gate_then<cluster::persisted_stm::make_snapshot_in_background()::$_3>(seastar::gate&, cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(seastar::broken_condition_variable const&)>(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)&, seastar::future_state<seastar::internal::monostate>&&), void>He had suggested I upgrade to 22.3.2 because there was a memory issue bug fixed recently.
TOPICS 53 TOTAL PARTITIONS 1,638
The node wasn't down when I upgraded.
I see the above warning (seastar - Exceptional future ignored) on other nodes. That is what I was trying to resolve across the cluster.
I just have the journal logs from other boxes. I've not tried to upgrade them at this point. I prefer to resolve this problem first before encountering it on other nodes.
kargh
commented
2 years ago Upgraded to 22.3.3 and here is the full log from when I started the service until it failed:
-- Logs begin at Thu 2022-11-17 08:45:50 EST. --
Nov 17 08:48:10 srv-01-03-404.iad1.trmr.io systemd[1]: Starting Redpanda, the fastest queue in the West....
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: System check 'Dir '/var/lib/redpanda/data' IRQs affinity set' failed. Required: true, Current false
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: System check 'NTP Synced' failed. Required: true, Current false
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: System check - PASSED
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: We'd love to hear about your experience with redpanda:
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: https://redpanda.com/feedback
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: Starting redpanda...
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: Running:
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: SHELL=/sbin/nologin USER=redpanda PATH=/opt/redpanda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin PWD=/ LANG=en_US.UTF-8 NOTIFY_SOCKET=/run/systemd/notify SHLVL=0 HOME=/var/lib/redpanda LOGNAME=redpanda START_ARGS=--check=true /opt/redpanda/bin/redpanda redpanda --redpanda-cfg /etc/redpanda/redpanda.yaml --lock-memory=false
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: Welcome to the Redpanda community!
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: Documentation: https://docs.redpanda.com - Product documentation site
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: GitHub Discussion: https://github.com/redpanda-data/redpanda/discussions - Longer, more involved discussions
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: GitHub Issues: https://github.com/redpanda-data/redpanda/issues - Report and track issues with the codebase
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: Support: https://support.redpanda.com - Contact the support team privately
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: Product Feedback: https://redpanda.com/feedback - Let us know how we can improve your experience
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: Slack: https://redpanda.com/slack - Chat about all things Redpanda. Join the conversation!
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: Twitter: https://twitter.com/redpandadata - All the latest Redpanda news!
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,740 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,740 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,740 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,741 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,829 [shard 0] main - application.cc:299 - Redpanda v22.3.3 - 41a324dfcf816dbf656de83744146e89ac3c2583-dirty
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,830 [shard 0] main - application.cc:307 - kernel=6.0.9-1.el7.elrepo.x86_64, nodename=srv-01-03-404.iad1.trmr.io, machine=x86_64
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,830 [shard 0] main - application.cc:235 - System resources: { cpus: 64, available memory: 233.625GiB, reserved memory: 16.354GiB}
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,830 [shard 0] main - application.cc:243 - File handle limit: 800000/800000
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,852 [shard 0] main - application.cc:539 - Cluster configuration properties:
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,852 [shard 0] main - application.cc:540 - (use `rpk cluster config edit` to change)
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.abort_index_segment_size:50000 - Capacity (in number of txns) of an abort index segment
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.abort_timed_out_transactions_interval_ms:10000 - How often look for the inactive transactions and abort them
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.admin_api_require_auth:0 - Whether admin API clients must provide HTTP Basic authentication headers
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.aggregate_metrics:0 - Enable aggregations of metrics returned by the prometheus '/metrics' endpoint. Metric aggregation is performed by summing the values of samples by labels. Aggregations are performed where it makes sense by the shard and/or partition labels.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.alter_topic_cfg_timeout_ms:5000 - Time to wait for entries replication in controller log when executing alter configuration requst
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.append_chunk_size:16384 - Size of direct write operations to disk in bytes
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.auto_create_topics_enabled:0 - Allow topic auto creation
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_access_key:{nullopt} - AWS access key
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_api_endpoint:{nullopt} - Optional API endpoint
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_api_endpoint_port:443 - TLS port override
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_bucket:{nullopt} - AWS bucket that should be used to store data
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_cache_check_interval:30000 - Timeout to check if cache eviction should be triggered
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_cache_size:21474836480 - Max size of archival cache
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_credentials_source:config_file - The source of credentials to connect to cloud services
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_disable_tls:0 - Disable TLS for all S3 connections
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_enable_compacted_topic_reupload:1 - Enable re-uploading data for compacted topics
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_enable_remote_read:0 - Default remote read config value for new topics
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_enable_remote_write:0 - Default remote write value for new topics
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_enabled:0 - Enable archival storage
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_housekeeping_interval_ms:300000 - Interval for cloud storage housekeeping tasks
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_initial_backoff_ms:100 - Initial backoff time for exponetial backoff algorithm (ms)
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_manifest_upload_timeout_ms:10000 - Manifest upload timeout (ms)
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_max_connection_idle_time_ms:5000 - Max https connection idle time (ms)
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_max_connections:20 - Max number of simultaneous connections to S3 per shard (includes connections used for both uploads and downloads)
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_max_materialized_segments_per_shard:{nullopt} - Maximum concurrent readers of remote data per CPU core. If unset, value of `topic_partitions_per_shard` multiplied by 2 is used.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_max_readers_per_shard:{nullopt} - Maximum concurrent readers of remote data per CPU core. If unset, value of `topic_partitions_per_shard` is used, i.e. one reader per partition if the shard is at its maximum partition capacity.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_max_segments_pending_deletion_per_partition:5000 - The per-partition limit for the number of segments pending deletion from the cloud. Segments can be deleted due to retention or compaction. If this limit is breached and deletion fails, then segments will be orphaned in the cloud and will have to be removed manually
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_metadata_sync_timeout_ms:10000 - Timeout for SI metadata synchronization
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_readreplica_manifest_sync_timeout_ms:30000 - Timeout to check if new data is available for partition in S3 for read replica
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_reconciliation_interval_ms:1000 - Interval at which the archival service runs reconciliation (ms)
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_region:{nullopt} - AWS region that houses the bucket used for storage
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_secret_key:{nullopt} - AWS secret key
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_segment_max_upload_interval_sec:{nullopt} - Time that segment can be kept locally without uploading it to the remote storage (sec)
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_segment_upload_timeout_ms:30000 - Log segment upload timeout (ms)
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_trust_file:{nullopt} - Path to certificate that should be used to validate server certificate during TLS handshake
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_upload_ctrl_d_coeff:0 - derivative coefficient for upload PID controller.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_upload_ctrl_max_shares:1000 - maximum number of IO and CPU shares that archival upload can use
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_upload_ctrl_min_shares:100 - minimum number of IO and CPU shares that archival upload can use
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_upload_ctrl_p_coeff:-2 - proportional coefficient for upload PID controller
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_upload_ctrl_update_interval_ms:60000 -
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_upload_loop_initial_backoff_ms:100 - Initial backoff interval when there is nothing to upload for a partition (ms)
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cloud_storage_upload_loop_max_backoff_ms:10000 - Max backoff interval when there is nothing to upload for a partition (ms)
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.cluster_id:{11292c39-9267-4cd3-9d74-d7f63648aee0} - Cluster identifier
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.compacted_log_segment_size:268435456 - How large in bytes should each compacted log segment be (default 256MiB)
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.compaction_ctrl_backlog_size:{nullopt} - target backlog size for compaction controller. if not set compaction target compaction backlog would be equal to
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.compaction_ctrl_d_coeff:0.2 - derivative coefficient for compaction PID controller.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.compaction_ctrl_i_coeff:0 - integral coefficient for compaction PID controller.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.compaction_ctrl_max_shares:1000 - maximum number of IO and CPU shares that compaction process can use
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.compaction_ctrl_min_shares:10 - minimum number of IO and CPU shares that compaction process can use
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.compaction_ctrl_p_coeff:-12.5 - proportional coefficient for compaction PID controller. This has to be negative since compaction backlog should decrease when number of compaction shares increases
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.compaction_ctrl_update_interval_ms:30000 -
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.controller_backend_housekeeping_interval_ms:1000 - Interval between iterations of controller backend housekeeping loop
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.controller_log_accummulation_rps_capacity_acls_and_users_operations:{nullopt} - Maximum capacity of rate limit accumulationin controller acls and users operations limit
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.controller_log_accummulation_rps_capacity_configuration_operations:{nullopt} - Maximum capacity of rate limit accumulationin controller configuration operations limit
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.controller_log_accummulation_rps_capacity_move_operations:{nullopt} - Maximum capacity of rate limit accumulationin controller move operations limit
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.controller_log_accummulation_rps_capacity_node_management_operations:{nullopt} - Maximum capacity of rate limit accumulationin controller node management operations limit
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.controller_log_accummulation_rps_capacity_topic_operations:{nullopt} - Maximum capacity of rate limit accumulationin controller topic operations limit
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.coproc_max_batch_size:32768 - Maximum amount of bytes to read from one topic read
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.coproc_max_inflight_bytes:10485760 - Maximum amountt of inflight bytes when sending data to wasm engine
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.coproc_max_ingest_bytes:655360 - Maximum amount of data to hold from input logs in memory
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.coproc_offset_flush_interval_ms:300000 - Interval for which all coprocessor offsets are flushed to disk
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.create_topic_timeout_ms:2000 - Timeout (ms) to wait for new topic creation
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.default_num_windows:10 - Default number of quota tracking windows
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.default_topic_partitions:10 - Default number of partitions per topic
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.default_topic_replications:3 - Default replication factor for new topics
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.default_window_sec:1000 - Default quota tracking window size in milliseconds
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.delete_retention_ms:86400000 - delete segments older than this - default 1 week
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.disable_batch_cache:0 - Disable batch cache in log manager
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.disable_metrics:0 - Disable registering metrics exposed on the internal metrics endpoint (/metrics)
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.disable_public_metrics:0 - Disable registering metrics exposed on the public metrics endpoint (/public_metrics)
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.election_timeout_ms:1500 - Election timeout expressed in milliseconds
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.enable_admin_api: -
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.enable_auto_rebalance_on_node_add:0 - Enable automatic partition rebalancing when new nodes are added
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,853 [shard 0] main - application.cc:515 - redpanda.enable_controller_log_rate_limiting:0 - Enables limiting of controller log write rate
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.enable_coproc:0 - Enable coprocessing mode
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.enable_idempotence:1 - Enable idempotent producer
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.enable_leader_balancer:1 - Enable automatic leadership rebalancing
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.enable_metrics_reporter:1 - Enable cluster metrics reporter
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.enable_pid_file:1 - Enable pid file. You probably don't want to change this.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.enable_rack_awareness:0 - Enables rack-aware replica assignment
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.enable_sasl:0 - Enable SASL authentication for Kafka connections, authorization is required. see also `kafka_enable_authorization`
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.enable_transactions:1 - Enable transactions
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.features_auto_enable:1 - Whether new feature flags may auto-activate after upgrades (true) or must wait for manual activation via the admin API (false)
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.fetch_max_bytes:57671680 - Maximum number of bytes returned in fetch request
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.fetch_reads_debounce_timeout:1 - Time to wait for next read in fetch request when requested min bytes wasn't reached
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.fetch_session_eviction_timeout_ms:60000 - Minimum time before which unused session will get evicted from sessions. Maximum time after which inactive session will be deleted is two time given configuration valuecache
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.full_raft_configuration_recovery_pattern:{} - Recover raft configuration on start for NTPs matching pattern
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.group_initial_rebalance_delay:3000 - Extra delay (ms) added to rebalance phase to wait for new members
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.group_max_session_timeout_ms:300000 - The maximum allowed session timeout for registered consumers. Longer timeouts give consumers more time to process messages in between heartbeats at the cost of a longer time to detect failures.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.group_min_session_timeout_ms:6000 - The minimum allowed session timeout for registered consumers. Shorter timeouts result in quicker failure detection at the cost of more frequent consumer heartbeating, which can overwhelm broker resources.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.group_new_member_join_timeout:30000 - Timeout for new member joins
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.group_topic_partitions:16 - Number of partitions in the internal group membership topic
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.health_manager_tick_interval:180000 - How often the health manager runs
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.health_monitor_max_metadata_age:10000 - Max age of metadata cached in the health monitor of non controller node
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.health_monitor_tick_interval:10000 - How often health monitor refresh cluster state
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.id_allocator_batch_size:1000 - Id allocator allocates messages in batches (each batch is a one log record) and then serves requests from memory without touching the log until the batch is exhausted.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.id_allocator_log_capacity:100 - Capacity of the id_allocator log in number of messages. Once it reached id_allocator_stm should compact the log.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.id_allocator_replication: -
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.internal_topic_replication_factor:3 - Target replication factor for internal topics
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.join_retry_timeout_ms:5000 - Time between cluster join retries in milliseconds
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.kafka_admin_topic_api_rate:{nullopt} - Target quota rate (partition mutations per default_window_sec)
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.kafka_batch_max_bytes:1048576 - Maximum size of a batch processed by server. If batch is compressed the limit applies to compressed batch size
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.kafka_connection_rate_limit:{nullopt} - Maximum connections per second for one core
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.kafka_connection_rate_limit_overrides:{} - Overrides for specific ips for maximum connections per second for one core
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.kafka_connections_max:{nullopt} - Maximum number of Kafka client connections per broker
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.kafka_connections_max_overrides:{} - Per-IP overrides of kafka connection count limit, list of <ip>:<count> strings
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.kafka_connections_max_per_ip:{nullopt} - Maximum number of Kafka client connections from each IP address, per broker
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.kafka_enable_authorization:{nullopt} - Enable authorization for Kafka connections. Values:- `nil`: Ignored. Authorization is enabled with `enable_sasl: true`; `true`: authorization is required; `false`: authorization is disabled. See also: `enable_sasl` and `kafka_api[].authentication_method`
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.kafka_group_recovery_timeout_ms:30000 - Kafka group recovery timeout expressed in milliseconds
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.kafka_max_bytes_per_fetch:67108864 - Limit fetch responses to this many bytes, even if total of partition bytes limits is higher
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.kafka_mtls_principal_mapping_rules:{{}} - Principal Mapping Rules for mTLS Authentication on the Kafka API
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.kafka_qdc_depth_alpha:0.8 - Smoothing factor for kafka queue depth control depth tracking.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.kafka_qdc_depth_update_ms:7000 - Update frequency for kafka queue depth control.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.kafka_qdc_enable:0 - Enable kafka queue depth control.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.kafka_qdc_idle_depth:10 - Queue depth when idleness is detected in kafka queue depth control.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.kafka_qdc_latency_alpha:0.002 - Smoothing parameter for kafka queue depth control latency tracking.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.kafka_qdc_max_depth:100 - Maximum queue depth used in kafka queue depth control.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.kafka_qdc_max_latency_ms:80 - Max latency threshold for kafka queue depth control depth tracking.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.kafka_qdc_min_depth:1 - Minimum queue depth used in kafka queue depth control.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.kafka_qdc_window_count:12 - Number of windows used in kafka queue depth control latency tracking.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.kafka_qdc_window_size_ms:1500 - Window size for kafka queue depth control latency tracking.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.kafka_request_max_bytes:104857600 - Maximum size of a single request processed via Kafka API
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.kafka_rpc_server_stream_recv_buf:{nullopt} - Userspace receive buffer max size in bytes
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.kafka_rpc_server_tcp_recv_buf:{nullopt} - Kafka server TCP receive buffer size in bytes.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.kafka_rpc_server_tcp_send_buf:{nullopt} - Kafka server TCP transmit buffer size in bytes.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.kvstore_flush_interval:10 - Key-value store flush interval (ms)
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.kvstore_max_segment_size:16777216 - Key-value maximum segment size (bytes)
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.leader_balancer_idle_timeout:120000 - Leadership rebalancing idle timeout
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.leader_balancer_mute_timeout:300000 - Leadership rebalancing mute timeout
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.leader_balancer_transfer_limit_per_shard:512 - Per shard limit for in progress leadership transfers
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.log_cleanup_policy:delete - Default topic cleanup policy
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.log_compaction_interval_ms:10000 - How often do we trigger background compaction
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.log_compression_type:producer - Default topic compression type
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.log_message_timestamp_type:CreateTime - Default topic messages timestamp type
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.log_segment_size:104857600 - Default log segment size in bytes for topics which do not set segment.bytes
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.log_segment_size_jitter_percent:5 - Random variation to the segment size limit used for each partition
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.log_segment_size_max:{nullopt} - Upper bound on topic segment.bytes: higher values will be clamped to this limit
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.log_segment_size_min:{nullopt} - Lower bound on topic segment.bytes: lower values will be clamped to this limit
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.max_compacted_log_segment_size:5368709120 - Max compacted segment size after consolidation
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.max_kafka_throttle_delay_ms:60000 - Fail-safe maximum throttle delay on kafka requests
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.max_version: -
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.members_backend_retry_ms:5000 - Time between members backend reconciliation loop retries
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.metadata_dissemination_interval_ms:3000 - Interaval for metadata dissemination batching
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.metadata_dissemination_retries:30 - Number of attempts of looking up a topic's meta data like shard before failing a request
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.metadata_dissemination_retry_delay_ms:320 - Delay before retry a topic lookup in a shard or other meta tables
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.metadata_status_wait_timeout_ms:2000 - Maximum time to wait in metadata request for cluster health to be refreshed
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.metrics_reporter_report_interval:86400000 - cluster metrics reporter report interval
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.metrics_reporter_tick_interval:60000 - Cluster metrics reporter tick interval
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.metrics_reporter_url:https://m.rp.vectorized.io/v2 - cluster metrics reporter url
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.min_version: -
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.node_management_operation_timeout_ms:5000 - Timeout for executing node management operations
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.node_status_interval:100 - Time interval between two node status messages. Node status messages establish liveness status outside of the Raft protocol.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.partition_autobalancing_concurrent_moves:50 - Number of partitions that can be reassigned at once
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.partition_autobalancing_max_disk_usage_percent:80 - Disk usage threshold that triggers moving partitions from the node
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.partition_autobalancing_mode:node_add - Partition autobalancing mode
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.partition_autobalancing_movement_batch_size_bytes:5368709120 - Total size of partitions that autobalancer is going to move in one batch
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.partition_autobalancing_node_availability_timeout_sec:900000 - Node unavailability timeout that triggers moving partitions from the node
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.partition_autobalancing_tick_interval_ms:30000 - Partition autobalancer tick interval
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.quota_manager_gc_sec:30000 - Quota manager GC frequency in milliseconds
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.raft_heartbeat_disconnect_failures:3 - After how many failed heartbeats to forcibly close an unresponsive TCP connection. Set to 0 to disable force disconnection.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.raft_heartbeat_interval_ms:150 - Milliseconds for raft leader heartbeats
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.raft_heartbeat_timeout_ms:3000 - raft heartbeat RPC timeout
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.raft_io_timeout_ms:10000 - Raft I/O timeout
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.raft_learner_recovery_rate:104857600 - Raft learner recovery rate limit in bytes per sec
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.raft_max_concurrent_append_requests_per_follower:16 - Maximum number of concurrent append entries requests sent by leader to one follower
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.raft_max_recovery_memory:{nullopt} - Max memory that can be used for reads in raft recovery process by default 15% of total memory
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.raft_recovery_default_read_size:524288 - default size of read issued during raft follower recovery
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.raft_replicate_batch_window_size:1048576 - Max size of requests cached for replication
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.raft_smp_max_non_local_requests:{nullopt} - Maximum number of x-core requests pending in Raft seastar::smp group. (for more details look at `seastar::smp_service_group` documentation)
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.raft_timeout_now_timeout_ms:1000 - Timeout for a timeout now request
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.raft_transfer_leader_recovery_timeout_ms:10000 - Timeout waiting for follower recovery when transferring leadership
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.readers_cache_eviction_timeout_ms:30000 - Duration after which inactive readers will be evicted from cache
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.reclaim_batch_cache_min_free:67108864 - Free memory limit that will be kept by batch cache background reclaimer
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.reclaim_growth_window:3000 - Length of time in which reclaim sizes grow
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.reclaim_max_size:4194304 - Maximum batch cache reclaim size
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.reclaim_min_size:131072 - Minimum batch cache reclaim size
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.reclaim_stable_window:10000 - Length of time above which growth is reset
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.recovery_append_timeout_ms:5000 - Timeout for append entries requests issued while updating stale follower
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.release_cache_on_segment_roll:0 - Free cache when segments roll
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.replicate_append_timeout_ms:3000 - Timeout for append entries requests issued while replicating entries
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.retention_bytes:{nullopt} - Default max bytes per partition on disk before triggering a compaction
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.retention_local_target_bytes_default:{nullopt} - Local retention size target for partitions of topics with cloud storage write enabled
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.retention_local_target_ms_default:86400000 - Local retention time target for partitions of topics with cloud storage write enabled
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.rm_sync_timeout_ms:10000 - Time to wait state catch up before rejecting a request
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.rm_violation_recovery_policy:crash - Describes how to recover from an invariant violation happened on the partition level
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.rpc_server_listen_backlog:{nullopt} - TCP connection queue length for Kafka server and internal RPC server
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.rpc_server_tcp_recv_buf:{nullopt} - Internal RPC TCP receive buffer size in bytes.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.rpc_server_tcp_send_buf:{nullopt} - Internal RPC TCP transmit buffer size in bytes.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.rps_limit_acls_and_users_operations:1000 - Rate limit for controller acls and users operations
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.rps_limit_configuration_operations:1000 - Rate limit for controller configuration operations
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.rps_limit_move_operations:1000 - Rate limit for controller move operations
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.rps_limit_node_management_operations:1000 - Rate limit for controller node management operations
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.rps_limit_topic_operations:1000 - Rate limit for controller topic operations
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.seed_server_meta_topic_partitions: -
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.segment_appender_flush_timeout_ms:1000 - Maximum delay until buffered data is written
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.segment_fallocation_step:33554432 - Size for segments fallocation
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.seq_table_min_size:1000 - Minimum size of the seq table non affected by compaction
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.storage_compaction_index_memory:134217728 - Maximum number of bytes that may be used on each shard by compactionindex writers
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.storage_max_concurrent_replay:1024 - Maximum number of partitions' logs that will be replayed concurrently at startup, or flushed concurrently on shutdown.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.storage_min_free_bytes:5368709120 - Threshold of minimum bytes free space before rejecting producers.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.storage_read_buffer_size:131072 - Size of each read buffer (one per in-flight read, per log segment)
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.storage_read_readahead_count:10 - How many additional reads to issue ahead of current read location
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.storage_space_alert_free_threshold_bytes:0 - Threshold of minimim bytes free space before setting storage space alert
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.storage_space_alert_free_threshold_percent:5 - Threshold of minimim percent free space before setting storage space alert
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.storage_strict_data_init:0 - Requires that an empty file named `.redpanda_data_dir` be present in the data directory. Redpanda will refuse to start if it is not found.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.storage_target_replay_bytes:10737418240 - Target bytes to replay from disk on startup after clean shutdown: controls frequency of snapshots and checkpoints
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.superusers:{} - List of superuser usernames
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.target_quota_byte_rate:2147483648 - Target quota byte rate (bytes per second) - 2GB default
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.tm_sync_timeout_ms:10000 - Time to wait state catch up before rejecting a request
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.tm_violation_recovery_policy:crash - Describes how to recover from an invariant violation happened on the transaction coordinator level
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.topic_fds_per_partition:{5} - Required file handles per partition when creating topics
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.topic_memory_per_partition:{1048576} - Required memory per partition when creating topics
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.topic_partitions_per_shard:7000 - Maximum number of partitions which may be allocated to one shard (CPU core)
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.topic_partitions_reserve_shard0:2 - Reserved partition slots on shard (CPU core) 0 on each node. If this is >= topic_partitions_per_core, no data partitions will be scheduled on shard 0
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.transaction_coordinator_cleanup_policy:delete - Cleanup policy for a transaction coordinator topic
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.transaction_coordinator_delete_retention_ms:86400000 - delete segments older than this - default 1 week
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.transaction_coordinator_log_segment_size:1073741824 - How large in bytes should each log segment be (default 1G)
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.transaction_coordinator_replication: -
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.transactional_id_expiration_ms:604800000 - Producer ids are expired once this time has elapsed after the last write with the given producer id.
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.tx_timeout_delay_ms:1000 - Delay before scheduling next check for timed out transactions
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.use_scheduling_groups: -
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.wait_for_leader_timeout_ms:5000 - Timeout (ms) to wait for leadership in metadata cache
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:515 - redpanda.zstd_decompress_workspace_bytes:8388608 - Size of the zstd decompression workspace
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:543 - Node configuration properties:
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,854 [shard 0] main - application.cc:544 - (use `rpk config set <cfg> <value>` to change)
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - redpanda.admin:{{:{host: 10.16.67.48, port: 9644}}} - Address and port of admin server
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - redpanda.admin_api_doc_dir:/usr/share/redpanda/admin-api-doc - Admin API doc directory
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - redpanda.admin_api_tls:{} - TLS configuration for admin HTTP server
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - redpanda.advertised_kafka_api:{} - Address of Kafka API published to the clients
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - redpanda.advertised_rpc_api:{nullopt} - Address of RPC endpoint published to other cluster members
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - redpanda.cloud_storage_cache_directory:{nullopt} - Directory for archival cache. Should be present when `cloud_storage_enabled` is present
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - redpanda.coproc_supervisor_server:{host: 127.0.0.1, port: 43189} - IpAddress and port for supervisor service
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - redpanda.dashboard_dir: -
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - redpanda.data_directory:{data_directory="/var/lib/redpanda/data"} - Place where redpanda will keep the data
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - redpanda.developer_mode:0 - Skips most of the checks performed at startup, not recomended for production use
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - redpanda.empty_seed_starts_cluster:1 - If enabled, requires exactly one node in a cluster-to-be to have its seed_servers list empty. Otherwise, seed_servers list cannot be empty, and must be the same in each node from that list
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - redpanda.enable_central_config: -
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - redpanda.kafka_api:{{:{host: 10.16.67.48, port: 9092}:{nullopt}}} - Address and port of an interface to listen for Kafka API requests
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - redpanda.kafka_api_tls:{} - TLS configuration for Kafka API endpoint
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - redpanda.node_id:{0} - Unique id identifying a node in the cluster. If missing, a unique id will be assigned for this node when it joins the cluster
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - redpanda.rack:{nullopt} - Rack identifier
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - redpanda.rpc_server:{host: 10.16.67.48, port: 33145} - IpAddress and port for RPC server
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - redpanda.rpc_server_tls:{ enabled: 0 key/cert files: {nullopt} ca file: {nullopt} client_auth_required: 0 } - TLS configuration for RPC server
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - redpanda.seed_servers:{} - List of the seed servers used to join current cluster. If the seed_server list is empty the node will be a cluster root and it will form a new cluster
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - pandaproxy.advertised_pandaproxy_api:{} - Rest API address and port to publish to client
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - pandaproxy.api_doc_dir:/usr/share/redpanda/proxy-api-doc - API doc directory
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - pandaproxy.client_cache_max_size:10 - The maximum number of kafka clients in the LRU cache
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - pandaproxy.client_keep_alive:300000 - Time in milliseconds that an idle connection may remain open
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - pandaproxy.consumer_instance_timeout_ms:300000 - How long to wait for an idle consumer before removing it
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - pandaproxy.pandaproxy_api:{{:{host: 0.0.0.0, port: 8082}:<nullopt>}} - Rest API listen address and port
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - pandaproxy.pandaproxy_api_tls:{} - TLS configuration for Pandaproxy api
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - pandaproxy_client.broker_tls:{ enabled: 0 key/cert files: {nullopt} ca file: {nullopt} client_auth_required: 0 } - TLS configuration for the brokers
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - pandaproxy_client.brokers:{{host: 10.16.67.48, port: 9092}} - List of address and port of the brokers
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - pandaproxy_client.client_identifier:{test_client} - Identifier to use within the kafka request header
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - pandaproxy_client.consumer_heartbeat_interval_ms:500 - Interval (in milliseconds) for consumer heartbeats
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - pandaproxy_client.consumer_rebalance_timeout_ms:2000 - Timeout (in milliseconds) for consumer rebalance
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - pandaproxy_client.consumer_request_max_bytes:1048576 - Max bytes to fetch per request
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - pandaproxy_client.consumer_request_timeout_ms:100 - Interval (in milliseconds) for consumer request timeout
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - pandaproxy_client.consumer_session_timeout_ms:300000 - Timeout (in milliseconds) for consumer session
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - pandaproxy_client.produce_batch_delay_ms:100 - Delay (in milliseconds) to wait before sending batch
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - pandaproxy_client.produce_batch_record_count:1000 - Number of records to batch before sending to broker
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - pandaproxy_client.produce_batch_size_bytes:1048576 - Number of bytes to batch before sending to broker
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - pandaproxy_client.retries:5 - Number of times to retry a request to a broker
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - pandaproxy_client.retry_base_backoff_ms:100 - Delay (in milliseconds) for initial retry backoff
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - pandaproxy_client.sasl_mechanism: - The SASL mechanism to use when connecting
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - pandaproxy_client.scram_password: - Password to use for SCRAM authentication mechanisms
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - pandaproxy_client.scram_username: - Username to use for SCRAM authentication mechanisms
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - schema_registry.api_doc_dir:/usr/share/redpanda/proxy-api-doc - API doc directory
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - schema_registry.schema_registry_api:{{:{host: 0.0.0.0, port: 8081}:<nullopt>}} - Schema Registry API listen address and port
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - schema_registry.schema_registry_api_tls:{} - TLS configuration for Schema Registry API
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - schema_registry.schema_registry_replication_factor:{nullopt} - Replication factor for internal _schemas topic. If unset, defaults to `default_topic_replication`
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - schema_registry_client.broker_tls:{ enabled: 0 key/cert files: {nullopt} ca file: {nullopt} client_auth_required: 0 } - TLS configuration for the brokers
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - schema_registry_client.brokers:{{host: 10.16.67.48, port: 9092}} - List of address and port of the brokers
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - schema_registry_client.client_identifier:{test_client} - Identifier to use within the kafka request header
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - schema_registry_client.consumer_heartbeat_interval_ms:500 - Interval (in milliseconds) for consumer heartbeats
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - schema_registry_client.consumer_rebalance_timeout_ms:2000 - Timeout (in milliseconds) for consumer rebalance
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - schema_registry_client.consumer_request_max_bytes:1048576 - Max bytes to fetch per request
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - schema_registry_client.consumer_request_timeout_ms:100 - Interval (in milliseconds) for consumer request timeout
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - schema_registry_client.consumer_session_timeout_ms:10000 - Timeout (in milliseconds) for consumer session
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - schema_registry_client.produce_batch_delay_ms:0 - Delay (in milliseconds) to wait before sending batch
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - schema_registry_client.produce_batch_record_count:0 - Number of records to batch before sending to broker
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - schema_registry_client.produce_batch_size_bytes:0 - Number of bytes to batch before sending to broker
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - schema_registry_client.retries:5 - Number of times to retry a request to a broker
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - schema_registry_client.retry_base_backoff_ms:100 - Delay (in milliseconds) for initial retry backoff
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - schema_registry_client.sasl_mechanism: - The SASL mechanism to use when connecting
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - schema_registry_client.scram_password: - Password to use for SCRAM authentication mechanisms
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,855 [shard 0] main - application.cc:515 - schema_registry_client.scram_username: - Username to use for SCRAM authentication mechanisms
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,868 [shard 0] syschecks - Writing pid file "/var/lib/redpanda/data/pid.lock"
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,878 [shard 0] syschecks - STATUS=checking environment (CPU, Mem)
Nov 17 08:48:12 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:12,879 [shard 0] storage - directories.h:33 - Checking `/var/lib/redpanda/data` for supported filesystems
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,391 [shard 0] syschecks - STATUS=Constructing storage services
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,392 [shard 0] syschecks - STATUS=Creating feature table
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,396 [shard 0] syschecks - STATUS=Constructing internal RPC services {{://10.16.67.48:33145:PLAINTEXT}, max_service_memory_per_core: 783915417, metrics_enabled:true}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,397 [shard 0] syschecks - STATUS=Starting storage services
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,414 [shard 48] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:934, committed_offset:947, dirty_offset:947}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/48_0/934-0-v1.log, (2278 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/48_0/934-0-v1.base_index, offsets:{934}, index:{header_bitflags:0, base_offset:{934}, max_offset:{947}, base_timestamp:{timestamp: 1668633097517}, max_timestamp:{timestamp: 1668692181805}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,414 [shard 6] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:1441, committed_offset:1450, dirty_offset:1450}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/6_0/1441-0-v1.log, (2738 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/6_0/1441-0-v1.base_index, offsets:{1441}, index:{header_bitflags:0, base_offset:{1441}, max_offset:{1450}, base_timestamp:{timestamp: 1668633097528}, max_timestamp:{timestamp: 1668692181805}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,414 [shard 48] kvstore - kvstore.cc:509 - Replaying segment with base offset 934
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,414 [shard 11] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:1160, committed_offset:1186, dirty_offset:1186}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/11_0/1160-0-v1.log, (9169 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/11_0/1160-0-v1.base_index, offsets:{1160}, index:{header_bitflags:0, base_offset:{1160}, max_offset:{1186}, base_timestamp:{timestamp: 1668633097526}, max_timestamp:{timestamp: 1668692181805}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,414 [shard 11] kvstore - kvstore.cc:509 - Replaying segment with base offset 1160
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,414 [shard 63] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:63073, committed_offset:63541, dirty_offset:63541}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/63_0/63073-0-v1.log, (71355 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/63_0/63073-0-v1.base_index, offsets:{63073}, index:{header_bitflags:0, base_offset:{63073}, max_offset:{63541}, base_timestamp:{timestamp: 1668633097536}, max_timestamp:{timestamp: 1668692181805}, index(3,3,3)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,414 [shard 45] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:14267, committed_offset:14449, dirty_offset:14449}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/45_0/14267-0-v1.log, (24302 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/45_0/14267-0-v1.base_index, offsets:{14267}, index:{header_bitflags:0, base_offset:{14267}, max_offset:{14449}, base_timestamp:{timestamp: 1668633097361}, max_timestamp:{timestamp: 1668692181805}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,414 [shard 6] kvstore - kvstore.cc:509 - Replaying segment with base offset 1441
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,414 [shard 63] kvstore - kvstore.cc:509 - Replaying segment with base offset 63073
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,414 [shard 45] kvstore - kvstore.cc:509 - Replaying segment with base offset 14267
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,414 [shard 10] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:1216, committed_offset:1248, dirty_offset:1248}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/10_0/1216-0-v1.log, (9936 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/10_0/1216-0-v1.base_index, offsets:{1216}, index:{header_bitflags:0, base_offset:{1216}, max_offset:{1248}, base_timestamp:{timestamp: 1668633097520}, max_timestamp:{timestamp: 1668692181805}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,414 [shard 10] kvstore - kvstore.cc:509 - Replaying segment with base offset 1216
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,414 [shard 36] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:4928, committed_offset:5020, dirty_offset:5020}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/36_0/4928-0-v1.log, (18447 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/36_0/4928-0-v1.base_index, offsets:{4928}, index:{header_bitflags:0, base_offset:{4928}, max_offset:{5020}, base_timestamp:{timestamp: 1668633097509}, max_timestamp:{timestamp: 1668692181805}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,414 [shard 36] kvstore - kvstore.cc:509 - Replaying segment with base offset 4928
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,414 [shard 7] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:340417, committed_offset:341733, dirty_offset:341733}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/7_0/340417-0-v1.log, (293768 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/7_0/340417-0-v1.base_index, offsets:{340417}, index:{header_bitflags:0, base_offset:{340417}, max_offset:{341733}, base_timestamp:{timestamp: 1668633097540}, max_timestamp:{timestamp: 1668692181805}, index(9,9,9)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,414 [shard 7] kvstore - kvstore.cc:509 - Replaying segment with base offset 340417
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,414 [shard 22] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:1186, committed_offset:1221, dirty_offset:1221}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/22_0/1186-0-v1.log, (18273 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/22_0/1186-0-v1.base_index, offsets:{1186}, index:{header_bitflags:0, base_offset:{1186}, max_offset:{1221}, base_timestamp:{timestamp: 1668633097523}, max_timestamp:{timestamp: 1668692181800}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,414 [shard 22] kvstore - kvstore.cc:509 - Replaying segment with base offset 1186
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,414 [shard 53] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:9460, committed_offset:9505, dirty_offset:9505}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/53_0/9460-0-v1.log, (11168 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/53_0/9460-0-v1.base_index, offsets:{9460}, index:{header_bitflags:0, base_offset:{9460}, max_offset:{9505}, base_timestamp:{timestamp: 1668633097521}, max_timestamp:{timestamp: 1668692181805}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,414 [shard 53] kvstore - kvstore.cc:509 - Replaying segment with base offset 9460
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,414 [shard 2] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:11509, committed_offset:11613, dirty_offset:11613}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/2_0/11509-0-v1.log, (19839 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/2_0/11509-0-v1.base_index, offsets:{11509}, index:{header_bitflags:0, base_offset:{11509}, max_offset:{11613}, base_timestamp:{timestamp: 1668633097553}, max_timestamp:{timestamp: 1668692181805}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,414 [shard 30] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:13660, committed_offset:13820, dirty_offset:13820}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/30_0/13660-0-v1.log, (29985 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/30_0/13660-0-v1.base_index, offsets:{13660}, index:{header_bitflags:0, base_offset:{13660}, max_offset:{13820}, base_timestamp:{timestamp: 1668633097516}, max_timestamp:{timestamp: 1668692181805}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,414 [shard 12] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:1869, committed_offset:1909, dirty_offset:1909}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/12_0/1869-0-v1.log, (11975 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/12_0/1869-0-v1.base_index, offsets:{1869}, index:{header_bitflags:0, base_offset:{1869}, max_offset:{1909}, base_timestamp:{timestamp: 1668633097520}, max_timestamp:{timestamp: 1668692181805}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,414 [shard 28] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:403919, committed_offset:406327, dirty_offset:406327}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/28_0/403919-0-v1.log, (436949 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/28_0/403919-0-v1.base_index, offsets:{403919}, index:{header_bitflags:0, base_offset:{403919}, max_offset:{406327}, base_timestamp:{timestamp: 1668633097517}, max_timestamp:{timestamp: 1668692181805}, index(14,14,14)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 37] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:5864, committed_offset:5942, dirty_offset:5942}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/37_0/5864-0-v1.log, (11201 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/37_0/5864-0-v1.base_index, offsets:{5864}, index:{header_bitflags:0, base_offset:{5864}, max_offset:{5942}, base_timestamp:{timestamp: 1668633097510}, max_timestamp:{timestamp: 1668692181805}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 40] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:115718, committed_offset:116858, dirty_offset:116858}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/40_0/115718-0-v1.log, (179547 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/40_0/115718-0-v1.base_index, offsets:{115718}, index:{header_bitflags:0, base_offset:{115718}, max_offset:{116858}, base_timestamp:{timestamp: 1668633097599}, max_timestamp:{timestamp: 1668692181805}, index(6,6,6)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 52] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:9394, committed_offset:9427, dirty_offset:9427}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/52_0/9394-0-v1.log, (5086 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/52_0/9394-0-v1.base_index, offsets:{9394}, index:{header_bitflags:0, base_offset:{9394}, max_offset:{9427}, base_timestamp:{timestamp: 1668633097522}, max_timestamp:{timestamp: 1668692181805}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 49] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:28764, committed_offset:28963, dirty_offset:28963}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/49_0/28764-0-v1.log, (26005 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/49_0/28764-0-v1.base_index, offsets:{28764}, index:{header_bitflags:0, base_offset:{28764}, max_offset:{28963}, base_timestamp:{timestamp: 1668633097497}, max_timestamp:{timestamp: 1668692181805}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 14] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:130180, committed_offset:131245, dirty_offset:131245}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/14_0/130180-0-v1.log, (208404 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/14_0/130180-0-v1.base_index, offsets:{130180}, index:{header_bitflags:0, base_offset:{130180}, max_offset:{131245}, base_timestamp:{timestamp: 1668633097516}, max_timestamp:{timestamp: 1668692181805}, index(7,7,7)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 13] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:132701, committed_offset:133773, dirty_offset:133773}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/13_0/132701-0-v1.log, (206385 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/13_0/132701-0-v1.base_index, offsets:{132701}, index:{header_bitflags:0, base_offset:{132701}, max_offset:{133773}, base_timestamp:{timestamp: 1668633097520}, max_timestamp:{timestamp: 1668692181805}, index(7,7,7)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 35] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:5080, committed_offset:5173, dirty_offset:5173}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/35_0/5080-0-v1.log, (18813 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/35_0/5080-0-v1.base_index, offsets:{5080}, index:{header_bitflags:0, base_offset:{5080}, max_offset:{5173}, base_timestamp:{timestamp: 1668633097518}, max_timestamp:{timestamp: 1668692181805}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 32] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:1301, committed_offset:1329, dirty_offset:1329}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/32_0/1301-0-v1.log, (8710 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/32_0/1301-0-v1.base_index, offsets:{1301}, index:{header_bitflags:0, base_offset:{1301}, max_offset:{1329}, base_timestamp:{timestamp: 1668633097522}, max_timestamp:{timestamp: 1668692181805}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 4] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:164917, committed_offset:165989, dirty_offset:165989}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/4_0/164917-0-v1.log, (137899 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/4_0/164917-0-v1.base_index, offsets:{164917}, index:{header_bitflags:0, base_offset:{164917}, max_offset:{165989}, base_timestamp:{timestamp: 1668633097521}, max_timestamp:{timestamp: 1668692181805}, index(5,5,5)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 39] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:5763, committed_offset:5828, dirty_offset:5828}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/39_0/5763-0-v1.log, (9051 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/39_0/5763-0-v1.base_index, offsets:{5763}, index:{header_bitflags:0, base_offset:{5763}, max_offset:{5828}, base_timestamp:{timestamp: 1668633097518}, max_timestamp:{timestamp: 1668692181805}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 3] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:166527, committed_offset:167602, dirty_offset:167602}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/3_0/166527-0-v1.log, (163343 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/3_0/166527-0-v1.base_index, offsets:{166527}, index:{header_bitflags:0, base_offset:{166527}, max_offset:{167602}, base_timestamp:{timestamp: 1668633097635}, max_timestamp:{timestamp: 1668692181805}, index(5,5,5)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 18] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:6309, committed_offset:6371, dirty_offset:6371}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/18_0/6309-0-v1.log, (15881 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/18_0/6309-0-v1.base_index, offsets:{6309}, index:{header_bitflags:0, base_offset:{6309}, max_offset:{6371}, base_timestamp:{timestamp: 1668633097529}, max_timestamp:{timestamp: 1668692181805}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 21] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:1126, committed_offset:1136, dirty_offset:1136}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/21_0/1126-0-v1.log, (6292 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/21_0/1126-0-v1.base_index, offsets:{1126}, index:{header_bitflags:0, base_offset:{1126}, max_offset:{1136}, base_timestamp:{timestamp: 1668633097524}, max_timestamp:{timestamp: 1668692181805}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 24] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:17383, committed_offset:17567, dirty_offset:17567}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/24_0/17383-0-v1.log, (49216 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/24_0/17383-0-v1.base_index, offsets:{17383}, index:{header_bitflags:0, base_offset:{17383}, max_offset:{17567}, base_timestamp:{timestamp: 1668633097519}, max_timestamp:{timestamp: 1668692181792}, index(2,2,2)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 5] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:147356, committed_offset:148314, dirty_offset:148314}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/5_0/147356-0-v1.log, (125244 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/5_0/147356-0-v1.base_index, offsets:{147356}, index:{header_bitflags:0, base_offset:{147356}, max_offset:{148314}, base_timestamp:{timestamp: 1668633097533}, max_timestamp:{timestamp: 1668692181805}, index(4,4,4)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 24] kvstore - kvstore.cc:509 - Replaying segment with base offset 17383
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 31] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:1273, committed_offset:1295, dirty_offset:1295}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/31_0/1273-0-v1.log, (4080 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/31_0/1273-0-v1.base_index, offsets:{1273}, index:{header_bitflags:0, base_offset:{1273}, max_offset:{1295}, base_timestamp:{timestamp: 1668633097524}, max_timestamp:{timestamp: 1668692181792}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 8] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:333444, committed_offset:334808, dirty_offset:334808}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/8_0/333444-0-v1.log, (304588 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/8_0/333444-0-v1.base_index, offsets:{333444}, index:{header_bitflags:0, base_offset:{333444}, max_offset:{334808}, base_timestamp:{timestamp: 1668633097519}, max_timestamp:{timestamp: 1668692181805}, index(10,10,10)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 52] kvstore - kvstore.cc:509 - Replaying segment with base offset 9394
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 15] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:127338, committed_offset:128472, dirty_offset:128472}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/15_0/127338-0-v1.log, (220598 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/15_0/127338-0-v1.base_index, offsets:{127338}, index:{header_bitflags:0, base_offset:{127338}, max_offset:{128472}, base_timestamp:{timestamp: 1668633097525}, max_timestamp:{timestamp: 1668692181805}, index(7,7,7)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 59] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:538841, committed_offset:542189, dirty_offset:542189}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/59_0/538841-0-v1.log, (424843 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/59_0/538841-0-v1.base_index, offsets:{538841}, index:{header_bitflags:0, base_offset:{538841}, max_offset:{542189}, base_timestamp:{timestamp: 1668633097511}, max_timestamp:{timestamp: 1668692181800}, index(13,13,13)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 9] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:338364, committed_offset:339721, dirty_offset:339721}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/9_0/338364-0-v1.log, (261532 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/9_0/338364-0-v1.base_index, offsets:{338364}, index:{header_bitflags:0, base_offset:{338364}, max_offset:{339721}, base_timestamp:{timestamp: 1668633097522}, max_timestamp:{timestamp: 1668692181805}, index(8,8,8)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 38] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:5661, committed_offset:5728, dirty_offset:5728}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/38_0/5661-0-v1.log, (11495 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/38_0/5661-0-v1.base_index, offsets:{5661}, index:{header_bitflags:0, base_offset:{5661}, max_offset:{5728}, base_timestamp:{timestamp: 1668633097510}, max_timestamp:{timestamp: 1668692181805}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 38] kvstore - kvstore.cc:509 - Replaying segment with base offset 5661
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 34] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:1361, committed_offset:1389, dirty_offset:1389}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/34_0/1361-0-v1.log, (8649 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/34_0/1361-0-v1.base_index, offsets:{1361}, index:{header_bitflags:0, base_offset:{1361}, max_offset:{1389}, base_timestamp:{timestamp: 1668633097521}, max_timestamp:{timestamp: 1668692181792}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 34] kvstore - kvstore.cc:509 - Replaying segment with base offset 1361
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 13] kvstore - kvstore.cc:509 - Replaying segment with base offset 132701
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 35] kvstore - kvstore.cc:509 - Replaying segment with base offset 5080
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 30] kvstore - kvstore.cc:509 - Replaying segment with base offset 13660
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 50] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:28889, committed_offset:29086, dirty_offset:29086}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/50_0/28889-0-v1.log, (31102 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/50_0/28889-0-v1.base_index, offsets:{28889}, index:{header_bitflags:0, base_offset:{28889}, max_offset:{29086}, base_timestamp:{timestamp: 1668633097523}, max_timestamp:{timestamp: 1668692181805}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 17] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:6573, committed_offset:6643, dirty_offset:6643}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/17_0/6573-0-v1.log, (14437 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/17_0/6573-0-v1.base_index, offsets:{6573}, index:{header_bitflags:0, base_offset:{6573}, max_offset:{6643}, base_timestamp:{timestamp: 1668633097577}, max_timestamp:{timestamp: 1668692181805}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 62] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:62551, committed_offset:63017, dirty_offset:63017}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/62_0/62551-0-v1.log, (77503 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/62_0/62551-0-v1.base_index, offsets:{62551}, index:{header_bitflags:0, base_offset:{62551}, max_offset:{63017}, base_timestamp:{timestamp: 1668633097623}, max_timestamp:{timestamp: 1668692181805}, index(3,3,3)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 17] kvstore - kvstore.cc:509 - Replaying segment with base offset 6573
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 62] kvstore - kvstore.cc:509 - Replaying segment with base offset 62551
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 16] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:129061, committed_offset:130171, dirty_offset:130171}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/16_0/129061-0-v1.log, (248585 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/16_0/129061-0-v1.base_index, offsets:{129061}, index:{header_bitflags:0, base_offset:{129061}, max_offset:{130171}, base_timestamp:{timestamp: 1668633097526}, max_timestamp:{timestamp: 1668692181805}, index(8,8,8)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 19] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:1365, committed_offset:1382, dirty_offset:1382}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/19_0/1365-0-v1.log, (6991 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/19_0/1365-0-v1.base_index, offsets:{1365}, index:{header_bitflags:0, base_offset:{1365}, max_offset:{1382}, base_timestamp:{timestamp: 1668633097529}, max_timestamp:{timestamp: 1668692181806}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 16] kvstore - kvstore.cc:509 - Replaying segment with base offset 129061
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 46] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:968, committed_offset:981, dirty_offset:981}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/46_0/968-0-v1.log, (1949 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/46_0/968-0-v1.base_index, offsets:{968}, index:{header_bitflags:0, base_offset:{968}, max_offset:{981}, base_timestamp:{timestamp: 1668633097498}, max_timestamp:{timestamp: 1668692181798}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 39] kvstore - kvstore.cc:509 - Replaying segment with base offset 5763
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 18] kvstore - kvstore.cc:509 - Replaying segment with base offset 6309
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 21] kvstore - kvstore.cc:509 - Replaying segment with base offset 1126
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 54] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:9548, committed_offset:9581, dirty_offset:9581}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/54_0/9548-0-v1.log, (5305 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/54_0/9548-0-v1.base_index, offsets:{9548}, index:{header_bitflags:0, base_offset:{9548}, max_offset:{9581}, base_timestamp:{timestamp: 1668633097509}, max_timestamp:{timestamp: 1668692181805}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 51] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:28689, committed_offset:28913, dirty_offset:28913}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/51_0/28689-0-v1.log, (34667 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/51_0/28689-0-v1.base_index, offsets:{28689}, index:{header_bitflags:0, base_offset:{28689}, max_offset:{28913}, base_timestamp:{timestamp: 1668633097516}, max_timestamp:{timestamp: 1668692181800}, index(2,2,2)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 54] kvstore - kvstore.cc:509 - Replaying segment with base offset 9548
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 51] kvstore - kvstore.cc:509 - Replaying segment with base offset 28689
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 40] kvstore - kvstore.cc:509 - Replaying segment with base offset 115718
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 31] kvstore - kvstore.cc:509 - Replaying segment with base offset 1273
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 2] kvstore - kvstore.cc:509 - Replaying segment with base offset 11509
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 5] kvstore - kvstore.cc:509 - Replaying segment with base offset 147356
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 47] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:998, committed_offset:1013, dirty_offset:1013}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/47_0/998-0-v1.log, (2433 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/47_0/998-0-v1.base_index, offsets:{998}, index:{header_bitflags:0, base_offset:{998}, max_offset:{1013}, base_timestamp:{timestamp: 1668633097524}, max_timestamp:{timestamp: 1668692181800}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 15] kvstore - kvstore.cc:509 - Replaying segment with base offset 127338
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 59] kvstore - kvstore.cc:509 - Replaying segment with base offset 538841
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 9] kvstore - kvstore.cc:509 - Replaying segment with base offset 338364
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 60] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:526391, committed_offset:529696, dirty_offset:529696}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/60_0/526391-0-v1.log, (418214 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/60_0/526391-0-v1.base_index, offsets:{526391}, index:{header_bitflags:0, base_offset:{526391}, max_offset:{529696}, base_timestamp:{timestamp: 1668633097513}, max_timestamp:{timestamp: 1668692181805}, index(13,13,13)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 60] kvstore - kvstore.cc:509 - Replaying segment with base offset 526391
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 43] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:24839, committed_offset:25032, dirty_offset:25032}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/43_0/24839-0-v1.log, (25688 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/43_0/24839-0-v1.base_index, offsets:{24839}, index:{header_bitflags:0, base_offset:{24839}, max_offset:{25032}, base_timestamp:{timestamp: 1668633097521}, max_timestamp:{timestamp: 1668692181805}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 14] kvstore - kvstore.cc:509 - Replaying segment with base offset 130180
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 61] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:60778, committed_offset:61220, dirty_offset:61220}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/61_0/60778-0-v1.log, (67143 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/61_0/60778-0-v1.base_index, offsets:{60778}, index:{header_bitflags:0, base_offset:{60778}, max_offset:{61220}, base_timestamp:{timestamp: 1668633097624}, max_timestamp:{timestamp: 1668692181798}, index(3,3,3)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 41] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:115911, committed_offset:117049, dirty_offset:117049}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/41_0/115911-0-v1.log, (173052 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/41_0/115911-0-v1.base_index, offsets:{115911}, index:{header_bitflags:0, base_offset:{115911}, max_offset:{117049}, base_timestamp:{timestamp: 1668633097530}, max_timestamp:{timestamp: 1668692181800}, index(6,6,6)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 61] kvstore - kvstore.cc:509 - Replaying segment with base offset 60778
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 32] kvstore - kvstore.cc:509 - Replaying segment with base offset 1301
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 50] kvstore - kvstore.cc:509 - Replaying segment with base offset 28889
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 1] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:11422, committed_offset:11533, dirty_offset:11533}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/1_0/11422-0-v1.log, (25074 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/1_0/11422-0-v1.base_index, offsets:{11422}, index:{header_bitflags:0, base_offset:{11422}, max_offset:{11533}, base_timestamp:{timestamp: 1668633097537}, max_timestamp:{timestamp: 1668692181805}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 23] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:17487, committed_offset:17664, dirty_offset:17664}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/23_0/17487-0-v1.log, (47657 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/23_0/17487-0-v1.base_index, offsets:{17487}, index:{header_bitflags:0, base_offset:{17487}, max_offset:{17664}, base_timestamp:{timestamp: 1668633097534}, max_timestamp:{timestamp: 1668692181793}, index(2,2,2)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 1] kvstore - kvstore.cc:509 - Replaying segment with base offset 11422
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 23] kvstore - kvstore.cc:509 - Replaying segment with base offset 17487
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 19] kvstore - kvstore.cc:509 - Replaying segment with base offset 1365
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 20] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:1106, committed_offset:1138, dirty_offset:1138}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/20_0/1106-0-v1.log, (15331 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/20_0/1106-0-v1.base_index, offsets:{1106}, index:{header_bitflags:0, base_offset:{1106}, max_offset:{1138}, base_timestamp:{timestamp: 1668633097533}, max_timestamp:{timestamp: 1668692181805}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 20] kvstore - kvstore.cc:509 - Replaying segment with base offset 1106
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 3] kvstore - kvstore.cc:509 - Replaying segment with base offset 166527
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 44] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:14202, committed_offset:14384, dirty_offset:14384}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/44_0/14202-0-v1.log, (24363 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/44_0/14202-0-v1.base_index, offsets:{14202}, index:{header_bitflags:0, base_offset:{14202}, max_offset:{14384}, base_timestamp:{timestamp: 1668633097510}, max_timestamp:{timestamp: 1668692181805}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 12] kvstore - kvstore.cc:509 - Replaying segment with base offset 1869
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 44] kvstore - kvstore.cc:509 - Replaying segment with base offset 14202
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 47] kvstore - kvstore.cc:509 - Replaying segment with base offset 998
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 49] kvstore - kvstore.cc:509 - Replaying segment with base offset 28764
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 42] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:114285, committed_offset:115429, dirty_offset:115429}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/42_0/114285-0-v1.log, (145031 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/42_0/114285-0-v1.base_index, offsets:{114285}, index:{header_bitflags:0, base_offset:{114285}, max_offset:{115429}, base_timestamp:{timestamp: 1668633097534}, max_timestamp:{timestamp: 1668692181805}, index(5,5,5)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 28] kvstore - kvstore.cc:509 - Replaying segment with base offset 403919
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 42] kvstore - kvstore.cc:509 - Replaying segment with base offset 114285
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 46] kvstore - kvstore.cc:509 - Replaying segment with base offset 968
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 8] kvstore - kvstore.cc:509 - Replaying segment with base offset 333444
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 37] kvstore - kvstore.cc:509 - Replaying segment with base offset 5864
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 43] kvstore - kvstore.cc:509 - Replaying segment with base offset 24839
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 41] kvstore - kvstore.cc:509 - Replaying segment with base offset 115911
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 4] kvstore - kvstore.cc:509 - Replaying segment with base offset 164917
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 29] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:13838, committed_offset:13999, dirty_offset:13999}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/29_0/13838-0-v1.log, (29248 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/29_0/13838-0-v1.base_index, offsets:{13838}, index:{header_bitflags:0, base_offset:{13838}, max_offset:{13999}, base_timestamp:{timestamp: 1668633097514}, max_timestamp:{timestamp: 1668692181805}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 29] kvstore - kvstore.cc:509 - Replaying segment with base offset 13838
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 25] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:16833, committed_offset:17015, dirty_offset:17015}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/25_0/16833-0-v1.log, (30146 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/25_0/16833-0-v1.base_index, offsets:{16833}, index:{header_bitflags:0, base_offset:{16833}, max_offset:{17015}, base_timestamp:{timestamp: 1668633097523}, max_timestamp:{timestamp: 1668692181805}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,415 [shard 33] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:1339, committed_offset:1367, dirty_offset:1367}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/33_0/1339-0-v1.log, (8856 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/33_0/1339-0-v1.base_index, offsets:{1339}, index:{header_bitflags:0, base_offset:{1339}, max_offset:{1367}, base_timestamp:{timestamp: 1668633097521}, max_timestamp:{timestamp: 1668692181792}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,416 [shard 33] kvstore - kvstore.cc:509 - Replaying segment with base offset 1339
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,416 [shard 25] kvstore - kvstore.cc:509 - Replaying segment with base offset 16833
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,416 [shard 27] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:812142, committed_offset:819026, dirty_offset:819026}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/27_0/812142-0-v1.log, (1229621 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/27_0/812142-0-v1.base_index, offsets:{812142}, index:{header_bitflags:0, base_offset:{812142}, max_offset:{819026}, base_timestamp:{timestamp: 1668633097515}, max_timestamp:{timestamp: 1668692181805}, index(38,38,38)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,416 [shard 26] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:812739, committed_offset:819469, dirty_offset:819469}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/26_0/812739-0-v1.log, (1409773 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/26_0/812739-0-v1.base_index, offsets:{812739}, index:{header_bitflags:0, base_offset:{812739}, max_offset:{819469}, base_timestamp:{timestamp: 1668633097525}, max_timestamp:{timestamp: 1668692181805}, index(43,43,43)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,416 [shard 26] kvstore - kvstore.cc:509 - Replaying segment with base offset 812739
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,416 [shard 27] kvstore - kvstore.cc:509 - Replaying segment with base offset 812142
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,417 [shard 56] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:551824, committed_offset:560024, dirty_offset:560024}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/56_0/551824-0-v1.log, (1049725 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/56_0/551824-0-v1.base_index, offsets:{551824}, index:{header_bitflags:0, base_offset:{551824}, max_offset:{560024}, base_timestamp:{timestamp: 1668633097574}, max_timestamp:{timestamp: 1668692181805}, index(32,32,32)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,417 [shard 57] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:542525, committed_offset:550499, dirty_offset:550499}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/57_0/542525-0-v1.log, (1021997 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/57_0/542525-0-v1.base_index, offsets:{542525}, index:{header_bitflags:0, base_offset:{542525}, max_offset:{550499}, base_timestamp:{timestamp: 1668633097591}, max_timestamp:{timestamp: 1668692181805}, index(32,32,32)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,417 [shard 57] kvstore - kvstore.cc:509 - Replaying segment with base offset 542525
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,417 [shard 56] kvstore - kvstore.cc:509 - Replaying segment with base offset 551824
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,417 [shard 58] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:535757, committed_offset:543667, dirty_offset:543667}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/58_0/535757-0-v1.log, (1011380 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/58_0/535757-0-v1.base_index, offsets:{535757}, index:{header_bitflags:0, base_offset:{535757}, max_offset:{543667}, base_timestamp:{timestamp: 1668633097519}, max_timestamp:{timestamp: 1668692181809}, index(31,31,31)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,417 [shard 58] kvstore - kvstore.cc:509 - Replaying segment with base offset 535757
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,417 [shard 55] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:558318, committed_offset:566744, dirty_offset:566744}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/55_0/558318-0-v1.log, (1358889 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/55_0/558318-0-v1.base_index, offsets:{558318}, index:{header_bitflags:0, base_offset:{558318}, max_offset:{566744}, base_timestamp:{timestamp: 1668633097520}, max_timestamp:{timestamp: 1668692181805}, index(42,42,42)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,417 [shard 55] kvstore - kvstore.cc:509 - Replaying segment with base offset 558318
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,435 [shard 0] storage - segment_set.cc:313 - Recovered: {offset_tracker:{term:0, base_offset:2581, committed_offset:2630, dirty_offset:2630}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/redpanda/kvstore/0_0/2581-0-v1.log, (31384 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/0_0/2581-0-v1.base_index, offsets:{2581}, index:{header_bitflags:0, base_offset:{2581}, max_offset:{2630}, base_timestamp:{timestamp: 1668633093922}, max_timestamp:{timestamp: 1668692181797}, index(1,1,1)}, step:32768, needs_persistence:0}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,435 [shard 0] kvstore - kvstore.cc:509 - Replaying segment with base offset 2581
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,435 [shard 0] kvstore - kvstore.cc:538 - Removing old segment with base offset 2579
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,436 [shard 0] main - application.cc:1814 - Upgrade in progress! This binary logical version 7, last feature table snapshot version 5
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,439 [shard 0] main - application.cc:1368 - Loaded stored node ID for node: 0
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,440 [shard 0] main - application.cc:1395 - Loaded existing UUID for node: {1cddf71a-283f-47c5-9169-2b6bce0de366}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,442 [shard 0] syschecks - STATUS=Starting internal RPC bootstrap service
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,450 [shard 0] main - application.cc:1428 - Started RPC server listening at {host: 10.16.67.48, port: 33145}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,452 [shard 0] cluster - cluster_discovery.cc:99 - Controller directory /var/lib/redpanda/data/redpanda/controller/0_0 not empty; assuming existing cluster exists
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,452 [shard 0] main - application.cc:1455 - Starting Redpanda with node_id 0, cluster UUID {nullopt}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,458 [shard 0] syschecks - STATUS=Initializing connection cache
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,459 [shard 0] syschecks - STATUS=Building shard-lookup tables
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,459 [shard 0] syschecks - STATUS=Intializing raft recovery throttle
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,459 [shard 0] syschecks - STATUS=Intializing raft group manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,460 [shard 0] syschecks - STATUS=Adding partition manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,461 [shard 0] main - application.cc:872 - Partition manager started
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,462 [shard 0] syschecks - STATUS=Creating cluster::controller
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,465 [shard 0] syschecks - STATUS=Creating kafka metadata cache
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,465 [shard 0] syschecks - STATUS=Adding kafka quota manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,465 [shard 0] syschecks - STATUS=Creating metadata dissemination service
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,466 [shard 0] syschecks - STATUS=Creating kafka group managers
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,466 [shard 0] syschecks - STATUS=Creating kafka group shard mapper
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,467 [shard 0] syschecks - STATUS=Creating kafka group router
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,467 [shard 0] syschecks - STATUS=Creating id allocator frontend
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,467 [shard 0] syschecks - STATUS=Creating group resource manager frontend
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,467 [shard 0] syschecks - STATUS=Creating partition resource manager frontend
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,468 [shard 0] syschecks - STATUS=Creating tx coordinator frontend
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,469 [shard 0] syschecks - STATUS=Starting kafka RPC {{://10.16.67.48:9092:PLAINTEXT}, max_service_memory_per_core: 1175873126, metrics_enabled:true}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,470 [shard 0] syschecks - STATUS=constructing http server
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,470 [shard 0] syschecks - STATUS=Starting the partition manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,471 [shard 0] syschecks - STATUS=Starting the coproc partition manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,471 [shard 0] syschecks - STATUS=Starting Raft group manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,471 [shard 0] syschecks - STATUS=Starting Kafka group manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,472 [shard 0] syschecks - STATUS=Starting controller
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,481 [shard 0] raft - [group_id:0, {redpanda/controller/0}] consensus.cc:1132 - Starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,482 [shard 0] raft - [group_id:0, {redpanda/controller/0}] consensus.cc:1419 - Recovered persistent state from kvstore: voted for: {id: {3}, revision: {0}}, term: 20
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,482 [shard 0] raft - [group_id:0, {redpanda/controller/0}] consensus.cc:1166 - Starting with voted_for {id: {3}, revision: {0}} term 20 initial_state false
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,485 [shard 0] raft - [group_id:0, {redpanda/controller/0}] consensus.cc:1209 - Read bootstrap state: data_seen 36 config_seen 0 eol false commit 1423 term 0 prev_idx 1422 prev_term 0 config_tracker -9223372036854775808 commit_base_tracker 1423 configurations {}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,485 [shard 0] raft - [group_id:0, {redpanda/controller/0}] consensus.cc:1210 - Current log offsets: {start_offset:0, committed_offset:1423, committed_offset_term:20, dirty_offset:1423, dirty_offset_term:20, last_term_start_offset:1387}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,485 [shard 0] raft - [group_id:0, {redpanda/controller/0}] consensus.cc:1234 - Truncating configurations at 1423
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,495 [shard 0] storage - segment.cc:655 - Creating new segment /var/lib/redpanda/data/redpanda/kvstore/0_0/2631-0-v1.log
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,508 [shard 0] raft - [group_id:0, {redpanda/controller/0}] consensus.cc:1332 - started raft, log offsets: {start_offset:0, committed_offset:1423, committed_offset_term:20, dirty_offset:1423, dirty_offset_term:20, last_term_start_offset:1387}, term: 20, configuration: {current: {voters: {{id: {0}, revision: {0}}, {id: {8}, revision: {0}}, {id: {9}, revision: {0}}, {id: {3}, revision: {0}}, {id: {1}, revision: {0}}, {id: {6}, revision: {0}}, {id: {4}, revision: {0}}, {id: {5}, revision: {0}}, {id: {7}, revision: {0}}, {id: {2}, revision: {0}}}, learners: {}}, old:{nullopt}, revision: 0, update: {nullopt}, version: 4, brokers: {{id: 0, kafka_advertised_listeners: {{:{host: 10.16.67.48, port: 9092}}}, rpc_address: {host: 10.16.67.48, port: 33145}, rack: {nullopt}, properties: {cores 64, mem_available 233, disk_available 14305}, membership_state: active}, {id: 8, kafka_advertised_listeners: {{:{host: 10.16.67.47, port: 9092}}}, rpc_address: {host: 10.16.67.47, port: 33145}, rack: {nullopt}, properties: {cores 64, mem_available 233, disk_available 14305}, membership_state: active}, {id: 9, kafka_advertised_listeners: {{:{host: 10.16.66.78, port: 9092}}}, rpc_address: {host: 10.16.66.78, port: 33145}, rack: {nullopt}, properties: {cores 64, mem_available 233, disk_available 14305}, membership_state: active}, {id: 3, kafka_advertised_listeners: {{:{host: 10.16.67.49, port: 9092}}}, rpc_address: {host: 10.16.67.49, port: 33145}, rack: {nullopt}, properties: {cores 64, mem_available 233, disk_available 14305}, membership_state: active}, {id: 1, kafka_advertised_listeners: {{:{host: 10.16.67.50, port: 9092}}}, rpc_address: {host: 10.16.67.50, port: 33145}, rack: {nullopt}, properties: {cores 64, mem_available 233, disk_available 14305}, membership_state: active}, {id: 6, kafka_advertised_listeners: {{:{host: 10.16.67.46, port: 9092}}}, rpc_address: {host: 10.16.67.46, port: 33145}, rack: {nullopt}, properties: {cores 64, mem_available 233, disk_available 14305}, membership_state: active}, {id: 4, kafka_advertised_listeners: {{:{host: 10.16.67.51, port: 9092}}}, rpc_address: {host: 10.16.67.51, port: 33145}, rack: {nullopt}, properties: {cores 64, mem_available 233, disk_available 14305}, membership_state: active}, {id: 5, kafka_advertised_listeners: {{:{host: 10.16.67.44, port: 9092}}}, rpc_address: {host: 10.16.67.44, port: 33145}, rack: {nullopt}, properties: {cores 64, mem_available 233, disk_available 14305}, membership_state: active}, {id: 7, kafka_advertised_listeners: {{:{host: 10.16.67.45, port: 9092}}}, rpc_address: {host: 10.16.67.45, port: 33145}, rack: {nullopt}, properties: {cores 64, mem_available 233, disk_available 14305}, membership_state: active}, {id: 2, kafka_advertised_listeners: {{:{host: 10.16.67.43, port: 9092}}}, rpc_address: {host: 10.16.67.43, port: 33145}, rack: {nullopt}, properties: {cores 64, mem_available 233, disk_available 14305}, membership_state: active}}}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 0] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 1] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 8] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 11] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 13] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 15] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 3] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 19] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 24] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 26] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 30] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 34] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 36] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 38] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 4] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 45] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 47] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 52] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 60] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 14] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 18] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 6] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 20] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 2] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 21] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 25] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 22] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 23] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 7] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 27] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 28] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 29] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 10] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 9] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 31] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 32] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 33] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 35] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 37] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 12] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 40] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 41] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 42] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 39] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 43] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 44] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 49] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 46] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 48] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 50] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 51] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 53] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 16] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 56] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 58] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 55] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 54] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 59] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 61] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 57] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 17] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 62] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,513 [shard 63] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,518 [shard 5] cluster - drain_manager.cc:20 - Drain manager starting
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,518 [shard 0] cluster - members_manager.cc:87 - starting cluster::members_manager...
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,519 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,519 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,521 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,521 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,521 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,521 [shard 0] cluster - node_status_backend.cc:61 - Node 8 has been discovered via members table
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,521 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,521 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,521 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,521 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,521 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,522 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,522 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,522 [shard 0] cluster - node_status_backend.cc:61 - Node 9 has been discovered via members table
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,523 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,523 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,523 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,523 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,523 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,523 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,523 [shard 0] cluster - node_status_backend.cc:61 - Node 3 has been discovered via members table
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,524 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,524 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,524 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,524 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,524 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,524 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,524 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,524 [shard 0] cluster - node_status_backend.cc:61 - Node 1 has been discovered via members table
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,525 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,525 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,525 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,525 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,525 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,525 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,525 [shard 0] cluster - node_status_backend.cc:61 - Node 6 has been discovered via members table
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,526 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,526 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,526 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,526 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,526 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,526 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,526 [shard 0] cluster - node_status_backend.cc:61 - Node 4 has been discovered via members table
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,526 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,526 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,527 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,527 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,527 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,527 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,527 [shard 0] cluster - node_status_backend.cc:61 - Node 5 has been discovered via members table
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,527 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,527 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,528 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,528 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,528 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,528 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,528 [shard 0] cluster - node_status_backend.cc:61 - Node 7 has been discovered via members table
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,528 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,528 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,528 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,528 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,529 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,529 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,529 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,529 [shard 0] cluster - node_status_backend.cc:61 - Node 2 has been discovered via members table
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,529 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,529 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,529 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,529 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,530 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,530 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,530 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,530 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,530 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,530 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,530 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,530 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,531 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,531 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,539 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,539 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,541 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,541 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,542 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,542 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,542 [shard 0] cluster - members_manager.cc:214 - Applying update to members_manager
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: INFO 2022-11-17 08:48:13,542 [shard 0] cluster - members_manager.cc:216 - Raft config update
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: ERROR 2022-11-17 08:48:13,908 [shard 0] assert - Assert failure: (/var/lib/buildkite-agent/builds/buildkite-amd64-xfs-builders-i-0219b56622333e416-1/redpanda/redpanda/src/v/cluster/scheduling/allocation_node.cc:81) 'domain_partitions > allocation_capacity{0} && domain_partitions <= _allocated_partitions' Unable to deallocate partition from core 27 in domain 0 at node {node: 8, max_partitions_per_core: 7000, state: active, partition_capacity: 447993, weights: [(2)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(1)(1)(1)(1)(1)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)], allocated: 5({0: 0, -1: 5})}
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: ERROR 2022-11-17 08:48:13,910 [shard 0] assert - Backtrace below:
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: 0x534cdc4 0x4504914 0x40202bd 0x412d744 0x50e9a6f 0x50ed747 0x50eab19 0x500b6b1 0x50097cf 0x1a6067e 0x53ff549 /opt/redpanda/lib/libc.so.6+0x2d58f /opt/redpanda/lib/libc.so.6+0x2d648 0x1a5b464
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: --------
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: seastar::continuation<seastar::internal::promise_base_with_type<std::__1::error_code>, auto cluster::topic_updates_dispatcher::apply_update(model::record_batch)::$_0::operator()<std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::topic_namespace, cluster::create_partitions_configuration_assignment, (signed char)5, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::ntp, cluster::cancel_moving_partition_replicas_cmd_data, (signed char)7, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::topic_namespace, std::__1::vector<cluster::move_topic_replicas_data, std::__1::allocator<cluster::move_topic_replicas_data> >, (signed char)8, (model::record_batch_type)6, (cluster::serde_opts)1> > >(std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::topic_namespace, cluster::create_partitions_configuration_assignment, (signed char)5, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::ntp, cluster::cancel_moving_partition_replicas_cmd_data, (signed char)7, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::topic_namespace, std::__1::vector<cluster::move_topic_replicas_data, std::__1::allocator<cluster::move_topic_replicas_data> >, (signed char)8, (model::record_batch_type)6, (cluster::serde_opts)1> >) const::'lambda'(cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6, (cluster::serde_opts)0>)::operator()(cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6, (cluster::serde_opts)0>) const::'lambda'(std::__1::error_code), seastar::future<std::__1::error_code> seastar::future<std::__1::error_code>::then_impl_nrvo<auto cluster::topic_updates_dispatcher::apply_update(model::record_batch)::$_0::operator()<std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::topic_namespace, cluster::create_partitions_configuration_assignment, (signed char)5, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::ntp, cluster::cancel_moving_partition_replicas_cmd_data, (signed char)7, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::topic_namespace, std::__1::vector<cluster::move_topic_replicas_data, std::__1::allocator<cluster::move_topic_replicas_data> >, (signed char)8, (model::record_batch_type)6, (cluster::serde_opts)1> > >(std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::topic_namespace, cluster::create_partitions_configuration_assignment, (signed char)5, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::ntp, cluster::cancel_moving_partition_replicas_cmd_data, (signed char)7, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::topic_namespace, std::__1::vector<cluster::move_topic_replicas_data, std::__1::allocator<cluster::move_topic_replicas_data> >, (signed char)8, (model::record_batch_type)6, (cluster::serde_opts)1> >) const::'lambda'(cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6, (cluster::serde_opts)0>)::operator()(cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6, (cluster::serde_opts)0>) const::'lambda'(std::__1::error_code), seastar::future<std::__1::error_code> >(std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::topic_namespace, cluster::create_partitions_configuration_assignment, (signed char)5, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::ntp, cluster::cancel_moving_partition_replicas_cmd_data, (signed char)7, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::topic_namespace, std::__1::vector<cluster::move_topic_replicas_data, std::__1::allocator<cluster::move_topic_replicas_data> >, (signed char)8, (model::record_batch_type)6, (cluster::serde_opts)1> >&&)::'lambda'(seastar::internal::promise_base_with_type<std::__1::error_code>&&, auto cluster::topic_updates_dispatcher::apply_update(model::record_batch)::$_0::operator()<std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::topic_namespace, cluster::create_partitions_configuration_assignment, (signed char)5, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::ntp, cluster::cancel_moving_partition_replicas_cmd_data, (signed char)7, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::topic_namespace, std::__1::vector<cluster::move_topic_replicas_data, std::__1::allocator<cluster::move_topic_replicas_data> >, (signed char)8, (model::record_batch_type)6, (cluster::serde_opts)1> > >(std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::topic_namespace, cluster::create_partitions_configuration_assignment, (signed char)5, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::ntp, cluster::cancel_moving_partition_replicas_cmd_data, (signed char)7, (model::record_batch_type)6, (cluster::serde_opts)0>, cluster::controller_command<model::topic_namespace, std::__1::vector<cluster::move_topic_replicas_data, std::__1::allocator<cluster::move_topic_replicas_data> >, (signed char)8, (model::record_batch_type)6, (cluster::serde_opts)1> >) const::'lambda'(cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6, (cluster::serde_opts)0>)::operator()(cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6, (cluster::serde_opts)0>) const::'lambda'(std::__1::error_code)&, seastar::future_state<std::__1::error_code>&&), std::__1::error_code>
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: --------
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: seastar::continuation<seastar::internal::promise_base_with_type<void>, raft::mux_state_machine<cluster::topic_updates_dispatcher, cluster::security_manager, cluster::members_manager, cluster::data_policy_manager, cluster::config_manager, cluster::feature_backend, cluster::bootstrap_backend>::apply(model::record_batch)::'lambda'()::operator()()::'lambda'(std::__1::error_code), seastar::future<void> seastar::future<std::__1::error_code>::then_impl_nrvo<raft::mux_state_machine<cluster::topic_updates_dispatcher, cluster::security_manager, cluster::members_manager, cluster::data_policy_manager, cluster::config_manager, cluster::feature_backend, cluster::bootstrap_backend>::apply(model::record_batch)::'lambda'()::operator()()::'lambda'(std::__1::error_code), seastar::future<void> >(raft::mux_state_machine<cluster::topic_updates_dispatcher, cluster::security_manager, cluster::members_manager, cluster::data_policy_manager, cluster::config_manager, cluster::feature_backend, cluster::bootstrap_backend>::apply(model::record_batch)::'lambda'()::operator()()::'lambda'(std::__1::error_code)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, raft::mux_state_machine<cluster::topic_updates_dispatcher, cluster::security_manager, cluster::members_manager, cluster::data_policy_manager, cluster::config_manager, cluster::feature_backend, cluster::bootstrap_backend>::apply(model::record_batch)::'lambda'()::operator()()::'lambda'(std::__1::error_code)&, seastar::future_state<std::__1::error_code>&&), std::__1::error_code>
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: --------
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::mux_state_machine<cluster::topic_updates_dispatcher, cluster::security_manager, cluster::members_manager, cluster::data_policy_manager, cluster::config_manager, cluster::feature_backend, cluster::bootstrap_backend>::apply(model::record_batch)::'lambda'()>(seastar::gate&, raft::mux_state_machine<cluster::topic_updates_dispatcher, cluster::security_manager, cluster::members_manager, cluster::data_policy_manager, cluster::config_manager, cluster::feature_backend, cluster::bootstrap_backend>::apply(model::record_batch)::'lambda'()&&)::'lambda'(), false>, seastar::futurize<raft::mux_state_machine<cluster::topic_updates_dispatcher, cluster::security_manager, cluster::members_manager, cluster::data_policy_manager, cluster::config_manager, cluster::feature_backend, cluster::bootstrap_backend>::apply(model::record_batch)::'lambda'()>::type seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::mux_state_machine<cluster::topic_updates_dispatcher, cluster::security_manager, cluster::members_manager, cluster::data_policy_manager, cluster::config_manager, cluster::feature_backend, cluster::bootstrap_backend>::apply(model::record_batch)::'lambda'()>(seastar::gate&, raft::mux_state_machine<cluster::topic_updates_dispatcher, cluster::security_manager, cluster::members_manager, cluster::data_policy_manager, cluster::config_manager, cluster::feature_backend, cluster::bootstrap_backend>::apply(model::record_batch)::'lambda'()&&)::'lambda'(), false> >(seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::mux_state_machine<cluster::topic_updates_dispatcher, cluster::security_manager, cluster::members_manager, cluster::data_policy_manager, cluster::config_manager, cluster::feature_backend, cluster::bootstrap_backend>::apply(model::record_batch)::'lambda'()>(seastar::gate&, raft::mux_state_machine<cluster::topic_updates_dispatcher, cluster::security_manager, cluster::members_manager, cluster::data_policy_manager, cluster::config_manager, cluster::feature_backend, cluster::bootstrap_backend>::apply(model::record_batch)::'lambda'()&&)::'lambda'(), false>&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::mux_state_machine<cluster::topic_updates_dispatcher, cluster::security_manager, cluster::members_manager, cluster::data_policy_manager, cluster::config_manager, cluster::feature_backend, cluster::bootstrap_backend>::apply(model::record_batch)::'lambda'()>(seastar::gate&, raft::mux_state_machine<cluster::topic_updates_dispatcher, cluster::security_manager, cluster::members_manager, cluster::data_policy_manager, cluster::config_manager, cluster::feature_backend, cluster::bootstrap_backend>::apply(model::record_batch)::'lambda'()&&)::'lambda'(), false>&, seastar::future_state<seastar::internal::monostate>&&), void>
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: --------
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: seastar::continuation<seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >, raft::state_machine::batch_applicator::operator()(model::record_batch)::$_5, seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::future<void>::then_impl_nrvo<raft::state_machine::batch_applicator::operator()(model::record_batch)::$_5, seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >(raft::state_machine::batch_applicator::operator()(model::record_batch)::$_5&&)::'lambda'(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, raft::state_machine::batch_applicator::operator()(model::record_batch)::$_5&, seastar::future_state<seastar::internal::monostate>&&), void>
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: --------
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: seastar::internal::repeater<auto model::record_batch_reader::impl::do_action<raft::state_machine::batch_applicator, auto model::record_batch_reader::impl::do_consume<raft::state_machine::batch_applicator>(raft::state_machine::batch_applicator&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::'lambda'(raft::state_machine::batch_applicator&)>(raft::state_machine::batch_applicator&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >, auto model::record_batch_reader::impl::do_consume<raft::state_machine::batch_applicator>(raft::state_machine::batch_applicator&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::'lambda'(raft::state_machine::batch_applicator&)&&)::'lambda'()>
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: --------
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: seastar::continuation<seastar::internal::promise_base_with_type<detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> > >, auto model::record_batch_reader::impl::do_action<raft::state_machine::batch_applicator, auto model::record_batch_reader::impl::do_consume<raft::state_machine::batch_applicator>(raft::state_machine::batch_applicator&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::'lambda'(raft::state_machine::batch_applicator&)>(raft::state_machine::batch_applicator&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >, auto model::record_batch_reader::impl::do_consume<raft::state_machine::batch_applicator>(raft::state_machine::batch_applicator&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::'lambda'(raft::state_machine::batch_applicator&)&&)::'lambda0'(), auto model::record_batch_reader::impl::do_consume<raft::state_machine::batch_applicator>(raft::state_machine::batch_applicator&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::'lambda'(raft::state_machine::batch_applicator&) seastar::future<void>::then_impl_nrvo<auto model::record_batch_reader::impl::do_action<raft::state_machine::batch_applicator, auto model::record_batch_reader::impl::do_consume<raft::state_machine::batch_applicator>(raft::state_machine::batch_applicator&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::'lambda'(raft::state_machine::batch_applicator&)>(raft::state_machine::batch_applicator&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >, auto model::record_batch_reader::impl::do_consume<raft::state_machine::batch_applicator>(raft::state_machine::batch_applicator&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::'lambda'(raft::state_machine::batch_applicator&)&&)::'lambda0'(), seastar::future<detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> > > >(raft::state_machine::batch_applicator&&)::'lambda'(seastar::internal::promise_base_with_type<detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> > >&&, auto model::record_batch_reader::impl::do_action<raft::state_machine::batch_applicator, auto model::record_batch_reader::impl::do_consume<raft::state_machine::batch_applicator>(raft::state_machine::batch_applicator&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::'lambda'(raft::state_machine::batch_applicator&)>(raft::state_machine::batch_applicator&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >, auto model::record_batch_reader::impl::do_consume<raft::state_machine::batch_applicator>(raft::state_machine::batch_applicator&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::'lambda'(raft::state_machine::batch_applicator&)&&)::'lambda0'()&, seastar::future_state<seastar::internal::monostate>&&), void>
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: --------
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: seastar::internal::do_with_state<std::__1::tuple<raft::state_machine::batch_applicator>, seastar::future<detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> > > >
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: --------
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: seastar::continuation<seastar::internal::promise_base_with_type<detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> > >, seastar::future<detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> > >::finally_body<auto model::record_batch_reader::consume<raft::state_machine::batch_applicator>(raft::state_machine::batch_applicator, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >) &&::'lambda'(), true>, seastar::futurize<raft::state_machine::batch_applicator>::type seastar::future<detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> > >::then_wrapped_nrvo<seastar::future<detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> > >, seastar::future<detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> > >::finally_body<auto model::record_batch_reader::consume<raft::state_machine::batch_applicator>(raft::state_machine::batch_applicator, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >) &&::'lambda'(), true> >(seastar::future<detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> > >::finally_body<auto model::record_batch_reader::consume<raft::state_machine::batch_applicator>(raft::state_machine::batch_applicator, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >) &&::'lambda'(), true>&&)::'lambda'(seastar::internal::promise_base_with_type<detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> > >&&, seastar::future<detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> > >::finally_body<auto model::record_batch_reader::consume<raft::state_machine::batch_applicator>(raft::state_machine::batch_applicator, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >) &&::'lambda'(), true>&, seastar::future_state<detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> > >&&), detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> > >
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: --------
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: seastar::continuation<seastar::internal::promise_base_with_type<void>, raft::state_machine::apply()::$_2::operator()(model::record_batch_reader) const::'lambda'(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >), seastar::future<void> seastar::future<detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> > >::then_impl_nrvo<raft::state_machine::apply()::$_2::operator()(model::record_batch_reader) const::'lambda'(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >), seastar::future<void> >(raft::state_machine::apply()::$_2::operator()(model::record_batch_reader) const::'lambda'(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, raft::state_machine::apply()::$_2::operator()(model::record_batch_reader) const::'lambda'(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)&, seastar::future_state<detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> > >&&), detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> > >
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: --------
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void> seastar::future<void>::handle_exception_type<raft::state_machine::apply()::$_7>(raft::state_machine::apply()::$_7&&)::'lambda'(raft::state_machine::apply()::$_7&&), seastar::futurize<raft::state_machine::apply()::$_7>::type seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void> seastar::future<void>::handle_exception_type<raft::state_machine::apply()::$_7>(raft::state_machine::apply()::$_7&&)::'lambda'(raft::state_machine::apply()::$_7&&)>(seastar::future<void> seastar::future<void>::handle_exception_type<raft::state_machine::apply()::$_7>(raft::state_machine::apply()::$_7&&)::'lambda'(raft::state_machine::apply()::$_7&&)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::future<void> seastar::future<void>::handle_exception_type<raft::state_machine::apply()::$_7>(raft::state_machine::apply()::$_7&&)::'lambda'(raft::state_machine::apply()::$_7&&)&, seastar::future_state<seastar::internal::monostate>&&), void>
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: --------
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void> seastar::future<void>::handle_exception_type<raft::state_machine::apply()::$_8>(raft::state_machine::apply()::$_8&&)::'lambda'(raft::state_machine::apply()::$_8&&), seastar::futurize<raft::state_machine::apply()::$_8>::type seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void> seastar::future<void>::handle_exception_type<raft::state_machine::apply()::$_8>(raft::state_machine::apply()::$_8&&)::'lambda'(raft::state_machine::apply()::$_8&&)>(seastar::future<void> seastar::future<void>::handle_exception_type<raft::state_machine::apply()::$_8>(raft::state_machine::apply()::$_8&&)::'lambda'(raft::state_machine::apply()::$_8&&)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::future<void> seastar::future<void>::handle_exception_type<raft::state_machine::apply()::$_8>(raft::state_machine::apply()::$_8&&)::'lambda'(raft::state_machine::apply()::$_8&&)&, seastar::future_state<seastar::internal::monostate>&&), void>
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: --------
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void> seastar::future<void>::handle_exception_type<raft::state_machine::apply()::$_9>(raft::state_machine::apply()::$_9&&)::'lambda'(raft::state_machine::apply()::$_9&&), seastar::futurize<raft::state_machine::apply()::$_9>::type seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void> seastar::future<void>::handle_exception_type<raft::state_machine::apply()::$_9>(raft::state_machine::apply()::$_9&&)::'lambda'(raft::state_machine::apply()::$_9&&)>(seastar::future<void> seastar::future<void>::handle_exception_type<raft::state_machine::apply()::$_9>(raft::state_machine::apply()::$_9&&)::'lambda'(raft::state_machine::apply()::$_9&&)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::future<void> seastar::future<void>::handle_exception_type<raft::state_machine::apply()::$_9>(raft::state_machine::apply()::$_9&&)::'lambda'(raft::state_machine::apply()::$_9&&)&, seastar::future_state<seastar::internal::monostate>&&), void>
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: --------
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void> seastar::future<void>::handle_exception<raft::state_machine::apply()::$_10>(raft::state_machine::apply()::$_10&&)::'lambda'(raft::state_machine::apply()::$_10&&), seastar::futurize<raft::state_machine::apply()::$_10>::type seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void> seastar::future<void>::handle_exception<raft::state_machine::apply()::$_10>(raft::state_machine::apply()::$_10&&)::'lambda'(raft::state_machine::apply()::$_10&&)>(seastar::future<void> seastar::future<void>::handle_exception<raft::state_machine::apply()::$_10>(raft::state_machine::apply()::$_10&&)::'lambda'(raft::state_machine::apply()::$_10&&)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::future<void> seastar::future<void>::handle_exception<raft::state_machine::apply()::$_10>(raft::state_machine::apply()::$_10&&)::'lambda'(raft::state_machine::apply()::$_10&&)&, seastar::future_state<seastar::internal::monostate>&&), void>
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: --------
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: seastar::internal::do_until_state<raft::state_machine::start()::$_3::operator()() const::'lambda'(), raft::state_machine::start()::$_3::operator()() const::'lambda0'()>
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: --------
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::state_machine::start()::$_3>(seastar::gate&, raft::state_machine::start()::$_3&&)::'lambda'(), false>, seastar::futurize<raft::state_machine::start()::$_3>::type seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::state_machine::start()::$_3>(seastar::gate&, raft::state_machine::start()::$_3&&)::'lambda'(), false> >(seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::state_machine::start()::$_3>(seastar::gate&, raft::state_machine::start()::$_3&&)::'lambda'(), false>&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::state_machine::start()::$_3>(seastar::gate&, raft::state_machine::start()::$_3&&)::'lambda'(), false>&, seastar::future_state<seastar::internal::monostate>&&), void>
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: --------
Nov 17 08:48:13 srv-01-03-404.iad1.trmr.io rpk[16743]: seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void> seastar::future<void>::handle_exception_type<auto ssx::spawn_with_gate_then<raft::state_machine::start()::$_3>(seastar::gate&, raft::state_machine::start()::$_3&&)::'lambda'(seastar::abort_requested_exception const&)>(raft::state_machine::start()::$_3&&)::'lambda'(raft::state_machine::start()::$_3&&), seastar::futurize<raft::state_machine::start()::$_3>::type seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void> seastar::future<void>::handle_exception_type<auto ssx::spawn_with_gate_then<raft::state_machine::start()::$_3>(seastar::gate&, raft::state_machine::start()::$_3&&)::'lambda'(seastar::abort_requested_exception const&)>(raft::state_machine::start()::$_3&&)::'lambda'(raft::state_machine::start()::$_3&&)>(seastar::future<void> seastar::future<void>::handle_exception_type<auto ssx::spawn_with_gate_then<raft::state_machine::start()::$_3>(seastar::gate&, raft::state_machine::start()::$_3&&)::'lambda'(seastar::abort_requested_exception const&)>(raft::state_machine::start()::$_3&&)::'lambda'(raft::state_machine::start()::$_3&&)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::future<void> seastar::future<void>::handle_exception_type<auto ssx::spawn_with_gate_then<raft::state_machine::start()::$_3>(seastar::gate&, raft::state_machine::start()::$_3&&)::'lambda'(seastar::abort_requested_exception const&)>(raft::state_machine::start()::$_3&&)::'lambda'(raft::state_machine::start()::$_3&&)&, seastar::future_state<seastar::internal::monostate>&&), void>
Nov 17 08:48:14 srv-01-03-404.iad1.trmr.io systemd[1]: redpanda.service: main process exited, code=killed, status=4/ILL
Nov 17 08:48:14 srv-01-03-404.iad1.trmr.io systemd[1]: Failed to start Redpanda, the fastest queue in the West..
Nov 17 08:48:14 srv-01-03-404.iad1.trmr.io systemd[1]: Unit redpanda.service entered failed state.
Nov 17 08:48:14 srv-01-03-404.iad1.trmr.io systemd[1]: redpanda.service failed.
Nov 17 08:48:14 srv-01-03-404.iad1.trmr.io systemd[1]: redpanda.service holdoff time over, scheduling restart.
Nov 17 08:48:14 srv-01-03-404.iad1.trmr.io systemd[1]: Stopped Redpanda, the fastest queue in the West..Also, here are the logs from 22.2.7 from when I issued the rpk cluster maintenance enable 0 --wait command and then stopped redpanda. Not sure if there will be anything of use or not.
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 0] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 1] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 2] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 8] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 13] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 3] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 26] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 12] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 37] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 15] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 29] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 5] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 57] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 38] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 30] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 58] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 19] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 18] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 47] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 41] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 36] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 53] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 39] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 54] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 6] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 27] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 31] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 40] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 20] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 24] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 33] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 32] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 4] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 23] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 7] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 50] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 43] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 22] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 44] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 42] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 17] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 9] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 59] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 28] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 52] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 55] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 16] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 14] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 25] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 51] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 48] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 49] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 34] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 35] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 46] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 60] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 63] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 45] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 11] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 62] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 56] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 10] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,633 [shard 61] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 21] cluster - members_table.cc:216 - marking node 0 in maintenance state
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 0] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 0] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 0] cluster - drain_manager.cc:256 - Node draining has completed on shard 0
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 1] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 8] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 5] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 4] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 11] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 2] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 16] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 26] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 30] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 33] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 32] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 38] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 42] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 10] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 47] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 50] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 56] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 21] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 59] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 63] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 15] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 11] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 25] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 24] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 29] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 22] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 13] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 27] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 3] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 16] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 28] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 23] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 23] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 36] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 34] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 34] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 23] raft - [group_id:6923, {kafka/rxdata-impnurl/2}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1254}} to {id: {9}, revision: {1254}} in term 18
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 35] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 40] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 39] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 9] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 9] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 44] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 44] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 45] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 45] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 45] cluster - drain_manager.cc:256 - Node draining has completed on shard 45
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 49] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 8] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 51] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 53] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 52] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 52] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 14] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 14] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 54] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 57] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 57] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 58] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 60] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 58] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 58] cluster - drain_manager.cc:256 - Node draining has completed on shard 58
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 21] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 6] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 59] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 6] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 6] cluster - drain_manager.cc:256 - Node draining has completed on shard 6
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 15] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 25] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 11] raft - [group_id:6674, {kafka/pixelverse_conv_eu/3}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1204}} to {id: {3}, revision: {1204}} in term 16
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 15] raft - [group_id:6689, {kafka/pixelverse_conv_us/8}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1206}} to {id: {5}, revision: {1206}} in term 22
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 24] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 15] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 25] raft - [group_id:6720, {kafka/rx-aerokeys-userdata/9}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1212}} to {id: {7}, revision: {1212}} in term 19
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 25] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 31] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 28] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 37] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 28] raft - [group_id:6940, {kafka/rxdata-impserve/9}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1256}} to {id: {3}, revision: {1256}} in term 13
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 36] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 43] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 30] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 36] raft - [group_id:6968, {kafka/rxdata-outstream_pread_event/7}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1262}} to {id: {7}, revision: {1262}} in term 15
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 23] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 35] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 9] raft - [group_id:6669, {kafka/pixelverse_conv_diagnostic/8}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1202}} to {id: {4}, revision: {1202}} in term 13
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 32] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 46] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 48] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 12] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 12] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 8] raft - [group_id:6875, {kafka/rxdata-dspbidrequest/4}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1244}} to {id: {5}, revision: {1244}} in term 22
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 53] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 10] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 47] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 47] cluster - drain_manager.cc:256 - Node draining has completed on shard 47
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 10] raft - [group_id:6672, {kafka/pixelverse_conv_eu/1}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1204}} to {id: {4}, revision: {1204}} in term 15
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 17] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 1] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 54] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 50] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 57] cluster - drain_manager.cc:256 - Node draining has completed on shard 57
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 19] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 60] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 61] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 62] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 62] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 62] cluster - drain_manager.cc:256 - Node draining has completed on shard 62
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 18] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 63] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 59] cluster - drain_manager.cc:256 - Node draining has completed on shard 59
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 18] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 29] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 11] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 18] raft - [group_id:6696, {kafka/rx-aerokeys-devicedata/5}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1208}} to {id: {9}, revision: {1208}} in term 17
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 18] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 27] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 27] raft - [group_id:6938, {kafka/rxdata-impserve/7}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1256}} to {id: {5}, revision: {1256}} in term 21
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 27] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 16] cluster - drain_manager.cc:256 - Node draining has completed on shard 16
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 31] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 26] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 7] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 34] raft - [group_id:7320, {kafka/rx-fc-usw/9}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1343}} to {id: {7}, revision: {1343}} in term 19
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 7] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 39] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 40] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 33] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 30] raft - [group_id:6738, {kafka/rx-cm-post/7}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1216}} to {id: {3}, revision: {1216}} in term 16
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 36] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 5] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 35] raft - [group_id:6963, {kafka/rxdata-outstream_pread_event/2}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1262}} to {id: {6}, revision: {1262}} in term 19
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 35] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 5] raft - [group_id:6655, {kafka/pixelverse_conv_ap/4}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1200}} to {id: {2}, revision: {1200}} in term 19
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 38] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 4] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 52] cluster - drain_manager.cc:256 - Node draining has completed on shard 52
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 12] raft - [group_id:6889, {kafka/rxdata-dvviewability/8}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1246}} to {id: {9}, revision: {1246}} in term 19
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 10] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 53] raft - [group_id:6814, {kafka/rx-uid-map-batch/3}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1232}} to {id: {4}, revision: {1232}} in term 19
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 14] raft - [group_id:7109, {kafka/summarystats-supplystat/8}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1290}} to {id: {2}, revision: {1290}} in term 23
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 55] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 60] cluster - drain_manager.cc:256 - Node draining has completed on shard 60
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 61] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 56] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 63] cluster - drain_manager.cc:256 - Node draining has completed on shard 63
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 13] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 3] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 31] cluster - drain_manager.cc:256 - Node draining has completed on shard 31
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 29] raft - [group_id:6946, {kafka/rxdata-lurl/5}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1258}} to {id: {7}, revision: {1258}} in term 17
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 37] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 28] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 43] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 43] cluster - drain_manager.cc:256 - Node draining has completed on shard 43
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 41] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 7] raft - [group_id:6661, {kafka/pixelverse_conv_diagnostic/0}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1202}} to {id: {9}, revision: {1202}} in term 16
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 33] raft - [group_id:7314, {kafka/rx-fc-usw/3}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1343}} to {id: {1}, revision: {1343}} in term 12
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 7] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 9] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 48] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 46] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 32] raft - [group_id:7313, {kafka/rx-fc-usw/2}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1343}} to {id: {3}, revision: {1343}} in term 12
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 32] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 38] cluster - drain_manager.cc:256 - Node draining has completed on shard 38
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 5] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 42] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 2] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 54] cluster - drain_manager.cc:256 - Node draining has completed on shard 54
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 2] raft - [group_id:6645, {kafka/dmp-cookie-sync/4}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1198}} to {id: {6}, revision: {1198}} in term 18
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 53] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 19] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 19] cluster - drain_manager.cc:256 - Node draining has completed on shard 19
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 17] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 56] cluster - drain_manager.cc:256 - Node draining has completed on shard 56
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,639 [shard 22] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 17] raft - [group_id:6694, {kafka/rx-aerokeys-devicedata/3}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1208}} to {id: {8}, revision: {1208}} in term 20
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 24] raft - [group_id:6929, {kafka/rxdata-impnurl/8}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1254}} to {id: {5}, revision: {1254}} in term 19
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 24] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 37] cluster - drain_manager.cc:256 - Node draining has completed on shard 37
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 40] cluster - drain_manager.cc:256 - Node draining has completed on shard 40
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 30] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 44] cluster - drain_manager.cc:256 - Node draining has completed on shard 44
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 46] cluster - drain_manager.cc:256 - Node draining has completed on shard 46
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 48] cluster - drain_manager.cc:256 - Node draining has completed on shard 48
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 49] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 4] raft - [group_id:6650, {kafka/dmp-cookie-sync/9}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1198}} to {id: {2}, revision: {1198}} in term 19
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 4] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 21] cluster - drain_manager.cc:256 - Node draining has completed on shard 21
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 55] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 26] raft - [group_id:6933, {kafka/rxdata-impserve/2}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1256}} to {id: {5}, revision: {1256}} in term 22
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 22] raft - [group_id:6712, {kafka/rx-aerokeys-userdata/1}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1212}} to {id: {9}, revision: {1212}} in term 24
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 26] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 51] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 33] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 8] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 42] cluster - drain_manager.cc:256 - Node draining has completed on shard 42
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 14] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 1] raft - [group_id:6852, {kafka/rxdata-bidresponse_extra/1}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1240}} to {id: {2}, revision: {1240}} in term 17
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 61] cluster - drain_manager.cc:256 - Node draining has completed on shard 61
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 1] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 17] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 34] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 13] raft - [group_id:6681, {kafka/pixelverse_conv_us/0}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1206}} to {id: {2}, revision: {1206}} in term 24
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 7] raft - [group_id:6661, {kafka/pixelverse_conv_diagnostic/0}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 16, dirty offset 6817
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 13] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 22] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 12] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 3] cluster - drain_manager.cc:256 - Node draining has completed on shard 3
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 50] cluster - drain_manager.cc:256 - Node draining has completed on shard 50
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 51] raft - [group_id:6808, {kafka/rx-uid-map/7}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1230}} to {id: {6}, revision: {1230}} in term 23
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 25] raft - [group_id:6720, {kafka/rx-aerokeys-userdata/9}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 19, dirty offset 18
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 39] cluster - drain_manager.cc:256 - Node draining has completed on shard 39
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 49] cluster - drain_manager.cc:256 - Node draining has completed on shard 49
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 55] cluster - drain_manager.cc:256 - Node draining has completed on shard 55
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 27] raft - [group_id:6938, {kafka/rxdata-impserve/7}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 21, dirty offset 2768205137
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 7] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 41] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 24] raft - [group_id:6929, {kafka/rxdata-impnurl/8}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 19, dirty offset 96728752
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 30] raft - [group_id:6738, {kafka/rx-cm-post/7}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 16, dirty offset 15
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 29] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 25] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 2] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 27] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 7] cluster - drain_manager.cc:256 - Node draining has completed on shard 7
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 24] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 41] cluster - drain_manager.cc:256 - Node draining has completed on shard 41
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 30] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 24] cluster - drain_manager.cc:256 - Node draining has completed on shard 24
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 30] cluster - drain_manager.cc:256 - Node draining has completed on shard 30
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 1] raft - [group_id:6852, {kafka/rxdata-bidresponse_extra/1}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 17, dirty offset 12342762
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 27] cluster - drain_manager.cc:256 - Node draining has completed on shard 27
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 22] raft - [group_id:6712, {kafka/rx-aerokeys-userdata/1}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 24, dirty offset 23
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 51] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,640 [shard 25] cluster - drain_manager.cc:256 - Node draining has completed on shard 25
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 28] raft - [group_id:6940, {kafka/rxdata-impserve/9}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 13, dirty offset 2768304980
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 1] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 28] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 1] cluster - drain_manager.cc:256 - Node draining has completed on shard 1
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 36] raft - [group_id:6968, {kafka/rxdata-outstream_pread_event/7}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 15, dirty offset 44804
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 5] raft - [group_id:6655, {kafka/pixelverse_conv_ap/4}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 19, dirty offset 18
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 22] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 14] raft - [group_id:7109, {kafka/summarystats-supplystat/8}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 23, dirty offset 6103
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 18] raft - [group_id:6696, {kafka/rx-aerokeys-devicedata/5}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 17, dirty offset 16
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 29] raft - [group_id:6946, {kafka/rxdata-lurl/5}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 17, dirty offset 433872858
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 11] raft - [group_id:6674, {kafka/pixelverse_conv_eu/3}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 16, dirty offset 15
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 10] raft - [group_id:6672, {kafka/pixelverse_conv_eu/1}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 15, dirty offset 14
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 17] raft - [group_id:6694, {kafka/rx-aerokeys-devicedata/3}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 20, dirty offset 19
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 5] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 5] cluster - drain_manager.cc:256 - Node draining has completed on shard 5
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 22] cluster - drain_manager.cc:256 - Node draining has completed on shard 22
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 14] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 29] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 11] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 29] cluster - drain_manager.cc:256 - Node draining has completed on shard 29
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 35] raft - [group_id:6963, {kafka/rxdata-outstream_pread_event/2}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 19, dirty offset 44978
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 32] raft - [group_id:7313, {kafka/rx-fc-usw/2}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 12, dirty offset 12
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 9] raft - [group_id:6669, {kafka/pixelverse_conv_diagnostic/8}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 13, dirty offset 6788
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 35] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 4] raft - [group_id:6650, {kafka/dmp-cookie-sync/9}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 19, dirty offset 67526875
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 14] cluster - drain_manager.cc:256 - Node draining has completed on shard 14
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 9] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 33] raft - [group_id:7314, {kafka/rx-fc-usw/3}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 12, dirty offset 15
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 12] raft - [group_id:6889, {kafka/rxdata-dvviewability/8}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 19, dirty offset 5587526
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 34] raft - [group_id:7320, {kafka/rx-fc-usw/9}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 19, dirty offset 19
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 10] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 10] cluster - drain_manager.cc:256 - Node draining has completed on shard 10
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 53] raft - [group_id:6814, {kafka/rx-uid-map-batch/3}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 19, dirty offset 12488437
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 26] raft - [group_id:6933, {kafka/rxdata-impserve/2}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 22, dirty offset 2768351744
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 53] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 36] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 36] cluster - drain_manager.cc:256 - Node draining has completed on shard 36
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 8] raft - [group_id:6875, {kafka/rxdata-dspbidrequest/4}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 22, dirty offset 1177612364
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 33] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 34] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 34] cluster - drain_manager.cc:256 - Node draining has completed on shard 34
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 18] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 2] raft - [group_id:6645, {kafka/dmp-cookie-sync/4}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 18, dirty offset 67526307
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 13] raft - [group_id:6681, {kafka/pixelverse_conv_us/0}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 24, dirty offset 1050
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 2] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 2] cluster - drain_manager.cc:256 - Node draining has completed on shard 2
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 13] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 35] cluster - drain_manager.cc:256 - Node draining has completed on shard 35
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 32] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 32] cluster - drain_manager.cc:256 - Node draining has completed on shard 32
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 11] cluster - drain_manager.cc:256 - Node draining has completed on shard 11
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 51] raft - [group_id:6808, {kafka/rx-uid-map/7}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 23, dirty offset 21423199
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 33] cluster - drain_manager.cc:256 - Node draining has completed on shard 33
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 12] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 51] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 12] cluster - drain_manager.cc:256 - Node draining has completed on shard 12
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 51] cluster - drain_manager.cc:256 - Node draining has completed on shard 51
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 13] cluster - drain_manager.cc:256 - Node draining has completed on shard 13
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 4] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 9] cluster - drain_manager.cc:256 - Node draining has completed on shard 9
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 28] cluster - drain_manager.cc:256 - Node draining has completed on shard 28
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 17] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 26] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 26] cluster - drain_manager.cc:256 - Node draining has completed on shard 26
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 4] cluster - drain_manager.cc:256 - Node draining has completed on shard 4
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 8] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 8] cluster - drain_manager.cc:256 - Node draining has completed on shard 8
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 15] raft - [group_id:6689, {kafka/pixelverse_conv_us/8}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 22, dirty offset 1049
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 17] cluster - drain_manager.cc:256 - Node draining has completed on shard 17
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 15] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 15] cluster - drain_manager.cc:256 - Node draining has completed on shard 15
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 18] cluster - drain_manager.cc:256 - Node draining has completed on shard 18
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 53] cluster - drain_manager.cc:256 - Node draining has completed on shard 53
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 23] raft - [group_id:6923, {kafka/rxdata-impnurl/2}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 18, dirty offset 96754064
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 23] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,641 [shard 23] cluster - drain_manager.cc:256 - Node draining has completed on shard 23
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,650 [shard 7] raft - [group_id:6661, {kafka/pixelverse_conv_diagnostic/0}] consensus.cc:1566 - Received vote request with larger term from node {id: {9}, revision: {1202}}, received 17, current 16
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,650 [shard 25] raft - [group_id:6720, {kafka/rx-aerokeys-userdata/9}] consensus.cc:1566 - Received vote request with larger term from node {id: {7}, revision: {1212}}, received 20, current 19
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,650 [shard 30] raft - [group_id:6738, {kafka/rx-cm-post/7}] consensus.cc:1566 - Received vote request with larger term from node {id: {3}, revision: {1216}}, received 17, current 16
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,650 [shard 24] raft - [group_id:6929, {kafka/rxdata-impnurl/8}] consensus.cc:1566 - Received vote request with larger term from node {id: {5}, revision: {1254}}, received 20, current 19
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,651 [shard 28] raft - [group_id:6940, {kafka/rxdata-impserve/9}] consensus.cc:1566 - Received vote request with larger term from node {id: {3}, revision: {1256}}, received 14, current 13
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,651 [shard 1] raft - [group_id:6852, {kafka/rxdata-bidresponse_extra/1}] consensus.cc:1566 - Received vote request with larger term from node {id: {2}, revision: {1240}}, received 18, current 17
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,651 [shard 5] raft - [group_id:6655, {kafka/pixelverse_conv_ap/4}] consensus.cc:1566 - Received vote request with larger term from node {id: {2}, revision: {1200}}, received 20, current 19
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,651 [shard 14] raft - [group_id:7109, {kafka/summarystats-supplystat/8}] consensus.cc:1566 - Received vote request with larger term from node {id: {2}, revision: {1290}}, received 24, current 23
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,651 [shard 36] raft - [group_id:6968, {kafka/rxdata-outstream_pread_event/7}] consensus.cc:1566 - Received vote request with larger term from node {id: {7}, revision: {1262}}, received 16, current 15
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,651 [shard 11] raft - [group_id:6674, {kafka/pixelverse_conv_eu/3}] consensus.cc:1566 - Received vote request with larger term from node {id: {3}, revision: {1204}}, received 17, current 16
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,651 [shard 18] raft - [group_id:6696, {kafka/rx-aerokeys-devicedata/5}] consensus.cc:1566 - Received vote request with larger term from node {id: {9}, revision: {1208}}, received 18, current 17
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,651 [shard 29] raft - [group_id:6946, {kafka/rxdata-lurl/5}] consensus.cc:1566 - Received vote request with larger term from node {id: {7}, revision: {1258}}, received 18, current 17
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,651 [shard 9] raft - [group_id:6669, {kafka/pixelverse_conv_diagnostic/8}] consensus.cc:1566 - Received vote request with larger term from node {id: {4}, revision: {1202}}, received 14, current 13
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,651 [shard 32] raft - [group_id:7313, {kafka/rx-fc-usw/2}] consensus.cc:1566 - Received vote request with larger term from node {id: {3}, revision: {1343}}, received 13, current 12
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,651 [shard 4] raft - [group_id:6650, {kafka/dmp-cookie-sync/9}] consensus.cc:1566 - Received vote request with larger term from node {id: {2}, revision: {1198}}, received 20, current 19
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,651 [shard 17] raft - [group_id:6694, {kafka/rx-aerokeys-devicedata/3}] consensus.cc:1566 - Received vote request with larger term from node {id: {8}, revision: {1208}}, received 21, current 20
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,651 [shard 23] raft - [group_id:6923, {kafka/rxdata-impnurl/2}] consensus.cc:1566 - Received vote request with larger term from node {id: {9}, revision: {1254}}, received 19, current 18
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,651 [shard 27] raft - [group_id:6938, {kafka/rxdata-impserve/7}] consensus.cc:1566 - Received vote request with larger term from node {id: {5}, revision: {1256}}, received 22, current 21
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,651 [shard 53] raft - [group_id:6814, {kafka/rx-uid-map-batch/3}] consensus.cc:1566 - Received vote request with larger term from node {id: {4}, revision: {1232}}, received 20, current 19
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,651 [shard 22] raft - [group_id:6712, {kafka/rx-aerokeys-userdata/1}] consensus.cc:1566 - Received vote request with larger term from node {id: {9}, revision: {1212}}, received 25, current 24
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,651 [shard 15] raft - [group_id:6689, {kafka/pixelverse_conv_us/8}] consensus.cc:1566 - Received vote request with larger term from node {id: {5}, revision: {1206}}, received 23, current 22
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,651 [shard 13] raft - [group_id:6681, {kafka/pixelverse_conv_us/0}] consensus.cc:1566 - Received vote request with larger term from node {id: {2}, revision: {1206}}, received 25, current 24
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,651 [shard 51] raft - [group_id:6808, {kafka/rx-uid-map/7}] consensus.cc:1566 - Received vote request with larger term from node {id: {6}, revision: {1230}}, received 24, current 23
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,651 [shard 2] raft - [group_id:6645, {kafka/dmp-cookie-sync/4}] consensus.cc:1566 - Received vote request with larger term from node {id: {6}, revision: {1198}}, received 19, current 18
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,651 [shard 8] raft - [group_id:6875, {kafka/rxdata-dspbidrequest/4}] consensus.cc:1566 - Received vote request with larger term from node {id: {5}, revision: {1244}}, received 23, current 22
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,651 [shard 33] raft - [group_id:7314, {kafka/rx-fc-usw/3}] consensus.cc:1566 - Received vote request with larger term from node {id: {1}, revision: {1343}}, received 13, current 12
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,651 [shard 26] raft - [group_id:6933, {kafka/rxdata-impserve/2}] consensus.cc:1566 - Received vote request with larger term from node {id: {5}, revision: {1256}}, received 23, current 22
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,655 [shard 20] cluster - drain_manager.cc:49 - Node draining is starting
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,655 [shard 12] raft - [group_id:6889, {kafka/rxdata-dvviewability/8}] consensus.cc:1566 - Received vote request with larger term from node {id: {9}, revision: {1246}}, received 20, current 19
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,655 [shard 20] cluster - drain_manager.cc:143 - Node draining has started
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,655 [shard 20] raft - [group_id:6705, {kafka/rx-aerokeys-ipdata/4}] consensus.cc:2935 - Starting leadership transfer from {id: {0}, revision: {1210}} to {id: {9}, revision: {1210}} in term 12
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,655 [shard 20] cluster - drain_manager.cc:207 - Draining leadership from 1 groups
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,655 [shard 20] raft - [group_id:6705, {kafka/rx-aerokeys-ipdata/4}] consensus.cc:187 - [leadership_transfer] Stepping down as leader in term 12, dirty offset 11
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,655 [shard 20] cluster - drain_manager.cc:239 - Draining leadership from 1 groups 1 succeeded
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,655 [shard 20] cluster - drain_manager.cc:256 - Node draining has completed on shard 20
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,661 [shard 10] raft - [group_id:6672, {kafka/pixelverse_conv_eu/1}] consensus.cc:1566 - Received vote request with larger term from node {id: {4}, revision: {1204}}, received 16, current 15
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,661 [shard 34] raft - [group_id:7320, {kafka/rx-fc-usw/9}] consensus.cc:1566 - Received vote request with larger term from node {id: {7}, revision: {1343}}, received 20, current 19
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,661 [shard 35] raft - [group_id:6963, {kafka/rxdata-outstream_pread_event/2}] consensus.cc:1566 - Received vote request with larger term from node {id: {6}, revision: {1262}}, received 20, current 19
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,665 [shard 20] raft - [group_id:6705, {kafka/rx-aerokeys-ipdata/4}] consensus.cc:1566 - Received vote request with larger term from node {id: {9}, revision: {1210}}, received 13, current 12
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,690 [shard 30] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rx-cm-post/7_1216/16-17-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,700 [shard 24] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rxdata-impnurl/8_1254/96728753-20-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,702 [shard 25] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rx-aerokeys-userdata/9_1212/19-20-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,705 [shard 9] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/pixelverse_conv_diagnostic/8_1202/6789-14-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,711 [shard 36] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rxdata-outstream_pread_event/7_1262/44805-16-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,713 [shard 27] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rxdata-impserve/7_1256/2768205138-22-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,713 [shard 35] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rxdata-outstream_pread_event/2_1262/44979-20-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,719 [shard 14] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/summarystats-supplystat/8_1290/6104-24-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,729 [shard 11] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/pixelverse_conv_eu/3_1204/16-17-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,729 [shard 26] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rxdata-impserve/2_1256/2768351745-23-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,735 [shard 17] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rx-aerokeys-devicedata/3_1208/20-21-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,736 [shard 20] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rx-aerokeys-ipdata/4_1210/12-13-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,753 [shard 33] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rx-fc-usw/3_1343/16-13-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,763 [shard 18] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rx-aerokeys-devicedata/5_1208/17-18-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,769 [shard 51] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rx-uid-map/7_1230/21423200-24-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,774 [shard 12] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rxdata-dvviewability/8_1246/5587527-20-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,774 [shard 8] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rxdata-dspbidrequest/4_1244/1177612365-23-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,778 [shard 4] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/dmp-cookie-sync/9_1198/67526876-20-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,781 [shard 10] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/pixelverse_conv_eu/1_1204/15-16-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,787 [shard 32] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rx-fc-usw/2_1343/13-13-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,789 [shard 5] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/pixelverse_conv_ap/4_1200/19-20-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,791 [shard 15] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/pixelverse_conv_us/8_1206/1050-23-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,793 [shard 2] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/dmp-cookie-sync/4_1198/67526308-19-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,796 [shard 28] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rxdata-impserve/9_1256/2768304981-14-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,799 [shard 34] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rx-fc-usw/9_1343/20-20-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,800 [shard 7] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/pixelverse_conv_diagnostic/0_1202/6818-17-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,820 [shard 22] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rx-aerokeys-userdata/1_1212/24-25-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,820 [shard 23] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rxdata-impnurl/2_1254/96754065-19-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,828 [shard 53] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rx-uid-map-batch/3_1232/12488438-20-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,828 [shard 13] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/pixelverse_conv_us/0_1206/1051-25-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,831 [shard 29] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rxdata-lurl/5_1258/433872859-18-v1.log
Nov 17 08:35:46 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:46,858 [shard 1] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rxdata-bidresponse_extra/1_1240/12342763-18-v1.log
Nov 17 08:35:47 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:47,304 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:35:48 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:48,304 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:35:49 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:49,304 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:35:50 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:50,304 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:35:51 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:51,304 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:35:52 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:52,304 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:35:53 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:53,304 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:35:54 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:54,140 [shard 58] storage - disk_log_impl.cc:1005 - remove_prefix_full_segments - tombstone & delete segment: {offset_tracker:{term:17, base_offset:8004665807, committed_offset:8004867551, dirty_offset:8004867551}, compacted_segment=0, finished_self_compaction=0, generation={0}, reader={/var/lib/redpanda/data/kafka/rx-usersync-us/9_1234/8004665807-17-v1.log, (107553213 bytes)}, writer=nullptr, cache={cache_size=0}, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/kafka/rx-usersync-us/9_1234/8004665807-17-v1.base_index, offsets:{8004665807}, index:{header_bitflags:0, base_offset:{8004665807}, max_offset:{8004867551}, base_timestamp:{timestamp: 1668605699066}, max_timestamp:{timestamp: 1668605746310}, index(486,486,486)}, step:32768, needs_persistence:0}}
Nov 17 08:35:54 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:54,140 [shard 58] storage - segment.cc:112 - removing: {"/var/lib/redpanda/data/kafka/rx-usersync-us/9_1234/8004665807-17-v1.log", "/var/lib/redpanda/data/kafka/rx-usersync-us/9_1234/8004665807-17-v1.base_index"}
Nov 17 08:35:54 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:54,170 [shard 58] offset_translator - ntp: {kafka/rx-usersync-us/9} - offset_translator.cc:277 - prefix_truncate at offset: 8004867551, new state: {base offset/delta: {8004867551}/17, map size: 1, last delta: 17}
Nov 17 08:35:54 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:54,304 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:35:54 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:54,458 [shard 62] storage - disk_log_impl.cc:1005 - remove_prefix_full_segments - tombstone & delete segment: {offset_tracker:{term:14, base_offset:254690063, committed_offset:254800087, dirty_offset:254800087}, compacted_segment=0, finished_self_compaction=0, generation={0}, reader={/var/lib/redpanda/data/kafka/rxdata-bidresponse/3_1238/254690063-14-v1.log, (105277585 bytes)}, writer=nullptr, cache={cache_size=0}, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/kafka/rxdata-bidresponse/3_1238/254690063-14-v1.base_index, offsets:{254690063}, index:{header_bitflags:0, base_offset:{254690063}, max_offset:{254800087}, base_timestamp:{timestamp: 1668430983677}, max_timestamp:{timestamp: 1668432947691}, index(3073,3073,3073)}, step:32768, needs_persistence:0}}
Nov 17 08:35:54 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:54,458 [shard 62] storage - segment.cc:112 - removing: {"/var/lib/redpanda/data/kafka/rxdata-bidresponse/3_1238/254690063-14-v1.log", "/var/lib/redpanda/data/kafka/rxdata-bidresponse/3_1238/254690063-14-v1.base_index"}
Nov 17 08:35:54 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:54,488 [shard 62] offset_translator - ntp: {kafka/rxdata-bidresponse/3} - offset_translator.cc:277 - prefix_truncate at offset: 254800087, new state: {base offset/delta: {254800087}/13, map size: 2, last delta: 14}
Nov 17 08:35:55 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:55,304 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:35:56 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:56,304 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:35:57 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:57,304 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:35:58 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:58,186 [shard 55] storage - disk_log_impl.cc:1005 - remove_prefix_full_segments - tombstone & delete segment: {offset_tracker:{term:13, base_offset:8005249365, committed_offset:8005437367, dirty_offset:8005437367}, compacted_segment=0, finished_self_compaction=0, generation={0}, reader={/var/lib/redpanda/data/kafka/rx-usersync-us/0_1234/8005249365-13-v1.log, (100440578 bytes)}, writer=nullptr, cache={cache_size=0}, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/kafka/rx-usersync-us/0_1234/8005249365-13-v1.base_index, offsets:{8005249365}, index:{header_bitflags:0, base_offset:{8005249365}, max_offset:{8005437367}, base_timestamp:{timestamp: 1668605699424}, max_timestamp:{timestamp: 1668605744327}, index(455,455,455)}, step:32768, needs_persistence:0}}
Nov 17 08:35:58 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:58,186 [shard 55] storage - segment.cc:112 - removing: {"/var/lib/redpanda/data/kafka/rx-usersync-us/0_1234/8005249365-13-v1.log", "/var/lib/redpanda/data/kafka/rx-usersync-us/0_1234/8005249365-13-v1.base_index"}
Nov 17 08:35:58 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:58,217 [shard 55] offset_translator - ntp: {kafka/rx-usersync-us/0} - offset_translator.cc:277 - prefix_truncate at offset: 8005437367, new state: {base offset/delta: {8005437367}/13, map size: 2, last delta: 14}
Nov 17 08:35:58 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:58,304 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:35:59 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:35:59,304 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:36:00 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:00,305 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:36:00 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:00,443 [shard 22] raft - [group_id:6712, {kafka/rx-aerokeys-userdata/1}] consensus.cc:1566 - Received vote request with larger term from node {id: {4}, revision: {1212}}, received 26, current 25
Nov 17 08:36:00 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:00,479 [shard 22] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rx-aerokeys-userdata/1_1212/25-26-v1.log
Nov 17 08:36:01 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:01,305 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:36:02 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:02,305 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:36:02 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:02,760 [shard 16] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rxdata-impbeacon/9_1248/749504034-17-v1.log
Nov 17 08:36:03 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:03,305 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:36:03 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:03,520 [shard 57] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rx-usersync-us/6_1234/8321194434-17-v1.log
Nov 17 08:36:04 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:04,305 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:36:05 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:05,168 [shard 55] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rx-usersync-us/0_1234/8319857959-14-v1.log
Nov 17 08:36:05 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:05,229 [shard 57] storage - disk_log_impl.cc:1005 - remove_prefix_full_segments - tombstone & delete segment: {offset_tracker:{term:17, base_offset:8007003919, committed_offset:8007206679, dirty_offset:8007206679}, compacted_segment=0, finished_self_compaction=0, generation={0}, reader={/var/lib/redpanda/data/kafka/rx-usersync-us/6_1234/8007003919-17-v1.log, (108004183 bytes)}, writer=nullptr, cache={cache_size=0}, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/kafka/rx-usersync-us/6_1234/8007003919-17-v1.base_index, offsets:{8007003919}, index:{header_bitflags:0, base_offset:{8007003919}, max_offset:{8007206679}, base_timestamp:{timestamp: 1668605717659}, max_timestamp:{timestamp: 1668605762096}, index(488,488,488)}, step:32768, needs_persistence:0}}
Nov 17 08:36:05 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:05,229 [shard 57] storage - segment.cc:112 - removing: {"/var/lib/redpanda/data/kafka/rx-usersync-us/6_1234/8007003919-17-v1.log", "/var/lib/redpanda/data/kafka/rx-usersync-us/6_1234/8007003919-17-v1.base_index"}
Nov 17 08:36:05 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:05,259 [shard 57] offset_translator - ntp: {kafka/rx-usersync-us/6} - offset_translator.cc:277 - prefix_truncate at offset: 8007206679, new state: {base offset/delta: {8007206679}/17, map size: 1, last delta: 17}
Nov 17 08:36:05 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:05,305 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:36:05 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:05,642 [shard 26] storage - disk_log_impl.cc:1005 - remove_prefix_full_segments - tombstone & delete segment: {offset_tracker:{term:18, base_offset:18984496149, committed_offset:18985202084, dirty_offset:18985202084}, compacted_segment=0, finished_self_compaction=0, generation={0}, reader={/var/lib/redpanda/data/kafka/rx-bidrequest-stream/4_1214/18984496149-18-v1.log, (109366968 bytes)}, writer=nullptr, cache={cache_size=0}, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/kafka/rx-bidrequest-stream/4_1214/18984496149-18-v1.base_index, offsets:{18984496149}, index:{header_bitflags:0, base_offset:{18984496149}, max_offset:{18985202084}, base_timestamp:{timestamp: 1668432812000}, max_timestamp:{timestamp: 1668432959000}, index(3297,3297,3297)}, step:32768, needs_persistence:0}}
Nov 17 08:36:05 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:05,642 [shard 26] storage - segment.cc:112 - removing: {"/var/lib/redpanda/data/kafka/rx-bidrequest-stream/4_1214/18984496149-18-v1.log", "/var/lib/redpanda/data/kafka/rx-bidrequest-stream/4_1214/18984496149-18-v1.base_index"}
Nov 17 08:36:05 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:05,673 [shard 26] offset_translator - ntp: {kafka/rx-bidrequest-stream/4} - offset_translator.cc:277 - prefix_truncate at offset: 18985202084, new state: {base offset/delta: {18985202084}/18, map size: 2, last delta: 19}
Nov 17 08:36:06 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:06,305 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:36:07 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:07,305 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:36:08 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:08,305 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:36:09 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:09,305 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:36:10 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:10,305 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:36:11 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:11,305 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:36:11 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:11,906 [shard 56] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rx-usersync-us/3_1234/8320901771-16-v1.log
Nov 17 08:36:12 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:12,305 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:36:12 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:12,948 [shard 26] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rx-bidrequest-stream/4_1214/20396372245-18-v1.log
Nov 17 08:36:13 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:13,305 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:36:14 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:14,305 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:36:15 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:15,211 [shard 56] storage - disk_log_impl.cc:1005 - remove_prefix_full_segments - tombstone & delete segment: {offset_tracker:{term:16, base_offset:8006309648, committed_offset:8006505235, dirty_offset:8006505235}, compacted_segment=0, finished_self_compaction=0, generation={0}, reader={/var/lib/redpanda/data/kafka/rx-usersync-us/3_1234/8006309648-16-v1.log, (104018962 bytes)}, writer=nullptr, cache={cache_size=0}, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/kafka/rx-usersync-us/3_1234/8006309648-16-v1.base_index, offsets:{8006309648}, index:{header_bitflags:0, base_offset:{8006309648}, max_offset:{8006505235}, base_timestamp:{timestamp: 1668605724143}, max_timestamp:{timestamp: 1668605769574}, index(467,467,467)}, step:32768, needs_persistence:0}}
Nov 17 08:36:15 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:15,211 [shard 56] storage - segment.cc:112 - removing: {"/var/lib/redpanda/data/kafka/rx-usersync-us/3_1234/8006309648-16-v1.log", "/var/lib/redpanda/data/kafka/rx-usersync-us/3_1234/8006309648-16-v1.base_index"}
Nov 17 08:36:15 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:15,241 [shard 56] offset_translator - ntp: {kafka/rx-usersync-us/3} - offset_translator.cc:277 - prefix_truncate at offset: 8006505235, new state: {base offset/delta: {8006505235}/15, map size: 1, last delta: 15}
Nov 17 08:36:15 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:15,305 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:36:16 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:16,218 [shard 27] storage - disk_log_impl.cc:1005 - remove_prefix_full_segments - tombstone & delete segment: {offset_tracker:{term:18, base_offset:2570288095, committed_offset:2570420954, dirty_offset:2570420954}, compacted_segment=0, finished_self_compaction=0, generation={0}, reader={/var/lib/redpanda/data/kafka/rxdata-impserve/7_1256/2570288095-18-v1.log, (102658275 bytes)}, writer=nullptr, cache={cache_size=0}, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/kafka/rxdata-impserve/7_1256/2570288095-18-v1.base_index, offsets:{2570288095}, index:{header_bitflags:0, base_offset:{2570288095}, max_offset:{2570420954}, base_timestamp:{timestamp: 1668432740789}, max_timestamp:{timestamp: 1668432966363}, index(2979,2979,2979)}, step:32768, needs_persistence:0}}
Nov 17 08:36:16 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:16,218 [shard 27] storage - segment.cc:112 - removing: {"/var/lib/redpanda/data/kafka/rxdata-impserve/7_1256/2570288095-18-v1.log", "/var/lib/redpanda/data/kafka/rxdata-impserve/7_1256/2570288095-18-v1.base_index"}
Nov 17 08:36:16 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:16,218 [shard 27] storage - disk_log_impl.cc:1005 - remove_prefix_full_segments - tombstone & delete segment: {offset_tracker:{term:12, base_offset:18984669590, committed_offset:18985342161, dirty_offset:18985342161}, compacted_segment=0, finished_self_compaction=0, generation={0}, reader={/var/lib/redpanda/data/kafka/rx-bidrequest-stream/6_1214/18984669590-12-v1.log, (103704564 bytes)}, writer=nullptr, cache={cache_size=0}, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/kafka/rx-bidrequest-stream/6_1214/18984669590-12-v1.base_index, offsets:{18984669590}, index:{header_bitflags:0, base_offset:{18984669590}, max_offset:{18985342161}, base_timestamp:{timestamp: 1668432827000}, max_timestamp:{timestamp: 1668432966000}, index(3127,3127,3127)}, step:32768, needs_persistence:0}}
Nov 17 08:36:16 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:16,218 [shard 27] storage - segment.cc:112 - removing: {"/var/lib/redpanda/data/kafka/rx-bidrequest-stream/6_1214/18984669590-12-v1.log", "/var/lib/redpanda/data/kafka/rx-bidrequest-stream/6_1214/18984669590-12-v1.base_index"}
Nov 17 08:36:16 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:16,248 [shard 27] offset_translator - ntp: {kafka/rxdata-impserve/7} - offset_translator.cc:277 - prefix_truncate at offset: 2570420954, new state: {base offset/delta: {2570420954}/16, map size: 5, last delta: 20}
Nov 17 08:36:16 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:16,248 [shard 27] offset_translator - ntp: {kafka/rx-bidrequest-stream/6} - offset_translator.cc:277 - prefix_truncate at offset: 18985342161, new state: {base offset/delta: {18985342161}/12, map size: 2, last delta: 13}
Nov 17 08:36:16 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:16,305 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:36:16 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:16,781 [shard 4] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rxdata-videoevent/2_1284/2365889976-18-v1.log
Nov 17 08:36:17 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:17,305 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:36:18 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:18,305 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:36:19 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:19,305 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:36:20 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:20,305 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,305 [shard 26] cluster - controller_backend.cc:678 - partition operation {type: update, ntp: {kafka/rx-bidrequest-stream/4}, offset: 1397, new_assignment: { id: 4, group_id: 6725, replicas: {{node_id: 9, shard: 39}, {node_id: 1, shard: 26}, {node_id: 6, shard: 26}} }, previous_replica_set: {{{node_id: 1, shard: 26}, {node_id: 6, shard: 26}, {node_id: 0, shard: 26}}}} result: Current node is not a leader for partition
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io systemd[1]: Stopping Redpanda, the fastest queue in the West....
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[4810]: Command "stop" is deprecated, use "rpk redpanda stop" instead
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,715 [shard 0] redpanda::main - application.cc:250 - Stopping...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,715 [shard 0] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,715 [shard 0] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 2] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 8] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 12] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 14] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 15] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 18] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 26] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 29] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 25] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 36] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 42] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 48] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 45] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 46] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 56] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 61] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 6] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 16] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 17] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 61] rpc - server.cc:262 - kafka rpc protocol - Shutting down 425 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 16] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 19] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 10] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 21] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 27] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 28] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 21] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 9] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 32] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 38] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 4] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 30] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 38] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 22] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 33] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 34] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 41] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 34] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 39] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 13] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 41] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 39] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 35] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 37] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 40] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 37] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 23] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 40] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 49] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 3] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 49] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 42] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 31] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 59] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 31] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 12] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 52] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 47] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 52] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 57] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 47] rpc - server.cc:262 - kafka rpc protocol - Shutting down 425 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 45] rpc - server.cc:262 - kafka rpc protocol - Shutting down 425 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 60] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 62] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 46] rpc - server.cc:262 - kafka rpc protocol - Shutting down 425 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 48] rpc - server.cc:262 - kafka rpc protocol - Shutting down 425 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 60] rpc - server.cc:262 - kafka rpc protocol - Shutting down 425 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 6] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 5] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 17] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 19] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 10] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 27] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 26] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 29] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 9] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 32] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 4] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 24] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 22] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 24] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 25] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 18] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 20] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 13] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 35] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 1] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 43] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 50] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 43] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 44] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 3] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 8] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 54] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 58] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 54] rpc - server.cc:262 - kafka rpc protocol - Shutting down 425 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 53] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 59] rpc - server.cc:262 - kafka rpc protocol - Shutting down 425 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 53] rpc - server.cc:262 - kafka rpc protocol - Shutting down 425 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 63] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 55] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,718 [shard 57] rpc - server.cc:262 - kafka rpc protocol - Shutting down 425 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,718 [shard 62] rpc - server.cc:262 - kafka rpc protocol - Shutting down 425 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 7] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 14] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 5] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,718 [shard 15] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 30] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 33] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 36] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 1] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,718 [shard 23] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 50] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 44] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 58] rpc - server.cc:262 - kafka rpc protocol - Shutting down 425 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 2] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,718 [shard 63] rpc - server.cc:262 - kafka rpc protocol - Shutting down 425 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,718 [shard 56] rpc - server.cc:262 - kafka rpc protocol - Shutting down 425 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 11] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,717 [shard 28] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,718 [shard 20] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,718 [shard 11] rpc - server.cc:262 - kafka rpc protocol - Shutting down 425 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,718 [shard 7] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,718 [shard 55] rpc - server.cc:262 - kafka rpc protocol - Shutting down 425 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,738 [shard 51] rpc - server.cc:253 - kafka rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,738 [shard 51] rpc - server.cc:262 - kafka rpc protocol - Shutting down 426 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,741 [shard 0] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,741 [shard 0] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 2 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,741 [shard 4] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,741 [shard 4] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 2 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,741 [shard 0] rpc - server.cc:135 - vectorized internal rpc protocol - Disconnected 10.16.67.44:54400 (applying protocol, parse error)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,741 [shard 0] rpc - server.cc:135 - vectorized internal rpc protocol - Disconnected 10.16.67.46:54016 (applying protocol, parse error)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,742 [shard 42] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,742 [shard 42] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 1 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,742 [shard 9] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,742 [shard 9] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 1 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,743 [shard 34] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,743 [shard 34] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 1 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,743 [shard 35] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,743 [shard 35] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 2 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,743 [shard 24] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,743 [shard 24] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 1 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,743 [shard 63] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,743 [shard 63] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 0 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,743 [shard 34] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.49:59874 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,743 [shard 49] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,743 [shard 5] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,743 [shard 49] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 2 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,743 [shard 5] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 0 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,743 [shard 61] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,743 [shard 61] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 3 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,743 [shard 18] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,743 [shard 18] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 2 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,744 [shard 58] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,744 [shard 58] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 2 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,744 [shard 17] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,744 [shard 17] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 1 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,744 [shard 29] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,744 [shard 29] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 2 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,744 [shard 21] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,744 [shard 21] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 1 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,744 [shard 45] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,744 [shard 45] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 1 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,744 [shard 20] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,744 [shard 20] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 1 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,744 [shard 39] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,744 [shard 39] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 0 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,744 [shard 22] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,744 [shard 22] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 1 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,744 [shard 56] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,744 [shard 56] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 2 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 15] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 25] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 15] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 1 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 25] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 1 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 6] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 6] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 1 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 27] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 27] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 0 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 2] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 12] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 2] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 0 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 12] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 2 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 58] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.51:54138 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 31] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 31] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 1 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 26] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 26] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 1 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 54] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 57] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 54] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 1 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 31] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.50:56543 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 43] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 44] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 43] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 0 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 44] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 1 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 57] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 1 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 60] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 60] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 0 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 8] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 31] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.50:56543 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 8] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 2 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 37] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 7] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 7] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 2 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 0] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.44:54400 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 37] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 1 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 0] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.44:54400 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 0] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.44:54400 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 46] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 46] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 0 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 28] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 28] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 0 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 50] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 45] rpc - simple_protocol.cc:92 - Disconnected 10.16.66.78:63725 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 50] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 0 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 32] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 30] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 32] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 0 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 41] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 30] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 1 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 56] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.49:59448 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 41] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 0 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 15] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.46:62095 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 19] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 15] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.46:62095 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 19] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 2 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 1] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 1] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 0 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 3] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 7] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.51:62279 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 36] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 3] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 1 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,745 [shard 36] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 2 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 35] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.44:55779 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 62] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 11] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 62] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 1 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 11] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 0 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 38] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 7] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.51:62279 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 38] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 2 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 47] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 47] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 0 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 59] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 59] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 1 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 36] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.50:59876 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 15] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.46:62095 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 4] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.43:54596 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 48] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 35] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.44:55779 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 15] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.46:62095 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 48] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 1 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 15] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.46:62095 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 15] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.46:62095 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 15] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.46:62095 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 15] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.46:62095 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 0] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.44:54400 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 48] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.47:63664 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 10] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 19] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.49:59411 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 10] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 1 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 36] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.50:59876 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 0] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.44:54400 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 55] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 55] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 0 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 15] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.46:62095 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 38] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.50:60070 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 52] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 52] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 1 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 4] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.43:54596 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 38] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.45:57254 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 4] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.43:54596 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 16] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 4] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.43:54596 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 16] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 0 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 4] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.43:54596 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 40] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 10] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.50:54282 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 4] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.43:54596 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 62] rpc - simple_protocol.cc:92 - Disconnected 10.16.66.78:54014 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 40] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 0 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 56] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.49:59448 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 0] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.46:54016 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 13] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 13] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 0 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 35] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.49:56739 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 9] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.49:62281 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 21] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.44:51221 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 19] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.49:59411 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 0] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.46:54016 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 14] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 0] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.46:54016 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 14] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 0 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 0] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.46:54016 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 53] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 0] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.46:54016 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 53] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 4 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 0] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.46:54016 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 21] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.44:51221 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 61] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.46:64189 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 23] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 23] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 0 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,746 [shard 8] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.43:53192 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,747 [shard 53] rpc - simple_protocol.cc:92 - Disconnected 10.16.66.78:62453 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,747 [shard 6] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.45:54598 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,747 [shard 49] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.46:57009 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,747 [shard 53] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.45:56501 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,747 [shard 49] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.46:57009 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,747 [shard 49] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.46:57009 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,747 [shard 12] rpc - simple_protocol.cc:92 - Disconnected 10.16.66.78:53004 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,747 [shard 6] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.45:54598 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,747 [shard 53] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.45:56501 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,747 [shard 49] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.46:57009 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,747 [shard 53] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.44:63221 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,747 [shard 53] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.44:63221 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,747 [shard 53] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.44:63221 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,747 [shard 53] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.44:63221 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,747 [shard 53] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.43:60341 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,747 [shard 53] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.43:60341 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,747 [shard 53] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.43:60341 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,747 [shard 53] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.43:60341 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,747 [shard 53] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.43:60341 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,749 [shard 33] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,749 [shard 33] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 0 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,759 [shard 51] rpc - server.cc:253 - vectorized internal rpc protocol - Stopping 1 listeners
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,759 [shard 51] rpc - server.cc:262 - vectorized internal rpc protocol - Shutting down 0 connections
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,763 [shard 19] rpc - simple_protocol.cc:92 - Disconnected 10.16.67.46:58067 (stream closed)
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 0] raft - [group_id:0, {redpanda/controller/0}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 0] raft - [group_id:7061, {kafka/rxdata-usersync/0}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 3] raft - [group_id:6648, {kafka/dmp-cookie-sync/7}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 4] raft - [group_id:6860, {kafka/rxdata-bidresponse_extra/9}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 2] raft - [group_id:7068, {kafka/rxdata-usersync/7}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 7] raft - [group_id:7084, {kafka/summarystats-dspstat/3}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 39] raft - [group_id:6978, {kafka/rxdata-providerdmpevent/7}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 44] raft - [group_id:6994, {kafka/rxdata-rg_application/3}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 59] raft - [group_id:7045, {kafka/rxdata-testad/4}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 12] raft - [group_id:6889, {kafka/rxdata-dvviewability/8}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 26] raft - [group_id:6933, {kafka/rxdata-impserve/2}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 28] raft - [group_id:6731, {kafka/rx-cm-post/0}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 32] raft - [group_id:6744, {kafka/rx-cm-tremor/3}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 46] raft - [group_id:6791, {kafka/rx-confiant-response/0}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 24] raft - [group_id:6718, {kafka/rx-aerokeys-userdata/7}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 51] raft - [group_id:7017, {kafka/rxdata-rg_report/6}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 53] raft - [group_id:6814, {kafka/rx-uid-map-batch/3}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 11] raft - [group_id:6885, {kafka/rxdata-dvviewability/4}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 23] raft - [group_id:6715, {kafka/rx-aerokeys-userdata/4}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 43] raft - [group_id:6993, {kafka/rxdata-rg_application/2}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 24] raft - [group_id:7285, {kafka/rx-fc-iad/4}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 25] raft - [group_id:6930, {kafka/rxdata-impnurl/9}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 7] raft - [group_id:6872, {kafka/rxdata-dspbidrequest/1}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 24] raft - [group_id:6929, {kafka/rxdata-impnurl/8}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 60] raft - [group_id:6838, {kafka/rxdata-bidrequest/7}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 18] raft - [group_id:6696, {kafka/rx-aerokeys-devicedata/5}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 5] raft - [group_id:6655, {kafka/pixelverse_conv_ap/4}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 60] raft - [group_id:7048, {kafka/rxdata-testad/7}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 14] raft - [group_id:6684, {kafka/pixelverse_conv_us/3}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 59] raft - [group_id:6834, {kafka/rxdata-bidrequest/3}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 33] raft - [group_id:6747, {kafka/rx-cm-tremor/6}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 17] raft - [group_id:6903, {kafka/rxdata-impconfirmation/2}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 35] raft - [group_id:6963, {kafka/rxdata-outstream_pread_event/2}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 33] raft - [group_id:7314, {kafka/rx-fc-usw/3}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 42] raft - [group_id:6989, {kafka/rxdata-pubopportunity/8}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 20] raft - [group_id:6913, {kafka/rxdata-impexpired/2}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 48] raft - [group_id:6799, {kafka/rx-confiant-response/8}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 50] raft - [group_id:6804, {kafka/rx-uid-map/3}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 10] raft - [group_id:7094, {kafka/summarystats-pubdealstat/3}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 52] raft - [group_id:7023, {kafka/rxdata-segmentusage/2}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 13] raft - [group_id:6681, {kafka/pixelverse_conv_us/0}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 57] raft - [group_id:7037, {kafka/rxdata-sspbidresponse/6}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 61] raft - [group_id:6841, {kafka/rxdata-bidresponse/0}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 15] raft - [group_id:6689, {kafka/pixelverse_conv_us/8}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 21] raft - [group_id:6919, {kafka/rxdata-impexpired/8}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 61] raft - [group_id:7052, {kafka/rxdata-thirdpartybeacon/1}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 47] raft - [group_id:6793, {kafka/rx-confiant-response/2}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 31] raft - [group_id:7307, {kafka/geouniques/6}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 34] raft - [group_id:6750, {kafka/rx-cm-tremor/9}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 45] raft - [group_id:6999, {kafka/rxdata-rg_application/8}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 40] raft - [group_id:6772, {kafka/rx-confiant-request-iad/1}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 41] raft - [group_id:6774, {kafka/rx-confiant-request-iad/3}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 54] raft - [group_id:7027, {kafka/rxdata-segmentusage/6}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 40] raft - [group_id:6982, {kafka/rxdata-pubopportunity/1}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 41] raft - [group_id:6985, {kafka/rxdata-pubopportunity/4}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 54] raft - [group_id:6817, {kafka/rx-uid-map-batch/6}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 9] raft - [group_id:6669, {kafka/pixelverse_conv_diagnostic/8}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 56] raft - [group_id:7035, {kafka/rxdata-sspbidresponse/4}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 36] raft - [group_id:7326, {kafka/rx-uid-map-batch/15}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 22] raft - [group_id:6712, {kafka/rx-aerokeys-userdata/1}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 56] raft - [group_id:6824, {kafka/rx-usersync-us/3}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 36] raft - [group_id:6968, {kafka/rxdata-outstream_pread_event/7}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 8] raft - [group_id:7088, {kafka/summarystats-dspstat/7}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 22] raft - [group_id:6920, {kafka/rxdata-impexpired/9}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 22] raft - [group_id:7136, {kafka/__consumer_offsets/15}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 8] raft - [group_id:6875, {kafka/rxdata-dspbidrequest/4}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 6] raft - [group_id:7081, {kafka/summarystats-dspstat/0}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 2] raft - [group_id:6645, {kafka/dmp-cookie-sync/4}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 4] raft - [group_id:7073, {kafka/rxdata-videoevent/2}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 27] raft - [group_id:7296, {kafka/rx-fc-iad2/5}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 19] raft - [group_id:6702, {kafka/rx-aerokeys-ipdata/1}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 6] raft - [group_id:6656, {kafka/pixelverse_conv_ap/5}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 51] raft - [group_id:6808, {kafka/rx-uid-map/7}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 27] raft - [group_id:6727, {kafka/rx-bidrequest-stream/6}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 6] raft - [group_id:6867, {kafka/rxdata-click/6}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 19] raft - [group_id:6912, {kafka/rxdata-impexpired/1}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 16] raft - [group_id:6900, {kafka/rxdata-impbeacon/9}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 49] raft - [group_id:7012, {kafka/rxdata-rg_report/1}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 19] raft - [group_id:7123, {kafka/__consumer_offsets/2}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 29] raft - [group_id:6736, {kafka/rx-cm-post/5}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 62] raft - [group_id:6844, {kafka/rxdata-bidresponse/3}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 18] raft - [group_id:6909, {kafka/rxdata-impconfirmation/8}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 29] raft - [group_id:7303, {kafka/geouniques/2}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 5] raft - [group_id:6865, {kafka/rxdata-click/4}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 30] raft - [group_id:7304, {kafka/geouniques/3}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 62] raft - [group_id:7055, {kafka/rxdata-thirdpartybeacon/4}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 12] raft - [group_id:7101, {kafka/summarystats-supplystat/0}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 63] raft - [group_id:6847, {kafka/rxdata-bidresponse/6}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 42] raft - [group_id:6779, {kafka/rx-confiant-request-iad/8}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 35] raft - [group_id:6756, {kafka/rx-confiant-request-ams/5}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 12] raft - [group_id:6677, {kafka/pixelverse_conv_eu/6}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 50] raft - [group_id:7013, {kafka/rxdata-rg_report/2}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 52] raft - [group_id:6813, {kafka/rx-uid-map-batch/2}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 13] raft - [group_id:6892, {kafka/rxdata-impbeacon/1}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 57] raft - [group_id:6827, {kafka/rx-usersync-us/6}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 38] raft - [group_id:6766, {kafka/rx-confiant-request-ap/5}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 13] raft - [group_id:7105, {kafka/summarystats-supplystat/4}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 21] raft - [group_id:7130, {kafka/__consumer_offsets/9}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 31] raft - [group_id:6740, {kafka/rx-cm-post/9}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 34] raft - [group_id:7320, {kafka/rx-fc-usw/9}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 45] raft - [group_id:6788, {kafka/rx-confiant-request-usw/7}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 26] raft - [group_id:6725, {kafka/rx-bidrequest-stream/4}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 9] raft - [group_id:7090, {kafka/summarystats-dspstat/9}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 9] raft - [group_id:6877, {kafka/rxdata-dspbidrequest/6}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 26] raft - [group_id:7290, {kafka/rx-fc-iad/9}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 36] raft - [group_id:6758, {kafka/rx-confiant-request-ams/7}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 8] raft - [group_id:6666, {kafka/pixelverse_conv_diagnostic/5}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 37] raft - [group_id:6972, {kafka/rxdata-providerdmpevent/1}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 32] raft - [group_id:7313, {kafka/rx-fc-usw/2}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 2] raft - [group_id:6854, {kafka/rxdata-bidresponse_extra/3}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 23] raft - [group_id:6923, {kafka/rxdata-impnurl/2}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 37] raft - [group_id:6762, {kafka/rx-confiant-request-ap/1}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 11] raft - [group_id:7099, {kafka/summarystats-pubdealstat/8}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 25] raft - [group_id:7289, {kafka/rx-fc-iad/8}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 37] raft - [group_id:7330, {kafka/rx-uid-map-batch/19}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 44] raft - [group_id:6784, {kafka/rx-confiant-request-usw/3}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 11] raft - [group_id:6674, {kafka/pixelverse_conv_eu/3}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 18] raft - [group_id:7121, {kafka/__consumer_offsets/0}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 17] raft - [group_id:6694, {kafka/rx-aerokeys-devicedata/3}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 20] raft - [group_id:7128, {kafka/__consumer_offsets/7}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 30] raft - [group_id:6738, {kafka/rx-cm-post/7}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 10] raft - [group_id:6882, {kafka/rxdata-dvviewability/1}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 63] raft - [group_id:7058, {kafka/rxdata-thirdpartybeacon/7}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 15] raft - [group_id:6898, {kafka/rxdata-impbeacon/7}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 38] raft - [group_id:6976, {kafka/rxdata-providerdmpevent/5}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 21] raft - [group_id:6709, {kafka/rx-aerokeys-ipdata/8}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 55] raft - [group_id:7031, {kafka/rxdata-sspbidresponse/0}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 53] raft - [group_id:7024, {kafka/rxdata-segmentusage/3}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 28] raft - [group_id:6940, {kafka/rxdata-impserve/9}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 28] raft - [group_id:7297, {kafka/rx-fc-iad2/6}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 55] raft - [group_id:6821, {kafka/rx-usersync-us/0}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 4] raft - [group_id:6650, {kafka/dmp-cookie-sync/9}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 43] raft - [group_id:6780, {kafka/rx-confiant-request-iad/9}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 39] raft - [group_id:6767, {kafka/rx-confiant-request-ap/6}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 27] raft - [group_id:6938, {kafka/rxdata-impserve/7}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 23] raft - [group_id:7281, {kafka/rx-fc-iad/0}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 49] raft - [group_id:6802, {kafka/rx-uid-map/1}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 3] raft - [group_id:6858, {kafka/rxdata-bidresponse_extra/7}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 5] raft - [group_id:7079, {kafka/rxdata-videoevent/8}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 3] raft - [group_id:7072, {kafka/rxdata-videoevent/1}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 20] raft - [group_id:6705, {kafka/rx-aerokeys-ipdata/4}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 30] raft - [group_id:6949, {kafka/rxdata-lurl/8}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 10] raft - [group_id:6672, {kafka/pixelverse_conv_eu/1}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 25] raft - [group_id:6720, {kafka/rx-aerokeys-userdata/9}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 16] raft - [group_id:6691, {kafka/rx-aerokeys-devicedata/0}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 7] raft - [group_id:6661, {kafka/pixelverse_conv_diagnostic/0}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,782 [shard 14] raft - [group_id:7109, {kafka/summarystats-supplystat/8}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,783 [shard 14] raft - [group_id:6895, {kafka/rxdata-impbeacon/4}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,783 [shard 29] raft - [group_id:6946, {kafka/rxdata-lurl/5}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,783 [shard 35] raft - [group_id:7321, {kafka/rx-uid-map-batch/10}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,781 [shard 1] raft - [group_id:7064, {kafka/rxdata-usersync/3}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,788 [shard 1] raft - [group_id:6852, {kafka/rxdata-bidresponse_extra/1}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,788 [shard 1] raft - [group_id:6642, {kafka/dmp-cookie-sync/1}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,796 [shard 58] raft - [group_id:7041, {kafka/rxdata-testad/0}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,796 [shard 58] raft - [group_id:6830, {kafka/rx-usersync-us/9}] consensus.cc:239 - Stopping
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: Reactor stalled for 97 ms on shard 61. Backtrace: 0x4ab0f1f 0x4ab0c32 0x42abf 0x44fbba2 0x194edf7 0x43a477a 0x43f031a 0x19ac73f 0x345f55c 0x194254a 0x4ace59f 0x4ad2277 0x4b15745 0x4a6f10f 0x91016 0x1166cf
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack: 0xffffffffffffff80 0xffffffffa13d02ff 0xffffffffa13d0511 0xffffffffc02c7b9d 0xffffffffc02c7cf2 0xffffffffa13b13c6 0xffffffffa13b20ca 0xffffffffa13b5338 0xffffffffa13b5b67 0xffffffffa1ab314b 0xffffffffa1c0009b
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: Reactor stalled for 97 ms on shard 13. Backtrace: 0x4ab0f1f 0x4ab0c32 0x42abf 0x44fbba2 0x194edf7 0x43a477a 0x43f031a 0x19ac73f 0x345f55c 0x194254a 0x4ace59f 0x4ad2277 0x4b15745 0x4a6f10f 0x91016 0x1166cf
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack: 0xffffffffffffff80 0xffffffffa150092e 0xffffffffc02e7f40 0xffffffffc02ef919 0xffffffffc02e80fb 0xffffffffc02e8630 0xffffffffc02d57d5 0xffffffffa136aa1e 0xffffffffa136ae45 0xffffffffa136b417 0xffffffffc02c7331 0xffffffffc02c7b6b 0xffffffffc02c7cf2 0xffffffffa13b13c6 0xffffffffa13b20ca 0xffffffffa13b5338 0xffffffffa13b5b67 0xffffffffa1ab314b 0xffffffffa1c0009b
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: Reactor stalled for 155 ms on shard 61. Backtrace: 0x4ab0f1f 0x4ab0c32 0x42abf 0x44fbba2 0x194edf7 0x43a477a 0x43f031a 0x19ac73f 0x345f55c 0x194254a 0x4ace59f 0x4ad2277 0x4b15745 0x4a6f10f 0x91016 0x1166cf
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: kernel callstack:
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 0] cluster - leader_balancer.cc:195 - Stopping Leader Balancer...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 0] cluster - partition_balancer_backend.cc:101 - stopping...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 0] cluster - metrics_reporter.cc:130 - Stopping Metrics Reporter...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 0] cluster - feature_manager.cc:159 - Stopping Feature Manager...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 0] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 3] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 19] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 56] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 37] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 5] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 54] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 32] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 18] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 47] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 26] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 27] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 59] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 43] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 29] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 44] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 40] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 7] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 57] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 11] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 15] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 23] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 38] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 35] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 24] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 30] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 28] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 36] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 6] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 21] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 45] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 9] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 16] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 50] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 51] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 52] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 8] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 2] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 33] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 63] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 60] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 25] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 39] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 58] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 13] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 4] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 42] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 41] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 46] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 62] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 20] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 34] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:21 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 10] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 49] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 61] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 48] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 31] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 1] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 14] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 12] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 53] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 55] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:21,994 [shard 22] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,000 [shard 17] cluster - health_monitor_frontend.cc:33 - Stopping Health Monitor Frontend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 0] cluster - health_monitor_backend.cc:118 - Stopping Health Monitor Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 0] cluster - health_manager.cc:57 - Stopping Health Manager...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 0] cluster - members_backend.cc:64 - Stopping Members Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 0] cluster - config_manager.cc:220 - Stopping Config Manager...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 0] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 2] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 5] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 6] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 4] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 15] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 21] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 22] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 10] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 44] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 53] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 31] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 23] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 8] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 20] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 29] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 47] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 33] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 36] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 51] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 56] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 24] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 61] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 14] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 11] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 16] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 37] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 38] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 46] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 49] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 7] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 41] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 54] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 25] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 57] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 19] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 39] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 26] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 63] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 32] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 30] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 18] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 45] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 1] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 13] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 55] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 42] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 52] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 34] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 40] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 3] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 59] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 50] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 48] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 12] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 60] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 43] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 62] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 27] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 28] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 58] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 35] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,002 [shard 9] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,008 [shard 17] cluster - controller_backend.cc:268 - Stopping Controller Backend...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 0] cluster - members_manager.cc:344 - stopping cluster::members_manager...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 0] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 3] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 6] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 12] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 13] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 15] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 16] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 7] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 24] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 23] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 31] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 10] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 4] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 34] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 38] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 43] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 47] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 52] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 57] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 17] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 62] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 63] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 25] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 19] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 60] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 21] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 22] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 1] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 28] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 30] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 27] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 29] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 32] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 33] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 42] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 11] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 35] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 36] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 2] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 37] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 14] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 39] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 40] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 41] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 5] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 48] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 50] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 51] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 45] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 49] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 46] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 55] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 53] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 8] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 54] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 44] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 56] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 58] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 59] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 61] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 26] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 20] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,014 [shard 9] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,020 [shard 18] cluster - drain_manager.cc:30 - Drain manager stopping
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 0] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 2] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 5] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 9] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 11] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 13] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 14] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 8] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 6] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 7] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 18] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 22] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 26] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 28] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 4] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 12] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 34] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 38] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 45] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 47] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 49] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 48] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 54] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 55] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 63] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 27] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 3] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 1] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 29] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 30] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 31] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 33] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 16] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 15] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 32] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 37] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 50] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 62] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 40] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 19] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 60] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 42] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 44] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 56] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 58] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 53] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 24] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 59] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 25] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 23] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 36] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 43] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 39] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 57] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 35] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 21] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 41] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 46] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 61] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 10] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 20] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 51] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,025 [shard 52] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,031 [shard 17] cluster - partition_leaders_table.cc:30 - Stopping Partition Leaders Table...
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 0] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/0_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 1] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/1_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 13] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/13_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 54] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/54_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 26] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/26_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 6] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/6_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 41] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/41_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 33] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/33_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 50] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/50_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 57] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/57_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 30] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/30_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 37] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/37_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 23] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/23_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 40] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/40_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 59] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/59_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 43] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/43_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 28] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/28_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 46] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/46_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 3] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/3_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 7] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/7_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 45] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/45_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 24] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/24_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 12] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/12_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 5] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/5_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 2] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/2_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 29] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/29_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 19] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/19_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 32] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/32_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 31] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/31_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 36] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/36_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 34] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/34_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 8] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/8_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 39] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/39_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 27] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/27_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 15] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/15_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 49] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/49_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 53] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/53_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 55] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/55_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 56] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/56_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 16] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/16_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 58] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/58_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 18] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/18_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 38] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/38_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 35] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/35_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 52] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/52_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 17] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/17_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 60] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/60_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 47] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/47_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 51] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/51_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 22] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/22_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 61] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/61_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 48] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/48_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 62] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/62_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 63] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/63_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 21] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/21_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 25] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/25_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 10] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/10_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 14] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/14_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 4] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/4_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 11] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/11_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,044 [shard 20] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/20_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 42] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/42_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 9] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/9_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,038 [shard 44] kvstore - kvstore.cc:106 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/44_0
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io rpk[5055]: INFO 2022-11-17 08:36:22,063 [shard 0] redpanda::main - application.cc:239 - Shutdown complete.
Nov 17 08:36:22 srv-01-03-404.iad1.trmr.io systemd[1]: Stopped Redpanda, the fastest queue in the West..Is it safe to do a yum uninstall then reinstall? I assume the config files and such will remain in place.
kargh
commented
2 years ago Now the entire cluster is unstable (I've not changed anything). This is in dmesg:
[Thu Nov 17 14:03:07 2022] reactor-27[63448]: segfault at 18 ip 0000564710ce56da sp 00007f7867df1b80 error 4 in redpanda[56470da45000+353c000]
[Thu Nov 17 14:03:07 2022] Code: b4 de f8 fb ff ff 48 81 c3 7f ff ff ff 49 89 9f a0 04 00 00 48 89 e3 4c 39 f3 74 24 0f 1f 40 00 4c 8b 2b 49 8b 45 00 4c 89 ef <ff> 50 18 49 8b 45 00 4c 89 ef ff 50 08 48 83 c3 08 4c 39 f3 75 e0
[Thu Nov 17 14:05:07 2022] reactor-60[13816]: segfault at 1b1de625 ip 000056422d4612e3 sp 00007f4d2f5ef8c0 error 4 in redpanda[56422a2d8000+353c000]
[Thu Nov 17 14:05:07 2022] Code: c1 e1 04 44 39 bc 0f 88 01 00 00 0f 82 77 11 00 00 4c 8d a4 0f 90 01 00 00 48 8b 9c 0f 90 01 00 00 48 85 db 0f 84 ed 04 00 00 <48> 8b 03 49 89 04 24 48 ff 8c 0f 98 01 00 00 e9 b0 0b 00 00 48 8b
[Thu Nov 17 14:05:49 2022] reactor-60[14270]: segfault at 1be5cc ip 0000559dd668b90c sp 00007f0385bf0f50 error 4 in redpanda[559dd33e2000+353c000]
[Thu Nov 17 14:05:49 2022] Code: ca 8d 00 00 83 f8 ff 74 46 89 c5 85 c0 74 38 89 eb 4c 8d 64 24 30 0f 1f 84 00 00 00 00 00 49 8b 7c 24 f0 49 8b 34 24 48 8b 07 <ff> 10 49 83 c4 20 48 ff cb 75 e9 0f 57 c0 0f 29 44 24 10 b3 01 81
[Thu Nov 17 14:06:04 2022] reactor-58[14775]: segfault at 4 ip 000055bc5b6142e3 sp 00007fd7fc9ef660 error 4 in redpanda[55bc5848b000+353c000]
[Thu Nov 17 14:06:04 2022] Code: c1 e1 04 44 39 bc 0f 88 01 00 00 0f 82 77 11 00 00 4c 8d a4 0f 90 01 00 00 48 8b 9c 0f 90 01 00 00 48 85 db 0f 84 ed 04 00 00 <48> 8b 03 49 89 04 24 48 ff 8c 0f 98 01 00 00 e9 b0 0b 00 00 48 8b
[Thu Nov 17 14:08:27 2022] traps: reactor-59[15564] general protection fault ip:561e0c0a243f sp:7fb0b69eff10 error:0 in redpanda[561e0b2a2000+353c000]
[Thu Nov 17 14:09:11 2022] traps: reactor-60[16370] general protection fault ip:5593696c42e3 sp:7f589b3ee300 error:0 in redpanda[55936653b000+353c000]
[Thu Nov 17 14:09:51 2022] traps: reactor-60[16841] general protection fault ip:563d99e25893 sp:7f80b2deead0 error:0 in redpanda[563d96c9c000+353c000]
[Thu Nov 17 14:11:27 2022] reactor-59[17285]: segfault at 4f2 ip 00005558986d82e3 sp 00007fece99ee380 error 4 in redpanda[55589554f000+353c000]
[Thu Nov 17 14:11:27 2022] Code: c1 e1 04 44 39 bc 0f 88 01 00 00 0f 82 77 11 00 00 4c 8d a4 0f 90 01 00 00 48 8b 9c 0f 90 01 00 00 48 85 db 0f 84 ed 04 00 00 <48> 8b 03 49 89 04 24 48 ff 8c 0f 98 01 00 00 e9 b0 0b 00 00 48 8b
[Thu Nov 17 14:12:08 2022] reactor-60[18213]: segfault at 1c90ea ip 0000561eec4d62e3 sp 00007fd6855ef8c0 error 4 in redpanda[561ee934d000+353c000]
[Thu Nov 17 14:12:08 2022] Code: c1 e1 04 44 39 bc 0f 88 01 00 00 0f 82 77 11 00 00 4c 8d a4 0f 90 01 00 00 48 8b 9c 0f 90 01 00 00 48 85 db 0f 84 ed 04 00 00 <48> 8b 03 49 89 04 24 48 ff 8c 0f 98 01 00 00 e9 b0 0b 00 00 48 8b
[Thu Nov 17 14:13:41 2022] traps: reactor-59[18645] general protection fault ip:5589e183837b sp:7f55ffbef4b0 error:0 in redpanda[5589de6f5000+353c000]
[Thu Nov 17 14:15:52 2022] traps: reactor-60[20232] general protection fault ip:555b71ab22e3 sp:7fa760dee790 error:0 in redpanda[555b6e929000+353c000]
[Thu Nov 17 14:16:29 2022] traps: reactor-60[20705] general protection fault ip:55d8db5f709a sp:7f556ebef840 error:0 in redpanda[55d8da7f3000+353c000]
[Thu Nov 17 14:18:03 2022] traps: reactor-59[21164] general protection fault ip:561a8084f2e3 sp:7fa15b1edf40 error:0 in redpanda[561a7d6c6000+353c000]
[Thu Nov 17 14:19:35 2022] reactor-59[21921]: segfault at 2c7 ip 00005587ed2dfdf8 sp 00007f5ee31efd50 error 4 in redpanda[5587ecd3f000+353c000]
[Thu Nov 17 14:19:35 2022] Code: 10 48 39 ef 74 10 48 8b 75 00 48 85 f6 74 0c 48 8b 5c 08 68 eb 14 48 89 c6 eb 63 48 89 6d 00 48 89 6c 08 68 48 89 eb 48 89 ee <4c> 8b 4e 08 48 89 7e 08 4c 89 4c 24 10 48 8b 33 48 89 3b 48 8b 5d
[Thu Nov 17 14:20:52 2022] reactor-60[23170]: segfault at 1c5a0e ip 000055e7ff500893 sp 00007ff3acbee790 error 4 in redpanda[55e7fc377000+353c000]
[Thu Nov 17 14:20:52 2022] Code: 0f 44 c9 c7 01 00 00 00 00 41 89 30 48 8b 4c 07 18 48 85 c9 74 b1 4c 8d 54 07 18 0f b7 7c 07 02 0f 1f 80 00 00 00 00 48 89 cb <48> 8b 09 48 89 2b 48 ff c2 ff c7 48 89 dd 48 85 c9 75 ea 49 c7 02@kargh - this redpanda.service: main process exited, code=killed, status=4/ILL what processor are you using?
kargh
commented
2 years ago Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 64
On-line CPU(s) list: 0-63
Thread(s) per core: 1
Core(s) per socket: 64
Socket(s): 1
NUMA node(s): 1
Vendor ID: AuthenticAMD
CPU family: 25
Model: 1
Model name: AMD EPYC 7763 64-Core Processor
Stepping: 1
CPU MHz: 3240.760
BogoMIPS: 4890.28
Virtualization: AMD-V
L1d cache: 32K
L1i cache: 32K
L2 cache: 512K
L3 cache: 32768K
NUMA node0 CPU(s): 0-63
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin brs arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca@kargh can you decomision one node since the cluster is unstable and run an FIO test on the same device to get a hardware profile (16KiB pages, seq read,write, 10G file) wondering what FIO latencies look like.
kargh
commented
2 years ago @emaxerrno Can you provide me quick and dirty instructions on that? It has been a long time since I played around with that.
kargh
commented
2 years ago Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io rpk[62143]: Segmentation fault on shard 58.
Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io rpk[62143]: Backtrace:
Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io rpk[62143]: 0x4ab1196
Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io rpk[62143]: 0x4b13ff6
Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io rpk[62143]: /opt/redpanda/lib/libc.so.6+0x42abf
Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io rpk[62143]: 0x4a212e2
Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io rpk[62143]: 0x4a2f025
Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io rpk[62143]: 0x49be4ce
Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io rpk[62143]: 0x44f8c65
Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io rpk[62143]: 0x44fa423
Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io rpk[62143]: 0x442482b
Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io rpk[62143]: 0x4419ce4
Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io rpk[62143]: 0x43ff4d0
Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io rpk[62143]: 0x4404aa3
Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io rpk[62143]: 0x43ffa94
Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io rpk[62143]: 0x441b55e
Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io rpk[62143]: 0x441dfbc
Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io rpk[62143]: 0x4625ccc
Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io rpk[62143]: 0x3441e09
Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io rpk[62143]: 0x344163a
Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io rpk[62143]: 0x4ace59f
Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io rpk[62143]: 0x4ad2277
Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io rpk[62143]: 0x4b15745
Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io rpk[62143]: 0x4a6f10f
Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io rpk[62143]: /opt/redpanda/lib/libc.so.6+0x91016
Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io rpk[62143]: /opt/redpanda/lib/libc.so.6+0x1166cf
Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io systemd[1]: redpanda.service: main process exited, code=killed, status=11/SEGV
Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io systemd[1]: Unit redpanda.service entered failed state.
Nov 17 14:29:34 srv-01-09-310.iad1.trmr.io systemd[1]: redpanda.service failed.
Nov 17 14:29:35 srv-01-09-310.iad1.trmr.io systemd[1]: redpanda.service holdoff time over, scheduling restart.
Nov 17 14:29:35 srv-01-09-310.iad1.trmr.io systemd[1]: Stopped Redpanda, the fastest queue in the West..@emaxerrno I found the stuff I need for the fio test.
kargh
commented
2 years ago @emaxerrno Running this. Let me know if you want a specific engine or anything. I can restart it with different parameters:
fio --size=10G --bs=16k --rw=readwrite --filename=/var/lib/redpanda/data/test.dat --name=SeqTest --runtime=600 --numjobs=1 --time_based --group_reporting
kargh
commented
2 years ago SeqTest: (g=0): rw=rw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [M(1)][100.0%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 00m:00s]
SeqTest: (groupid=0, jobs=1): err= 0: pid=28734: Thu Nov 17 14:55:03 2022
read: IOPS=159k, BW=2486MiB/s (2607MB/s)(1460GiB/601342msec)
clat (nsec): min=641, max=4805.0k, avg=1291.16, stdev=3192.57
lat (nsec): min=671, max=4806.0k, avg=1317.03, stdev=3192.75
clat percentiles (nsec):
| 1.00th=[ 956], 5.00th=[ 1048], 10.00th=[ 1064], 20.00th=[ 1096],
| 30.00th=[ 1160], 40.00th=[ 1176], 50.00th=[ 1192], 60.00th=[ 1208],
| 70.00th=[ 1256], 80.00th=[ 1368], 90.00th=[ 1560], 95.00th=[ 2040],
| 99.00th=[ 2352], 99.50th=[ 2512], 99.90th=[ 3312], 99.95th=[ 3728],
| 99.99th=[39168]
bw ( MiB/s): min= 17, max= 5193, per=100.00%, avg=2986.51, stdev=1055.98, samples=1001
iops : min= 1144, max=332368, avg=191136.34, stdev=67583.01, samples=1001
write: IOPS=159k, BW=2486MiB/s (2607MB/s)(1460GiB/601342msec)
clat (nsec): min=1192, max=501396, avg=2829.37, stdev=1108.78
lat (nsec): min=1252, max=501466, avg=2897.68, stdev=1110.21
clat percentiles (nsec):
| 1.00th=[ 1576], 5.00th=[ 1640], 10.00th=[ 1640], 20.00th=[ 1688],
| 30.00th=[ 2800], 40.00th=[ 2864], 50.00th=[ 2896], 60.00th=[ 2960],
| 70.00th=[ 3152], 80.00th=[ 3440], 90.00th=[ 3664], 95.00th=[ 4080],
| 99.00th=[ 5920], 99.50th=[ 6688], 99.90th=[ 7904], 99.95th=[ 8512],
| 99.99th=[11072]
bw ( MiB/s): min= 17, max= 5181, per=100.00%, avg=2986.78, stdev=1056.37, samples=1001
iops : min= 1148, max=331616, avg=191153.66, stdev=67607.49, samples=1001
lat (nsec) : 750=0.07%, 1000=0.65%
lat (usec) : 2=60.36%, 4=36.17%, 10=2.74%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=10.57%, sys=89.14%, ctx=688789, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=95673882,95682561,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=2486MiB/s (2607MB/s), 2486MiB/s-2486MiB/s (2607MB/s-2607MB/s), io=1460GiB (1568GB), run=601342-601342msec
WRITE: bw=2486MiB/s (2607MB/s), 2486MiB/s-2486MiB/s (2607MB/s-2607MB/s), io=1460GiB (1568GB), run=601342-601342msec
Disk stats (read/write):
md0: ios=169001/4620401, merge=0/0, ticks=68270/766698858, in_queue=766767128, util=25.78%, aggrios=21125/1156355, aggrmerge=0/0, aggrticks=8399/11310365, aggrin_queue=11318764, aggrutil=25.01%
nvme3n1: ios=18537/1155697, merge=0/0, ticks=7023/11307406, in_queue=11314429, util=22.22%
nvme7n1: ios=18059/1160316, merge=0/0, ticks=7161/11554948, in_queue=11562108, util=22.31%
nvme0n1: ios=25626/1153543, merge=0/0, ticks=9329/11974729, in_queue=11984057, util=22.38%
nvme4n1: ios=22202/1155866, merge=0/0, ticks=7565/11307487, in_queue=11315051, util=22.57%
nvme1n1: ios=19085/1153543, merge=0/0, ticks=7693/12017526, in_queue=12025219, util=22.42%
nvme5n1: ios=21216/1155866, merge=0/0, ticks=10610/8896171, in_queue=8906782, util=22.32%
nvme2n1: ios=24798/1155697, merge=0/0, ticks=10096/11630921, in_queue=11641018, util=25.01%
nvme6n1: ios=19478/1160316, merge=0/0, ticks=7718/11793732, in_queue=11801450, util=22.53%The remaining nodes in the cluster appear to have stabilized -- at least they have stopped crashing and restarting.
We have noticed that the last two times we have pushed out updates to our application, it has had connection issues with RP. The app does a reload which creates new connections, I believe, to all the services. It appears the crash/restart cycle might have started when the update was pushed out. Maybe the nodes are getting smashed with connection requests. We'll be investigating that on our end.
emaxerrno
commented
2 years ago @kargh thanks for all these notes.... this is even MORE suspicious if that is the case. hmmmm... any insight ont he app behavior.
btw eng team has today and tomorrow off, so they'll be online next week.
emaxerrno
commented
2 years ago this is definitely a bug w/ redpanda Segmentation fault on shard 58 cc: @piyushredpanda and @mattschumpert
@kargh - any 'demo' app we can run on our CI to test? seems like the app is able to trigger a bug somewhere. maybe compression buffers, maybe large payloads.... anything you can help us get to the bottom would be helpful.
emaxerrno
commented
2 years ago last q @kargh what is the CPU to memory ratio, is it at least 2G/core?
kargh
commented
2 years ago @emaxerrno 4G/core. 64 cores (logical processors are disabled). 256GB memory per node.
What is odd is that the connection issue persists until we reload the app a second time. Then everything seems to run fine. It almost might be the volume of servers that get updated that could be causing the issue. That said, it looks like all the nodes might have crashed at the same time. Looking at the memory graphs, they all go from about 75% usage to 25% usage at 14:02. The app release was finished at 13:52.
As far as a demo app, I'm not really sure what I could provide. I'll have to find out. Problem is that it wouldn't actually run without all the required services.
I'm hoping it remains stable over the weekend or even through next week.
kargh
commented
2 years ago The app does create new connections. The old processes and their connections die off and new ones are spawned. We've recently moved from a lot of really small bidders (2x10, 2x14, 2x18 cores boxes) to a large amount of 1x64 core boxes that handle a much larger volume of traffic. During out updates we were updating 15 servers at a time. It is possible that the volume of newly created connections was too much. We are going to change it from 15 to 5.
The thing is, I'm only seeing about 170 connections per RP node from a single bidder. Even if 15 of them kill/create new connections at a time that shouldn't be enough to cause an issue or instability.
emaxerrno
commented
2 years ago this sounds ideal tbh. so definitely a bug on our side. let us dig next week.
kargh
commented
2 years ago Just a bit more info. The application is just a nodejs app using node-rdkafka to handle kafka duties.
nodejs v16.15.0 node-rdkafka v2.12.0
So nothing home baked for connection management or communicating with the cluster.
travisdowns
commented
2 years ago @kargh - can you provide the exact version of repdanda used when this output occurred?
Sorry there a few different versions involved so I just want to be clear since to decode the backtrace we need version-specific debug into.
travisdowns
commented
2 years ago Based on the backtrace you provided in slack, you were seeing OOMs due to #6854, which is fixed in 22.3. So that's a good reason to upgrade. Can you tell us a bit about the topic structure? E.g., are topics created and deleted often? Does it happen that a topic receives a lot of traffic for a period then later almost no traffic (but is not deleted)? Those would lead to the type of OOM you saw.
kargh
commented
2 years ago @travisdowns Topics are rarely created and never deleted. There are some topics that receive a bunch of traffic once or twice a week (this is something new that they added recently).
I would like to upgrade but unfortunately RP won't start when I try.
kargh
commented
2 years ago @kargh - can you provide the exact version of repdanda used when this output occurred?
Sorry there a few different versions involved so I just want to be clear since to decode the backtrace we need version-specific debug into.
22.2.7
kargh
commented
2 years ago @travisdowns Just so we are on the same page, what is your definition of 'a lot of traffic'?
kargh
commented
2 years ago BTW, I know that 22.3.2 will run on these servers. I built a 3 node cluster in another DC on Tuesday with that version. Granted, it is not in production and not taking any traffic yet but RP did start and I have created all the topics. So while there could be some issues with the AMD CPU (which wouldn't surprise me considering I've ran into other issues with these chips) but it is not likely the sole issue that I'm having with the production cluster.
kargh
commented
2 years ago Another thing of note. The server I was upgrading was node 0. It is the seed server for the rest of the cluster. I'm not sure if that has any bearing on the upgrade issues I'm experiencing.
kargh
commented
2 years ago Looks like #7351 might fix the 22.3.1 upgrade issue. I'll be ready to test that as soon as it is merged.
travisdowns
commented
2 years ago @travisdowns Just so we are on the same page, what is your definition of 'a lot of traffic'?
About 10 GB or more. I.e., if those topics which only receive traffic once a week or so get more than that (even added up over several weeks) it could trigger the out of memory issue you saw.
travisdowns
commented
2 years ago Looks like #7351 might fix the 22.3.1 upgrade issue. I'll be ready to test that as soon as it is merged.
It may. It definitely causes a similar problem. However, since it is race condition it usually only triggers intermittently, but in your case I understood that you see a failure to start every time. It is possible that the race condition simply occurs every time (especially since it seems like you have 64 cores, increasing the chance of this occurring), but it is also possible that it's a different issue related to the state of your existing logs, which didn't trigger an assert previously (but still existed) but which now triggers an assert.
travisdowns
commented
2 years ago I would like to upgrade but unfortunately RP won't start when I try.
Yup, I got it: just wanted to provide also some background on the OOM issue which drove the upgrade attempt in the first place.
kargh
commented
2 years ago @travisdowns Just so we are on the same page, what is your definition of 'a lot of traffic'?
About 10 GB or more. I.e., if those topics which only receive traffic once a week or so get more than that (even added up over several weeks) it could trigger the out of memory issue you saw.
Ah, nothing close to that. Jumped to about 650MB total for the entire cluster.
kargh
commented
2 years ago Looks like #7351 might fix the 22.3.1 upgrade issue. I'll be ready to test that as soon as it is merged.
It may. It definitely causes a similar problem. However, since it is race condition it usually only triggers intermittently, but in your case I understood that you see a failure to start every time. It is possible that the race condition simply occurs every time (especially since it seems like you have 64 cores, increasing the chance of this occurring), but it is also possible that it's a different issue related to the state of your existing logs, which didn't trigger an assert previously (but still existed) but which now triggers an assert.
Only one way to find out!
FYI - It is always the same assert message:
Nov 18 11:21:08 srv-01-03-404.iad1.trmr.io rpk[9395]: ERROR 2022-11-18 11:21:08,227 [shard 0] assert - Assert failure: (/var/lib/buildkite-agent/builds/buildkite-amd64-xfs-builders-i-06c4550a08f45c0ae-1/redpanda/redpanda/src/v/cluster/scheduling/allocation_node.cc:81) 'domain_partitions > allocation_capacity{0} && domain_partitions <= _allocated_partitions' Unable to deallocate partition from core 27 in domain 0 at node {node: 8, max_partitions_per_core: 7000, state: active, partition_capacity: 447993, weights: [(2)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(1)(1)(1)(1)(1)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)], allocated: 5({0: 0, -1: 5})}Always core 27 in domain 0 at node { node: 8...
travisdowns
commented
2 years ago @kargh is there only a single common host being used here, or is the "upgrade" happening on a different host, possibly with a different number of cores?
kargh
commented
2 years ago @travisdowns the upgrade is only happening on the same host (node 0). I've not tried upgrading any other host.
All hosts are identical servers. Same number of cores, memory, drives, etc.
kargh
commented
2 years ago And 22.2.7 is crashing again.
kargh
commented
2 years ago Every node is now in an endless crash loop. The logs are getting spammed with things such as this:
Nov 18 22:37:27 srv-01-03-404.iad1.trmr.io rpk[59281]: --------
Nov 18 22:37:27 srv-01-03-404.iad1.trmr.io rpk[59281]: seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void> seastar::future<void>::handle_exception_type<auto ssx::spawn_with_gate_then<cluster::persisted_stm::make_snapshot_in_background()::$_3>(seastar::gate&, cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(seastar::broken_condition_variable const&)>(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(cluster::persisted_stm::make_snapshot_in_background()::$_3&&), seastar::futurize<cluster::persisted_stm::make_snapshot_in_background()::$_3>::type seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void> seastar::future<void>::handle_exception_type<auto ssx::spawn_with_gate_then<cluster::persisted_stm::make_snapshot_in_background()::$_3>(seastar::gate&, cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(seastar::broken_condition_variable const&)>(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)>(seastar::future<void> seastar::future<void>::handle_exception_type<auto ssx::spawn_with_gate_then<cluster::persisted_stm::make_snapshot_in_background()::$_3>(seastar::gate&, cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(seastar::broken_condition_variable const&)>(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::future<void> seastar::future<void>::handle_exception_type<auto ssx::spawn_with_gate_then<cluster::persisted_stm::make_snapshot_in_background()::$_3>(seastar::gate&, cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(seastar::broken_condition_variable const&)>(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)&, seastar::future_state<seastar::internal::monostate>&&), void>
Nov 18 22:37:27 srv-01-03-404.iad1.trmr.io rpk[59281]: INFO 2022-11-18 22:37:27,246 [shard 14] r/heartbeat - heartbeat_manager.cc:209 - Closed unresponsive connection to 1
Nov 18 22:37:27 srv-01-03-404.iad1.trmr.io rpk[59281]: WARN 2022-11-18 22:37:27,249 [shard 14] raft - [follower: {id: {1}, revision: {1248}}] [group_id:6895, {kafka/rxdata-impbeacon/4}] - recovery_stm.cc:435 - recovery append entries error: rpc::errc::disconnected_endpoint(node down)
Nov 18 22:37:27 srv-01-03-404.iad1.trmr.io rpk[59281]: INFO 2022-11-18 22:37:27,249 [shard 61] r/heartbeat - heartbeat_manager.cc:209 - Closed unresponsive connection to 1
Nov 18 22:37:27 srv-01-03-404.iad1.trmr.io rpk[59281]: INFO 2022-11-18 22:37:27,249 [shard 14] raft - [group_id:6895, {kafka/rxdata-impbeacon/4}] consensus.cc:539 - Node {id: {1}, revision: {1248}} recovery cancelled (rpc::errc::disconnected_endpoint(node down))
Nov 18 22:37:27 srv-01-03-404.iad1.trmr.io rpk[59281]: INFO 2022-11-18 22:37:27,258 [shard 26] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rxdata-impserve/2_1256/2872337596-33-v1.log
Nov 18 22:37:27 srv-01-03-404.iad1.trmr.io rpk[59281]: INFO 2022-11-18 22:37:27,258 [shard 9] r/heartbeat - heartbeat_manager.cc:209 - Closed unresponsive connection to 1
Nov 18 22:37:27 srv-01-03-404.iad1.trmr.io rpk[59281]: INFO 2022-11-18 22:37:27,276 [shard 17] r/heartbeat - heartbeat_manager.cc:209 - Closed unresponsive connection to 1
Nov 18 22:37:27 srv-01-03-404.iad1.trmr.io rpk[59281]: INFO 2022-11-18 22:37:27,289 [shard 55] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rx-usersync-us/0_1234/8629374553-55-v1.log
Nov 18 22:37:27 srv-01-03-404.iad1.trmr.io rpk[59281]: INFO 2022-11-18 22:37:27,321 [shard 59] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rxdata-bidrequest/3_1236/523306929-31-v1.log
Nov 18 22:37:27 srv-01-03-404.iad1.trmr.io rpk[59281]: INFO 2022-11-18 22:37:27,351 [shard 13] rpc - Disconnected from server {host: 10.16.67.49, port: 33145}: std::__1::system_error (error system:104, read: Connection reset by peer)
Nov 18 22:37:27 srv-01-03-404.iad1.trmr.io rpk[59281]: INFO 2022-11-18 22:37:27,351 [shard 55] rpc - Disconnected from server {host: 10.16.67.49, port: 33145}: std::__1::system_error (error system:104, read: Connection reset by peer)
Nov 18 22:37:27 srv-01-03-404.iad1.trmr.io rpk[59281]: INFO 2022-11-18 22:37:27,351 [shard 7] rpc - Disconnected from server {host: 10.16.67.49, port: 33145}: std::__1::system_error (error system:104, read: Connection reset by peer)
Nov 18 22:37:27 srv-01-03-404.iad1.trmr.io rpk[59281]: INFO 2022-11-18 22:37:27,352 [shard 29] rpc - Disconnected from server {host: 10.16.67.49, port: 33145}: std::__1::system_error (error system:104, read: Connection reset by peer)
Nov 18 22:37:27 srv-01-03-404.iad1.trmr.io rpk[59281]: INFO 2022-11-18 22:37:27,354 [shard 58] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rx-usersync-us/9_1234/8637243084-44-v1.log
Nov 18 22:37:27 srv-01-03-404.iad1.trmr.io rpk[59281]: INFO 2022-11-18 22:37:27,362 [shard 9] rpc - server.cc:135 - vectorized internal rpc protocol - Disconnected 10.16.67.49:50057 (applying protocol, Connection reset by peer)
Nov 18 22:37:27 srv-01-03-404.iad1.trmr.io rpk[59281]: INFO 2022-11-18 22:37:27,362 [shard 28] rpc - Disconnected from server {host: 10.16.67.49, port: 33145}: std::__1::system_error (error system:104, read: Connection reset by peer)
Nov 18 22:37:27 srv-01-03-404.iad1.trmr.io rpk[59281]: INFO 2022-11-18 22:37:27,368 [shard 13] rpc - Disconnected from server {host: 10.16.67.49, port: 33145}: std::__1::system_error (error system:104, read: Connection reset by peer)
Nov 18 22:37:27 srv-01-03-404.iad1.trmr.io rpk[59281]: INFO 2022-11-18 22:37:27,368 [shard 7] rpc - Disconnected from server {host: 10.16.67.49, port: 33145}: std::__1::system_error (error system:104, read: Connection reset by peer)
Nov 18 22:37:27 srv-01-03-404.iad1.trmr.io rpk[59281]: WARN 2022-11-18 22:37:27,370 [shard 59] seastar - Exceptional future ignored: std::bad_alloc (std::bad_alloc), backtrace: 0x4d1d6c4 0x4a1298b 0x3dfb960 0x4ace59f 0x4ad2277 0x4b15745 0x4a6f10f /opt/redpanda/lib/libc.so.6+0x91016 /opt/redpanda/lib/libc.so.6+0x1166cfNov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: Reactor stalled for 94 ms on shard 60. Backtrace: 0x4ab0f1f 0x4ab0c32 0x42abf 0x450bde9 0x44fbfcf 0x4391bdb 0x4a1b3ac 0x4a2143c 0x4a30c11 0x3e83653 0x3decb62 0x3df481e 0x3dee862 0x3dedc98 0x43c59a9 0x43c56b6 0x43c6174 0x4ace59f 0x4ad2277 0x4b15745 0x4a6f10f 0x91016 0x1166cf
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa84f276e 0xffffffffa81607e9 0xffffffffa807fbbb 0xffffffffa807fd5d 0xffffffffa82c152e 0xffffffffa82a819d 0xffffffffa82abdd7 0xffffffffa82b0703 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa84f276e 0xffffffffa81607e9 0xffffffffa807fbbb 0xffffffffa807fd5d 0xffffffffa82c152e 0xffffffffa82a819d 0xffffffffa82abdd7 0xffffffffa82b0703 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1170 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1170 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1170 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1170 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1170 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1170 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1170 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa82d30ad 0xffffffffa82d6f07 0xffffffffa82d799b 0xffffffffa82f9606 0xffffffffa8318862 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1170 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa84b1685 0xffffffffa84a247e 0xffffffffa8492a2c 0xffffffffa8492f15 0xffffffffa84932ef 0xffffffffa849363a 0xffffffffa83cf672 0xffffffffa83cfac2 0xffffffffa83d022e 0xffffffffa83d0511 0xffffffffc0759f6a 0xffffffffc075a135 0xffffffffa83b177f 0xffffffffa83b2027 0xffffffffa83b5338 0xffffffffa83b5b67 0xffffffffa8ab314b 0xffffffffa8c0009b
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa83ac8c6 0xffffffffa83482a0 0xffffffffa8348607 0xffffffffa83486aa 0xffffffffa8ab314b 0xffffffffa8c0009b
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffc057beb0 0xffffffffc057d4b2 0xffffffffa84a1e79 0xffffffffa8493b20 0xffffffffa8493d9a 0xffffffffa83d0295 0xffffffffa83d0511 0xffffffffc0759f6a 0xffffffffc075a135 0xffffffffa83b177f 0xffffffffa83b2027 0xffffffffa83b5338 0xffffffffa83b5b67 0xffffffffa8ab314b 0xffffffffa8c0009b
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1170 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1170 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa82a379c 0xffffffffa82a3c8c 0xffffffffa82a580e 0xffffffffa82a5a84 0xffffffffa84ed656 0xffffffffa848e41d 0xffffffffa83cfa56 0xffffffffa83d022e 0xffffffffa83d0511 0xffffffffc0759f6a 0xffffffffc075a135 0xffffffffa83b177f 0xffffffffa83b2027 0xffffffffa83b5338 0xffffffffa83b5b67 0xffffffffa8ab314b 0xffffffffa8c0009b
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1170 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa84b53d0 0xffffffffa84b567e 0xffffffffa848c082 0xffffffffa848db81 0xffffffffa83cf729 0xffffffffa83cfa19 0xffffffffa83d022e 0xffffffffa83d0511 0xffffffffc0759f6a 0xffffffffc075a135 0xffffffffa83b177f 0xffffffffa83b2027 0xffffffffa83b5338 0xffffffffa83b5b67 0xffffffffa8ab314b 0xffffffffa8c0009b
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1170 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa82a379c 0xffffffffa82a3c8c 0xffffffffa82a580e 0xffffffffa82a5a84 0xffffffffa84ed656 0xffffffffa848e41d 0xffffffffa83cfa56 0xffffffffa83d022e 0xffffffffa83d0511 0xffffffffc0759f6a 0xffffffffc075a135 0xffffffffa83b177f 0xffffffffa83b2027 0xffffffffa83b5338 0xffffffffa83b5b67 0xffffffffa8ab314b 0xffffffffa8c0009b
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa8493279 0xffffffffc0a093f2 0xffffffffc0a0b16d 0xffffffffa87db0b0 0xffffffffa87db24d 0xffffffffa84929fb 0xffffffffa8492f15 0xffffffffa84932ef 0xffffffffa849363a 0xffffffffa83cf672 0xffffffffa83cfac2 0xffffffffa83d022e 0xffffffffa83d0511 0xffffffffc0759f6a 0xffffffffc075a135 0xffffffffa83b177f 0xffffffffa83b2027 0xffffffffa83b5338 0xffffffffa83b5b67 0xffffffffa8ab314b 0xffffffffa8c0009b
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854ca93 0xffffffffa83b5b26 0xffffffffa8ab314b 0xffffffffa8c0009b
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1170 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1170 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:06,609 [shard 44] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rx-confiant-request-usw/4_1226/18821239-64-v1.log
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:06,610 [shard 27] raft - [group_id:7295, {kafka/rx-fc-iad2/4}] vote_stm.cc:279 - became the leader term: 56
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:06,612 [shard 24] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rx-aerokeys-userdata/8_1212/55-58-v1.log
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:06,631 [shard 60] raft - [group_id:6838, {kafka/rxdata-bidrequest/7}] vote_stm.cc:77 - vote reply from {id: {7}, revision: {1236}} - {term:{32}, target_node_id{id: {5}, revision: {1236}}, vote_granted: 1, log_ok:1}
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: WARN 2022-11-18 22:40:06,631 [shard 60] seastar - Exceptional future ignored: std::bad_alloc (std::bad_alloc), backtrace: 0x4d1d6c4 0x4a1298b 0x3dfb960 0x4ace59f 0x4ad2277 0x4b15745 0x4a6f10f /opt/redpanda/lib/libc.so.6+0x91016 /opt/redpanda/lib/libc.so.6+0x1166cf
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: --------
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void> seastar::future<void>::handle_exception_type<auto ssx::spawn_with_gate_then<cluster::persisted_stm::make_snapshot_in_background()::$_3>(seastar::gate&, cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(seastar::broken_condition_variable const&)>(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(cluster::persisted_stm::make_snapshot_in_background()::$_3&&), seastar::futurize<cluster::persisted_stm::make_snapshot_in_background()::$_3>::type seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void> seastar::future<void>::handle_exception_type<auto ssx::spawn_with_gate_then<cluster::persisted_stm::make_snapshot_in_background()::$_3>(seastar::gate&, cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(seastar::broken_condition_variable const&)>(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)>(seastar::future<void> seastar::future<void>::handle_exception_type<auto ssx::spawn_with_gate_then<cluster::persisted_stm::make_snapshot_in_background()::$_3>(seastar::gate&, cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(seastar::broken_condition_variable const&)>(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::future<void> seastar::future<void>::handle_exception_type<auto ssx::spawn_with_gate_then<cluster::persisted_stm::make_snapshot_in_background()::$_3>(seastar::gate&, cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(seastar::broken_condition_variable const&)>(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)&, seastar::future_state<seastar::internal::monostate>&&), void>
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:06,632 [shard 47] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rx-confiant-response/3_1228/1703499-47-v1.log
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:06,648 [shard 50] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rx-uid-map/4_1230/22226950-71-v1.log
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:06,665 [shard 28] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rx-cm-post/0_1216/44-51-v1.log
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:06,729 [shard 10] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rxdata-dspbidrequest/9_1244/1221661334-47-v1.log
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: WARN 2022-11-18 22:40:06,868 [shard 10] seastar - Exceptional future ignored: std::bad_alloc (std::bad_alloc), backtrace: 0x4d1d6c4 0x4a1298b 0x3dfb960 0x4ace59f 0x4ad2277 0x4b15745 0x4a6f10f /opt/redpanda/lib/libc.so.6+0x91016 /opt/redpanda/lib/libc.so.6+0x1166cf
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: --------
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void> seastar::future<void>::handle_exception_type<auto ssx::spawn_with_gate_then<cluster::persisted_stm::make_snapshot_in_background()::$_3>(seastar::gate&, cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(seastar::broken_condition_variable const&)>(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(cluster::persisted_stm::make_snapshot_in_background()::$_3&&), seastar::futurize<cluster::persisted_stm::make_snapshot_in_background()::$_3>::type seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void> seastar::future<void>::handle_exception_type<auto ssx::spawn_with_gate_then<cluster::persisted_stm::make_snapshot_in_background()::$_3>(seastar::gate&, cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(seastar::broken_condition_variable const&)>(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)>(seastar::future<void> seastar::future<void>::handle_exception_type<auto ssx::spawn_with_gate_then<cluster::persisted_stm::make_snapshot_in_background()::$_3>(seastar::gate&, cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(seastar::broken_condition_variable const&)>(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::future<void> seastar::future<void>::handle_exception_type<auto ssx::spawn_with_gate_then<cluster::persisted_stm::make_snapshot_in_background()::$_3>(seastar::gate&, cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(seastar::broken_condition_variable const&)>(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)&, seastar::future_state<seastar::internal::monostate>&&), void>
Nov 18 22:40:06 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:06,903 [shard 60] raft - [group_id:6838, {kafka/rxdata-bidrequest/7}] vote_stm.cc:279 - became the leader term: 32
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:07,084 [shard 27] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rx-bidrequest-stream/7_1214/21190070168-46-v1.log
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: Reactor stalled for 110 ms on shard 27. Backtrace: 0x4ab0f1f 0x4ab0c32 0x42abf 0x450a89f 0x44fbfcf 0x4391bdb 0x4a1b3ac 0x4a2143c 0x4a30c11 0x3e83653 0x3decb62 0x3df481e 0x3dee862 0x3dedc98 0x43c59a9 0x43c56b6 0x43c6174 0x4ace59f 0x4ad2277 0x4b15745 0x4a6f10f 0x91016 0x1166cf
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa84f2763 0xffffffffa81607e9 0xffffffffa807fbbb 0xffffffffa807fd5d 0xffffffffa82c152e 0xffffffffa82a819d 0xffffffffa82abdd7 0xffffffffa82b0703 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1170 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1170 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1170 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1170 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1170 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1170 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1170 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1170 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1170 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1170 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1170 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1170 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa83caa69 0xffffffffa83cff06 0xffffffffa83d0511 0xffffffffc075ab9d 0xffffffffc075acf2 0xffffffffa83b13c6 0xffffffffa83b20ca 0xffffffffa83b5338 0xffffffffa83b5b67 0xffffffffa8ab314b 0xffffffffa8c0009b
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa848c03d 0xffffffffa848db81 0xffffffffa83cf729 0xffffffffa83cfa19 0xffffffffa83d022e 0xffffffffa83d0511 0xffffffffc0759f6a 0xffffffffc075a135 0xffffffffa83b177f 0xffffffffa83b2027 0xffffffffa83b5338 0xffffffffa83b5b67 0xffffffffa8ab314b 0xffffffffa8c0009b
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1134 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1170 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa832eba6 0xffffffffa8277a06 0xffffffffa8277b6d 0xffffffffa8272168 0xffffffffa8276e6f 0xffffffffa83164e6 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa82697b9 0xffffffffc0a0b0e6 0xffffffffa87db0b0 0xffffffffa87db24d 0xffffffffa84929fb 0xffffffffa8492f15 0xffffffffa84932ef 0xffffffffa849363a 0xffffffffa83cf672 0xffffffffa83cfac2 0xffffffffa83d022e 0xffffffffa83d0511 0xffffffffc0759f6a 0xffffffffc075a135 0xffffffffa83b177f 0xffffffffa83b2027 0xffffffffa83b5338 0xffffffffa83b5b67 0xffffffffa8ab314b 0xffffffffa8c0009b
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa88365b5 0xffffffffa8836617 0xffffffffa8836687 0xffffffffa8836715 0xffffffffa88367ec 0xffffffffa883685f 0xffffffffa8ab314b 0xffffffffa8c0009b
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa84ece15 0xffffffffa848e5bc 0xffffffffa83cfa56 0xffffffffa83d022e 0xffffffffa83d0511 0xffffffffc0759f6a 0xffffffffc075a135 0xffffffffa83b177f 0xffffffffa83b2027 0xffffffffa83b5338 0xffffffffa83b5b67 0xffffffffa8ab314b 0xffffffffa8c0009b
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854bfe8 0xffffffffa8921b87 0xffffffffa8921e8d 0xffffffffa895e303 0xffffffffa8833fc2 0xffffffffa883431c 0xffffffffa8836731 0xffffffffa88367ec 0xffffffffa883685f 0xffffffffa8ab314b 0xffffffffa8c0009b
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa81369c6 0xffffffffa8ab3158 0xffffffffa8c0009b
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack: 0xffffffffffffff80 0xffffffffa854b907 0xffffffffa82b1134 0xffffffffa83163fa 0xffffffffa8318879 0xffffffffa82b0725 0xffffffffa82b080b 0xffffffffa807c7fe 0xffffffffa8ab729c 0xffffffffa8c00b77
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: Reactor stalled for 167 ms on shard 27. Backtrace: 0x4ab0f1f 0x4ab0c32 0x42abf 0x44fbe42 0x4391bdb 0x4a1b3ac 0x4a2143c 0x4a30c11 0x3e83653 0x3decb62 0x3df481e 0x3dee862 0x3dedc98 0x43c59a9 0x43c56b6 0x43c6174 0x4ace59f 0x4ad2277 0x4b15745 0x4a6f10f 0x91016 0x1166cf
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: Reactor stalled for 275 ms on shard 27. Backtrace: 0x4ab0f1f 0x4ab0c32 0x42abf 0x450bf1d 0x44fbfcf 0x4391bdb 0x4a1b3ac 0x4a2143c 0x4a30c11 0x3e83653 0x3decb62 0x3df481e 0x3dee862 0x3dedc98 0x43c59a9 0x43c56b6 0x43c6174 0x4ace59f 0x4ad2277 0x4b15745 0x4a6f10f 0x91016 0x1166cf
Nov 18 22:40:07 srv-01-08-309.iad1.trmr.io rpk[54634]: kernel callstack:
Nov 18 22:40:08 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:08,421 [shard 57] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rx-usersync-us/6_1234/8827742883-53-v1.log
Nov 18 22:40:09 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:09,171 [shard 21] rpc - Disconnected from server {host: 10.16.67.49, port: 33145}: std::__1::system_error (error system:104, read: Connection reset by peer)
Nov 18 22:40:09 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:09,171 [shard 59] rpc - Disconnected from server {host: 10.16.67.49, port: 33145}: std::__1::system_error (error system:104, read: Connection reset by peer)
Nov 18 22:40:09 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:09,173 [shard 18] rpc - Disconnected from server {host: 10.16.67.49, port: 33145}: std::__1::system_error (error system:104, read: Connection reset by peer)
Nov 18 22:40:09 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:09,173 [shard 48] rpc - Disconnected from server {host: 10.16.67.49, port: 33145}: std::__1::system_error (error system:104, read: Connection reset by peer)
Nov 18 22:40:09 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:09,173 [shard 19] rpc - Disconnected from server {host: 10.16.67.49, port: 33145}: std::__1::system_error (error system:104, read: Connection reset by peer)
Nov 18 22:40:09 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:09,173 [shard 55] rpc - Disconnected from server {host: 10.16.67.49, port: 33145}: std::__1::system_error (error system:104, read: Connection reset by peer)
Nov 18 22:40:09 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:09,203 [shard 19] rpc - Disconnected from server {host: 10.16.67.49, port: 33145}: std::__1::system_error (error system:104, read: Connection reset by peer)
Nov 18 22:40:09 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:09,552 [shard 7] storage - disk_log_impl.cc:1005 - remove_prefix_full_segments - tombstone & delete segment: {offset_tracker:{term:20, base_offset:1137681689, committed_offset:1137795840, dirty_offset:1137795840}, compacted_segment=0, finished_self_compaction=0, generation={0}, reader={/var/lib/redpanda/data/kafka/rxdata-dspbidrequest/0_1244/1137681689-20-v1.log, (105487674 bytes)}, writer=nullptr, cache={cache_size=0}, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/kafka/rxdata-dspbidrequest/0_1244/1137681689-20-v1.base_index, offsets:{1137681689}, index:{header_bitflags:0, base_offset:{1137681689}, max_offset:{1137795840}, base_timestamp:{timestamp: 1668569666303}, max_timestamp:{timestamp: 1668570007198}, index(3025,3025,3025)}, step:32768, needs_persistence:0}}
Nov 18 22:40:09 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:09,552 [shard 7] storage - segment.cc:112 - removing: {"/var/lib/redpanda/data/kafka/rxdata-dspbidrequest/0_1244/1137681689-20-v1.log", "/var/lib/redpanda/data/kafka/rxdata-dspbidrequest/0_1244/1137681689-20-v1.base_index"}
Nov 18 22:40:09 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:09,583 [shard 7] offset_translator - ntp: {kafka/rxdata-dspbidrequest/0} - offset_translator.cc:277 - prefix_truncate at offset: 1137795840, new state: {base offset/delta: {1137795840}/19, map size: 25, last delta: 43}
Nov 18 22:40:09 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:09,598 [shard 61] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.66.141:23407 (shutting down, Broken pipe)
Nov 18 22:40:09 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:09,598 [shard 9] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.66.76:38672 (shutting down, Broken pipe)
Nov 18 22:40:09 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:09,599 [shard 48] kafka - connection_context.cc:371 - Disconnected 10.16.67.58:10392 (Broken pipe)
Nov 18 22:40:09 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:09,599 [shard 41] kafka - connection_context.cc:371 - Disconnected 10.16.65.51:59380 (Broken pipe)
Nov 18 22:40:09 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:09,599 [shard 46] kafka - connection_context.cc:371 - Disconnected 10.16.67.72:33098 (Broken pipe)
Nov 18 22:40:09 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:09,599 [shard 48] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.67.58:10392 (shutting down, Broken pipe)
Nov 18 22:40:09 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:09,600 [shard 27] kafka - connection_context.cc:371 - Disconnected 10.16.66.158:49288 (Broken pipe)
Nov 18 22:40:09 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:09,600 [shard 27] kafka - connection_context.cc:371 - Disconnected 10.16.66.158:49288 (Broken pipe)
Nov 18 22:40:09 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:09,602 [shard 27] kafka - connection_context.cc:371 - Disconnected 10.16.66.158:49288 (Broken pipe)
Nov 18 22:40:09 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:09,602 [shard 33] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.67.22:25120 (shutting down, Broken pipe)
Nov 18 22:40:09 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:09,602 [shard 27] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.66.158:49288 (shutting down, Broken pipe)
Nov 18 22:40:09 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:09,605 [shard 9] kafka - connection_context.cc:371 - Disconnected 10.16.66.76:38672 (stream closed)
Nov 18 22:40:09 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:09,605 [shard 33] kafka - connection_context.cc:371 - Disconnected 10.16.67.22:25120 (stream closed)
Nov 18 22:40:09 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:09,605 [shard 61] kafka - connection_context.cc:371 - Disconnected 10.16.66.141:23407 (stream closed)
Nov 18 22:40:09 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:09,605 [shard 48] kafka - connection_context.cc:371 - Disconnected 10.16.66.204:37868 (Broken pipe)
Nov 18 22:40:09 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:09,605 [shard 48] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.66.204:37868 (shutting down, Broken pipe)
Nov 18 22:40:09 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:09,606 [shard 15] kafka - connection_context.cc:371 - Disconnected 10.16.67.70:47286 (Broken pipe)
Nov 18 22:40:09 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:09,607 [shard 33] kafka - connection_context.cc:371 - Disconnected 10.16.67.22:25120 (stream closed)
Nov 18 22:40:10 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:10,655 [shard 58] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rx-usersync-us/9_1234/8823716823-57-v1.log
Nov 18 22:40:10 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:10,750 [shard 13] raft - [group_id:6891, {kafka/rxdata-impbeacon/0}] consensus.cc:1566 - Received vote request with larger term from node {id: {7}, revision: {1248}}, received 45, current 44
Nov 18 22:40:10 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:10,850 [shard 13] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rxdata-impbeacon/0_1248/780914858-45-v1.log
Nov 18 22:40:10 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:10,934 [shard 20] raft - [group_id:7129, {kafka/__consumer_offsets/8}] vote_stm.cc:77 - vote reply from {id: {9}, revision: {1321}} - {term:{45}, target_node_id{id: {5}, revision: {1321}}, vote_granted: 1, log_ok:1}
Nov 18 22:40:10 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:10,934 [shard 20] raft - [group_id:7129, {kafka/__consumer_offsets/8}] vote_stm.cc:263 - becoming the leader term:45
Nov 18 22:40:10 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:10,936 [shard 20] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/__consumer_offsets/8_1321/40-45-v1.log
Nov 18 22:40:10 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:10,978 [shard 20] raft - [group_id:7129, {kafka/__consumer_offsets/8}] vote_stm.cc:279 - became the leader term: 45
Nov 18 22:40:11 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:11,229 [shard 11] raft - [group_id:6674, {kafka/pixelverse_conv_eu/3}] vote_stm.cc:77 - vote reply from {id: {0}, revision: {1204}} - {term:{57}, target_node_id{id: {5}, revision: {1204}}, vote_granted: 1, log_ok:1}
Nov 18 22:40:11 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:11,229 [shard 11] raft - [group_id:6674, {kafka/pixelverse_conv_eu/3}] vote_stm.cc:263 - becoming the leader term:57
Nov 18 22:40:11 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:11,230 [shard 11] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/pixelverse_conv_eu/3_1204/54-57-v1.log
Nov 18 22:40:11 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:11,266 [shard 21] raft - [group_id:6917, {kafka/rxdata-impexpired/6}] consensus.cc:1566 - Received vote request with larger term from node {id: {8}, revision: {1252}}, received 72, current 71
Nov 18 22:40:11 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:11,297 [shard 11] raft - [group_id:6674, {kafka/pixelverse_conv_eu/3}] vote_stm.cc:279 - became the leader term: 57
Nov 18 22:40:11 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:11,347 [shard 21] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rxdata-impexpired/6_1252/7373691-72-v1.log
Nov 18 22:40:11 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:11,352 [shard 26] storage - disk_log_impl.cc:1005 - remove_prefix_full_segments - tombstone & delete segment: {offset_tracker:{term:16, base_offset:19744756210, committed_offset:19745468370, dirty_offset:19745468370}, compacted_segment=0, finished_self_compaction=0, generation={0}, reader={/var/lib/redpanda/data/kafka/rx-bidrequest-stream/3_1214/19744756210-16-v1.log, (105801450 bytes)}, writer=nullptr, cache={cache_size=0}, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/kafka/rx-bidrequest-stream/3_1214/19744756210-16-v1.base_index, offsets:{19744756210}, index:{header_bitflags:0, base_offset:{19744756210}, max_offset:{19745468370}, base_timestamp:{timestamp: 1668569875000}, max_timestamp:{timestamp: 1668570006000}, index(3190,3190,3190)}, step:32768, needs_persistence:0}}
Nov 18 22:40:11 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:11,352 [shard 26] storage - segment.cc:112 - removing: {"/var/lib/redpanda/data/kafka/rx-bidrequest-stream/3_1214/19744756210-16-v1.log", "/var/lib/redpanda/data/kafka/rx-bidrequest-stream/3_1214/19744756210-16-v1.base_index"}
Nov 18 22:40:11 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:11,395 [shard 26] offset_translator - ntp: {kafka/rx-bidrequest-stream/3} - offset_translator.cc:277 - prefix_truncate at offset: 19745468370, new state: {base offset/delta: {19745468370}/16, map size: 36, last delta: 51}
Nov 18 22:40:11 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:11,632 [shard 29] raft - [group_id:6733, {kafka/rx-cm-post/2}] consensus.cc:1566 - Received vote request with larger term from node {id: {2}, revision: {1216}}, received 57, current 56
Nov 18 22:40:11 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:11,750 [shard 25] storage - disk_log_impl.cc:1005 - remove_prefix_full_segments - tombstone & delete segment: {offset_tracker:{term:15, base_offset:19745089483, committed_offset:19745789148, dirty_offset:19745789148}, compacted_segment=0, finished_self_compaction=0, generation={0}, reader={/var/lib/redpanda/data/kafka/rx-bidrequest-stream/0_1214/19745089483-15-v1.log, (103913983 bytes)}, writer=nullptr, cache={cache_size=0}, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/kafka/rx-bidrequest-stream/0_1214/19745089483-15-v1.base_index, offsets:{19745089483}, index:{header_bitflags:0, base_offset:{19745089483}, max_offset:{19745789148}, base_timestamp:{timestamp: 1668569877000}, max_timestamp:{timestamp: 1668570005000}, index(3134,3134,3134)}, step:32768, needs_persistence:0}}
Nov 18 22:40:11 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:11,750 [shard 25] storage - segment.cc:112 - removing: {"/var/lib/redpanda/data/kafka/rx-bidrequest-stream/0_1214/19745089483-15-v1.log", "/var/lib/redpanda/data/kafka/rx-bidrequest-stream/0_1214/19745089483-15-v1.base_index"}
Nov 18 22:40:11 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:11,790 [shard 25] offset_translator - ntp: {kafka/rx-bidrequest-stream/0} - offset_translator.cc:277 - prefix_truncate at offset: 19745789148, new state: {base offset/delta: {19745789148}/15, map size: 41, last delta: 55}
Nov 18 22:40:11 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:11,797 [shard 29] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rx-cm-post/2_1216/53-57-v1.log
Nov 18 22:40:11 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:11,893 [shard 5] raft - [group_id:6654, {kafka/pixelverse_conv_ap/3}] vote_stm.cc:77 - vote reply from {id: {8}, revision: {1200}} - {term:{67}, target_node_id{id: {5}, revision: {1200}}, vote_granted: 1, log_ok:1}
Nov 18 22:40:11 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:11,893 [shard 5] raft - [group_id:6654, {kafka/pixelverse_conv_ap/3}] vote_stm.cc:263 - becoming the leader term:67
Nov 18 22:40:11 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:11,894 [shard 5] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/pixelverse_conv_ap/3_1200/63-67-v1.log
Nov 18 22:40:11 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:11,979 [shard 5] raft - [group_id:6654, {kafka/pixelverse_conv_ap/3}] vote_stm.cc:279 - became the leader term: 67
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,276 [shard 9] raft - [group_id:7091, {kafka/summarystats-pubdealstat/0}] vote_stm.cc:77 - vote reply from {id: {4}, revision: {1288}} - {term:{49}, target_node_id{id: {5}, revision: {1288}}, vote_granted: 1, log_ok:1}
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,276 [shard 9] raft - [group_id:7091, {kafka/summarystats-pubdealstat/0}] vote_stm.cc:263 - becoming the leader term:49
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,277 [shard 9] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/summarystats-pubdealstat/0_1288/8195-49-v1.log
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,302 [shard 12] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.67.109:45674 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,303 [shard 48] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.25.8.80:8750 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,303 [shard 37] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.67.100:60278 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,303 [shard 39] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.66.115:16046 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,303 [shard 13] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.67.107:13176 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,303 [shard 11] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.67.129:41596 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,303 [shard 55] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.25.0.75:16796 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,303 [shard 39] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.67.86:42926 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,303 [shard 17] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.65.66:10250 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,303 [shard 46] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.66.181:62087 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,303 [shard 10] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.25.0.71:50428 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,303 [shard 24] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.67.71:22974 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,303 [shard 35] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.67.129:41458 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,303 [shard 22] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.66.76:48136 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 61] kafka - connection_context.cc:371 - Disconnected 10.16.67.96:26514 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 42] kafka - connection_context.cc:371 - Disconnected 10.16.67.76:37630 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 23] kafka - connection_context.cc:371 - Disconnected 10.16.67.124:21726 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 43] kafka - connection_context.cc:371 - Disconnected 10.16.66.169:61660 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 22] kafka - connection_context.cc:371 - Disconnected 10.16.66.76:48136 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 13] kafka - connection_context.cc:371 - Disconnected 10.16.67.107:13176 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 1] kafka - connection_context.cc:371 - Disconnected 10.16.67.122:11692 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 6] kafka - connection_context.cc:371 - Disconnected 10.16.66.138:6787 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 39] kafka - connection_context.cc:371 - Disconnected 10.16.66.115:16046 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 12] kafka - connection_context.cc:371 - Disconnected 10.16.67.109:45674 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 3] kafka - connection_context.cc:371 - Disconnected 10.25.4.79:23365 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 44] kafka - connection_context.cc:371 - Disconnected 10.16.67.65:7284 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 53] kafka - connection_context.cc:371 - Disconnected 10.16.67.72:21342 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 48] kafka - connection_context.cc:371 - Disconnected 10.25.8.80:8750 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 10] kafka - connection_context.cc:371 - Disconnected 10.16.66.110:13080 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 11] kafka - connection_context.cc:371 - Disconnected 10.16.67.129:41596 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 39] kafka - connection_context.cc:371 - Disconnected 10.16.67.117:57240 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 46] kafka - connection_context.cc:371 - Disconnected 10.16.66.110:32538 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 46] kafka - connection_context.cc:371 - Disconnected 10.16.66.181:62087 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 55] kafka - connection_context.cc:371 - Disconnected 10.25.0.75:16796 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 58] kafka - connection_context.cc:371 - Disconnected 10.16.65.65:40482 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 41] kafka - connection_context.cc:371 - Disconnected 10.16.67.30:12106 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 55] kafka - connection_context.cc:371 - Disconnected 10.16.66.91:56196 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 55] kafka - connection_context.cc:371 - Disconnected 10.16.67.118:28824 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 39] kafka - connection_context.cc:371 - Disconnected 10.16.67.86:42926 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 37] kafka - connection_context.cc:371 - Disconnected 10.16.67.100:60278 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 22] kafka - connection_context.cc:371 - Disconnected 10.16.67.111:31712 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 17] kafka - connection_context.cc:371 - Disconnected 10.16.65.66:10250 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 37] kafka - connection_context.cc:371 - Disconnected 10.16.66.141:61882 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 22] kafka - connection_context.cc:371 - Disconnected 10.25.4.63:37134 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 63] kafka - connection_context.cc:371 - Disconnected 10.16.66.183:19732 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 38] kafka - connection_context.cc:371 - Disconnected 10.16.67.33:46840 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,304 [shard 35] kafka - connection_context.cc:371 - Disconnected 10.16.67.129:41458 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,305 [shard 58] kafka - connection_context.cc:371 - Disconnected 10.16.67.30:12116 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,305 [shard 47] kafka - connection_context.cc:371 - Disconnected 10.16.67.99:16732 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,305 [shard 10] kafka - connection_context.cc:371 - Disconnected 10.16.67.96:58984 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,305 [shard 15] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.67.70:47286 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,305 [shard 16] kafka - connection_context.cc:371 - Disconnected 10.16.65.46:32446 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,305 [shard 10] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.67.96:58984 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,305 [shard 60] kafka - connection_context.cc:371 - Disconnected 10.25.0.62:46110 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,305 [shard 15] kafka - connection_context.cc:371 - Disconnected 10.16.67.70:47286 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,305 [shard 7] kafka - connection_context.cc:371 - Disconnected 10.16.66.141:24145 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,305 [shard 4] kafka - connection_context.cc:371 - Disconnected 10.25.0.39:48946 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,305 [shard 60] kafka - connection_context.cc:371 - Disconnected 10.25.8.78:57386 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,305 [shard 21] kafka - connection_context.cc:371 - Disconnected 10.16.67.96:26468 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,305 [shard 3] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.65.53:37422 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,305 [shard 10] kafka - connection_context.cc:371 - Disconnected 10.25.0.71:50428 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,305 [shard 24] kafka - connection_context.cc:371 - Disconnected 10.16.67.71:22974 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,305 [shard 21] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.67.96:26468 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,305 [shard 23] kafka - connection_context.cc:371 - Disconnected 10.16.65.71:60228 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,305 [shard 46] kafka - connection_context.cc:371 - Disconnected 10.16.67.72:33098 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,305 [shard 30] kafka - connection_context.cc:371 - Disconnected 10.16.65.67:58692 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,305 [shard 24] kafka - connection_context.cc:371 - Disconnected 10.16.67.33:46938 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,305 [shard 56] kafka - connection_context.cc:371 - Disconnected 10.16.67.118:28438 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,305 [shard 58] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.67.30:12116 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,306 [shard 7] kafka - connection_context.cc:371 - Disconnected 10.16.67.32:61718 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,306 [shard 7] kafka - connection_context.cc:371 - Disconnected 10.16.66.76:41826 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,306 [shard 41] kafka - connection_context.cc:371 - Disconnected 10.16.67.30:12106 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,306 [shard 47] kafka - connection_context.cc:371 - Disconnected 10.16.67.99:16732 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,306 [shard 41] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.67.30:12106 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,306 [shard 47] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.67.99:16732 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,306 [shard 27] kafka - connection_context.cc:371 - Disconnected 10.16.67.19:39612 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,306 [shard 39] kafka - connection_context.cc:371 - Disconnected 10.16.67.117:57240 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,306 [shard 24] kafka - connection_context.cc:371 - Disconnected 10.16.67.33:46938 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,306 [shard 24] kafka - connection_context.cc:371 - Disconnected 10.16.67.27:28630 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,306 [shard 27] kafka - connection_context.cc:371 - Disconnected 10.16.67.19:39430 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,306 [shard 46] kafka - connection_context.cc:371 - Disconnected 10.16.66.159:61565 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,306 [shard 33] kafka - connection_context.cc:371 - Disconnected 10.16.66.104:28514 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,306 [shard 39] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.67.117:57240 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,306 [shard 33] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.66.104:28514 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,306 [shard 60] kafka - connection_context.cc:371 - Disconnected 10.16.67.85:37684 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,306 [shard 60] kafka - connection_context.cc:371 - Disconnected 10.16.67.118:28560 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,306 [shard 3] kafka - connection_context.cc:371 - Disconnected 10.25.12.68:23496 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,306 [shard 7] kafka - connection_context.cc:371 - Disconnected 10.25.0.71:50464 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,306 [shard 58] kafka - connection_context.cc:371 - Disconnected 10.16.67.30:12116 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,306 [shard 57] kafka - connection_context.cc:371 - Disconnected 10.16.67.124:21558 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,307 [shard 56] kafka - connection_context.cc:371 - Disconnected 10.16.66.191:32808 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,307 [shard 25] kafka - connection_context.cc:371 - Disconnected 10.16.67.77:17912 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,307 [shard 41] kafka - connection_context.cc:371 - Disconnected 10.16.67.76:44540 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,307 [shard 3] kafka - connection_context.cc:371 - Disconnected 10.16.65.53:37422 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,308 [shard 30] kafka - connection_context.cc:371 - Disconnected 10.16.65.67:58692 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,308 [shard 30] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.65.67:58692 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,308 [shard 31] kafka - connection_context.cc:371 - Disconnected 10.16.67.122:11870 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,308 [shard 59] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.66.87:14335 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,308 [shard 26] kafka - connection_context.cc:371 - Disconnected 10.16.67.17:29976 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,308 [shard 23] kafka - connection_context.cc:371 - Disconnected 10.16.65.71:60228 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,308 [shard 39] kafka - connection_context.cc:371 - Disconnected 10.25.4.11:44364 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,308 [shard 46] kafka - connection_context.cc:371 - Disconnected 10.16.67.72:33098 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,308 [shard 54] kafka - connection_context.cc:371 - Disconnected 10.16.67.33:47058 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,308 [shard 62] kafka - connection_context.cc:371 - Disconnected 10.16.67.83:35654 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,308 [shard 23] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.65.71:60228 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,308 [shard 46] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.67.72:33098 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,308 [shard 7] kafka - connection_context.cc:371 - Disconnected 10.25.0.71:50464 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,308 [shard 7] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.25.0.71:50464 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,308 [shard 59] kafka - connection_context.cc:371 - Disconnected 10.16.66.87:14335 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,308 [shard 46] kafka - connection_context.cc:371 - Disconnected 10.16.66.159:61565 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,308 [shard 42] kafka - connection_context.cc:371 - Disconnected 10.16.67.116:25250 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,308 [shard 24] kafka - connection_context.cc:371 - Disconnected 10.16.67.33:46938 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,308 [shard 24] kafka - connection_context.cc:371 - Disconnected 10.16.67.27:28630 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,309 [shard 27] kafka - connection_context.cc:371 - Disconnected 10.16.67.19:39430 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,309 [shard 24] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.67.33:46938 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,309 [shard 46] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.66.159:61565 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,309 [shard 24] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.67.27:28630 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,309 [shard 3] kafka - connection_context.cc:371 - Disconnected 10.25.12.68:23496 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,309 [shard 56] kafka - connection_context.cc:371 - Disconnected 10.16.66.191:32808 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,309 [shard 3] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.25.12.68:23496 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,309 [shard 57] kafka - connection_context.cc:371 - Disconnected 10.16.67.124:21558 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,309 [shard 60] kafka - connection_context.cc:371 - Disconnected 10.16.67.118:28560 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,310 [shard 39] kafka - connection_context.cc:371 - Disconnected 10.25.4.11:44364 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,310 [shard 25] kafka - connection_context.cc:371 - Disconnected 10.16.67.77:17912 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,310 [shard 26] kafka - connection_context.cc:371 - Disconnected 10.16.66.114:29008 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,310 [shard 26] kafka - connection_context.cc:371 - Disconnected 10.16.67.17:29976 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,310 [shard 54] kafka - connection_context.cc:371 - Disconnected 10.16.67.33:47058 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,310 [shard 62] kafka - connection_context.cc:371 - Disconnected 10.16.67.83:35654 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,310 [shard 54] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.67.33:47058 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,311 [shard 27] kafka - connection_context.cc:371 - Disconnected 10.16.67.19:39430 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,311 [shard 59] kafka - connection_context.cc:371 - Disconnected 10.25.4.48:41332 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,311 [shard 60] kafka - connection_context.cc:371 - Disconnected 10.16.67.118:28560 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,311 [shard 57] kafka - connection_context.cc:371 - Disconnected 10.16.67.124:21558 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,311 [shard 57] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.67.124:21558 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,311 [shard 27] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.67.19:39430 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,311 [shard 39] kafka - connection_context.cc:371 - Disconnected 10.25.4.11:44364 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,311 [shard 60] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.67.118:28560 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,312 [shard 39] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.25.4.11:44364 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,312 [shard 26] kafka - connection_context.cc:371 - Disconnected 10.16.66.114:29008 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,312 [shard 26] kafka - connection_context.cc:371 - Disconnected 10.16.67.17:29976 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,312 [shard 26] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.67.17:29976 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,312 [shard 25] kafka - connection_context.cc:371 - Disconnected 10.16.67.77:17912 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,313 [shard 49] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.67.19:39978 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,313 [shard 25] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.16.67.77:17912 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,318 [shard 59] rpc - server.cc:135 - kafka rpc protocol - Disconnected 10.25.4.48:41332 (shutting down, Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,322 [shard 49] kafka - connection_context.cc:371 - Disconnected 10.16.67.19:39978 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,323 [shard 9] kafka - connection_context.cc:371 - Disconnected 10.16.65.52:52438 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,323 [shard 49] kafka - connection_context.cc:371 - Disconnected 10.16.67.19:39978 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,325 [shard 6] kafka - connection_context.cc:371 - Disconnected 10.16.66.191:33636 (Broken pipe)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,325 [shard 59] kafka - connection_context.cc:371 - Disconnected 10.25.4.48:41332 (stream closed)
Nov 18 22:40:12 srv-01-08-309.iad1.trmr.io rpk[54634]: INFO 2022-11-18 22:40:12,325 [shard 27] kafka - connection_context.cc:371 - Disconnected 10.16.67.26:44746 (Broken pipe)Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: Reactor stalled for 115 ms on shard 27. Backtrace: 0x4ab0f1f 0x4ab0c32 0x42abf 0x450de59 0x450b0e6 0x44fbfcf 0x4391bdb 0x4a1b3ac 0x4a2143c 0x4a30c11 0x3e83653 0x3decb62 0x3df481e 0x3dee862 0x3dedc98 0x43c59a9 0x43c56b6 0x43c6174 0x4ace59f 0x4ad2277 0x4b15745 0x4a6f10f 0x91016 0x1166cf
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffb886302e 0xffffffffb8e000df 0xffffffffb809c8bf 0xffffffffb8ab48be 0xffffffffb8c00d67 0xffffffffb8ac881d 0xffffffffb80fc43a 0xffffffffb80fc483 0xffffffffc0445151 0xffffffffb87db0b0 0xffffffffb87db24d 0xffffffffb84929fb 0xffffffffb8492f15 0xffffffffb84932ef 0xffffffffb849363a 0xffffffffb83cf672 0xffffffffb83cfac2 0xffffffffb83d022e 0xffffffffb83d0511 0xffffffffc05e3f6a 0xffffffffc05e4135 0xffffffffb83b177f 0xffffffffb83b2027 0xffffffffb83b5338 0xffffffffb83b5b67 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffb8e00072 0xffffffffb809c8bf 0xffffffffb8ab487a 0xffffffffb8c00d67
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffb81064f5 0xffffffffc05e3d92 0xffffffffc05e3ee6 0xffffffffc05e4135 0xffffffffb83b177f 0xffffffffb83b2027 0xffffffffb83b5338 0xffffffffb83b5b67 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffb87db20e 0xffffffffb84929fb 0xffffffffb8492f15 0xffffffffb84932ef 0xffffffffb849363a 0xffffffffb83cf672 0xffffffffb83cfac2 0xffffffffb83d022e 0xffffffffb83d0511 0xffffffffc05e3f6a 0xffffffffc05e4135 0xffffffffb83b177f 0xffffffffb83b2027 0xffffffffb83b5338 0xffffffffb83b5b67 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffb892da96 0xffffffffb893f779 0xffffffffb8941ae1 0xffffffffb890d827 0xffffffffb890daa6 0xffffffffb890db3f 0xffffffffb890dd39 0xffffffffb890dedd 0xffffffffb890e19a 0xffffffffb886218a 0xffffffffb886238a 0xffffffffb8862afa 0xffffffffc049e3cc 0xffffffffb8862c81 0xffffffffb88631c2 0xffffffffb8e000df 0xffffffffb809c8bf 0xffffffffb8ab487a 0xffffffffb8c00d67
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffb8ac87e5 0xffffffffb83ac95e 0xffffffffb83482a0 0xffffffffb8348607 0xffffffffb83486aa 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffb8250f1b 0xffffffffb82517f8 0xffffffffb83615c5 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffb853f00d 0xffffffffb853f0ed 0xffffffffb84b38e7 0xffffffffb84b53df 0xffffffffb84b567e 0xffffffffb848c082 0xffffffffb848daff 0xffffffffb83cf729 0xffffffffb83cfa19 0xffffffffb83d022e 0xffffffffb83d0511 0xffffffffc05e4b9d 0xffffffffc05e4cf2 0xffffffffb83b13c6 0xffffffffb83b20ca 0xffffffffb83b5338 0xffffffffb83b5b67 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: Reactor stalled for 80 ms on shard 28. Backtrace: 0x4ab0f1f 0x4ab0c32 0x42abf 0x44fbdea 0x4391bdb 0x4a1b3ac 0x4a2143c 0x4a30c11 0x3e83653 0x3decb62 0x3df481e 0x3dee862 0x3dedc98 0x43c59a9 0x43c56b6 0x43c6174 0x4ace59f 0x4ad2277 0x4b15745 0x4a6f10f 0x91016 0x1166cf
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffb8500959 0xffffffffc060c83e 0xffffffffc06050fb 0xffffffffc0605630 0xffffffffc05f27d5 0xffffffffb836aa1e 0xffffffffb836ceff 0xffffffffc05e3f9c 0xffffffffc05e4135 0xffffffffb83b177f 0xffffffffb83b2027 0xffffffffb83b5338 0xffffffffb83b5b67 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffc0442f5f 0xffffffffc04432e4 0xffffffffc044516d 0xffffffffb87db0b0 0xffffffffb87db24d 0xffffffffb84929fb 0xffffffffb8492f15 0xffffffffb84932ef 0xffffffffb849363a 0xffffffffb83cf672 0xffffffffb83cfac2 0xffffffffb83d022e 0xffffffffb83d0511 0xffffffffc05e3f6a 0xffffffffc05e4135 0xffffffffb83b177f 0xffffffffb83b2027 0xffffffffb83b5338 0xffffffffb83b5b67 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffb8ac87e5 0xffffffffb83ac95e 0xffffffffb83482a0 0xffffffffb8348607 0xffffffffb83486aa 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffb854b907 0xffffffffb82b1170 0xffffffffb83163fa 0xffffffffb8318879 0xffffffffb82b0725 0xffffffffb82b080b 0xffffffffb807c7fe 0xffffffffb8ab729c 0xffffffffb8c00b77
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffb8493111 0xffffffffb849363a 0xffffffffb83cf672 0xffffffffb83cfac2 0xffffffffb83d022e 0xffffffffb83d0511 0xffffffffc05e3f6a 0xffffffffc05e4135 0xffffffffb83b177f 0xffffffffb83b2027 0xffffffffb83b5338 0xffffffffb83b5b67 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffc05d5ada 0xffffffffc05f27cb 0xffffffffb836aa1e 0xffffffffb836ae45 0xffffffffb836b417 0xffffffffc05e4331 0xffffffffc05e4b6b 0xffffffffc05e4cf2 0xffffffffb83b13c6 0xffffffffb83b20ca 0xffffffffb83b5338 0xffffffffb83b5b67 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffc060b3d5 0xffffffffc06050fb 0xffffffffc0605630 0xffffffffc05f27d5 0xffffffffb836aa1e 0xffffffffb836ae45 0xffffffffb836b417 0xffffffffc05e4331 0xffffffffc05e4b6b 0xffffffffc05e4cf2 0xffffffffb83b13c6 0xffffffffb83b20ca 0xffffffffb83b5338 0xffffffffb83b5b67 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffb828fee5 0xffffffffb8267414 0xffffffffb82674de 0xffffffffb83cfe64 0xffffffffb83d0511 0xffffffffc05e3f6a 0xffffffffc05e4135 0xffffffffb83b177f 0xffffffffb83b2027 0xffffffffb83b5338 0xffffffffb83b5b67 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffb809c748 0xffffffffb885fade 0xffffffffc0f7eb6f 0xffffffffc0f807d3 0xffffffffb885f56e 0xffffffffb885fefa 0xffffffffc0fac0da 0xffffffffb885f56e 0xffffffffb885fefa 0xffffffffb886893d 0xffffffffb89115bd 0xffffffffb891279e 0xffffffffb89128ba 0xffffffffb89129da 0xffffffffb8913bfe 0xffffffffb8913fba 0xffffffffb8914275 0xffffffffb8935c97 0xffffffffb893747a 0xffffffffb8937c67 0xffffffffb8920618 0xffffffffb892143e 0xffffffffb8921e8d 0xffffffffb895e303 0xffffffffb8833fc2 0xffffffffb883431c 0xffffffffb8836731 0xffffffffb88367ec 0xffffffffb883685f 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffb84936ce 0xffffffffb83cf672 0xffffffffb83cfac2 0xffffffffb83d022e 0xffffffffb83d0511 0xffffffffc05e4b9d 0xffffffffc05e4cf2 0xffffffffb83b13c6 0xffffffffb83b20ca 0xffffffffb83b5338 0xffffffffb83b5b67 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffb8839b01 0xffffffffb8839fe1 0xffffffffb89213b2 0xffffffffb8921e8d 0xffffffffb895e303 0xffffffffb8833fc2 0xffffffffb883431c 0xffffffffb8836731 0xffffffffb88367ec 0xffffffffb883685f 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffb830b56e 0xffffffffb8269987 0xffffffffb82697c4 0xffffffffb848da93 0xffffffffb83cf729 0xffffffffb83cfa19 0xffffffffb83d022e 0xffffffffb83d0511 0xffffffffc05e3f6a 0xffffffffc05e4135 0xffffffffb83b177f 0xffffffffb83b2027 0xffffffffb83b5338 0xffffffffb83b5b67 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffb8067793 0xffffffffb80675b6 0xffffffffb81606f4 0xffffffffb81607e9 0xffffffffb807fbbb 0xffffffffb807fd5d 0xffffffffb82c152e 0xffffffffb82a819d 0xffffffffb82abdd7 0xffffffffb82b0703 0xffffffffb82b080b 0xffffffffb807c7fe 0xffffffffb8ab729c 0xffffffffb8c00b77
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffc049bc0e 0xffffffffc049e368 0xffffffffb8862c81 0xffffffffb88631c2 0xffffffffb8e000df 0xffffffffb809c8bf 0xffffffffb8ab48be 0xffffffffb8c00d67 0xffffffffb854b907 0xffffffffb82b1170 0xffffffffb83163fa 0xffffffffb8318879 0xffffffffb82b0725 0xffffffffb82b080b 0xffffffffb807c7fe 0xffffffffb8ab729c 0xffffffffb8c00b77
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: Reactor stalled for 77 ms on shard 59. Backtrace: 0x4ab0f1f 0x4ab0c32 0x42abf 0x3ededc2 0x3edea12 0x3e7d0cf 0x3e77983 0x3441b97 0x344163a 0x4ace59f 0x4ad2277 0x4b15745 0x4a6f10f 0x91016 0x1166cf
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffb8ab77b8 0xffffffffb8ab3126 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffb849367e 0xffffffffb83cf672 0xffffffffb83cfac2 0xffffffffb83d022e 0xffffffffb83d0511 0xffffffffc05e3f6a 0xffffffffc05e4135 0xffffffffb83b177f 0xffffffffb83b2027 0xffffffffb83b5338 0xffffffffb83b5b67 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffb8ac87e5 0xffffffffb83abdb0 0xffffffffb83abfec 0xffffffffb83ac264 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffb836ee89 0xffffffffb83b1d8e 0xffffffffb83b5338 0xffffffffb83b5b67 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffb848e49a 0xffffffffb83cfa56 0xffffffffb83d022e 0xffffffffb83d0511 0xffffffffc05e3f6a 0xffffffffc05e4135 0xffffffffb83b177f 0xffffffffb83b2027 0xffffffffb83b5338 0xffffffffb83b5b67 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffc05e40a8 0xffffffffb83b177f 0xffffffffb83b2027 0xffffffffb83b5338 0xffffffffb83b5b67 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffb854bfdc 0xffffffffb884a559 0xffffffffb884a5e8 0xffffffffb884a8b8 0xffffffffb8922b0c 0xffffffffb8923914 0xffffffffb895e434 0xffffffffb8833c61 0xffffffffb8833cff 0xffffffffb8345c62 0xffffffffb834818b 0xffffffffb8348525 0xffffffffb834857a 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffb83098e1 0xffffffffb8269757 0xffffffffb82697c4 0xffffffffc0904bb9 0xffffffffc09064b2 0xffffffffb84a1e79 0xffffffffb8493b20 0xffffffffb8493d9a 0xffffffffb83d0295 0xffffffffb83d0511 0xffffffffc05e3f6a 0xffffffffc05e4135 0xffffffffb83b177f 0xffffffffb83b2027 0xffffffffb83b5338 0xffffffffb83b5b67 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffb842fbfb 0xffffffffb8345535 0xffffffffb83480b0 0xffffffffb8348525 0xffffffffb834857a 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffc0440106 0xffffffffc044017c 0xffffffffc0442e7f 0xffffffffc04432e4 0xffffffffc044516d 0xffffffffb87db0b0 0xffffffffb87db24d 0xffffffffb84929fb 0xffffffffb8492f15 0xffffffffb84932ef 0xffffffffb849363a 0xffffffffb83cf672 0xffffffffb83cfac2 0xffffffffb83d022e 0xffffffffb83d0511 0xffffffffc05e3f6a 0xffffffffc05e4135 0xffffffffb83b177f 0xffffffffb83b2027 0xffffffffb83b5338 0xffffffffb83b5b67 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffb849301a 0xffffffffb84932ef 0xffffffffb849363a 0xffffffffb83cf672 0xffffffffb83cfac2 0xffffffffb83d022e 0xffffffffb83d0511 0xffffffffc05e3f6a 0xffffffffc05e4135 0xffffffffb83b177f 0xffffffffb83b2027 0xffffffffb83b5338 0xffffffffb83b5b67 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffb83cf605 0xffffffffb83d022e 0xffffffffb83d0511 0xffffffffc05e3f6a 0xffffffffc05e4135 0xffffffffb83b177f 0xffffffffb83b2027 0xffffffffb83b5338 0xffffffffb83b5b67 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack: 0xffffffffffffff80 0xffffffffc090513b 0xffffffffc09064b2 0xffffffffb84a1e79 0xffffffffb8493b20 0xffffffffb8493d9a 0xffffffffb83d0295 0xffffffffb83d0511 0xffffffffc05e3f6a 0xffffffffc05e4135 0xffffffffb83b177f 0xffffffffb83b2027 0xffffffffb83b5338 0xffffffffb83b5b67 0xffffffffb8ab314b 0xffffffffb8c0009b
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: Reactor stalled for 181 ms on shard 27. Backtrace: 0x4ab0f1f 0x4ab0c32 0x42abf 0x44fbed5 0x4391bdb 0x4a1b3ac 0x4a2143c 0x4a30c11 0x3e83653 0x3decb62 0x3df481e 0x3dee862 0x3dedc98 0x43c59a9 0x43c56b6 0x43c6174 0x4ace59f 0x4ad2277 0x4b15745 0x4a6f10f 0x91016 0x1166cf
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: Reactor stalled for 138 ms on shard 28. Backtrace: 0x4ab0f1f 0x4ab0c32 0x42abf 0x450a89b 0x44fbfcf 0x4391bdb 0x4a1b3ac 0x4a2143c 0x4a30c11 0x3e83653 0x3decb62 0x3df481e 0x3dee862 0x3dedc98 0x43c59a9 0x43c56b6 0x43c6174 0x4ace59f 0x4ad2277 0x4b15745 0x4a6f10f 0x91016 0x1166cf
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: Reactor stalled for 135 ms on shard 59. Backtrace: 0x4ab0f1f 0x4ab0c32 0x42abf 0x3eded5f 0x3edea12 0x3e7d0cf 0x3e77983 0x3441b97 0x344163a 0x4ace59f 0x4ad2277 0x4b15745 0x4a6f10f 0x91016 0x1166cf
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: Reactor stalled for 288 ms on shard 27. Backtrace: 0x4ab0f1f 0x4ab0c32 0x42abf 0x450dc0c 0x450b0e6 0x44fbfcf 0x4391bdb 0x4a1b3ac 0x4a2143c 0x4a30c11 0x3e83653 0x3decb62 0x3df481e 0x3dee862 0x3dedc98 0x43c59a9 0x43c56b6 0x43c6174 0x4ace59f 0x4ad2277 0x4b15745 0x4a6f10f 0x91016 0x1166cf
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: Reactor stalled for 245 ms on shard 28. Backtrace: 0x4ab0f1f 0x4ab0c32 0x42abf 0x450be2a 0x44fbfcf 0x4391bdb 0x4a1b3ac 0x4a2143c 0x4a30c11 0x3e83653 0x3decb62 0x3df481e 0x3dee862 0x3dedc98 0x43c59a9 0x43c56b6 0x43c6174 0x4ace59f 0x4ad2277 0x4b15745 0x4a6f10f 0x91016 0x1166cf
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: kernel callstack:
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,301 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,322 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,343 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,363 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: WARN 2022-11-18 22:40:02,369 [shard 27] seastar - Exceptional future ignored: std::bad_alloc (std::bad_alloc), backtrace: 0x4d1d6c4 0x4a1298b 0x3dfb960 0x4ace59f 0x4ad2277 0x4b15745 0x4a6f10f /opt/redpanda/lib/libc.so.6+0x91016 /opt/redpanda/lib/libc.so.6+0x1166cf
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: --------
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void> seastar::future<void>::handle_exception_type<auto ssx::spawn_with_gate_then<cluster::persisted_stm::make_snapshot_in_background()::$_3>(seastar::gate&, cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(seastar::broken_condition_variable const&)>(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(cluster::persisted_stm::make_snapshot_in_background()::$_3&&), seastar::futurize<cluster::persisted_stm::make_snapshot_in_background()::$_3>::type seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void> seastar::future<void>::handle_exception_type<auto ssx::spawn_with_gate_then<cluster::persisted_stm::make_snapshot_in_background()::$_3>(seastar::gate&, cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(seastar::broken_condition_variable const&)>(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)>(seastar::future<void> seastar::future<void>::handle_exception_type<auto ssx::spawn_with_gate_then<cluster::persisted_stm::make_snapshot_in_background()::$_3>(seastar::gate&, cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(seastar::broken_condition_variable const&)>(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::future<void> seastar::future<void>::handle_exception_type<auto ssx::spawn_with_gate_then<cluster::persisted_stm::make_snapshot_in_background()::$_3>(seastar::gate&, cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(seastar::broken_condition_variable const&)>(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)&, seastar::future_state<seastar::internal::monostate>&&), void>
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: WARN 2022-11-18 22:40:02,379 [shard 28] seastar - Exceptional future ignored: std::bad_alloc (std::bad_alloc), backtrace: 0x4d1d6c4 0x4a1298b 0x3dfb960 0x4ace59f 0x4ad2277 0x4b15745 0x4a6f10f /opt/redpanda/lib/libc.so.6+0x91016 /opt/redpanda/lib/libc.so.6+0x1166cf
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: --------
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void> seastar::future<void>::handle_exception_type<auto ssx::spawn_with_gate_then<cluster::persisted_stm::make_snapshot_in_background()::$_3>(seastar::gate&, cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(seastar::broken_condition_variable const&)>(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(cluster::persisted_stm::make_snapshot_in_background()::$_3&&), seastar::futurize<cluster::persisted_stm::make_snapshot_in_background()::$_3>::type seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void> seastar::future<void>::handle_exception_type<auto ssx::spawn_with_gate_then<cluster::persisted_stm::make_snapshot_in_background()::$_3>(seastar::gate&, cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(seastar::broken_condition_variable const&)>(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)>(seastar::future<void> seastar::future<void>::handle_exception_type<auto ssx::spawn_with_gate_then<cluster::persisted_stm::make_snapshot_in_background()::$_3>(seastar::gate&, cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(seastar::broken_condition_variable const&)>(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::future<void> seastar::future<void>::handle_exception_type<auto ssx::spawn_with_gate_then<cluster::persisted_stm::make_snapshot_in_background()::$_3>(seastar::gate&, cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(seastar::broken_condition_variable const&)>(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)::'lambda'(cluster::persisted_stm::make_snapshot_in_background()::$_3&&)&, seastar::future_state<seastar::internal::monostate>&&), void>
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,383 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,404 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,424 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,444 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,464 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,484 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,504 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,523 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,543 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,562 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,583 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,603 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,623 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,644 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,665 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,685 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,706 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,727 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,747 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,768 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,788 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,810 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,831 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,851 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,872 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,893 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,913 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,934 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,955 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,975 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:02 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:02,997 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,017 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,038 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,059 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,080 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,100 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,122 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,142 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,163 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,184 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,205 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,225 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,246 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,267 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,288 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,309 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,330 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,351 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,371 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,392 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,414 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,434 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,455 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,476 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,497 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,518 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,523 [shard 4] storage - disk_log_impl.cc:1005 - remove_prefix_full_segments - tombstone & delete segment: {offset_tracker:{term:14, base_offset:2286935605, committed_offset:2287256646, dirty_offset:2287256646}, compacted_segment=0, finished_self_compaction=0, generation={0}, reader={/var/lib/redpanda/data/kafka/rxdata-videoevent/3_1284/2286935605-14-v1.log, (108108551 bytes)}, writer=nullptr, cache={cache_size=0}, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/kafka/rxdata-videoevent/3_1284/2286935605-14-v1.base_index, offsets:{2286935605}, index:{header_bitflags:0, base_offset:{2286935605}, max_offset:{2287256646}, base_timestamp:{timestamp: 1668569508429}, max_timestamp:{timestamp: 1668569993353}, index(3228,3228,3228)}, step:32768, needs_persistence:0}}
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,523 [shard 4] storage - segment.cc:112 - removing: {"/var/lib/redpanda/data/kafka/rxdata-videoevent/3_1284/2286935605-14-v1.log", "/var/lib/redpanda/data/kafka/rxdata-videoevent/3_1284/2286935605-14-v1.base_index"}
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,539 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,553 [shard 4] offset_translator - ntp: {kafka/rxdata-videoevent/3} - offset_translator.cc:277 - prefix_truncate at offset: 2287256646, new state: {base offset/delta: {2287256646}/12, map size: 46, last delta: 57}
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,560 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,580 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,601 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,622 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,643 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,664 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,684 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,705 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,726 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,746 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,767 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,788 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,809 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,829 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,850 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,870 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,890 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,910 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,930 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,949 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,969 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:03 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:03,990 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,011 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,031 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,052 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,073 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,094 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,115 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,136 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,156 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,177 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,198 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,219 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,239 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,260 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,281 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,302 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,323 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,344 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,365 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,385 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,406 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,427 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,448 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,469 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,489 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,510 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,531 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,552 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,572 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,593 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,614 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,635 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,656 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,677 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,697 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,718 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,739 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,760 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,780 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,801 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,822 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,836 [shard 28] raft - [group_id:6940, {kafka/rxdata-impserve/9}] vote_stm.cc:77 - vote reply from {id: {2}, revision: {1256}} - {term:{70}, target_node_id{id: {3}, revision: {1256}}, vote_granted: 1, log_ok:1}
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,837 [shard 28] raft - [group_id:6940, {kafka/rxdata-impserve/9}] vote_stm.cc:263 - becoming the leader term:70
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,837 [shard 28] storage - segment.cc:630 - Creating new segment /var/lib/redpanda/data/kafka/rxdata-impserve/9_1256/2888454580-70-v1.log
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,837 [shard 28] raft - [group_id:6940, {kafka/rxdata-impserve/9}] vote_stm.cc:77 - vote reply from {id: {0}, revision: {1256}} - {term:{70}, target_node_id{id: {3}, revision: {1256}}, vote_granted: 1, log_ok:1}
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,843 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,864 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,884 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,905 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,926 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,946 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,967 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:04 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:04,988 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,004 [shard 28] raft - [group_id:6940, {kafka/rxdata-impserve/9}] vote_stm.cc:279 - became the leader term: 70
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,009 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,030 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,051 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,072 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,092 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,113 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,133 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,154 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,175 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,195 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,216 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,236 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,257 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,277 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,297 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,316 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,336 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,355 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,375 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,395 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,415 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,436 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,456 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,477 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,497 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,518 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,534 [shard 58] storage - disk_log_impl.cc:1005 - remove_prefix_full_segments - tombstone & delete segment: {offset_tracker:{term:21, base_offset:493196923, committed_offset:493217920, dirty_offset:493217920}, compacted_segment=0, finished_self_compaction=0, generation={0}, reader={/var/lib/redpanda/data/kafka/rxdata-bidrequest/1_1236/493196923-21-v1.log, (101820322 bytes)}, writer=nullptr, cache={cache_size=0}, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/kafka/rxdata-bidrequest/1_1236/493196923-21-v1.base_index, offsets:{493196923}, index:{header_bitflags:0, base_offset:{493196923}, max_offset:{493217920}, base_timestamp:{timestamp: 1668569882158}, max_timestamp:{timestamp: 1668569997170}, index(2826,2826,2826)}, step:32768, needs_persistence:0}}
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,534 [shard 58] storage - segment.cc:112 - removing: {"/var/lib/redpanda/data/kafka/rxdata-bidrequest/1_1236/493196923-21-v1.log", "/var/lib/redpanda/data/kafka/rxdata-bidrequest/1_1236/493196923-21-v1.base_index"}
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,538 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,559 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,567 [shard 58] offset_translator - ntp: {kafka/rxdata-bidrequest/1} - offset_translator.cc:277 - prefix_truncate at offset: 493217920, new state: {base offset/delta: {493217920}/21, map size: 63, last delta: 83}
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,579 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,600 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,621 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,642 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,663 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,683 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,704 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,725 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,745 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,766 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,786 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,807 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,828 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,849 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,869 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,890 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,910 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,931 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,952 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,973 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:05 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:05,993 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:06 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:06,014 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:06 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:06,034 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:06 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:06,055 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:06 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:06,075 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:06 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:06,096 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:06 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:06,116 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:06 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:06,136 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:06 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:06,157 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:06 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:06,178 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:06 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:06,199 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:06 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:06,219 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:06 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:06,240 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:06 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:06,260 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:06 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:06,281 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:06 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:06,302 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:06 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:06,323 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:06 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:06,343 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:06 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:06,364 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:06 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:06,384 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)
Nov 18 22:40:06 srv-01-04-404.iad1.trmr.io rpk[10694]: INFO 2022-11-18 22:40:06,405 [shard 59] cluster - state_machine.cc:141 - State machine for ntp={kafka/rxdata-bidrequest/5} handles std::bad_alloc (std::bad_alloc)Just an fyi - I had to blow away the cluster and all data and start over on 22.3.3. All the nodes started crashing Saturday evening. I wasn't able to keep any of them up for more than 15 seconds.
Hopefully whatever perfect storm of issues hit us doesn't happen with 22.3.3.
jcsp
commented
1 year ago v22.3.4 had two fixes for this area of code, so I think we can close this one out.
Version & Environment
Redpanda version: (use
rpk version): Current: 22.2.7 Upgraded to 22.3.2 then downgraded to 22.3.1, both have issues.OS: CentOS 7.9 Kernel: 5.19.8-1.el7.elrepo.x86_64
What went wrong?
After upgrading to 22.3.2, I observed the following errors in the journal. App would not start:
I then downgraded from 22.3.2 to 22.3.1 and attempted to start the app again. This was in the log:
Downgraded again, back on 22.2.7 and while that started, I did see this in the log:
What should have happened instead?
Application should start after the upgrade.
How to reproduce the issue?
Additional information
I suspect you won't see the same issues that I am experiencing but hopefully there is enough details to help track it down. I was asked to upgrade because I was observing the following issues originally:
From dmesg: