search

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.45k stars 579 forks source link

second node errors trying to join, seastar::abort_requested_exception (abort requested) #2667

Closed mzeier closed 2 years ago

mzeier commented 2 years ago

I have a single node cluster and I'm trying to join a second node.

I have a modified version of the Ansible playbook but it's basically the same shell commands. Before the shell commands are run, I dump out the commands and pause.

The first node is up and I'm able to look at rpk cluster metadata and create topics. The second node fails with the following log messages:

Oct 17 18:23:59 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[16420]: ERROR 2021-10-17 18:23:59,077 [shard 0] cluster - controller_backend.cc:346 - Error while reconciling topics - seastar::abort_requested_exception (abort requested)

I'll dump the whole config output of the second instance in the thread.

first node (id=1):

TASK [redpanda : configure redpanda - dry-run] ***********************************************************************************************************************************
ok: [redpanda-2c-dev3-1] => {
    "msg": [
        "rpk config set cluster_id 'redpanda-dev3'",
        "rpk config set organization 'redpanda-dev3'",
        "rpk config set redpanda.advertised_kafka_api '{",
        "  address: 10.199.182.189,",
        "  port: 9092",
        "}' --format yaml",
        "sudo rpk mode production",
        "",
        "sudo rpk config bootstrap \\",
        "  --id 1 \\",
        "  --self 10.199.182.189 \\",
        "  --ips 10.199.182.189",
        ""
    ]
}
[redpanda : Pause to verify commands]
Verify rpk startup commands.:
root@redpanda-2c-dev3-1-i-08028ef782c5597fd:/etc/redpanda# cat redpanda.yaml
cluster_id: redpanda-dev3
config_file: /etc/redpanda/redpanda.yaml
node_uuid: kF6BpQCnVDzJbap4xLaQirPFEGMKnvzq5qzgSb7BN8prgseDg
organization: redpanda-dev3
pandaproxy: {}
redpanda:
  admin:
  - address: 10.199.182.189
    port: 9644
  advertised_kafka_api:
  - address: 10.199.182.189
    port: 9092
  advertised_rpc_api:
    address: 10.199.182.189
    port: 33145
  auto_create_topics_enabled: true
  data_directory: /var/lib/redpanda/data
  developer_mode: false
  kafka_api:
  - address: 10.199.182.189
    port: 9092
  node_id: 1
  rpc_server:
    address: 10.199.182.189
    port: 33145
  seed_servers: []
rpk:
  coredump_dir: /var/lib/redpanda/coredump
  enable_memory_locking: false
  enable_usage_stats: true
  overprovisioned: false
  tune_aio_events: true
  tune_ballast_file: true
  tune_clocksource: true
  tune_coredump: false
  tune_cpu: true
  tune_disk_irq: true
  tune_disk_nomerges: true
  tune_disk_scheduler: true
  tune_disk_write_cache: true
  tune_fstrim: true
  tune_network: true
  tune_swappiness: true
  tune_transparent_hugepages: false
schema_registry: {}
root@redpanda-2c-dev3-1-i-08028ef782c5597fd:/etc/redpanda# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:fb:d8:bc:6a:f3 brd ff:ff:ff:ff:ff:ff
    inet 10.199.182.189/18 brd 10.199.191.255 scope global dynamic ens3
       valid_lft 2914sec preferred_lft 2914sec
    inet6 fe80::8fb:d8ff:febc:6af3/64 scope link
       valid_lft forever preferred_lft forever

second node (id=2):

TASK [redpanda : configure redpanda - dry-run] ***********************************************************************************************************************************
ok: [redpanda-2c-dev3-2] => {
    "msg": [
        "rpk config set cluster_id 'redpanda-dev3'",
        "rpk config set organization 'redpanda-dev3'",
        "rpk config set redpanda.advertised_kafka_api '{",
        "  address: 10.199.190.135,",
        "  port: 9092",
        "}' --format yaml",
        "sudo rpk mode production",
        "",
        "sudo rpk config bootstrap \\",
        "  --id 2 \\",
        "  --self 10.199.190.135 \\",
        "  --ips 10.199.182.189",
        ""
    ]
}
[redpanda : Pause to verify commands]
Verify rpk startup commands.:
root@redpanda-2c-dev3-2-i-0912a924afbc15f1d:/etc/redpanda# cat redpanda.yaml
cluster_id: redpanda-dev3
config_file: /etc/redpanda/redpanda.yaml
node_uuid: SC1wPAmyHnzpHx3dWuYJU9acQpv4i9Y2reXjPn2k8MbDWCKKE
organization: redpanda-dev3
pandaproxy: {}
redpanda:
  admin:
  - address: 10.199.190.135
    port: 9644
  advertised_kafka_api:
  - address: 10.199.190.135
    port: 9092
  advertised_rpc_api:
    address: 10.199.190.135
    port: 33145
  auto_create_topics_enabled: true
  data_directory: /var/lib/redpanda/data
  developer_mode: false
  kafka_api:
  - address: 10.199.190.135
    port: 9092
  node_id: 2
  rpc_server:
    address: 10.199.190.135
    port: 33145
  seed_servers:
  - host:
      address: 10.199.182.189
      port: 33145
rpk:
  coredump_dir: /var/lib/redpanda/coredump
  enable_memory_locking: false
  enable_usage_stats: true
  overprovisioned: false
  tune_aio_events: true
  tune_ballast_file: true
  tune_clocksource: true
  tune_coredump: false
  tune_cpu: true
  tune_disk_irq: true
  tune_disk_nomerges: true
  tune_disk_scheduler: true
  tune_disk_write_cache: true
  tune_fstrim: true
  tune_network: true
  tune_swappiness: true
  tune_transparent_hugepages: false
schema_registry: {}
root@redpanda-2c-dev3-2-i-0912a924afbc15f1d:/etc/redpanda# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:a5:d4:d6:59:f1 brd ff:ff:ff:ff:ff:ff
    inet 10.199.190.135/18 brd 10.199.191.255 scope global dynamic ens3
       valid_lft 2264sec preferred_lft 2264sec
    inet6 fe80::8a5:d4ff:fed6:59f1/64 scope link
       valid_lft forever preferred_lft forever
mzeier commented 2 years ago

debug messages:

Oct 17 18:30:56 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: Detecting the current cloud vendor and VM
Oct 17 18:30:56 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: Detected vendor 'aws' and VM type 'i3.2xlarge'
Oct 17 18:30:56 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: System check 'Ballast file present' failed. Required: true, Current false
Oct 17 18:30:56 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: System check 'Dir '/var/lib/redpanda/data' IRQs affinity set' failed. Required: true, Current false
Oct 17 18:30:56 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: System check 'I/O config file present' failed. Required: true, Current false
Oct 17 18:30:56 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: System check 'NTP Synced' failed. Required: true, Current false
Oct 17 18:30:56 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: System check 'Swap enabled' failed. Required: true, Current false
Oct 17 18:30:56 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: System check - PASSED
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: We'd love to hear about your experience with redpanda:
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: https://vectorized.io/feedback
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: Starting redpanda...
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: Running:
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: PWD=/ LOGNAME=redpanda HOME=/var/lib/redpanda LANG=C.UTF-8 START_ARGS=--check=true INVOCATION_ID=66e758d997da489cbfc89b49f2da1ff3 USER=redpanda NOTIFY_SOCKET=/run/systemd/notify SHLVL=0 JOURNAL_STREAM=9:100138 PATH=/opt/redpanda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin /opt/redpanda/bin/redpanda redpanda --redpanda-cfg /etc/redpanda/redpanda.yaml --lock-memory=false
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: Welcome to the Redpanda community!
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: Slack: https://vectorized.io/slack - is the main way the community interacts with one another in real time :)
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: Twitter: https://twitter.com/vectorizedio - come say hi!
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: Github Discussion: https://github.com/vectorizedio/redpanda/discussions - is preferred for longer, async, thoughtful discussions
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: GitHub Issues: https://github.com/vectorizedio/redpanda/issues - is reserved only for actual issues. Please use the GitHub for discussions.
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: Documentation: https://vectorized.io/docs/ - official docs site
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: Support: https://support.vectorized.io/ - to share private information with the production support vectorized team
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: Product Feedback: https://vectorized.io/feedback/ - let us know how we can improve your experience
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO   redpanda::main - application.cc:137 - Redpanda v21.9.5 - e19c3f2ae726e673f97af68535062250c11b4574
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO   redpanda::main - application.cc:145 - kernel=5.4.0-1045-aws, nodename=redpanda-2c-dev3-2-i-0912a924afbc15f1d, machine=x86_64
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,308 [shard 0] redpanda::main - application.cc:132 - System resources: { cpus: 8, available memory: 55.672GiB, reserved memory: 3.897GiB}
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,309 [shard 0] redpanda::main - application.cc:300 - Configuration:
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: cluster_id: redpanda-dev3
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: config_file: /etc/redpanda/redpanda.yaml
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: node_uuid: SC1wPAmyHnzpHx3dWuYJU9acQpv4i9Y2reXjPn2k8MbDWCKKE
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: organization: redpanda-dev3
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: pandaproxy: {}
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: redpanda:
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:   admin:
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:     - address: 10.199.182.189
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:       port: 9644
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:   advertised_kafka_api:
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:     - address: 10.199.190.135
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:       port: 9092
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:   advertised_rpc_api:
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:     address: 10.199.190.135
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:     port: 33145
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:   auto_create_topics_enabled: true
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:   data_directory: /var/lib/redpanda/data
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:   developer_mode: false
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:   kafka_api:
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:     - address: 10.199.182.189
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:       port: 9092
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:   node_id: 2
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:   rpc_server:
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:     address: 10.199.182.189
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:     port: 33145
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:   seed_servers: []
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: rpk:
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:   coredump_dir: /var/lib/redpanda/coredump
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:   enable_memory_locking: false
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:   enable_usage_stats: true
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:   overprovisioned: false
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:   tune_aio_events: true
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:   tune_ballast_file: true
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:   tune_clocksource: true
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:   tune_coredump: false
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:   tune_cpu: true
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:   tune_disk_irq: true
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:   tune_disk_nomerges: true
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:   tune_disk_scheduler: true
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:   tune_disk_write_cache: true
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:   tune_fstrim: true
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:   tune_network: true
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:   tune_swappiness: true
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]:   tune_transparent_hugepages: false
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: schema_registry: {}
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,309 [shard 0] redpanda::main - application.cc:304 - Use `rpk config set <cfg> <value>` to change values below:
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.advertised_kafka_api:{{:{host: 10.199.190.135, port: 9092}}}        - Address of Kafka API published to the clients
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.leader_balancer_idle_timeout:120000        - Leadership rebalancing idle timeout
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.full_raft_configuration_recovery_pattern:{}        - Recovery raft configuration on start for NTPs matching pattern
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.kafka_qdc_min_depth:1        - Minimum queue depth used in kafka queue depth control.
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.kafka_qdc_idle_depth:10        - Queue depth when idleness is detected in kafka queue depth control.
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.health_manager_tick_interval:180000        - How often the health manager runs
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.kafka_qdc_max_latency_ms:80        - Max latency threshold for kafka queue depth control depth tracking.
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.kafka_qdc_depth_alpha:0.8        - Smoothing factor for kafka queue depth control depth tracking.
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.kafka_qdc_enable:0        - Enable kafka queue depth control.
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.kafka_qdc_window_count:12        - Number of windows used in kafka queue depth control latency tracking.
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.superusers:{}        - List of superuser usernames
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.cloud_storage_max_connection_idle_time_ms:5000        - Max https connection idle time (ms)
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.cloud_storage_manifest_upload_timeout_ms:10000        - Manifest upload timeout (ms)
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.cloud_storage_segment_upload_timeout_ms:30000        - Log segment upload timeout (ms)
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.cloud_storage_trust_file:{nullopt}        - Path to certificate that should be used to validate server certificate during TLS handshake
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.cloud_storage_api_endpoint_port:443        - TLS port override
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.cloud_storage_disable_tls:0        - Disable TLS for all S3 connections
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.cloud_storage_reconciliation_interval_ms:10000        - Interval at which the archival service runs reconciliation (ms)
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.cloud_storage_api_endpoint:{nullopt}        - Optional API endpoint
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.cloud_storage_secret_key:{nullopt}        - AWS secret key
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.members_backend_retry_ms:5000        - Time between members backend reconciliation loop retries
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.compaction_ctrl_max_shares:1000        - maximum number of IO and CPU shares that compaction process can use
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.compaction_ctrl_min_shares:10        - minimum number of IO and CPU shares that compaction process can use
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.compaction_ctrl_d_coeff:0.2        - derivative coefficient for compaction PID controller.
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.compaction_ctrl_i_coeff:0        - integral coefficient for compaction PID controller.
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.kafka_qdc_depth_update_ms:7000        - Update frequency for kafka queue depth control.
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.compaction_ctrl_p_coeff:-12.5        - proportional coefficient for compaction PID controller. This has to be negative since compaction backlog should decrease when number of compaction shares increases
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.compaction_ctrl_update_interval_ms:30000        -
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.enable_auto_rebalance_on_node_add:0        - Enable automatic partition rebalancing when new nodes are added
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.node_management_operation_timeout_ms:5000        - Timeout for executing node management operations
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.leader_balancer_mute_timeout:300000        - Leadership rebalancing mute timeout
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.id_allocator_batch_size:1000        - Id allocator allocates messages in batches (each batch is a one log record) and then serves requests from memory without touching the log until the batch is exhausted.
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.id_allocator_log_capacity:100        - Capacity of the id_allocator log in number of messages. Once it reached id_allocator_stm should compact the log.
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.segment_appender_flush_timeout_ms:1000        - Maximum delay until buffered data is written
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.retention_bytes:{nullopt}        - max bytes per partition on disk before triggering a compaction
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,310 [shard 0] redpanda::main - application.cc:309 - redpanda.join_retry_timeout_ms:5000        - Time between cluster join retries in milliseconds
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.enable_admin_api:1        - Enable the admin API
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.raft_io_timeout_ms:10000        - Raft I/O timeout
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.zstd_decompress_workspace_bytes:8388608        - Size of the zstd decompression workspace
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.compaction_ctrl_backlog_size:{nullopt}        - target backlog size for compaction controller. if not set compaction target compaction backlog would be equal to
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.auto_create_topics_enabled:1        - Allow topic auto creation
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.reclaim_stable_window:10000        - Length of time above which growth is reset
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.fetch_session_eviction_timeout_ms:60000        - Minimum time before which unused session will get evicted from sessions. Maximum time after which inactive session will be deleted is two time given configuration valuecache
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.raft_transfer_leader_recovery_timeout_ms:10000        - Timeout waiting for follower recovery when transferring leadership
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.reclaim_max_size:4194304        - Maximum batch cache reclaim size
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.raft_smp_max_non_local_requests:13034000        - Maximum number of x-core requests pending in Raft seastar::smp group. (for more details look at `seastar::smp_service_group` documentation)
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.raft_timeout_now_timeout_ms:1000        - Timeout for a timeout now request
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.recovery_append_timeout_ms:5000        - Timeout for append entries requests issued while updating stale follower
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.internal_topic_replication_factor:3        - Target replication factor for internal topics
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.replicate_append_timeout_ms:3000        - Timeout for append entries requests issued while replicating entries
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.log_segment_size:1073741824        - How large in bytes should each log segment be (default 1G)
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.disable_batch_cache:0        - Disable batch cache in log manager
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.election_timeout_ms:1500        - Election timeout expressed in milliseconds
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.data_directory:{data_directory="/var/lib/redpanda/data"}        - Place where redpanda will keep the data
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.default_topic_partitions:1        - Default number of partitions per topic
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.wait_for_leader_timeout_ms:5000        - Timeout (ms) to wait for leadership in metadata cache
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.kafka_qdc_max_depth:100        - Maximum queue depth used in kafka queue depth control.
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.cluster_id:{nullopt}        - Cluster identifier
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.transaction_coordinator_delete_retention_ms:604800000        - delete segments older than this - default 1 week
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.transaction_coordinator_cleanup_policy:delete        - Cleanup policy for a transaction coordinator topic
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.max_kafka_throttle_delay_ms:60000        - Fail-safe maximum throttle delay on kafka requests
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.enable_idempotence:0        - Enable idempotent producer
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.id_allocator_replication:1        - Replication factor for an id allocator topic
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.abort_timed_out_transactions_interval_ms:60000        - How often look for the inactive transactions and abort them
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.default_topic_replications:1        - Default replication factor for new topics
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.kafka_api:{{:{host: 10.199.182.189, port: 9092}}}        - Address and port of an interface to listen for Kafka API requests
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.rpc_server_listen_backlog:{nullopt}        - TCP connection queue length for Kafka server and internal RPC server
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.group_topic_partitions:1        - Number of partitions in the internal group membership topic
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.log_compaction_interval_ms:10000        - How often do we trigger background compaction
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.cloud_storage_enabled:0        - Enable archival storage
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.delete_retention_ms:604800000        - delete segments older than this - default 1 week
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.abort_index_segment_size:50000        - Capacity (in number of txns) of an abort index segment
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.cloud_storage_bucket:{nullopt}        - AWS bucket that should be used to store data
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.enable_transactions:0        - Enable transactions
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.cloud_storage_access_key:{nullopt}        - AWS access key
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.tx_timeout_delay_ms:1000        - Delay before scheduling next check for timed out transactions
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.transactional_id_expiration_ms:604800000        - Producer ids are expired once this time has elapsed after the last write with the given producer id.
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.fetch_max_bytes:57671680        - Maximum number of bytes returned in fetch request
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.fetch_reads_debounce_timeout:1        - Time to wait for next read in fetch request when requested min bytes wasn't reached
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.enable_coproc:0        - Enable coprocessing mode
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.log_compression_type:producer        - Default topic compression type
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.log_message_timestamp_type:CreateTime        - Default topic messages timestamp type
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.kafka_qdc_latency_alpha:0.002        - Smoothing parameter for kafka queue depth control latency tracking.
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.node_id:{2}        - Unique id identifying a node in the cluster
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.log_cleanup_policy:delete        - Default topic cleanup policy
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.transaction_coordinator_log_segment_size:1073741824        - How large in bytes should each log segment be (default 1G)
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.tm_violation_recovery_policy:violation_recovery_policy: 0        - Describes how to recover from an invariant violation happened on the transaction coordinator level
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.coproc_max_batch_size:32768        - Maximum amount of bytes to read from one topic read
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.metadata_dissemination_retry_delay_ms:320        - Delay before retry a topic lookup in a shard or other meta tables
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.transaction_coordinator_replication:1        - Replication factor for a transaction coordinator topic
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.group_new_member_join_timeout:30000        - Timeout for new member joins
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.kafka_qdc_window_size_ms:1500        - Window size for kafka queue depth control latency tracking.
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.admin_api_tls:{}        - TLS configuration for admin HTTP server
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.group_max_session_timeout_ms:300000        - The maximum allowed session timeout for registered consumers. Longer timeouts give consumers more time to process messages in between heartbeats at the cost of a longer time to detect failures. Default quota tracking window size in milliseconds
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.enable_pid_file:1        - Enable pid file. You probably don't want to change this.
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.kafka_group_recovery_timeout_ms:30000        - Kafka group recovery timeout expressed in milliseconds
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.reclaim_growth_window:3000        - Length of time in which reclaim sizes grow
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.dashboard_dir:{nullopt}        - serve http dashboard on / url
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.group_min_session_timeout_ms:6000        - The minimum allowed session timeout for registered consumers. Shorter timeouts result in quicker failure detection at the cost of more frequent consumer heartbeating, which can overwhelm broker resources.
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.rack:{nullopt}        - Rack identifier
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.reclaim_min_size:131072        - Minimum batch cache reclaim size
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.cloud_storage_max_connections:20        - Max number of simultaneous uploads to S3
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.raft_replicate_batch_window_size:1048576        - Max size of requests cached for replication
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.rpc_server_tls:{ enabled: 0 key/cert files: {nullopt} ca file: {nullopt} client_auth_required: 0 }        - TLS configuration for RPC server
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.coproc_max_ingest_bytes:655360        - Maximum amount of data to hold from input logs in memory
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.alter_topic_cfg_timeout_ms:5000        - Time to wait for entries replication in controller log when executing alter configuration requst
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.rm_violation_recovery_policy:violation_recovery_policy: 0        - Describes how to recover from an invariant violation happened on the partition level
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.quota_manager_gc_sec:30000        - Quota manager GC frequency in milliseconds
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.controller_backend_housekeeping_interval_ms:1000        - Interval between iterations of controller backend housekeeping loop
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.tm_sync_timeout_ms:10000        - Time to wait state catch up before rejecting a request
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.raft_max_concurrent_append_requests_per_follower:16        - Maximum number of concurrent append entries requests sent by leader to one follower
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.seed_server_meta_topic_partitions:7        - Number of partitions in internal raft metadata topic
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.metadata_dissemination_interval_ms:3000        - Interaval for metadata dissemination batching
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.default_num_windows:10        - Default number of quota tracking windows
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.use_scheduling_groups:0        - Manage CPU scheduling
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.raft_heartbeat_timeout_ms:3000        - raft heartbeat RPC timeout
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.min_version:0        - minimum redpanda compat version
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.kafka_api_tls:{}        - TLS configuration for Kafka API endpoint
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.raft_learner_recovery_rate:104857600        - Raft learner recovery rate limit in bytes per sec
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.seed_servers:{}        - List of the seed servers used to join current cluster. If the seed_server list is empty the node will be a cluster root and it will form a new cluster
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.max_version:1        - max redpanda compat version
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.raft_heartbeat_disconnect_failures:3        - After how many failed heartbeats to forcibly close an unresponsive TCP connection.  Set to 0 to disable force disconnection.
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.advertised_rpc_api:{{host: 10.199.190.135, port: 33145}}        - Address of RPC endpoint published to other cluster members
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.enable_sasl:0        - Enable SASL authentication for Kafka connections.
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.kvstore_max_segment_size:16777216        - Key-value maximum segment size (bytes)
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.group_initial_rebalance_delay:300        - Extra delay (ms) added to rebalance phase to wait for new members
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.admin:{{:{host: 10.199.182.189, port: 9644}}}        - Address and port of admin server
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.coproc_offset_flush_interval_ms:300000        - Interval for which all coprocessor offsets are flushed to disk
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.coproc_supervisor_server:{host: 127.0.0.1, port: 43189}        - IpAddress and port for supervisor service
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.target_quota_byte_rate:2147483648        - Target quota byte rate (bytes per second) - 2GB default
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.create_topic_timeout_ms:2000        - Timeout (ms) to wait for new topic creation
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.metadata_dissemination_retries:30        - Number of attempts of looking up a topic's meta data like shard before failing a request
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.rpc_server_tcp_send_buf:{nullopt}        - TCP transmit buffer size in bytes.
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.admin_api_doc_dir:/usr/share/redpanda/admin-api-doc        - Admin API doc directory
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.rpc_server:{host: 10.199.182.189, port: 33145}        - IpAddress and port for RPC server
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.disable_metrics:0        - Disable registering metrics
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.coproc_max_inflight_bytes:10485760        - Maximum amountt of inflight bytes when sending data to wasm engine
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.developer_mode:0        - Skips most of the checks performed at startup, not recomended for production use
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.release_cache_on_segment_roll:0        - Free cache when segments roll
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.kvstore_flush_interval:10        - Key-value store flush interval (ms)
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.max_compacted_log_segment_size:5368709120        - Max compacted segment size after consolidation
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.rm_sync_timeout_ms:10000        - Time to wait state catch up before rejecting a request
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.default_window_sec:1000        - Default quota tracking window size in milliseconds
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.raft_heartbeat_interval_ms:150        - Milliseconds for raft leader heartbeats
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.readers_cache_eviction_timeout_ms:30000        - Duration after which inactive readers will be evicted from cache
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.cloud_storage_region:{nullopt}        - AWS region that houses the bucket used for storage
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.compacted_log_segment_size:268435456        - How large in bytes should each compacted log segment be (default 256MiB)
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.enable_leader_balancer:1        - Enable automatic leadership rebalancing
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.cloud_storage_initial_backoff_ms:100        - Initial backoff time for exponetial backoff algorithm (ms)
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - redpanda.rpc_server_tcp_recv_buf:{nullopt}        - TCP receive buffer size in bytes.
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - pandaproxy.advertised_pandaproxy_api:{}        - Rest API address and port to publish to client
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - pandaproxy.api_doc_dir:/usr/share/redpanda/proxy-api-doc        - API doc directory
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - pandaproxy.pandaproxy_api_tls:{}        - TLS configuration for Pandaproxy api
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - pandaproxy.pandaproxy_api:{{:{host: 0.0.0.0, port: 8082}}}        - Rest API listen address and port
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - pandaproxy_client.sasl_mechanism:        - The SASL mechanism to use when connecting
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - pandaproxy_client.consumer_heartbeat_interval_ms:500        - Interval (in milliseconds) for consumer heartbeats
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - pandaproxy_client.consumer_rebalance_timeout_ms:2000        - Timeout (in milliseconds) for consumer rebalance
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - pandaproxy_client.consumer_request_max_bytes:1048576        - Max bytes to fetch per request
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - pandaproxy_client.consumer_request_timeout_ms:100        - Interval (in milliseconds) for consumer request timeout
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - pandaproxy_client.scram_username:        - Username to use for SCRAM authentication mechanisms
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - pandaproxy_client.produce_batch_delay_ms:100        - Delay (in milliseconds) to wait before sending batch
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - pandaproxy_client.produce_batch_size_bytes:1048576        - Number of bytes to batch before sending to broker
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - pandaproxy_client.consumer_session_timeout_ms:10000        - Timeout (in milliseconds) for consumer session
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - pandaproxy_client.produce_batch_record_count:1000        - Number of records to batch before sending to broker
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - pandaproxy_client.retry_base_backoff_ms:100        - Delay (in milliseconds) for initial retry backoff
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - pandaproxy_client.retries:5        - Number of times to retry a request to a broker
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - pandaproxy_client.scram_password:        - Password to use for SCRAM authentication mechanisms
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - pandaproxy_client.broker_tls:{ enabled: 0 key/cert files: {nullopt} ca file: {nullopt} client_auth_required: 0 }        - TLS configuration for the brokers
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - pandaproxy_client.brokers:{{host: 10.199.182.189, port: 9092}}        - List of address and port of the brokers
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - schema_registry.api_doc_dir:/usr/share/redpanda/proxy-api-doc        - API doc directory
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - schema_registry.schema_registry_replication_factor:{nullopt}        - Replication factor for internal _schemas topic.  If unset, defaults to `default_topic_replication`
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - schema_registry.schema_registry_api_tls:{}        - TLS configuration for Schema Registry API
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - schema_registry.schema_registry_api:{{:{host: 0.0.0.0, port: 8081}}}        - Schema Registry API listen address and port
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - schema_registry_client.sasl_mechanism:        - The SASL mechanism to use when connecting
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - schema_registry_client.consumer_heartbeat_interval_ms:500        - Interval (in milliseconds) for consumer heartbeats
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - schema_registry_client.consumer_rebalance_timeout_ms:2000        - Timeout (in milliseconds) for consumer rebalance
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - schema_registry_client.consumer_request_max_bytes:1048576        - Max bytes to fetch per request
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - schema_registry_client.consumer_request_timeout_ms:100        - Interval (in milliseconds) for consumer request timeout
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - schema_registry_client.scram_username:        - Username to use for SCRAM authentication mechanisms
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - schema_registry_client.produce_batch_delay_ms:0        - Delay (in milliseconds) to wait before sending batch
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - schema_registry_client.produce_batch_size_bytes:0        - Number of bytes to batch before sending to broker
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - schema_registry_client.consumer_session_timeout_ms:10000        - Timeout (in milliseconds) for consumer session
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - schema_registry_client.produce_batch_record_count:0        - Number of records to batch before sending to broker
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - schema_registry_client.retry_base_backoff_ms:100        - Delay (in milliseconds) for initial retry backoff
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - schema_registry_client.retries:5        - Number of times to retry a request to a broker
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - schema_registry_client.scram_password:        - Password to use for SCRAM authentication mechanisms
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - schema_registry_client.broker_tls:{ enabled: 0 key/cert files: {nullopt} ca file: {nullopt} client_auth_required: 0 }        - TLS configuration for the brokers
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] redpanda::main - application.cc:309 - schema_registry_client.brokers:{{host: 10.199.182.189, port: 9092}}        - List of address and port of the brokers
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,311 [shard 0] syschecks - Writing pid file "/var/lib/redpanda/data/pid.lock"
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,312 [shard 0] syschecks - STATUS=checking environment (CPU, Mem)
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,318 [shard 0] storage - directories.h:33 - Checking `/var/lib/redpanda/data` for supported filesystems
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,840 [shard 0] syschecks - STATUS=Adding raft client cache
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,841 [shard 0] syschecks - STATUS=Building shard-lookup tables
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,841 [shard 0] syschecks - STATUS=Intializing storage services
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,841 [shard 0] syschecks - STATUS=Intializing raft recovery throttle
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,841 [shard 0] syschecks - STATUS=Intializing raft group manager
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,841 [shard 0] syschecks - STATUS=Adding partition manager
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,841 [shard 0] redpanda::main - application.cc:590 - Partition manager started
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,841 [shard 0] syschecks - STATUS=Creating cluster::controller
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,841 [shard 0] syschecks - STATUS=Creating kafka metadata cache
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,841 [shard 0] syschecks - STATUS=Creating metadata dissemination service
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,841 [shard 0] syschecks - STATUS=Creating partition manager
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,841 [shard 0] syschecks - STATUS=Creating kafka group shard mapper
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,841 [shard 0] syschecks - STATUS=Creating kafka group router
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,841 [shard 0] syschecks - STATUS=Adding kafka quota manager
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,842 [shard 0] syschecks - STATUS=Starting internal RPC {{://10.199.182.189:33145:PLAINTEXT}, max_service_memory_per_core: 2241645772, metrics_enabled:true}
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,844 [shard 0] syschecks - STATUS=Creating id allocator frontend
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,844 [shard 0] syschecks - STATUS=Creating group resource manager frontend
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,844 [shard 0] syschecks - STATUS=Creating partition resource manager frontend
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,844 [shard 0] syschecks - STATUS=Creating tx coordinator frontend
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,844 [shard 0] syschecks - STATUS=Starting kafka RPC {{://10.199.182.189:9092:PLAINTEXT}, max_service_memory_per_core: 2241645772, metrics_enabled:true}
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,845 [shard 0] syschecks - STATUS=constructing http server
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,845 [shard 0] syschecks - STATUS=Staring storage services
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,852 [shard 0] storage - segment_set.cc:239 - Recovered: {offset_tracker:{term:0, base_offset:0, committed_offset:8, dirty_offset:8}, compacted_segment=0, finished_self_compaction=0, reader={/var/lib/redpanda/data/redpanda/kvstore/0_0/0-0-v1.log, (1119 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/kvstore/0_0/0-0-v1.base_index, offsets:{0}, index:{header_size:64, header_checksum:14917501839194195956, header_bitflags:0, base_offset:{0}, max_offset:{8}, base_timestamp:{timestamp: 1634495039017}, max_timestamp:{timestamp: 1634495039094}, index(1,1,1)}, step:32768, needs_persistence:0}}
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,852 [shard 0] kvstore - kvstore.cc:497 - Replaying segment with base offset 0
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,855 [shard 0] syschecks - STATUS=Starting the partition manager
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,855 [shard 0] syschecks - STATUS=Starting Raft group manager
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,855 [shard 0] syschecks - STATUS=Starting Kafka group manager
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,855 [shard 0] syschecks - STATUS=Starting controller
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,855 [shard 0] cluster - raft0_utils.h:32 - Current node is cluster root
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,860 [shard 0] storage - segment_set.cc:239 - Recovered: {offset_tracker:{term:1, base_offset:0, committed_offset:0, dirty_offset:0}, compacted_segment=0, finished_self_compaction=0, reader={/var/lib/redpanda/data/redpanda/controller/0_0/0-1-v1.log, (177 bytes)}, writer=nullptr, cache={cache_size=0}, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/controller/0_0/0-1-v1.base_index, offsets:{0}, index:{header_size:48, header_checksum:16158495887184811764, header_bitflags:0, base_offset:{0}, max_offset:{0}, base_timestamp:{timestamp: 1634495039073}, max_timestamp:{timestamp: 1634495039073}, index(0,0,0)}, step:32768, needs_persistence:0}}
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,861 [shard 0] raft - [group_id:0, {redpanda/controller/0}] consensus.cc:951 - Starting
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,861 [shard 0] raft - [group_id:0, {redpanda/controller/0}] consensus.cc:1224 - Recovered persistent state from kvstore: voted for: {id: {2}, revision: {0}}, term: 1
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,861 [shard 0] raft - [group_id:0, {redpanda/controller/0}] consensus.cc:983 - Starting with voted_for {id: {2}, revision: {0}} term 1 initial_state false
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,861 [shard 0] raft - [group_id:0, {redpanda/controller/0}] consensus.cc:1008 - Read bootstrap state: data_seen 0 config_seen 0 eol false commit 0 term 0 prev_idx 0 prev_term 0 config_tracker -9223372036854775808 commit_base_tracker -9223372036854775808 configurations {}
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,861 [shard 0] raft - [group_id:0, {redpanda/controller/0}] consensus.cc:1009 - Current log offsets: {start_offset:0, committed_offset:0, committed_offset_term:1, dirty_offset:0, dirty_offset_term:1, last_term_start_offset:0}
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,861 [shard 0] raft - [group_id:0, {redpanda/controller/0}] consensus.cc:1027 - Truncating configurations at 0
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,871 [shard 0] storage - segment.cc:628 - Creating new segment /var/lib/redpanda/data/redpanda/kvstore/0_0/9-0-v1.log
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,884 [shard 0] raft - [group_id:0, {redpanda/controller/0}] consensus.cc:1123 - started raft, log offsets: {start_offset:0, committed_offset:0, committed_offset_term:1, dirty_offset:0, dirty_offset_term:1, last_term_start_offset:0}, term: 1, configuration: {current: {voters: {{id: {2}, revision: {0}}}, learners: {}}, old:{nullopt}, revision: 0, brokers: {{id: 2, kafka_advertised_listeners: {{:{host: 10.199.190.135, port: 9092}}}, rpc_address: {host: 10.199.190.135, port: 33145}, rack: {nullopt}, properties: {cores 8, mem_available 0, disk_available 0}, membership_state: active}}}
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,884 [shard 0] cluster - members_manager.cc:72 - starting cluster::members_manager...
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,884 [shard 0] cluster - state_machine.cc:29 - Starting state machine for ntp={redpanda/controller/0}
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,884 [shard 0] cluster - leader_balancer.cc:92 - Leader balancer: controller leadership lost
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,884 [shard 0] syschecks - STATUS=Starting metadata dissination service
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,885 [shard 0] syschecks - STATUS=Starting RPC
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: ERROR 2021-10-17 18:30:57,887 [shard 0] cluster - controller_backend.cc:346 - Error while reconciling topics - seastar::abort_requested_exception (abort requested)
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: ERROR 2021-10-17 18:30:57,887 [shard 1] cluster - controller_backend.cc:346 - Error while reconciling topics - seastar::abort_requested_exception (abort requested)
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: ERROR 2021-10-17 18:30:57,887 [shard 3] cluster - controller_backend.cc:346 - Error while reconciling topics - seastar::abort_requested_exception (abort requested)
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: ERROR 2021-10-17 18:30:57,887 [shard 4] cluster - controller_backend.cc:346 - Error while reconciling topics - seastar::abort_requested_exception (abort requested)
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: ERROR 2021-10-17 18:30:57,887 [shard 5] cluster - controller_backend.cc:346 - Error while reconciling topics - seastar::abort_requested_exception (abort requested)
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: ERROR 2021-10-17 18:30:57,887 [shard 7] cluster - controller_backend.cc:346 - Error while reconciling topics - seastar::abort_requested_exception (abort requested)
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: ERROR 2021-10-17 18:30:57,887 [shard 6] cluster - controller_backend.cc:346 - Error while reconciling topics - seastar::abort_requested_exception (abort requested)
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: ERROR 2021-10-17 18:30:57,887 [shard 2] cluster - controller_backend.cc:346 - Error while reconciling topics - seastar::abort_requested_exception (abort requested)
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,888 [shard 0] cluster - state_machine.cc:139 - State machine for ntp={redpanda/controller/0} handles raft::offset_monitor::wait_aborted (offset monitor wait aborted)
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,888 [shard 0] cluster - members_manager.cc:290 - stopping cluster::members_manager...
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,889 [shard 0] raft - [group_id:0, {redpanda/controller/0}] consensus.cc:158 - Stopping
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,894 [shard 0] raft - [group_id:0, {redpanda/controller/0}] vote_stm.cc:246 - became the leader term:2
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: WARN  2021-10-17 18:30:57,894 [shard 0] raft - [group_id:0, {redpanda/controller/0}] consensus.cc:818 - Error returned from voting process seastar::gate_closed_exception (gate closed)
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,895 [shard 7] kvstore - kvstore.cc:105 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/7_0
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,895 [shard 6] kvstore - kvstore.cc:105 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/6_0
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,895 [shard 5] kvstore - kvstore.cc:105 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/5_0
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,895 [shard 1] kvstore - kvstore.cc:105 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/1_0
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,895 [shard 3] kvstore - kvstore.cc:105 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/3_0
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,895 [shard 2] kvstore - kvstore.cc:105 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/2_0
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,895 [shard 0] kvstore - kvstore.cc:105 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/0_0
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,895 [shard 4] kvstore - kvstore.cc:105 - Stopping kvstore: dir /var/lib/redpanda/data/redpanda/kvstore/4_0
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,896 [shard 0] rpc - server.cc:166 - protocol not set - Stopping 0 listeners
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,896 [shard 0] rpc - server.cc:175 - protocol not set - Shutting down 0 connections
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,896 [shard 1] rpc - server.cc:166 - protocol not set - Stopping 0 listeners
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,896 [shard 5] rpc - server.cc:166 - protocol not set - Stopping 0 listeners
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,896 [shard 7] rpc - server.cc:166 - protocol not set - Stopping 0 listeners
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,896 [shard 5] rpc - server.cc:175 - protocol not set - Shutting down 0 connections
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,896 [shard 2] rpc - server.cc:166 - protocol not set - Stopping 0 listeners
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,896 [shard 3] rpc - server.cc:166 - protocol not set - Stopping 0 listeners
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,896 [shard 7] rpc - server.cc:175 - protocol not set - Shutting down 0 connections
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,896 [shard 1] rpc - server.cc:175 - protocol not set - Shutting down 0 connections
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,896 [shard 6] rpc - server.cc:166 - protocol not set - Stopping 0 listeners
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,896 [shard 6] rpc - server.cc:175 - protocol not set - Shutting down 0 connections
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,896 [shard 2] rpc - server.cc:175 - protocol not set - Shutting down 0 connections
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,896 [shard 3] rpc - server.cc:175 - protocol not set - Shutting down 0 connections
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,896 [shard 4] rpc - server.cc:166 - protocol not set - Stopping 0 listeners
Oct 17 18:30:57 redpanda-2c-dev3-2-i-0912a924afbc15f1d rpk[17334]: INFO  2021-10-17 18:30:57,897 [shard 4] rpc - server.cc:175 - protocol not set - Shutting down 0 connections
Oct 17 18:46:07 redpanda-2c-dev3-2-i-0912a924afbc15f1d systemd[1]: redpanda.service: Killing process 17334 (redpanda) with signal SIGKILL.
emaxerrno commented 2 years ago

cc: @rkruze

mzeier commented 2 years ago

Resolved. Looks like I inadvertently configured the secondary node as a seed node originally.