search

siderolabs / talos

Talos Linux is a modern Linux distribution built for Kubernetes.
https://www.talos.dev
Mozilla Public License 2.0
6.88k stars 553 forks source link

Upgrade hangs on 'task startAllServices (1/1): service "etcd" to be "up"' #5794

Closed scrayos closed 2 years ago

scrayos commented 2 years ago

Bug Report

Description

When I tried to upgrade a control-plane-node from version 1.0.6 to 1.1.0 (also happened during the upgrade from 1.0.5 to 1.0.6), the control plane node could not re-enter the cluster. I was able to observe the state of the cluster with talosctl dmesg -f during the upgrade and I've attached it below.

The only components that are installed on the cluster are cilium (in strict mode), rook-ceph and rook-ceph-cluster. All of them are installed through helm.

Configuration

This is the configuration for the control-plane-nodes (without the secrets of course):

version: 'v1alpha1'
debug: false
persist: true
machine:
  type: 'controlplane'
  token: '<token>'
  ca:
    crt: '<certificate>'
    key: '<key>'
  network:
    nameservers:
    - '1.1.1.1'
    - '1.0.0.1'
    - '8.8.8.8'
    - '8.8.4.4'
  install:
    disk: '/dev/sda'
    image: 'ghcr.io/siderolabs/installer:v1.1.0'
    bootloader: true
    wipe: true
  time:
    disabled: false
    servers:
    - 'time.cloudflare.com'
    - 'time.google.com'
    - 'pool.ntp.org'
  sysctls:
    kernel.sysrq: 0
  features:
    # we do not need or use RBAC for the Talos API, only for Kubernetes
    rbac: false
cluster:
  id: '<id>'
  secret: '<secret>'
  controlPlane:
    endpoint: 'https://control.example.com:6443'
    localAPIServerPort: 6443
  clusterName: 'example-production'
  network:
    cni:
      # set no CNI, as we set it with helm to Cilium (eBPF)
      name: 'none'
    dnsDomain: 'cluster.local'
    podSubnets:
    - '10.244.0.0/16'
    serviceSubnets:
    - '10.96.0.0/12'
  token: '<token>'
  aescbcEncryptionSecret: '<secret>'
  ca:
    crt: '<certificate>'
    key: '<key>'
  aggregatorCA:
    crt: '<certificate>'
    key: '<key>'
  serviceAccount:
    key: '<key>'
  apiServer:
    image: 'k8s.gcr.io/kube-apiserver:v1.23.6'
    extraArgs:
      oidc-issuer-url: 'https://auth.example.com/auth/realms/master'
      oidc-client-id: 'kubernetes'
      oidc-username-claim: 'sub'
      oidc-username-prefix: 'oidc:'
      oidc-groups-claim: 'groups'
      oidc-groups-prefix: 'oidc:'
    certSANs:
    - 'control.scrayos.net'
    # pod security policy is deprecated, so we disable it
    disablePodSecurityPolicy: true
    admissionControl:
    - name: 'PodSecurity'
      configuration:
        apiVersion: 'pod-security.admission.config.k8s.io/v1alpha1'
        kind: 'PodSecurityConfiguration'
        defaults:
          audit: 'restricted'
          audit-version: 'latest'
          enforce: 'baseline'
          enforce-version: 'latest'
          warn: 'restricted'
          warn-version: 'latest'
        exemptions:
          namespaces:
          - 'kube-system'
          - 'rook-ceph'
          runtimeClasses: []
          usernames: []
  controllerManager:
    image: 'k8s.gcr.io/kube-controller-manager:v1.23.6'
  proxy:
    # we disable the kube-proxy because we exclusively use cilium for cluster networking
    disabled: true
  scheduler:
    image: 'k8s.gcr.io/kube-scheduler:v1.23.6'
  discovery:
    enabled: true
    registries:
      kubernetes: {}
      service: {}
  etcd:
    image: 'gcr.io/etcd-development/etcd:v3.5.3'
    ca:
      crt: '<crt>'
      key: '<key>'
  coreDNS:
    image: 'docker.io/coredns/coredns:1.9.2'
  extraManifests: []
  extraManifestHeaders: {}
  inlineManifests: []
  allowSchedulingOnMasters: false

I also tried deploying a new server without the explicit version pins (except for the installer version) but that also blocked during etcd. During the update from 1.0.5 to 1.0.6 I was only able to "fix" it by deploying a completely new cluster (and then freshly bootstraping etcd).

Logs

The log from the talos_log.txt

Environment

smira commented 2 years ago

you might need to check etcd logs on all nodes to see what is up.

what is happening:

So you probably might need to inspect talosctl etcd members for sanity and talosctl logs etcd.

To retry joining you can do talosctl reset -n IP --system-labels-to-wipe=EPHEMERAL --reboot=true --graceful=false

frezbo commented 2 years ago

are you upgrading all controlplane nodes at once?

<ipv4-address>: user: warning: [2022-06-22T16:18:53.745298313Z]: [talos] task startAllServices (1/1): service "etcd" to be "up"
<ipv4-address>: user: warning: [2022-06-22T16:19:08.745172313Z]: [talos] task startAllServices (1/1): service "etcd" to be "up"
<ipv4-address>: user: warning: [2022-06-22T16:19:18.781830313Z]: [talos] failed promoting member: 3 error(s) occurred:
<ipv4-address>: user: warning: [2022-06-22T16:19:18.789519313Z]:  etcdserver: can only promote a learner member which is in sync with leader
<ipv4-address>: user: warning: [2022-06-22T16:19:18.794144313Z]:  etcdserver: rpc not supported for learner
<ipv4-address>: user: warning: [2022-06-22T16:19:18.796576313Z]:  timeout

also i see this in the logs, seems it cannot connect to the other etcd members, possibly some network issues

scrayos commented 2 years ago

I was upgrading a single control-plane node with talosctl -n <ip> upgrade --image ghcr.io/siderolabs/installer:v1.1.0.

But yeah, a network problem seems certainly possible. But I wouldn't really know where it's coming from. I closely followed the official guide for hcloud/hetzner cloud and deployed them with a load-balancer. The only difference is probably that I disabled the cni, because we use Cilium in strict mode.

frezbo commented 2 years ago

I was upgrading a single control-plane node with talosctl -n upgrade --image ghcr.io/siderolabs/installer:v1.1.0.

the cli shouldn;t allow you to upgrade a single control plane node unless --preserve is set, did you by any chance used --force?

So what happened here is that the upgrade process wiped the ephermeral data, but since there was no other control planes to re-create the etcd data it failed

smira commented 2 years ago

hmmm... it looks like etcd joined something according to the logs

smira commented 2 years ago

so I assumed there's something running

scrayos commented 2 years ago

the cli shouldn;t allow you to upgrade a single control plane node unless --preserve is set, did you by any chance used --force?

No, I did not use --force. In fact, I copied the excact command that I executed, but that's good to know. My talosconfig has one "endpoint" and four "nodes" like:

endpoint:
- 'controlplane-ip1'
nodes:
- 'worker-ip1'
- 'controlplane-ip1'
- 'controlplane-ip2'
- 'controlplane-ip3'

maybe that is wrong? I think the docs are a little bit thin regarding the addition of more nodes than the initial cp-node to the talosconfig.

Also, the docs regarding the upgrade are a little bit misleading in my opinion. If I remember correctly, the controlplane nodes are also upgraded individually in the explanation video and the command example also shows it with the --nodes <ip> flag. And the thing that ultimately made me upgrade the one controlplane node individually was the last answer of the FAQ:

Q. Can I break my cluster by upgrading everything at once?

A. Maybe - it’s not recommended.

Nothing prevents the user from sending near-simultaneous upgrades to each node of the cluster - and while Talos Linux and Kubernetes can generally deal with this situation, other components of the cluster may not be able to recover from more than one node rebooting at a time. (e.g. any software that maintains a quorum or state across nodes, such as Rook/Ceph)

frezbo commented 2 years ago

I was upgrading a single control-plane node with talosctl -n upgrade --image ghcr.io/siderolabs/installer:v1.1.0.

apologies, I mistakenly assumed from the above message you had a single control plane node.

What does talosctl logs etcd on the node that failed to upgrade shows?

scrayos commented 2 years ago

Sadly, I've redeployed the server already, but I tried to reproduce the problem with three new controlplane nodes and it seems like there is a broader problem. I am now unable to get the third server to join the etcd cluster (even without the update). The servers reused the same IP addresses of the previous cluster – maybe that is a problem with the Talos Discovery Service and a long ttl?

The server seems to have problem connecting to the Talos Discovery Service: talos-no-connection-log.txt (it is a v1.0.6 server and I've deployed it exactly as described in the official docs for hetzner cloud)

smira commented 2 years ago

The only real problem I see in the log is that your control plane endpoint doesn't work.

frezbo commented 2 years ago

maybe that is a problem with the Talos Discovery Service and a long ttl

the default ttl is 30mins, it's recommended to create a new set of machine config for a new cluster and not re-use the existing ones

smira commented 2 years ago

even with Discovery Service TTL, it doesn't make much sense.

during the upgrade, Talos controlplane node leaves etcd cluster, performs an upgrade, reboots and rejoins etcd back.

In order to rejoin etcd, Talos 1.0 requires connection to the Kubernetes control plane endpoint; Talos 1.1+ can also use discovery service data.

From your logs it seems that control plane endpoint is up, I wonder if that might be related.

reitermarkus commented 2 years ago

I have the exact same problem upgrading the last node in my cluster from 1.0.6 to 1.1.0.

talos-cp-1: user: warning: [2022-06-30T08:28:12.975244856Z]: [talos] service[etcd](Running): Health check failed: etcdserver: rpc not supported for learner
talos-cp-1: user: warning: [2022-06-30T08:28:18.529949856Z]: [talos] task startAllServices (1/1): service "etcd" to be "up"
talos-cp-1: user: warning: [2022-06-30T08:28:33.501711856Z]: [talos] task startAllServices (1/1): service "etcd" to be "up"
talos-cp-1: user: warning: [2022-06-30T08:28:48.493170856Z]: [talos] task startAllServices (1/1): service "etcd" to be "up"
reitermarkus commented 2 years ago

Here's the output of talosctl logs etcd for the corresponding node:

``` talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:56.919Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_CIPHER_SUITES","variable-value":"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:56.927Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["/usr/local/bin/etcd","--advertise-client-urls=https://10.0.0.41:2379","--auto-tls=false","--cert-file=/system/secrets/etcd/peer.crt","--client-cert-auth=true","--data-dir=/var/lib/etcd","--experimental-initial-corrupt-check=true","--key-file=/system/secrets/etcd/peer.key","--listen-client-urls=https://0.0.0.0:2379","--listen-peer-urls=https://0.0.0.0:2380","--name=talos-cp-1","--peer-auto-tls=false","--peer-cert-file=/system/secrets/etcd/peer.crt","--peer-client-cert-auth=true","--peer-key-file=/system/secrets/etcd/peer.key","--peer-trusted-ca-file=/system/secrets/etcd/ca.crt","--trusted-ca-file=/system/secrets/etcd/ca.crt"]} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:56.928Z","caller":"etcdmain/etcd.go:94","msg":"detected default host for advertise","host":"10.0.0.41"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:56.928Z","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/var/lib/etcd","dir-type":"member"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:56.928Z","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["https://0.0.0.0:2380"]} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:56.928Z","caller":"embed/etcd.go:479","msg":"starting with peer TLS","tls-info":"cert = /system/secrets/etcd/peer.crt, key = /system/secrets/etcd/peer.key, client-cert=, client-key=, trusted-ca = /system/secrets/etcd/ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":["TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"]} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:56.938Z","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["https://0.0.0.0:2379"]} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:56.938Z","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.4","git-sha":"08407ff76","go-version":"go1.16.15","go-os":"linux","go-arch":"amd64","max-cpu-set":24,"max-cpu-available":24,"member-initialized":true,"name":"talos-cp-1","data-dir":"/var/lib/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://10.0.0.41:2380"],"listen-peer-urls":["https://0.0.0.0:2380"],"advertise-client-urls":["https://10.0.0.41:2379"],"listen-client-urls":["https://0.0.0.0:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-size-bytes":2147483648,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","auto-compaction-mode":"periodic","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:57.790Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/etcd/member/snap/db","took":"851.996292ms"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:57.888Z","caller":"etcdserver/server.go:508","msg":"recovered v2 store from snapshot","snapshot-index":47939802,"snapshot-size":"91 kB"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:57.888Z","caller":"etcdserver/server.go:521","msg":"recovered v3 backend from snapshot","backend-size-bytes":177070080,"backend-size":"177 MB","backend-size-in-use-bytes":39116800,"backend-size-in-use":"39 MB"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:57.971Z","caller":"etcdserver/raft.go:483","msg":"restarting local member","cluster-id":"ed3ea90b36a924d0","local-member-id":"615fc079356c731c","commit-index":47952072} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:57.972Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"615fc079356c731c switched to configuration voters=(4124907066329562007 9918038437416413608) learners=(7016538371286397724)"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:57.972Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"615fc079356c731c became follower at term 570"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:57.972Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 615fc079356c731c [peers: [393e9d1f44127b97,89a3f3692ae029a8], term: 570, commit: 47952072, applied: 47939802, lastindex: 47952072, lastterm: 570]"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:57.972Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:57.972Z","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"ed3ea90b36a924d0","local-member-id":"615fc079356c731c","recovered-remote-peer-id":"393e9d1f44127b97","recovered-remote-peer-urls":["https://10.0.0.42:2380"]} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:57.972Z","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"ed3ea90b36a924d0","local-member-id":"615fc079356c731c","recovered-remote-peer-id":"615fc079356c731c","recovered-remote-peer-urls":["https://10.0.0.41:2380"]} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:57.972Z","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"ed3ea90b36a924d0","local-member-id":"615fc079356c731c","recovered-remote-peer-id":"89a3f3692ae029a8","recovered-remote-peer-urls":["https://10.0.0.43:2380"]} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:57.972Z","caller":"membership/cluster.go:287","msg":"set cluster version from store","cluster-version":"3.5"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:57.976Z","caller":"auth/store.go:1220","msg":"simple token is not cryptographically signed"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:57.978Z","caller":"mvcc/kvstore.go:345","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":40724912} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.046Z","caller":"mvcc/kvstore.go:415","msg":"kvstore restored","current-rev":40727346} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.049Z","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.053Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"393e9d1f44127b97"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.053Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"615fc079356c731c","remote-peer-id":"393e9d1f44127b97"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.053Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"615fc079356c731c","remote-peer-id":"393e9d1f44127b97"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.053Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"615fc079356c731c","remote-peer-id":"393e9d1f44127b97"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.054Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"393e9d1f44127b97"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.054Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"615fc079356c731c","remote-peer-id":"393e9d1f44127b97"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.054Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"615fc079356c731c","remote-peer-id":"393e9d1f44127b97","remote-peer-urls":["https://10.0.0.42:2380"]} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.054Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"89a3f3692ae029a8"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.054Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"615fc079356c731c","remote-peer-id":"89a3f3692ae029a8"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.054Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"615fc079356c731c","remote-peer-id":"393e9d1f44127b97"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.054Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"615fc079356c731c","remote-peer-id":"89a3f3692ae029a8"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.054Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"615fc079356c731c","remote-peer-id":"89a3f3692ae029a8"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.054Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"89a3f3692ae029a8"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.055Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"615fc079356c731c","remote-peer-id":"89a3f3692ae029a8"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.055Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"615fc079356c731c","remote-peer-id":"89a3f3692ae029a8","remote-peer-urls":["https://10.0.0.43:2380"]} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.055Z","caller":"etcdserver/corrupt.go:46","msg":"starting initial corruption check","local-member-id":"615fc079356c731c","timeout":"7s"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.055Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"615fc079356c731c","remote-peer-id":"89a3f3692ae029a8"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.059Z","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"393e9d1f44127b97"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.059Z","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"615fc079356c731c","remote-peer-id":"393e9d1f44127b97"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.059Z","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"615fc079356c731c","remote-peer-id":"393e9d1f44127b97"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.063Z","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"89a3f3692ae029a8"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.063Z","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"615fc079356c731c","remote-peer-id":"89a3f3692ae029a8"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.064Z","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"615fc079356c731c","remote-peer-id":"89a3f3692ae029a8"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.068Z","caller":"etcdserver/corrupt.go:289","msg":"failed hash kv request","local-member-id":"615fc079356c731c","requested-revision":40727346,"remote-peer-endpoint":"https://10.0.0.42:2380","error":"etcdserver: mvcc: required revision has been compacted"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.075Z","caller":"etcdserver/corrupt.go:289","msg":"failed hash kv request","local-member-id":"615fc079356c731c","requested-revision":40727346,"remote-peer-endpoint":"https://10.0.0.43:2380","error":"etcdserver: mvcc: required revision has been compacted"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.075Z","caller":"etcdserver/corrupt.go:116","msg":"initial corruption checking passed; no corruption","local-member-id":"615fc079356c731c"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.075Z","caller":"etcdserver/server.go:842","msg":"starting etcd server","local-member-id":"615fc079356c731c","local-server-version":"3.5.4","cluster-id":"ed3ea90b36a924d0","cluster-version":"3.5"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.075Z","caller":"etcdserver/server.go:752","msg":"starting initial election tick advance","election-ticks":10} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.080Z","caller":"embed/etcd.go:688","msg":"starting with client TLS","tls-info":"cert = /system/secrets/etcd/peer.crt, key = /system/secrets/etcd/peer.key, client-cert=, client-key=, trusted-ca = /system/secrets/etcd/ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":["TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"]} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.080Z","caller":"embed/etcd.go:277","msg":"now serving peer/client/metrics","local-member-id":"615fc079356c731c","initial-advertise-peer-urls":["http://10.0.0.41:2380"],"listen-peer-urls":["https://0.0.0.0:2380"],"advertise-client-urls":["https://10.0.0.41:2379"],"listen-client-urls":["https://0.0.0.0:2379"],"listen-metrics-urls":[]} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.080Z","caller":"embed/etcd.go:581","msg":"serving peer traffic","address":"[::]:2380"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.080Z","caller":"embed/etcd.go:553","msg":"cmux::serve","address":"[::]:2380"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.082Z","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"615fc079356c731c","to":"393e9d1f44127b97","stream-type":"stream Message"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.082Z","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"615fc079356c731c","to":"89a3f3692ae029a8","stream-type":"stream Message"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.082Z","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream Message","local-member-id":"615fc079356c731c","remote-peer-id":"393e9d1f44127b97"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.082Z","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream Message","local-member-id":"615fc079356c731c","remote-peer-id":"89a3f3692ae029a8"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.082Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 615fc079356c731c elected leader 89a3f3692ae029a8 at term 570"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.083Z","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"615fc079356c731c","to":"393e9d1f44127b97","stream-type":"stream MsgApp v2"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.083Z","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","local-member-id":"615fc079356c731c","remote-peer-id":"393e9d1f44127b97"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.083Z","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"615fc079356c731c","to":"89a3f3692ae029a8","stream-type":"stream MsgApp v2"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.083Z","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","local-member-id":"615fc079356c731c","remote-peer-id":"89a3f3692ae029a8"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.126Z","caller":"etcdserver/server.go:775","msg":"initialized peer connections; fast-forwarding election ticks","local-member-id":"615fc079356c731c","forward-ticks":8,"forward-duration":"800ms","election-ticks":10,"election-timeout":"1s","active-remote-members":2} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.620Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"1.68354ms","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.620Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"32.912µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.620Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"31.411µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.620Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"35.258µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.620Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"43.041µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.620Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"28.607µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.620Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.786µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.627Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"36.16µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.627Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"33.933µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.627Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"34.049µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.627Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"35.195µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.627Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.974µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.627Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"31.261µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.627Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"71.401µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.627Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.894µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.628Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"41.219µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.628Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"39.644µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.628Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.68µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.628Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"45.802µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.628Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"32.179µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.628Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.783µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.628Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.021µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.628Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"31.75µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.628Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.765µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.628Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"4.829µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.628Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.194µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.629Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"32.679µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.641Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":40726595} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.793Z","caller":"mvcc/kvstore_compaction.go:57","msg":"finished scheduled compaction","compact-revision":40726595,"took":"146.701623ms"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.870Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"15.628µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.870Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.535µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.870Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.156µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.870Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.249µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.870Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.73µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.870Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"4.614µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.870Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.391µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.870Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.146µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.870Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"4.451µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.870Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.796µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.870Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"4.905µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.870Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.323µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.870Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.906µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.870Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.344µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.870Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.256µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.870Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.303µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.870Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"4.63µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.870Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.22µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.870Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.245µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.870Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.272µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.870Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"4.579µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.870Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"29.475µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.871Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.1µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.871Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.977µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.871Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.396µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:47:58.871Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.454µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.907Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":40728026} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.921Z","caller":"etcdserver/server.go:2042","msg":"published local member to cluster through raft","local-member-id":"615fc079356c731c","local-member-attributes":"{Name:talos-cp-1 ClientURLs:[https://10.0.0.41:2379]}","request-path":"/0/members/615fc079356c731c/attributes","cluster-id":"ed3ea90b36a924d0","publish-timeout":"7s"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.921Z","caller":"embed/serve.go:98","msg":"ready to serve client requests"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.921Z","caller":"etcdmain/main.go:44","msg":"notifying init daemon"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.921Z","caller":"etcdmain/main.go:50","msg":"successfully notified init daemon"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:58.922Z","caller":"embed/serve.go:188","msg":"serving client traffic securely","address":"[::]:2379"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:47:59.072Z","caller":"mvcc/kvstore_compaction.go:57","msg":"finished scheduled compaction","compact-revision":40728026,"took":"156.243706ms"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:52:28.818Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"59.529µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:52:28.819Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.106µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:52:28.819Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"29.264µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:52:28.819Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"30.183µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:52:28.819Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.78µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:52:28.819Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"66.227µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:52:28.819Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"29.019µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:52:28.819Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.696µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:52:28.819Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"27.878µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:52:28.819Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"36.363µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:52:28.819Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.588µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:52:28.819Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"28.883µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:52:28.820Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"36.244µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:52:28.820Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.601µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:52:28.820Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.288µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:52:28.820Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.011µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:52:28.820Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"4.637µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:52:28.820Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"4.566µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:52:28.820Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.427µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:52:28.823Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.943844ms","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:52:28.888Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"22.531µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:52:29.606Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"522.988884ms","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure:<>>","response":"size:20"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:52:29.606Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"54.978µs","request":"header: lease_revoke:","response":"size:31","error":"lease not found"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:52:30.057Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"157.562377ms","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure:<>>","response":"size:20"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:52:30.667Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"140.891829ms","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure:<>>","response":"size:20"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:52:35.192Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"163.940365ms","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure:<>>","response":"size:20"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:52:39.973Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"492.178809ms","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure:<>>","response":"size:20"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:52:59.578Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"133.398242ms","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure:<>>","response":"size:20"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:53:00.512Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"441.811136ms","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure:<>>","response":"size:20"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:53:00.794Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"143.239318ms","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure:<>>","response":"size:20"} talos-cp-1: {"level":"warn","ts":"2022-06-30T08:53:31.198Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"138.513838ms","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure:<>>","response":"size:20"} talos-cp-1: {"level":"info","ts":"2022-06-30T08:54:00.063Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":40729685} talos-cp-1: {"level":"info","ts":"2022-06-30T08:54:00.217Z","caller":"mvcc/kvstore_compaction.go:57","msg":"finished scheduled compaction","compact-revision":40729685,"took":"151.085458ms"} ```
smira commented 2 years ago

from the log above it looks like etcd is copying the data from other nodes, and it was still doing that after 30 seconds which is completely normal. etcd should become healthy and promote itself to a full member

reitermarkus commented 2 years ago

The health check is still reporting as failed on this node:

NODE         SERVICE      STATE     HEALTH   LAST CHANGE   LAST EVENT
talos-cp-1   apid         Running   OK       13m28s ago    Health check successful
talos-cp-1   containerd   Running   OK       13m34s ago    Health check successful
talos-cp-1   cri          Running   OK       13m32s ago    Health check successful
talos-cp-1   etcd         Running   Fail     13m24s ago    Health check failed: etcdserver: rpc not supported for learner
talos-cp-1   kubelet      Running   OK       13m30s ago    Health check successful
talos-cp-1   machined     Running   ?        13m40s ago    Service started as goroutine
talos-cp-1   trustd       Running   OK       13m32s ago    Health check successful
talos-cp-1   udevd        Running   OK       13m34s ago    Health check successful
smira commented 2 years ago

so the root cause I guess is whatever makes etcd so slow to join the cluster.... (probably?)

I don't have any specific idea about it. What you could do is an attempt to re-join the cluster with:

talosctl reset --system-labels-to-wipe=STATE,EPHEMERAL --reboot --graceful=false

if you could capture the full dmesg after the reboot, that will be perfect

reitermarkus commented 2 years ago

Okay, I did that, here is the dmesg output:

``` talos-cp-1: kern: notice: [2022-06-30T12:55:07.922370274Z]: Linux version 5.15.48-talos (@buildkitsandbox) (gcc (GCC) 11.2.0, GNU ld (GNU Binutils) 2.37) #1 SMP Thu Jun 16 18:48:47 UTC 2022 talos-cp-1: kern: info: [2022-06-30T12:55:07.922370274Z]: Command line: BOOT_IMAGE=/A/vmlinuz talos.platform=metal talos.config=none console=ttyS0 console=tty0 init_on_alloc=1 slab_nomerge pti=on consoleblank=0 nvme_core.io_timeout=4294967295 random.trust_cpu=on printk.devkmsg=on ima_template=ima-ng ima_appraise=fix ima_hash=sha512 talos-cp-1: kern: info: [2022-06-30T12:55:07.922370274Z]: x86/fpu: x87 FPU will use FXSAVE talos-cp-1: kern: info: [2022-06-30T12:55:07.922370274Z]: signal: max sigframe size: 1440 talos-cp-1: kern: info: [2022-06-30T12:55:07.922370274Z]: BIOS-provided physical RAM map: talos-cp-1: kern: info: [2022-06-30T12:55:07.922370274Z]: BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable talos-cp-1: kern: info: [2022-06-30T12:55:07.922370274Z]: BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved talos-cp-1: kern: info: [2022-06-30T12:55:07.922370274Z]: BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved talos-cp-1: kern: info: [2022-06-30T12:55:07.922370274Z]: BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable talos-cp-1: kern: info: [2022-06-30T12:55:07.922370274Z]: BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved talos-cp-1: kern: info: [2022-06-30T12:55:07.922370274Z]: BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved talos-cp-1: kern: info: [2022-06-30T12:55:07.922370274Z]: BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved talos-cp-1: kern: info: [2022-06-30T12:55:07.922370274Z]: BIOS-e820: [mem 0x0000000100000000-0x000000063fffffff] usable talos-cp-1: kern: info: [2022-06-30T12:55:07.922370274Z]: NX (Execute Disable) protection: active talos-cp-1: kern: info: [2022-06-30T12:55:07.922370274Z]: SMBIOS 2.8 present. talos-cp-1: kern: info: [2022-06-30T12:55:07.922370274Z]: DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.15.0-0-g2dd4b9b3f840-prebuilt.qemu.org 04/01/2014 talos-cp-1: kern: info: [2022-06-30T12:55:07.922370274Z]: Hypervisor detected: KVM talos-cp-1: kern: info: [2022-06-30T12:55:07.922370274Z]: kvm-clock: Using msrs 4b564d01 and 4b564d00 talos-cp-1: kern: info: [2022-06-30T12:55:07.922370274Z]: kvm-clock: cpu 0, msr 1f2197001, primary cpu clock talos-cp-1: kern: info: [2022-06-30T12:55:07.922372274Z]: kvm-clock: using sched offset of 14854427422942 cycles talos-cp-1: kern: info: [2022-06-30T12:55:07.922376274Z]: clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns talos-cp-1: kern: info: [2022-06-30T12:55:07.922385274Z]: tsc: Detected 2599.996 MHz processor talos-cp-1: kern: debug: [2022-06-30T12:55:07.922900274Z]: e820: update [mem 0x00000000-0x00000fff] usable ==> reserved talos-cp-1: kern: debug: [2022-06-30T12:55:07.922903274Z]: e820: remove [mem 0x000a0000-0x000fffff] usable talos-cp-1: kern: info: [2022-06-30T12:55:07.922906274Z]: last_pfn = 0x640000 max_arch_pfn = 0x400000000 talos-cp-1: kern: info: [2022-06-30T12:55:07.922940274Z]: x86/PAT: Configuration [0-7]: WB WC UC- UC WB WC UC- UC talos-cp-1: kern: info: [2022-06-30T12:55:07.922951274Z]: last_pfn = 0xbffda max_arch_pfn = 0x400000000 talos-cp-1: kern: info: [2022-06-30T12:55:07.929334274Z]: found SMP MP-table at [mem 0x000f5b60-0x000f5b6f] talos-cp-1: kern: info: [2022-06-30T12:55:07.929856274Z]: RAMDISK: [mem 0x310f5000-0x34871fff] talos-cp-1: kern: info: [2022-06-30T12:55:07.929886274Z]: ACPI: Early table checksum verification disabled talos-cp-1: kern: info: [2022-06-30T12:55:07.929903274Z]: ACPI: RSDP 0x00000000000F58E0 000014 (v00 BOCHS ) talos-cp-1: kern: info: [2022-06-30T12:55:07.929911274Z]: ACPI: RSDT 0x00000000BFFE2960 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001) talos-cp-1: kern: info: [2022-06-30T12:55:07.929922274Z]: ACPI: FACP 0x00000000BFFE2442 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) talos-cp-1: kern: info: [2022-06-30T12:55:07.929928274Z]: ACPI: DSDT 0x00000000BFFDF040 003402 (v01 BOCHS BXPC 00000001 BXPC 00000001) talos-cp-1: kern: info: [2022-06-30T12:55:07.929933274Z]: ACPI: FACS 0x00000000BFFDF000 000040 talos-cp-1: kern: info: [2022-06-30T12:55:07.929936274Z]: ACPI: APIC 0x00000000BFFE24B6 000130 (v01 BOCHS BXPC 00000001 BXPC 00000001) talos-cp-1: kern: info: [2022-06-30T12:55:07.929939274Z]: ACPI: SSDT 0x00000000BFFE25E6 0000CA (v01 BOCHS VMGENID 00000001 BXPC 00000001) talos-cp-1: kern: info: [2022-06-30T12:55:07.929942274Z]: ACPI: HPET 0x00000000BFFE26B0 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) talos-cp-1: kern: info: [2022-06-30T12:55:07.929944274Z]: ACPI: SRAT 0x00000000BFFE26E8 000250 (v01 BOCHS BXPC 00000001 BXPC 00000001) talos-cp-1: kern: info: [2022-06-30T12:55:07.929947274Z]: ACPI: WAET 0x00000000BFFE2938 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) talos-cp-1: kern: info: [2022-06-30T12:55:07.929949274Z]: ACPI: Reserving FACP table memory at [mem 0xbffe2442-0xbffe24b5] talos-cp-1: kern: info: [2022-06-30T12:55:07.929951274Z]: ACPI: Reserving DSDT table memory at [mem 0xbffdf040-0xbffe2441] talos-cp-1: kern: info: [2022-06-30T12:55:07.929952274Z]: ACPI: Reserving FACS table memory at [mem 0xbffdf000-0xbffdf03f] talos-cp-1: kern: info: [2022-06-30T12:55:07.929953274Z]: ACPI: Reserving APIC table memory at [mem 0xbffe24b6-0xbffe25e5] talos-cp-1: kern: info: [2022-06-30T12:55:07.929954274Z]: ACPI: Reserving SSDT table memory at [mem 0xbffe25e6-0xbffe26af] talos-cp-1: kern: info: [2022-06-30T12:55:07.929954274Z]: ACPI: Reserving HPET table memory at [mem 0xbffe26b0-0xbffe26e7] talos-cp-1: kern: info: [2022-06-30T12:55:07.929955274Z]: ACPI: Reserving SRAT table memory at [mem 0xbffe26e8-0xbffe2937] talos-cp-1: kern: info: [2022-06-30T12:55:07.929956274Z]: ACPI: Reserving WAET table memory at [mem 0xbffe2938-0xbffe295f] talos-cp-1: kern: info: [2022-06-30T12:55:07.930024274Z]: SRAT: PXM 0 -> APIC 0x00 -> Node 0 talos-cp-1: kern: info: [2022-06-30T12:55:07.930027274Z]: SRAT: PXM 0 -> APIC 0x01 -> Node 0 talos-cp-1: kern: info: [2022-06-30T12:55:07.930028274Z]: SRAT: PXM 0 -> APIC 0x02 -> Node 0 talos-cp-1: kern: info: [2022-06-30T12:55:07.930028274Z]: SRAT: PXM 0 -> APIC 0x03 -> Node 0 talos-cp-1: kern: info: [2022-06-30T12:55:07.930029274Z]: SRAT: PXM 0 -> APIC 0x04 -> Node 0 talos-cp-1: kern: info: [2022-06-30T12:55:07.930030274Z]: SRAT: PXM 0 -> APIC 0x05 -> Node 0 talos-cp-1: kern: info: [2022-06-30T12:55:07.930030274Z]: SRAT: PXM 0 -> APIC 0x06 -> Node 0 talos-cp-1: kern: info: [2022-06-30T12:55:07.930031274Z]: SRAT: PXM 0 -> APIC 0x07 -> Node 0 talos-cp-1: kern: info: [2022-06-30T12:55:07.930032274Z]: SRAT: PXM 0 -> APIC 0x08 -> Node 0 talos-cp-1: kern: info: [2022-06-30T12:55:07.930033274Z]: SRAT: PXM 0 -> APIC 0x09 -> Node 0 talos-cp-1: kern: info: [2022-06-30T12:55:07.930033274Z]: SRAT: PXM 0 -> APIC 0x0a -> Node 0 talos-cp-1: kern: info: [2022-06-30T12:55:07.930034274Z]: SRAT: PXM 0 -> APIC 0x0b -> Node 0 talos-cp-1: kern: info: [2022-06-30T12:55:07.930035274Z]: SRAT: PXM 1 -> APIC 0x10 -> Node 1 talos-cp-1: kern: info: [2022-06-30T12:55:07.930035274Z]: SRAT: PXM 1 -> APIC 0x11 -> Node 1 talos-cp-1: kern: info: [2022-06-30T12:55:07.930036274Z]: SRAT: PXM 1 -> APIC 0x12 -> Node 1 talos-cp-1: kern: info: [2022-06-30T12:55:07.930037274Z]: SRAT: PXM 1 -> APIC 0x13 -> Node 1 talos-cp-1: kern: info: [2022-06-30T12:55:07.930037274Z]: SRAT: PXM 1 -> APIC 0x14 -> Node 1 talos-cp-1: kern: info: [2022-06-30T12:55:07.930038274Z]: SRAT: PXM 1 -> APIC 0x15 -> Node 1 talos-cp-1: kern: info: [2022-06-30T12:55:07.930039274Z]: SRAT: PXM 1 -> APIC 0x16 -> Node 1 talos-cp-1: kern: info: [2022-06-30T12:55:07.930039274Z]: SRAT: PXM 1 -> APIC 0x17 -> Node 1 talos-cp-1: kern: info: [2022-06-30T12:55:07.930040274Z]: SRAT: PXM 1 -> APIC 0x18 -> Node 1 talos-cp-1: kern: info: [2022-06-30T12:55:07.930041274Z]: SRAT: PXM 1 -> APIC 0x19 -> Node 1 talos-cp-1: kern: info: [2022-06-30T12:55:07.930042274Z]: SRAT: PXM 1 -> APIC 0x1a -> Node 1 talos-cp-1: kern: info: [2022-06-30T12:55:07.930043274Z]: SRAT: PXM 1 -> APIC 0x1b -> Node 1 talos-cp-1: kern: info: [2022-06-30T12:55:07.930046274Z]: ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x0009ffff] talos-cp-1: kern: info: [2022-06-30T12:55:07.930048274Z]: ACPI: SRAT: Node 0 PXM 0 [mem 0x00100000-0xbfffffff] talos-cp-1: kern: info: [2022-06-30T12:55:07.930050274Z]: ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x33fffffff] talos-cp-1: kern: info: [2022-06-30T12:55:07.930051274Z]: ACPI: SRAT: Node 1 PXM 1 [mem 0x340000000-0x63fffffff] talos-cp-1: kern: info: [2022-06-30T12:55:07.930052274Z]: NUMA: Node 0 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0xbfffffff] -> [mem 0x00000000-0xbfffffff] talos-cp-1: kern: info: [2022-06-30T12:55:07.930054274Z]: NUMA: Node 0 [mem 0x00000000-0xbfffffff] + [mem 0x100000000-0x33fffffff] -> [mem 0x00000000-0x33fffffff] talos-cp-1: kern: info: [2022-06-30T12:55:07.930061274Z]: NODE_DATA(0) allocated [mem 0x33fffc000-0x33fffffff] talos-cp-1: kern: info: [2022-06-30T12:55:07.930074274Z]: NODE_DATA(1) allocated [mem 0x63ffe8000-0x63ffebfff] talos-cp-1: kern: info: [2022-06-30T12:55:07.930148274Z]: Zone ranges: talos-cp-1: kern: info: [2022-06-30T12:55:07.930149274Z]: DMA [mem 0x0000000000001000-0x0000000000ffffff] talos-cp-1: kern: info: [2022-06-30T12:55:07.930151274Z]: DMA32 [mem 0x0000000001000000-0x00000000ffffffff] talos-cp-1: kern: info: [2022-06-30T12:55:07.930153274Z]: Normal [mem 0x0000000100000000-0x000000063fffffff] talos-cp-1: kern: info: [2022-06-30T12:55:07.930154274Z]: Movable zone start for each node talos-cp-1: kern: info: [2022-06-30T12:55:07.930155274Z]: Early memory node ranges talos-cp-1: kern: info: [2022-06-30T12:55:07.930155274Z]: node 0: [mem 0x0000000000001000-0x000000000009efff] talos-cp-1: kern: info: [2022-06-30T12:55:07.930158274Z]: node 0: [mem 0x0000000000100000-0x00000000bffd9fff] talos-cp-1: kern: info: [2022-06-30T12:55:07.930159274Z]: node 0: [mem 0x0000000100000000-0x000000033fffffff] talos-cp-1: kern: info: [2022-06-30T12:55:07.930160274Z]: node 1: [mem 0x0000000340000000-0x000000063fffffff] talos-cp-1: kern: info: [2022-06-30T12:55:07.930163274Z]: Initmem setup node 0 [mem 0x0000000000001000-0x000000033fffffff] talos-cp-1: kern: info: [2022-06-30T12:55:07.930167274Z]: Initmem setup node 1 [mem 0x0000000340000000-0x000000063fffffff] talos-cp-1: kern: info: [2022-06-30T12:55:07.930175274Z]: On node 0, zone DMA: 1 pages in unavailable ranges talos-cp-1: kern: info: [2022-06-30T12:55:07.930211274Z]: On node 0, zone DMA: 97 pages in unavailable ranges talos-cp-1: kern: info: [2022-06-30T12:55:07.961064274Z]: On node 0, zone Normal: 38 pages in unavailable ranges talos-cp-1: kern: info: [2022-06-30T12:55:07.991215274Z]: ACPI: PM-Timer IO Port: 0x608 talos-cp-1: kern: info: [2022-06-30T12:55:07.991239274Z]: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) talos-cp-1: kern: info: [2022-06-30T12:55:07.991287274Z]: IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 talos-cp-1: kern: info: [2022-06-30T12:55:07.991291274Z]: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) talos-cp-1: kern: info: [2022-06-30T12:55:07.991293274Z]: ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) talos-cp-1: kern: info: [2022-06-30T12:55:07.991294274Z]: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) talos-cp-1: kern: info: [2022-06-30T12:55:07.991297274Z]: ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) talos-cp-1: kern: info: [2022-06-30T12:55:07.991298274Z]: ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) talos-cp-1: kern: info: [2022-06-30T12:55:07.991302274Z]: ACPI: Using ACPI (MADT) for SMP configuration information talos-cp-1: kern: info: [2022-06-30T12:55:07.991303274Z]: ACPI: HPET id: 0x8086a201 base: 0xfed00000 talos-cp-1: kern: info: [2022-06-30T12:55:07.991315274Z]: smpboot: Allowing 24 CPUs, 0 hotplug CPUs talos-cp-1: kern: info: [2022-06-30T12:55:07.991369274Z]: [mem 0xc0000000-0xfeffbfff] available for PCI devices talos-cp-1: kern: info: [2022-06-30T12:55:07.991371274Z]: Booting paravirtualized kernel on KVM talos-cp-1: kern: info: [2022-06-30T12:55:07.991382274Z]: clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns talos-cp-1: kern: info: [2022-06-30T12:55:08.003268274Z]: setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:24 nr_node_ids:2 talos-cp-1: kern: info: [2022-06-30T12:55:08.010228274Z]: percpu: Embedded 55 pages/cpu s185744 r8192 d31344 u262144 talos-cp-1: kern: debug: [2022-06-30T12:55:08.010250274Z]: pcpu-alloc: s185744 r8192 d31344 u262144 alloc=1*2097152 talos-cp-1: kern: debug: [2022-06-30T12:55:08.010256274Z]: pcpu-alloc: [0] 00 01 02 03 04 05 06 07 [0] 08 09 10 11 -- -- -- -- talos-cp-1: kern: debug: [2022-06-30T12:55:08.010271274Z]: pcpu-alloc: [1] 12 13 14 15 16 17 18 19 [1] 20 21 22 23 -- -- -- -- talos-cp-1: kern: info: [2022-06-30T12:55:08.010312274Z]: kvm-guest: stealtime: cpu 0, msr 333a1c0c0 talos-cp-1: kern: info: [2022-06-30T12:55:08.010330274Z]: Built 2 zonelists, mobility grouping on. Total pages: 6192858 talos-cp-1: kern: info: [2022-06-30T12:55:08.010333274Z]: Policy zone: Normal talos-cp-1: kern: notice: [2022-06-30T12:55:08.010338274Z]: Kernel command line: BOOT_IMAGE=/A/vmlinuz talos.platform=metal talos.config=none console=ttyS0 console=tty0 init_on_alloc=1 slab_nomerge pti=on consoleblank=0 nvme_core.io_timeout=4294967295 random.trust_cpu=on printk.devkmsg=on ima_template=ima-ng ima_appraise=fix ima_hash=sha512 talos-cp-1: kern: notice: [2022-06-30T12:55:08.010592274Z]: Unknown kernel command line parameters "BOOT_IMAGE=/A/vmlinuz pti=on", will be passed to user space. talos-cp-1: kern: info: [2022-06-30T12:55:08.011023274Z]: mem auto-init: stack:byref_all(zero), heap alloc:on, heap free:off talos-cp-1: kern: info: [2022-06-30T12:55:08.166334274Z]: Memory: 24581100K/25165280K available (30745K kernel code, 3908K rwdata, 8260K rodata, 2252K init, 2612K bss, 583920K reserved, 0K cma-reserved) talos-cp-1: kern: info: [2022-06-30T12:55:08.166716274Z]: SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=24, Nodes=2 talos-cp-1: kern: info: [2022-06-30T12:55:08.166734274Z]: Kernel/User page tables isolation: enabled talos-cp-1: kern: info: [2022-06-30T12:55:08.167260274Z]: rcu: Hierarchical RCU implementation. talos-cp-1: kern: info: [2022-06-30T12:55:08.167261274Z]: rcu: \x09RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=24. talos-cp-1: kern: info: [2022-06-30T12:55:08.167263274Z]: \x09Tracing variant of Tasks RCU enabled. talos-cp-1: kern: info: [2022-06-30T12:55:08.167266274Z]: rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. talos-cp-1: kern: info: [2022-06-30T12:55:08.167267274Z]: rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=24 talos-cp-1: kern: info: [2022-06-30T12:55:08.169104274Z]: NR_IRQS: 33024, nr_irqs: 616, preallocated irqs: 16 talos-cp-1: kern: info: [2022-06-30T12:55:08.176298274Z]: Console: colour VGA+ 80x25 talos-cp-1: kern: info: [2022-06-30T12:55:08.212255274Z]: printk: console [tty0] enabled talos-cp-1: kern: info: [2022-06-30T12:55:08.265354274Z]: printk: console [ttyS0] enabled talos-cp-1: kern: info: [2022-06-30T12:55:08.265915274Z]: ACPI: Core revision 20210730 talos-cp-1: kern: info: [2022-06-30T12:55:08.267088274Z]: clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns talos-cp-1: kern: info: [2022-06-30T12:55:08.268208274Z]: APIC: Switch to symmetric I/O mode setup talos-cp-1: kern: info: [2022-06-30T12:55:08.269132274Z]: x2apic enabled talos-cp-1: kern: info: [2022-06-30T12:55:08.269744274Z]: Switched APIC routing to physical x2apic. talos-cp-1: kern: info: [2022-06-30T12:55:08.271346274Z]: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 talos-cp-1: kern: info: [2022-06-30T12:55:08.272039274Z]: clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x257a386c905, max_idle_ns: 440795250529 ns talos-cp-1: kern: info: [2022-06-30T12:55:08.273182274Z]: Calibrating delay loop (skipped) preset value.. 5199.99 BogoMIPS (lpj=10399984) talos-cp-1: kern: info: [2022-06-30T12:55:08.274122274Z]: pid_max: default: 32768 minimum: 301 talos-cp-1: kern: info: [2022-06-30T12:55:08.274752274Z]: LSM: Security Framework initializing talos-cp-1: kern: info: [2022-06-30T12:55:08.275299274Z]: Yama: becoming mindful. talos-cp-1: kern: info: [2022-06-30T12:55:08.291926274Z]: Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes, vmalloc) talos-cp-1: kern: info: [2022-06-30T12:55:08.300537274Z]: Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes, vmalloc) talos-cp-1: kern: info: [2022-06-30T12:55:08.301426274Z]: Mount-cache hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) talos-cp-1: kern: info: [2022-06-30T12:55:08.302535274Z]: Mountpoint-cache hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) talos-cp-1: kern: info: [2022-06-30T12:55:08.304056274Z]: Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 talos-cp-1: kern: info: [2022-06-30T12:55:08.304670274Z]: Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 talos-cp-1: kern: info: [2022-06-30T12:55:08.305187274Z]: Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization talos-cp-1: kern: info: [2022-06-30T12:55:08.306139274Z]: Spectre V2 : Mitigation: Retpolines talos-cp-1: kern: info: [2022-06-30T12:55:08.306673274Z]: Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch talos-cp-1: kern: info: [2022-06-30T12:55:08.307602274Z]: Speculative Store Bypass: Vulnerable talos-cp-1: kern: info: [2022-06-30T12:55:08.308142274Z]: MDS: Vulnerable: Clear CPU buffers attempted, no microcode talos-cp-1: kern: info: [2022-06-30T12:55:08.317149274Z]: Freeing SMP alternatives memory: 72K talos-cp-1: kern: info: [2022-06-30T12:55:08.425999274Z]: smpboot: CPU0: Intel Common KVM processor (family: 0xf, model: 0x6, stepping: 0x1) talos-cp-1: kern: info: [2022-06-30T12:55:08.427251274Z]: Performance Events: unsupported Netburst CPU model 6 no PMU driver, software events only. talos-cp-1: kern: info: [2022-06-30T12:55:08.429248274Z]: rcu: Hierarchical SRCU implementation. talos-cp-1: kern: info: [2022-06-30T12:55:08.430469274Z]: smp: Bringing up secondary CPUs ... talos-cp-1: kern: info: [2022-06-30T12:55:08.431197274Z]: x86: Booting SMP configuration: talos-cp-1: kern: info: [2022-06-30T12:55:08.431708274Z]: .... node #0, CPUs: #1 talos-cp-1: kern: info: [2022-06-30T12:55:08.022241274Z]: kvm-clock: cpu 1, msr 1f2197041, secondary cpu clock talos-cp-1: kern: info: [2022-06-30T12:55:08.445194274Z]: kvm-guest: stealtime: cpu 1, msr 333a5c0c0 talos-cp-1: kern: warning: [2022-06-30T12:55:08.446517274Z]: #2 talos-cp-1: kern: info: [2022-06-30T12:55:08.022241274Z]: kvm-clock: cpu 2, msr 1f2197081, secondary cpu clock talos-cp-1: kern: info: [2022-06-30T12:55:08.461028274Z]: kvm-guest: stealtime: cpu 2, msr 333a9c0c0 talos-cp-1: kern: warning: [2022-06-30T12:55:08.461361274Z]: #3 talos-cp-1: kern: info: [2022-06-30T12:55:08.022241274Z]: kvm-clock: cpu 3, msr 1f21970c1, secondary cpu clock talos-cp-1: kern: info: [2022-06-30T12:55:08.475808274Z]: kvm-guest: stealtime: cpu 3, msr 333adc0c0 talos-cp-1: kern: warning: [2022-06-30T12:55:08.477400274Z]: #4 talos-cp-1: kern: info: [2022-06-30T12:55:08.022241274Z]: kvm-clock: cpu 4, msr 1f2197101, secondary cpu clock talos-cp-1: kern: info: [2022-06-30T12:55:08.489207274Z]: kvm-guest: stealtime: cpu 4, msr 333b1c0c0 talos-cp-1: kern: warning: [2022-06-30T12:55:08.490515274Z]: #5 talos-cp-1: kern: info: [2022-06-30T12:55:08.022241274Z]: kvm-clock: cpu 5, msr 1f2197141, secondary cpu clock talos-cp-1: kern: info: [2022-06-30T12:55:08.505197274Z]: kvm-guest: stealtime: cpu 5, msr 333b5c0c0 talos-cp-1: kern: warning: [2022-06-30T12:55:08.506555274Z]: #6 talos-cp-1: kern: info: [2022-06-30T12:55:08.022241274Z]: kvm-clock: cpu 6, msr 1f2197181, secondary cpu clock talos-cp-1: kern: info: [2022-06-30T12:55:08.520293274Z]: kvm-guest: stealtime: cpu 6, msr 333b9c0c0 talos-cp-1: kern: warning: [2022-06-30T12:55:08.521373274Z]: #7 talos-cp-1: kern: info: [2022-06-30T12:55:08.022241274Z]: kvm-clock: cpu 7, msr 1f21971c1, secondary cpu clock talos-cp-1: kern: info: [2022-06-30T12:55:08.533190274Z]: kvm-guest: stealtime: cpu 7, msr 333bdc0c0 talos-cp-1: kern: warning: [2022-06-30T12:55:08.534458274Z]: #8 talos-cp-1: kern: info: [2022-06-30T12:55:08.022241274Z]: kvm-clock: cpu 8, msr 1f2197201, secondary cpu clock talos-cp-1: kern: info: [2022-06-30T12:55:08.549189274Z]: kvm-guest: stealtime: cpu 8, msr 333c1c0c0 talos-cp-1: kern: warning: [2022-06-30T12:55:08.550538274Z]: #9 talos-cp-1: kern: info: [2022-06-30T12:55:08.022241274Z]: kvm-clock: cpu 9, msr 1f2197241, secondary cpu clock talos-cp-1: kern: info: [2022-06-30T12:55:08.564756274Z]: kvm-guest: stealtime: cpu 9, msr 333c5c0c0 talos-cp-1: kern: warning: [2022-06-30T12:55:08.565409274Z]: #10 talos-cp-1: kern: info: [2022-06-30T12:55:08.022241274Z]: kvm-clock: cpu 10, msr 1f2197281, secondary cpu clock talos-cp-1: kern: info: [2022-06-30T12:55:08.579590274Z]: kvm-guest: stealtime: cpu 10, msr 333c9c0c0 talos-cp-1: kern: warning: [2022-06-30T12:55:08.581330274Z]: #11 talos-cp-1: kern: info: [2022-06-30T12:55:08.022241274Z]: kvm-clock: cpu 11, msr 1f21972c1, secondary cpu clock talos-cp-1: kern: info: [2022-06-30T12:55:08.593190274Z]: kvm-guest: stealtime: cpu 11, msr 333cdc0c0 talos-cp-1: kern: warning: [2022-06-30T12:55:08.594514274Z]: talos-cp-1: kern: info: [2022-06-30T12:55:08.594514274Z]: .... node #1, CPUs: #12 talos-cp-1: kern: info: [2022-06-30T12:55:08.022241274Z]: kvm-clock: cpu 12, msr 1f2197301, secondary cpu clock talos-cp-1: kern: info: [2022-06-30T12:55:08.022241274Z]: smpboot: CPU 12 Converting physical 0 to logical die 1 talos-cp-1: kern: info: [2022-06-30T12:55:08.609259274Z]: kvm-guest: stealtime: cpu 12, msr 633a1c0c0 talos-cp-1: kern: warning: [2022-06-30T12:55:08.613367274Z]: #13 talos-cp-1: kern: info: [2022-06-30T12:55:08.022241274Z]: kvm-clock: cpu 13, msr 1f2197341, secondary cpu clock talos-cp-1: kern: info: [2022-06-30T12:55:08.625190274Z]: kvm-guest: stealtime: cpu 13, msr 633a5c0c0 talos-cp-1: kern: warning: [2022-06-30T12:55:08.626559274Z]: #14 talos-cp-1: kern: info: [2022-06-30T12:55:08.022241274Z]: kvm-clock: cpu 14, msr 1f2197381, secondary cpu clock talos-cp-1: kern: info: [2022-06-30T12:55:08.641191274Z]: kvm-guest: stealtime: cpu 14, msr 633a9c0c0 talos-cp-1: kern: warning: [2022-06-30T12:55:08.642635274Z]: #15 talos-cp-1: kern: info: [2022-06-30T12:55:08.022241274Z]: kvm-clock: cpu 15, msr 1f21973c1, secondary cpu clock talos-cp-1: kern: info: [2022-06-30T12:55:08.656858274Z]: kvm-guest: stealtime: cpu 15, msr 633adc0c0 talos-cp-1: kern: warning: [2022-06-30T12:55:08.657401274Z]: #16 talos-cp-1: kern: info: [2022-06-30T12:55:08.022241274Z]: kvm-clock: cpu 16, msr 1f2197401, secondary cpu clock talos-cp-1: kern: info: [2022-06-30T12:55:08.671757274Z]: kvm-guest: stealtime: cpu 16, msr 633b1c0c0 talos-cp-1: kern: warning: [2022-06-30T12:55:08.673380274Z]: #17 talos-cp-1: kern: info: [2022-06-30T12:55:08.022241274Z]: kvm-clock: cpu 17, msr 1f2197441, secondary cpu clock talos-cp-1: kern: info: [2022-06-30T12:55:08.685207274Z]: kvm-guest: stealtime: cpu 17, msr 633b5c0c0 talos-cp-1: kern: warning: [2022-06-30T12:55:08.686614274Z]: #18 talos-cp-1: kern: info: [2022-06-30T12:55:08.022241274Z]: kvm-clock: cpu 18, msr 1f2197481, secondary cpu clock talos-cp-1: kern: info: [2022-06-30T12:55:08.701190274Z]: kvm-guest: stealtime: cpu 18, msr 633b9c0c0 talos-cp-1: kern: warning: [2022-06-30T12:55:08.702699274Z]: #19 talos-cp-1: kern: info: [2022-06-30T12:55:08.022241274Z]: kvm-clock: cpu 19, msr 1f21974c1, secondary cpu clock talos-cp-1: kern: info: [2022-06-30T12:55:08.716622274Z]: kvm-guest: stealtime: cpu 19, msr 633bdc0c0 talos-cp-1: kern: warning: [2022-06-30T12:55:08.717388274Z]: #20 talos-cp-1: kern: info: [2022-06-30T12:55:08.022241274Z]: kvm-clock: cpu 20, msr 1f2197501, secondary cpu clock talos-cp-1: kern: info: [2022-06-30T12:55:08.731844274Z]: kvm-guest: stealtime: cpu 20, msr 633c1c0c0 talos-cp-1: kern: warning: [2022-06-30T12:55:08.733376274Z]: #21 talos-cp-1: kern: info: [2022-06-30T12:55:08.022241274Z]: kvm-clock: cpu 21, msr 1f2197541, secondary cpu clock talos-cp-1: kern: info: [2022-06-30T12:55:08.745192274Z]: kvm-guest: stealtime: cpu 21, msr 633c5c0c0 talos-cp-1: kern: warning: [2022-06-30T12:55:08.746624274Z]: #22 talos-cp-1: kern: info: [2022-06-30T12:55:08.022241274Z]: kvm-clock: cpu 22, msr 1f2197581, secondary cpu clock talos-cp-1: kern: info: [2022-06-30T12:55:08.761190274Z]: kvm-guest: stealtime: cpu 22, msr 633c9c0c0 talos-cp-1: kern: warning: [2022-06-30T12:55:08.762699274Z]: #23 talos-cp-1: kern: info: [2022-06-30T12:55:08.022241274Z]: kvm-clock: cpu 23, msr 1f21975c1, secondary cpu clock talos-cp-1: kern: info: [2022-06-30T12:55:08.776553274Z]: kvm-guest: stealtime: cpu 23, msr 633cdc0c0 talos-cp-1: kern: info: [2022-06-30T12:55:08.777357274Z]: smp: Brought up 2 nodes, 24 CPUs talos-cp-1: kern: info: [2022-06-30T12:55:08.777931274Z]: smpboot: Max logical packages: 2 talos-cp-1: kern: info: [2022-06-30T12:55:08.778482274Z]: smpboot: Total of 24 processors activated (124799.80 BogoMIPS) talos-cp-1: kern: info: [2022-06-30T12:55:08.782245274Z]: devtmpfs: initialized talos-cp-1: kern: info: [2022-06-30T12:55:08.782245274Z]: clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns talos-cp-1: kern: info: [2022-06-30T12:55:08.785330274Z]: futex hash table entries: 8192 (order: 7, 524288 bytes, vmalloc) talos-cp-1: kern: info: [2022-06-30T12:55:08.786402274Z]: PM: RTC time: 12:55:08, date: 2022-06-30 talos-cp-1: kern: info: [2022-06-30T12:55:08.787569274Z]: NET: Registered PF_NETLINK/PF_ROUTE protocol family talos-cp-1: kern: info: [2022-06-30T12:55:08.788488274Z]: audit: initializing netlink subsys (disabled) talos-cp-1: kern: notice: [2022-06-30T12:55:08.789275274Z]: audit: type=2000 audit(1656593708.743:1): state=initialized audit_enabled=0 res=1 talos-cp-1: kern: info: [2022-06-30T12:55:08.789323274Z]: thermal_sys: Registered thermal governor 'step_wise' talos-cp-1: kern: info: [2022-06-30T12:55:08.790306274Z]: thermal_sys: Registered thermal governor 'user_space' talos-cp-1: kern: info: [2022-06-30T12:55:08.791163274Z]: cpuidle: using governor menu talos-cp-1: kern: info: [2022-06-30T12:55:08.791163274Z]: ACPI: bus type PCI registered talos-cp-1: kern: info: [2022-06-30T12:55:08.791473274Z]: acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 talos-cp-1: kern: info: [2022-06-30T12:55:08.793384274Z]: dca service started, version 1.12.1 talos-cp-1: kern: info: [2022-06-30T12:55:08.794083274Z]: PCI: Using configuration type 1 for base access talos-cp-1: kern: info: [2022-06-30T12:55:08.801247274Z]: HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages talos-cp-1: kern: info: [2022-06-30T12:55:08.802090274Z]: cryptd: max_cpu_qlen set to 1000 talos-cp-1: kern: info: [2022-06-30T12:55:08.805321274Z]: ACPI: Added _OSI(Module Device) talos-cp-1: kern: info: [2022-06-30T12:55:08.805806274Z]: ACPI: Added _OSI(Processor Device) talos-cp-1: kern: info: [2022-06-30T12:55:08.806380274Z]: ACPI: Added _OSI(3.0 _SCP Extensions) talos-cp-1: kern: info: [2022-06-30T12:55:08.806982274Z]: ACPI: Added _OSI(Processor Aggregator Device) talos-cp-1: kern: info: [2022-06-30T12:55:08.807679274Z]: ACPI: Added _OSI(Linux-Dell-Video) talos-cp-1: kern: info: [2022-06-30T12:55:08.808260274Z]: ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio) talos-cp-1: kern: info: [2022-06-30T12:55:08.808927274Z]: ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics) talos-cp-1: kern: info: [2022-06-30T12:55:08.810335274Z]: ACPI: 2 ACPI AML tables successfully acquired and loaded talos-cp-1: kern: info: [2022-06-30T12:55:08.813406274Z]: ACPI: Interpreter enabled talos-cp-1: kern: info: [2022-06-30T12:55:08.813933274Z]: ACPI: PM: (supports S0 S3 S5) talos-cp-1: kern: info: [2022-06-30T12:55:08.813933274Z]: ACPI: Using IOAPIC for interrupt routing talos-cp-1: kern: info: [2022-06-30T12:55:08.813933274Z]: PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug talos-cp-1: kern: info: [2022-06-30T12:55:08.815122274Z]: ACPI: Enabled 3 GPEs in block 00 to 0F talos-cp-1: kern: info: [2022-06-30T12:55:08.820563274Z]: ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) talos-cp-1: kern: info: [2022-06-30T12:55:08.821208274Z]: acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] SUBSYSTEM=acpi DEVICE=+acpi:PNP0A03:00 talos-cp-1: kern: warning: [2022-06-30T12:55:08.822228274Z]: acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. SUBSYSTEM=acpi DEVICE=+acpi:PNP0A03:00 talos-cp-1: kern: info: [2022-06-30T12:55:08.823679274Z]: acpiphp: Slot [3] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.824294274Z]: acpiphp: Slot [4] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.824858274Z]: acpiphp: Slot [5] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.825195274Z]: acpiphp: Slot [6] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.825825274Z]: acpiphp: Slot [7] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.826372274Z]: acpiphp: Slot [8] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.826920274Z]: acpiphp: Slot [9] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.827469274Z]: acpiphp: Slot [10] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.828083274Z]: acpiphp: Slot [11] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.828654274Z]: acpiphp: Slot [12] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.829193274Z]: acpiphp: Slot [13] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.829821274Z]: acpiphp: Slot [14] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.830370274Z]: acpiphp: Slot [15] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.830923274Z]: acpiphp: Slot [16] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.831514274Z]: acpiphp: Slot [17] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.832103274Z]: acpiphp: Slot [18] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.832693274Z]: acpiphp: Slot [19] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.833211274Z]: acpiphp: Slot [20] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.833809274Z]: acpiphp: Slot [21] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.834356274Z]: acpiphp: Slot [22] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.834904274Z]: acpiphp: Slot [23] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.835537274Z]: acpiphp: Slot [24] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.836080274Z]: acpiphp: Slot [25] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.836667274Z]: acpiphp: Slot [26] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.837200274Z]: acpiphp: Slot [27] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.837779274Z]: acpiphp: Slot [28] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.838333274Z]: acpiphp: Slot [29] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.838887274Z]: PCI host bridge to bus 0000:00 talos-cp-1: kern: warning: [2022-06-30T12:55:08.839531274Z]: pci_bus 0000:00: Unknown NUMA node; performance will be reduced SUBSYSTEM=pci_bus DEVICE=+pci_bus:0000:00 talos-cp-1: kern: info: [2022-06-30T12:55:08.840361274Z]: pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] SUBSYSTEM=pci_bus DEVICE=+pci_bus:0000:00 talos-cp-1: kern: info: [2022-06-30T12:55:08.841177274Z]: pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] SUBSYSTEM=pci_bus DEVICE=+pci_bus:0000:00 talos-cp-1: kern: info: [2022-06-30T12:55:08.841985274Z]: pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] SUBSYSTEM=pci_bus DEVICE=+pci_bus:0000:00 talos-cp-1: kern: info: [2022-06-30T12:55:08.842932274Z]: pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] SUBSYSTEM=pci_bus DEVICE=+pci_bus:0000:00 talos-cp-1: kern: info: [2022-06-30T12:55:08.843959274Z]: pci_bus 0000:00: root bus resource [mem 0x640000000-0x6bfffffff window] SUBSYSTEM=pci_bus DEVICE=+pci_bus:0000:00 talos-cp-1: kern: info: [2022-06-30T12:55:08.845177274Z]: pci_bus 0000:00: root bus resource [bus 00-ff] SUBSYSTEM=pci_bus DEVICE=+pci_bus:0000:00 talos-cp-1: kern: info: [2022-06-30T12:55:08.846004274Z]: pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 SUBSYSTEM=pci DEVICE=+pci:0000:00:00.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.847572274Z]: pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 SUBSYSTEM=pci DEVICE=+pci:0000:00:01.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.849311274Z]: pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 SUBSYSTEM=pci DEVICE=+pci:0000:00:01.1 talos-cp-1: kern: info: [2022-06-30T12:55:08.853178274Z]: pci 0000:00:01.1: reg 0x20: [io 0xe140-0xe14f] SUBSYSTEM=pci DEVICE=+pci:0000:00:01.1 talos-cp-1: kern: info: [2022-06-30T12:55:08.855138274Z]: pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] SUBSYSTEM=pci DEVICE=+pci:0000:00:01.1 talos-cp-1: kern: info: [2022-06-30T12:55:08.856059274Z]: pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] SUBSYSTEM=pci DEVICE=+pci:0000:00:01.1 talos-cp-1: kern: info: [2022-06-30T12:55:08.856847274Z]: pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] SUBSYSTEM=pci DEVICE=+pci:0000:00:01.1 talos-cp-1: kern: info: [2022-06-30T12:55:08.857177274Z]: pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] SUBSYSTEM=pci DEVICE=+pci:0000:00:01.1 talos-cp-1: kern: info: [2022-06-30T12:55:08.858168274Z]: pci 0000:00:01.2: [8086:7020] type 00 class 0x0c0300 SUBSYSTEM=pci DEVICE=+pci:0000:00:01.2 talos-cp-1: kern: info: [2022-06-30T12:55:08.862143274Z]: pci 0000:00:01.2: reg 0x20: [io 0xe100-0xe11f] SUBSYSTEM=pci DEVICE=+pci:0000:00:01.2 talos-cp-1: kern: info: [2022-06-30T12:55:08.863921274Z]: pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 SUBSYSTEM=pci DEVICE=+pci:0000:00:01.3 talos-cp-1: kern: info: [2022-06-30T12:55:08.865084274Z]: pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI SUBSYSTEM=pci DEVICE=+pci:0000:00:01.3 talos-cp-1: kern: info: [2022-06-30T12:55:08.865184274Z]: pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB SUBSYSTEM=pci DEVICE=+pci:0000:00:01.3 talos-cp-1: kern: info: [2022-06-30T12:55:08.866328274Z]: pci 0000:00:02.0: [1234:1111] type 00 class 0x030000 SUBSYSTEM=pci DEVICE=+pci:0000:00:02.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.873973274Z]: pci 0000:00:02.0: reg 0x10: [mem 0xfd000000-0xfdffffff pref] SUBSYSTEM=pci DEVICE=+pci:0000:00:02.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.879153274Z]: pci 0000:00:02.0: reg 0x18: [mem 0xfea50000-0xfea50fff] SUBSYSTEM=pci DEVICE=+pci:0000:00:02.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.887433274Z]: pci 0000:00:02.0: reg 0x30: [mem 0xfea40000-0xfea4ffff pref] SUBSYSTEM=pci DEVICE=+pci:0000:00:02.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.888369274Z]: pci 0000:00:02.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff] SUBSYSTEM=pci DEVICE=+pci:0000:00:02.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.889984274Z]: pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 SUBSYSTEM=pci DEVICE=+pci:0000:00:0a.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.892368274Z]: pci 0000:00:0a.0: reg 0x10: [io 0xe000-0xe07f] SUBSYSTEM=pci DEVICE=+pci:0000:00:0a.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.894881274Z]: pci 0000:00:0a.0: reg 0x14: [mem 0xfea51000-0xfea51fff] SUBSYSTEM=pci DEVICE=+pci:0000:00:0a.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.901178274Z]: pci 0000:00:0a.0: reg 0x20: [mem 0xfe400000-0xfe403fff 64bit pref] SUBSYSTEM=pci DEVICE=+pci:0000:00:0a.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.905989274Z]: pci 0000:00:0b.0: [1af4:1001] type 00 class 0x010000 SUBSYSTEM=pci DEVICE=+pci:0000:00:0b.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.908435274Z]: pci 0000:00:0b.0: reg 0x10: [io 0xe080-0xe0ff] SUBSYSTEM=pci DEVICE=+pci:0000:00:0b.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.910101274Z]: pci 0000:00:0b.0: reg 0x14: [mem 0xfea52000-0xfea52fff] SUBSYSTEM=pci DEVICE=+pci:0000:00:0b.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.914067274Z]: pci 0000:00:0b.0: reg 0x20: [mem 0xfe404000-0xfe407fff 64bit pref] SUBSYSTEM=pci DEVICE=+pci:0000:00:0b.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.919249274Z]: pci 0000:00:12.0: [1af4:1000] type 00 class 0x020000 SUBSYSTEM=pci DEVICE=+pci:0000:00:12.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.921178274Z]: pci 0000:00:12.0: reg 0x10: [io 0xe120-0xe13f] SUBSYSTEM=pci DEVICE=+pci:0000:00:12.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.922807274Z]: pci 0000:00:12.0: reg 0x14: [mem 0xfea53000-0xfea53fff] SUBSYSTEM=pci DEVICE=+pci:0000:00:12.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.926590274Z]: pci 0000:00:12.0: reg 0x20: [mem 0xfe408000-0xfe40bfff 64bit pref] SUBSYSTEM=pci DEVICE=+pci:0000:00:12.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.928673274Z]: pci 0000:00:12.0: reg 0x30: [mem 0xfea00000-0xfea3ffff pref] SUBSYSTEM=pci DEVICE=+pci:0000:00:12.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.930125274Z]: pci 0000:00:1e.0: [1b36:0001] type 01 class 0x060400 SUBSYSTEM=pci DEVICE=+pci:0000:00:1e.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.933524274Z]: pci 0000:00:1e.0: reg 0x10: [mem 0xfea54000-0xfea540ff 64bit] SUBSYSTEM=pci DEVICE=+pci:0000:00:1e.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.936258274Z]: pci 0000:00:1f.0: [1b36:0001] type 01 class 0x060400 SUBSYSTEM=pci DEVICE=+pci:0000:00:1f.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.938075274Z]: pci 0000:00:1f.0: reg 0x10: [mem 0xfea55000-0xfea550ff 64bit] SUBSYSTEM=pci DEVICE=+pci:0000:00:1f.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.941317274Z]: pci_bus 0000:01: extended config space not accessible SUBSYSTEM=pci_bus DEVICE=+pci_bus:0000:01 talos-cp-1: kern: info: [2022-06-30T12:55:08.942180274Z]: acpiphp: Slot [0] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.942730274Z]: acpiphp: Slot [1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.943273274Z]: acpiphp: Slot [2] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.943894274Z]: acpiphp: Slot [3-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.944516274Z]: acpiphp: Slot [4-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.945204274Z]: acpiphp: Slot [5-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.945817274Z]: acpiphp: Slot [6-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.946381274Z]: acpiphp: Slot [7-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.946979274Z]: acpiphp: Slot [8-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.947505274Z]: acpiphp: Slot [9-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.948184274Z]: acpiphp: Slot [10-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.948817274Z]: acpiphp: Slot [11-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.953215274Z]: acpiphp: Slot [12-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.953845274Z]: acpiphp: Slot [13-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.954385274Z]: acpiphp: Slot [14-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.955002274Z]: acpiphp: Slot [15-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.955633274Z]: acpiphp: Slot [16-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.956246274Z]: acpiphp: Slot [17-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.956878274Z]: acpiphp: Slot [18-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.957210274Z]: acpiphp: Slot [19-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.957794274Z]: acpiphp: Slot [20-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.958370274Z]: acpiphp: Slot [21-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.958977274Z]: acpiphp: Slot [22-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.959566274Z]: acpiphp: Slot [23-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.960200274Z]: acpiphp: Slot [24-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.961195274Z]: acpiphp: Slot [25-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.961775274Z]: acpiphp: Slot [26-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.962351274Z]: acpiphp: Slot [27-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.962957274Z]: acpiphp: Slot [28-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.963475274Z]: acpiphp: Slot [29-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.964166274Z]: acpiphp: Slot [30] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.964790274Z]: acpiphp: Slot [31] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.965382274Z]: pci 0000:00:1e.0: PCI bridge to [bus 01] SUBSYSTEM=pci DEVICE=+pci:0000:00:1e.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.966082274Z]: pci 0000:00:1e.0: bridge window [io 0xd000-0xdfff] SUBSYSTEM=pci DEVICE=+pci:0000:00:1e.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.966870274Z]: pci 0000:00:1e.0: bridge window [mem 0xfe800000-0xfe9fffff] SUBSYSTEM=pci DEVICE=+pci:0000:00:1e.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.967737274Z]: pci 0000:00:1e.0: bridge window [mem 0xfe200000-0xfe3fffff 64bit pref] SUBSYSTEM=pci DEVICE=+pci:0000:00:1e.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.969513274Z]: pci_bus 0000:02: extended config space not accessible SUBSYSTEM=pci_bus DEVICE=+pci_bus:0000:02 talos-cp-1: kern: info: [2022-06-30T12:55:08.970340274Z]: acpiphp: Slot [0-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.970950274Z]: acpiphp: Slot [1-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.971465274Z]: acpiphp: Slot [2-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.972148274Z]: acpiphp: Slot [3-2] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.972779274Z]: acpiphp: Slot [4-2] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.973195274Z]: acpiphp: Slot [5-2] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.973813274Z]: acpiphp: Slot [6-2] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.974349274Z]: acpiphp: Slot [7-2] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.974944274Z]: acpiphp: Slot [8-2] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.975508274Z]: acpiphp: Slot [9-2] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.976178274Z]: acpiphp: Slot [10-2] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.976805274Z]: acpiphp: Slot [11-2] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.977232274Z]: acpiphp: Slot [12-2] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.977843274Z]: acpiphp: Slot [13-2] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.978432274Z]: acpiphp: Slot [14-2] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.979045274Z]: acpiphp: Slot [15-2] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.979643274Z]: acpiphp: Slot [16-2] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.980297274Z]: acpiphp: Slot [17-2] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.981204274Z]: acpiphp: Slot [18-2] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.981825274Z]: acpiphp: Slot [19-2] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.982388274Z]: acpiphp: Slot [20-2] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.983000274Z]: acpiphp: Slot [21-2] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.983559274Z]: acpiphp: Slot [22-2] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.984237274Z]: acpiphp: Slot [23-2] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.984849274Z]: acpiphp: Slot [24-2] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.985201274Z]: acpiphp: Slot [25-2] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.985848274Z]: acpiphp: Slot [26-2] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.986410274Z]: acpiphp: Slot [27-2] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.986980274Z]: acpiphp: Slot [28-2] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.987626274Z]: acpiphp: Slot [29-2] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.988254274Z]: acpiphp: Slot [30-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.989196274Z]: acpiphp: Slot [31-1] registered talos-cp-1: kern: info: [2022-06-30T12:55:08.990025274Z]: pci 0000:00:1f.0: PCI bridge to [bus 02] SUBSYSTEM=pci DEVICE=+pci:0000:00:1f.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.990669274Z]: pci 0000:00:1f.0: bridge window [io 0xc000-0xcfff] SUBSYSTEM=pci DEVICE=+pci:0000:00:1f.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.991407274Z]: pci 0000:00:1f.0: bridge window [mem 0xfe600000-0xfe7fffff] SUBSYSTEM=pci DEVICE=+pci:0000:00:1f.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.992363274Z]: pci 0000:00:1f.0: bridge window [mem 0xfe000000-0xfe1fffff 64bit pref] SUBSYSTEM=pci DEVICE=+pci:0000:00:1f.0 talos-cp-1: kern: info: [2022-06-30T12:55:08.994512274Z]: ACPI: PCI: Interrupt link LNKA configured for IRQ 10 talos-cp-1: kern: info: [2022-06-30T12:55:08.995347274Z]: ACPI: PCI: Interrupt link LNKB configured for IRQ 10 talos-cp-1: kern: info: [2022-06-30T12:55:08.996284274Z]: ACPI: PCI: Interrupt link LNKC configured for IRQ 11 talos-cp-1: kern: info: [2022-06-30T12:55:08.997276274Z]: ACPI: PCI: Interrupt link LNKD configured for IRQ 11 talos-cp-1: kern: info: [2022-06-30T12:55:08.998121274Z]: ACPI: PCI: Interrupt link LNKS configured for IRQ 9 talos-cp-1: kern: info: [2022-06-30T12:55:09.001316274Z]: iommu: Default domain type: Translated talos-cp-1: kern: info: [2022-06-30T12:55:09.001316274Z]: iommu: DMA domain TLB invalidation policy: lazy mode talos-cp-1: kern: info: [2022-06-30T12:55:09.002038274Z]: pci 0000:00:02.0: vgaarb: setting as boot VGA device SUBSYSTEM=pci DEVICE=+pci:0000:00:02.0 talos-cp-1: kern: info: [2022-06-30T12:55:09.002038274Z]: pci 0000:00:02.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none SUBSYSTEM=pci DEVICE=+pci:0000:00:02.0 talos-cp-1: kern: info: [2022-06-30T12:55:09.003085274Z]: pci 0000:00:02.0: vgaarb: bridge control possible SUBSYSTEM=pci DEVICE=+pci:0000:00:02.0 talos-cp-1: kern: info: [2022-06-30T12:55:09.003874274Z]: vgaarb: loaded talos-cp-1: kern: notice: [2022-06-30T12:55:09.005420274Z]: SCSI subsystem initialized talos-cp-1: kern: debug: [2022-06-30T12:55:09.005970274Z]: libata version 3.00 loaded. talos-cp-1: kern: info: [2022-06-30T12:55:09.005970274Z]: ACPI: bus type USB registered talos-cp-1: kern: info: [2022-06-30T12:55:09.005970274Z]: usbcore: registered new interface driver usbfs talos-cp-1: kern: info: [2022-06-30T12:55:09.006592274Z]: usbcore: registered new interface driver hub talos-cp-1: kern: info: [2022-06-30T12:55:09.007343274Z]: usbcore: registered new device driver usb talos-cp-1: kern: info: [2022-06-30T12:55:09.009203274Z]: pps_core: LinuxPPS API ver. 1 registered talos-cp-1: kern: info: [2022-06-30T12:55:09.009887274Z]: pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti talos-cp-1: kern: info: [2022-06-30T12:55:09.011052274Z]: PTP clock support registered talos-cp-1: kern: info: [2022-06-30T12:55:09.011641274Z]: EDAC MC: Ver: 3.0.0 talos-cp-1: kern: info: [2022-06-30T12:55:09.013556274Z]: NET: Registered PF_ATMPVC protocol family talos-cp-1: kern: info: [2022-06-30T12:55:09.013556274Z]: NET: Registered PF_ATMSVC protocol family talos-cp-1: kern: info: [2022-06-30T12:55:09.013871274Z]: NetLabel: Initializing talos-cp-1: kern: info: [2022-06-30T12:55:09.014356274Z]: NetLabel: domain hash size = 128 talos-cp-1: kern: info: [2022-06-30T12:55:09.014911274Z]: NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO talos-cp-1: kern: info: [2022-06-30T12:55:09.015727274Z]: NetLabel: unlabeled traffic allowed by default talos-cp-1: kern: info: [2022-06-30T12:55:09.017228274Z]: PCI: Using ACPI for IRQ routing talos-cp-1: kern: debug: [2022-06-30T12:55:09.017228274Z]: PCI: pci_cache_line_size set to 64 bytes talos-cp-1: kern: debug: [2022-06-30T12:55:09.017333274Z]: e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] talos-cp-1: kern: debug: [2022-06-30T12:55:09.017337274Z]: e820: reserve RAM buffer [mem 0xbffda000-0xbfffffff] talos-cp-1: kern: info: [2022-06-30T12:55:09.017520274Z]: hpet: 3 channels of 0 reserved for per-cpu timers talos-cp-1: kern: info: [2022-06-30T12:55:09.018236274Z]: hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 talos-cp-1: kern: info: [2022-06-30T12:55:09.018845274Z]: hpet0: 3 comparators, 64-bit 100.000000 MHz counter talos-cp-1: kern: info: [2022-06-30T12:55:09.025176274Z]: clocksource: Switched to clocksource kvm-clock talos-cp-1: kern: notice: [2022-06-30T12:55:09.049318274Z]: VFS: Disk quotas dquot_6.6.0 talos-cp-1: kern: info: [2022-06-30T12:55:09.049916274Z]: VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) talos-cp-1: kern: info: [2022-06-30T12:55:09.050778274Z]: pnp: PnP ACPI init talos-cp-1: kern: debug: [2022-06-30T12:55:09.051264274Z]: pnp 00:02: [dma 2] SUBSYSTEM=pnp DEVICE=+pnp:00:02 talos-cp-1: kern: info: [2022-06-30T12:55:09.051463274Z]: pnp: PnP ACPI: found 4 devices talos-cp-1: kern: info: [2022-06-30T12:55:09.060203274Z]: clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns talos-cp-1: kern: info: [2022-06-30T12:55:09.061322274Z]: NET: Registered PF_INET protocol family talos-cp-1: kern: info: [2022-06-30T12:55:09.063552274Z]: IP idents hash table entries: 262144 (order: 9, 2097152 bytes, vmalloc) talos-cp-1: kern: info: [2022-06-30T12:55:09.066896274Z]: tcp_listen_portaddr_hash hash table entries: 16384 (order: 6, 262144 bytes, vmalloc) talos-cp-1: kern: info: [2022-06-30T12:55:09.068079274Z]: Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, vmalloc) talos-cp-1: kern: info: [2022-06-30T12:55:09.070138274Z]: TCP established hash table entries: 262144 (order: 9, 2097152 bytes, vmalloc) talos-cp-1: kern: info: [2022-06-30T12:55:09.071871274Z]: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes, vmalloc) talos-cp-1: kern: info: [2022-06-30T12:55:09.072796274Z]: TCP: Hash tables configured (established 262144 bind 65536) talos-cp-1: kern: info: [2022-06-30T12:55:09.074004274Z]: UDP hash table entries: 16384 (order: 7, 524288 bytes, vmalloc) talos-cp-1: kern: info: [2022-06-30T12:55:09.075082274Z]: UDP-Lite hash table entries: 16384 (order: 7, 524288 bytes, vmalloc) talos-cp-1: kern: info: [2022-06-30T12:55:09.076096274Z]: NET: Registered PF_UNIX/PF_LOCAL protocol family talos-cp-1: kern: info: [2022-06-30T12:55:09.077078274Z]: RPC: Registered named UNIX socket transport module. talos-cp-1: kern: info: [2022-06-30T12:55:09.077811274Z]: RPC: Registered udp transport module. talos-cp-1: kern: info: [2022-06-30T12:55:09.078409274Z]: RPC: Registered tcp transport module. talos-cp-1: kern: info: [2022-06-30T12:55:09.078957274Z]: RPC: Registered tcp NFSv4.1 backchannel transport module. talos-cp-1: kern: info: [2022-06-30T12:55:09.079952274Z]: pci 0000:00:1e.0: PCI bridge to [bus 01] SUBSYSTEM=pci DEVICE=+pci:0000:00:1e.0 talos-cp-1: kern: info: [2022-06-30T12:55:09.080596274Z]: pci 0000:00:1e.0: bridge window [io 0xd000-0xdfff] SUBSYSTEM=pci DEVICE=+pci:0000:00:1e.0 talos-cp-1: kern: info: [2022-06-30T12:55:09.082319274Z]: pci 0000:00:1e.0: bridge window [mem 0xfe800000-0xfe9fffff] SUBSYSTEM=pci DEVICE=+pci:0000:00:1e.0 talos-cp-1: kern: info: [2022-06-30T12:55:09.083611274Z]: pci 0000:00:1e.0: bridge window [mem 0xfe200000-0xfe3fffff 64bit pref] SUBSYSTEM=pci DEVICE=+pci:0000:00:1e.0 talos-cp-1: kern: info: [2022-06-30T12:55:09.085641274Z]: pci 0000:00:1f.0: PCI bridge to [bus 02] SUBSYSTEM=pci DEVICE=+pci:0000:00:1f.0 talos-cp-1: kern: info: [2022-06-30T12:55:09.086241274Z]: pci 0000:00:1f.0: bridge window [io 0xc000-0xcfff] SUBSYSTEM=pci DEVICE=+pci:0000:00:1f.0 talos-cp-1: kern: info: [2022-06-30T12:55:09.087753274Z]: pci 0000:00:1f.0: bridge window [mem 0xfe600000-0xfe7fffff] SUBSYSTEM=pci DEVICE=+pci:0000:00:1f.0 talos-cp-1: kern: info: [2022-06-30T12:55:09.089103274Z]: pci 0000:00:1f.0: bridge window [mem 0xfe000000-0xfe1fffff 64bit pref] SUBSYSTEM=pci DEVICE=+pci:0000:00:1f.0 talos-cp-1: kern: info: [2022-06-30T12:55:09.091097274Z]: pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] SUBSYSTEM=pci_bus DEVICE=+pci_bus:0000:00 talos-cp-1: kern: info: [2022-06-30T12:55:09.092104274Z]: pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] SUBSYSTEM=pci_bus DEVICE=+pci_bus:0000:00 talos-cp-1: kern: info: [2022-06-30T12:55:09.092819274Z]: pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] SUBSYSTEM=pci_bus DEVICE=+pci_bus:0000:00 talos-cp-1: kern: info: [2022-06-30T12:55:09.093571274Z]: pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] SUBSYSTEM=pci_bus DEVICE=+pci_bus:0000:00 talos-cp-1: kern: info: [2022-06-30T12:55:09.094325274Z]: pci_bus 0000:00: resource 8 [mem 0x640000000-0x6bfffffff window] SUBSYSTEM=pci_bus DEVICE=+pci_bus:0000:00 talos-cp-1: kern: info: [2022-06-30T12:55:09.095084274Z]: pci_bus 0000:01: resource 0 [io 0xd000-0xdfff] SUBSYSTEM=pci_bus DEVICE=+pci_bus:0000:01 talos-cp-1: kern: info: [2022-06-30T12:55:09.095710274Z]: pci_bus 0000:01: resource 1 [mem 0xfe800000-0xfe9fffff] SUBSYSTEM=pci_bus DEVICE=+pci_bus:0000:01 talos-cp-1: kern: info: [2022-06-30T12:55:09.096398274Z]: pci_bus 0000:01: resource 2 [mem 0xfe200000-0xfe3fffff 64bit pref] SUBSYSTEM=pci_bus DEVICE=+pci_bus:0000:01 talos-cp-1: kern: info: [2022-06-30T12:55:09.097257274Z]: pci_bus 0000:02: resource 0 [io 0xc000-0xcfff] SUBSYSTEM=pci_bus DEVICE=+pci_bus:0000:02 talos-cp-1: kern: info: [2022-06-30T12:55:09.097924274Z]: pci_bus 0000:02: resource 1 [mem 0xfe600000-0xfe7fffff] SUBSYSTEM=pci_bus DEVICE=+pci_bus:0000:02 talos-cp-1: kern: info: [2022-06-30T12:55:09.098614274Z]: pci_bus 0000:02: resource 2 [mem 0xfe000000-0xfe1fffff 64bit pref] SUBSYSTEM=pci_bus DEVICE=+pci_bus:0000:02 talos-cp-1: kern: info: [2022-06-30T12:55:09.099547274Z]: pci 0000:00:01.0: PIIX3: Enabling Passive Release SUBSYSTEM=pci DEVICE=+pci:0000:00:01.0 talos-cp-1: kern: info: [2022-06-30T12:55:09.100206274Z]: pci 0000:00:00.0: Limiting direct PCI/PCI transfers SUBSYSTEM=pci DEVICE=+pci:0000:00:00.0 talos-cp-1: kern: info: [2022-06-30T12:55:09.100877274Z]: pci 0000:00:01.0: Activating ISA DMA hang workarounds SUBSYSTEM=pci DEVICE=+pci:0000:00:01.0 talos-cp-1: kern: info: [2022-06-30T12:55:09.113913274Z]: ACPI: \x5c_SB_.LNKD: Enabled at IRQ 11 talos-cp-1: kern: info: [2022-06-30T12:55:09.126779274Z]: pci 0000:00:01.2: quirk_usb_early_handoff+0x0/0x720 took 24548 usecs SUBSYSTEM=pci DEVICE=+pci:0000:00:01.2 talos-cp-1: kern: info: [2022-06-30T12:55:09.127703274Z]: PCI: CLS 0 bytes, default 64 talos-cp-1: kern: info: [2022-06-30T12:55:09.128200274Z]: PCI-DMA: Using software bounce buffering for IO (SWIOTLB) talos-cp-1: kern: info: [2022-06-30T12:55:09.128549274Z]: Unpacking initramfs... talos-cp-1: kern: info: [2022-06-30T12:55:09.129007274Z]: software IO TLB: mapped [mem 0x00000000bbfda000-0x00000000bffda000] (64MB) talos-cp-1: kern: err: [2022-06-30T12:55:09.130668274Z]: kvm: no hardware support talos-cp-1: kern: info: [2022-06-30T12:55:09.131166274Z]: has_svm: not amd or hygon talos-cp-1: kern: err: [2022-06-30T12:55:09.131675274Z]: kvm: no hardware support talos-cp-1: kern: info: [2022-06-30T12:55:09.132156274Z]: clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x257a386c905, max_idle_ns: 440795250529 ns talos-cp-1: kern: notice: [2022-06-30T12:55:09.211630274Z]: Initialise system trusted keyrings talos-cp-1: kern: info: [2022-06-30T12:55:09.212368274Z]: workingset: timestamp_bits=40 max_order=23 bucket_order=0 talos-cp-1: kern: info: [2022-06-30T12:55:09.215329274Z]: squashfs: version 4.0 (2009/01/31) Phillip Lougher talos-cp-1: kern: notice: [2022-06-30T12:55:09.216366274Z]: NFS: Registering the id_resolver key type talos-cp-1: kern: notice: [2022-06-30T12:55:09.217015274Z]: Key type id_resolver registered talos-cp-1: kern: notice: [2022-06-30T12:55:09.217587274Z]: Key type id_legacy registered talos-cp-1: kern: info: [2022-06-30T12:55:09.218138274Z]: nfs4filelayout_init: NFSv4 File Layout Driver Registering... talos-cp-1: kern: info: [2022-06-30T12:55:09.218925274Z]: nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering... talos-cp-1: kern: notice: [2022-06-30T12:55:09.220381274Z]: Key type cifs.spnego registered talos-cp-1: kern: notice: [2022-06-30T12:55:09.220949274Z]: Key type cifs.idmap registered talos-cp-1: kern: info: [2022-06-30T12:55:09.221560274Z]: fuse: init (API version 7.34) talos-cp-1: kern: info: [2022-06-30T12:55:09.222290274Z]: SGI XFS with ACLs, security attributes, quota, no debug enabled talos-cp-1: kern: info: [2022-06-30T12:55:09.223902274Z]: ceph: loaded (mds proto 32) talos-cp-1: kern: info: [2022-06-30T12:55:09.233049274Z]: NET: Registered PF_ALG protocol family talos-cp-1: kern: info: [2022-06-30T12:55:09.233719274Z]: xor: measuring software checksum speed talos-cp-1: kern: info: [2022-06-30T12:55:09.234772274Z]: prefetch64-sse : 24728 MB/sec talos-cp-1: kern: info: [2022-06-30T12:55:09.235853274Z]: generic_sse : 17481 MB/sec talos-cp-1: kern: info: [2022-06-30T12:55:09.236370274Z]: xor: using function: prefetch64-sse (24728 MB/sec) talos-cp-1: kern: notice: [2022-06-30T12:55:09.240166274Z]: Key type asymmetric registered talos-cp-1: kern: notice: [2022-06-30T12:55:09.240719274Z]: Asymmetric key parser 'x509' registered talos-cp-1: kern: info: [2022-06-30T12:55:09.241390274Z]: Block layer SCSI generic (bsg) driver version 0.4 loaded (major 249) talos-cp-1: kern: info: [2022-06-30T12:55:09.242492274Z]: io scheduler mq-deadline registered talos-cp-1: kern: info: [2022-06-30T12:55:09.243082274Z]: io scheduler kyber registered talos-cp-1: kern: info: [2022-06-30T12:55:09.244992274Z]: IPMI message handler: version 39.2 talos-cp-1: kern: info: [2022-06-30T12:55:09.245602274Z]: ipmi device interface talos-cp-1: kern: info: [2022-06-30T12:55:09.246292274Z]: ipmi_si: IPMI System Interface driver talos-cp-1: kern: warning: [2022-06-30T12:55:09.246971274Z]: ipmi_si: Unable to find any System Interface(s) talos-cp-1: kern: info: [2022-06-30T12:55:09.247668274Z]: IPMI poweroff: Copyright (C) 2004 MontaVista Software - IPMI Powerdown via sys_reboot talos-cp-1: kern: info: [2022-06-30T12:55:09.248939274Z]: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 talos-cp-1: kern: info: [2022-06-30T12:55:09.250092274Z]: ACPI: button: Power Button [PWRF] talos-cp-1: kern: info: [2022-06-30T12:55:09.252028274Z]: ioatdma: Intel(R) QuickData Technology Driver 5.00 talos-cp-1: kern: info: [2022-06-30T12:55:09.265236274Z]: ACPI: \x5c_SB_.LNKB: Enabled at IRQ 10 talos-cp-1: kern: info: [2022-06-30T12:55:09.279445274Z]: ACPI: \x5c_SB_.LNKC: Enabled at IRQ 11 talos-cp-1: kern: info: [2022-06-30T12:55:09.294096274Z]: Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled talos-cp-1: kern: info: [2022-06-30T12:55:09.295628274Z]: Non-volatile memory driver v1.3 talos-cp-1: kern: info: [2022-06-30T12:55:09.296222274Z]: Linux agpgart interface v0.103 talos-cp-1: kern: info: [2022-06-30T12:55:09.296999274Z]: [drm] amdgpu kernel modesetting enabled. talos-cp-1: kern: info: [2022-06-30T12:55:09.305582274Z]: loop: module loaded talos-cp-1: kern: notice: [2022-06-30T12:55:09.357490274Z]: virtio_blk virtio0: [vda] 67108864 512-byte logical blocks (34.4 GB/32.0 GiB) SUBSYSTEM=virtio DEVICE=+virtio:virtio0 talos-cp-1: kern: info: [2022-06-30T12:55:09.361612274Z]: vda: vda1 vda2 vda3 vda4 vda5 vda6 talos-cp-1: kern: notice: [2022-06-30T12:55:09.427624274Z]: virtio_blk virtio1: [vdb] 536870912 512-byte logical blocks (275 GB/256 GiB) SUBSYSTEM=virtio DEVICE=+virtio:virtio1 talos-cp-1: kern: info: [2022-06-30T12:55:09.431872274Z]: rbd: loaded (major 252) talos-cp-1: kern: info: [2022-06-30T12:55:09.432434274Z]: Guest personality initialized and is inactive talos-cp-1: kern: info: [2022-06-30T12:55:09.433245274Z]: VMCI host device registered (name=vmci, major=10, minor=126) talos-cp-1: kern: info: [2022-06-30T12:55:09.434063274Z]: Initialized host personality talos-cp-1: kern: info: [2022-06-30T12:55:09.434632274Z]: Loading iSCSI transport class v2.0-870. talos-cp-1: kern: notice: [2022-06-30T12:55:09.435747274Z]: iscsi: registered transport (tcp) talos-cp-1: kern: info: [2022-06-30T12:55:09.436329274Z]: Adaptec aacraid driver 1.2.1[50983]-custom talos-cp-1: kern: info: [2022-06-30T12:55:09.436998274Z]: isci: Intel(R) C600 SAS Controller Driver - version 1.2.0 talos-cp-1: kern: info: [2022-06-30T12:55:09.437853274Z]: Microchip SmartPQI Driver (v2.1.10-020) talos-cp-1: kern: info: [2022-06-30T12:55:09.438525274Z]: megasas: 07.717.02.00-rc1 talos-cp-1: kern: info: [2022-06-30T12:55:09.439103274Z]: mpt3sas version 39.100.00.00 loaded talos-cp-1: kern: info: [2022-06-30T12:55:09.439915274Z]: VMware PVSCSI driver - version 1.0.7.0-k talos-cp-1: kern: info: [2022-06-30T12:55:09.440577274Z]: hv_vmbus: registering driver hv_storvsc talos-cp-1: kern: debug: [2022-06-30T12:55:09.442240274Z]: ata_piix 0000:00:01.1: version 2.13 SUBSYSTEM=pci DEVICE=+pci:0000:00:01.1 talos-cp-1: kern: info: [2022-06-30T12:55:09.443344274Z]: scsi host0: ata_piix SUBSYSTEM=scsi DEVICE=+scsi:host0 talos-cp-1: kern: info: [2022-06-30T12:55:09.444116274Z]: scsi host1: ata_piix SUBSYSTEM=scsi DEVICE=+scsi:host1 talos-cp-1: kern: info: [2022-06-30T12:55:09.444669274Z]: ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xe140 irq 14 talos-cp-1: kern: info: [2022-06-30T12:55:09.445502274Z]: ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xe148 irq 15 talos-cp-1: kern: info: [2022-06-30T12:55:09.447866274Z]: wireguard: WireGuard 1.0.0 loaded. See www.wireguard.com for information. talos-cp-1: kern: info: [2022-06-30T12:55:09.448862274Z]: wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld . All Rights Reserved. talos-cp-1: kern: info: [2022-06-30T12:55:09.450739274Z]: tun: Universal TUN/TAP device driver, 1.6 talos-cp-1: kern: info: [2022-06-30T12:55:09.454124274Z]: e100: Intel(R) PRO/100 Network Driver talos-cp-1: kern: info: [2022-06-30T12:55:09.454747274Z]: e100: Copyright(c) 1999-2006 Intel Corporation talos-cp-1: kern: info: [2022-06-30T12:55:09.455453274Z]: e1000: Intel(R) PRO/1000 Network Driver talos-cp-1: kern: info: [2022-06-30T12:55:09.456079274Z]: e1000: Copyright (c) 1999-2006 Intel Corporation. talos-cp-1: kern: info: [2022-06-30T12:55:09.456797274Z]: e1000e: Intel(R) PRO/1000 Network Driver talos-cp-1: kern: info: [2022-06-30T12:55:09.457445274Z]: e1000e: Copyright(c) 1999 - 2015 Intel Corporation. talos-cp-1: kern: info: [2022-06-30T12:55:09.458268274Z]: igb: Intel(R) Gigabit Ethernet Network Driver talos-cp-1: kern: info: [2022-06-30T12:55:09.458968274Z]: igb: Copyright (c) 2007-2014 Intel Corporation. talos-cp-1: kern: info: [2022-06-30T12:55:09.459655274Z]: Intel(R) 2.5G Ethernet Linux Driver talos-cp-1: kern: info: [2022-06-30T12:55:09.460277274Z]: Copyright(c) 2018 Intel Corporation. talos-cp-1: kern: info: [2022-06-30T12:55:09.460865274Z]: igbvf: Intel(R) Gigabit Virtual Function Network Driver talos-cp-1: kern: info: [2022-06-30T12:55:09.461670274Z]: igbvf: Copyright (c) 2009 - 2012 Intel Corporation. talos-cp-1: kern: info: [2022-06-30T12:55:09.462388274Z]: ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver talos-cp-1: kern: info: [2022-06-30T12:55:09.463155274Z]: ixgbe: Copyright (c) 1999-2016 Intel Corporation. talos-cp-1: kern: info: [2022-06-30T12:55:09.464200274Z]: ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver talos-cp-1: kern: info: [2022-06-30T12:55:09.465126274Z]: ixgbevf: Copyright (c) 2009 - 2018 Intel Corporation. talos-cp-1: kern: info: [2022-06-30T12:55:09.466027274Z]: i40e: Intel(R) Ethernet Connection XL710 Network Driver talos-cp-1: kern: info: [2022-06-30T12:55:09.466766274Z]: i40e: Copyright (c) 2013 - 2019 Intel Corporation. talos-cp-1: kern: info: [2022-06-30T12:55:09.467583274Z]: ixgb: Intel(R) PRO/10GbE Network Driver talos-cp-1: kern: info: [2022-06-30T12:55:09.468230274Z]: ixgb: Copyright (c) 1999-2008 Intel Corporation. talos-cp-1: kern: info: [2022-06-30T12:55:09.468878274Z]: iavf: Intel(R) Ethernet Adaptive Virtual Function Network Driver talos-cp-1: kern: info: [2022-06-30T12:55:09.469782274Z]: Copyright (c) 2013 - 2018 Intel Corporation. talos-cp-1: kern: info: [2022-06-30T12:55:09.470601274Z]: sky2: driver version 1.30 talos-cp-1: kern: info: [2022-06-30T12:55:09.471524274Z]: QLogic 1/10 GbE Converged/Intelligent Ethernet Driver v5.3.66 talos-cp-1: kern: info: [2022-06-30T12:55:09.472368274Z]: QLogic/NetXen Network Driver v4.0.82 talos-cp-1: kern: info: [2022-06-30T12:55:09.472997274Z]: QLogic FastLinQ 4xxxx Core Module qed talos-cp-1: kern: info: [2022-06-30T12:55:09.473630274Z]: qede init: QLogic FastLinQ 4xxxx Ethernet Driver qede talos-cp-1: kern: info: [2022-06-30T12:55:09.474395274Z]: VMware vmxnet3 virtual NIC driver - version 1.6.0.0-k-NAPI talos-cp-1: kern: info: [2022-06-30T12:55:09.475251274Z]: usbcore: registered new interface driver r8152 talos-cp-1: kern: info: [2022-06-30T12:55:09.475985274Z]: hv_vmbus: registering driver hv_netvsc talos-cp-1: kern: info: [2022-06-30T12:55:09.476543274Z]: Fusion MPT base driver 3.04.20 talos-cp-1: kern: info: [2022-06-30T12:55:09.477038274Z]: Copyright (c) 1999-2008 LSI Corporation talos-cp-1: kern: info: [2022-06-30T12:55:09.477686274Z]: Fusion MPT SAS Host driver 3.04.20 talos-cp-1: kern: info: [2022-06-30T12:55:09.478294274Z]: ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver talos-cp-1: kern: info: [2022-06-30T12:55:09.479117274Z]: ehci-pci: EHCI PCI platform driver talos-cp-1: kern: info: [2022-06-30T12:55:09.479723274Z]: usbcore: registered new interface driver cdc_acm talos-cp-1: kern: info: [2022-06-30T12:55:09.480461274Z]: cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters talos-cp-1: kern: info: [2022-06-30T12:55:09.481463274Z]: usbcore: registered new interface driver usb-storage talos-cp-1: kern: info: [2022-06-30T12:55:09.482230274Z]: usbcore: registered new interface driver ch341 talos-cp-1: kern: info: [2022-06-30T12:55:09.482964274Z]: usbserial: USB Serial support registered for ch341-uart talos-cp-1: kern: info: [2022-06-30T12:55:09.483785274Z]: usbcore: registered new interface driver cp210x talos-cp-1: kern: info: [2022-06-30T12:55:09.484523274Z]: usbserial: USB Serial support registered for cp210x talos-cp-1: kern: info: [2022-06-30T12:55:09.485317274Z]: usbcore: registered new interface driver ftdi_sio talos-cp-1: kern: info: [2022-06-30T12:55:09.486070274Z]: usbserial: USB Serial support registered for FTDI USB Serial Device talos-cp-1: kern: info: [2022-06-30T12:55:09.487028274Z]: usbcore: registered new interface driver pl2303 talos-cp-1: kern: info: [2022-06-30T12:55:09.487756274Z]: usbserial: USB Serial support registered for pl2303 talos-cp-1: kern: info: [2022-06-30T12:55:09.488575274Z]: i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 talos-cp-1: kern: info: [2022-06-30T12:55:09.490419274Z]: serio: i8042 KBD port at 0x60,0x64 irq 1 talos-cp-1: kern: info: [2022-06-30T12:55:09.491055274Z]: serio: i8042 AUX port at 0x60,0x64 irq 12 talos-cp-1: kern: info: [2022-06-30T12:55:09.491713274Z]: hv_vmbus: registering driver hyperv_keyboard talos-cp-1: kern: info: [2022-06-30T12:55:09.492510274Z]: mousedev: PS/2 mouse device common for all mice talos-cp-1: kern: info: [2022-06-30T12:55:09.493483274Z]: rtc_cmos 00:03: RTC can wake from S4 SUBSYSTEM=pnp DEVICE=+pnp:00:03 talos-cp-1: kern: info: [2022-06-30T12:55:09.494639274Z]: input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 talos-cp-1: kern: info: [2022-06-30T12:55:09.496256274Z]: rtc_cmos 00:03: registered as rtc0 SUBSYSTEM=pnp DEVICE=+pnp:00:03 talos-cp-1: kern: info: [2022-06-30T12:55:09.496900274Z]: rtc_cmos 00:03: alarms up to one day, y3k, 242 bytes nvram, hpet irqs SUBSYSTEM=pnp DEVICE=+pnp:00:03 talos-cp-1: kern: warning: [2022-06-30T12:55:09.498238274Z]: device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. talos-cp-1: kern: info: [2022-06-30T12:55:09.499914274Z]: device-mapper: ioctl: 4.45.0-ioctl (2021-03-22) initialised: dm-devel@redhat.com talos-cp-1: kern: info: [2022-06-30T12:55:09.501246274Z]: intel_pstate: CPU model not supported talos-cp-1: kern: info: [2022-06-30T12:55:09.502226274Z]: sdhci: Secure Digital Host Controller Interface driver talos-cp-1: kern: info: [2022-06-30T12:55:09.502983274Z]: sdhci: Copyright(c) Pierre Ossman talos-cp-1: kern: info: [2022-06-30T12:55:09.503626274Z]: sdhci-pltfm: SDHCI platform and OF driver helper talos-cp-1: kern: info: [2022-06-30T12:55:09.504680274Z]: hid: raw HID events driver (C) Jiri Kosina talos-cp-1: kern: info: [2022-06-30T12:55:09.505842274Z]: usbcore: registered new interface driver usbhid talos-cp-1: kern: info: [2022-06-30T12:55:09.506553274Z]: usbhid: USB HID core driver talos-cp-1: kern: info: [2022-06-30T12:55:09.507241274Z]: hv_utils: Registering HyperV Utility Driver talos-cp-1: kern: info: [2022-06-30T12:55:09.507918274Z]: hv_vmbus: registering driver hv_utils talos-cp-1: kern: info: [2022-06-30T12:55:09.508556274Z]: NET: Registered PF_LLC protocol family talos-cp-1: kern: info: [2022-06-30T12:55:09.509194274Z]: GACT probability NOT on talos-cp-1: kern: info: [2022-06-30T12:55:09.509717274Z]: Mirror/redirect action on talos-cp-1: kern: info: [2022-06-30T12:55:09.510279274Z]: Simple TC action Loaded talos-cp-1: kern: info: [2022-06-30T12:55:09.510970274Z]: netem: version 1.3 talos-cp-1: kern: info: [2022-06-30T12:55:09.511494274Z]: u32 classifier talos-cp-1: kern: info: [2022-06-30T12:55:09.511871274Z]: input device check on talos-cp-1: kern: info: [2022-06-30T12:55:09.512333274Z]: Actions configured talos-cp-1: kern: info: [2022-06-30T12:55:09.516070274Z]: xt_time: kernel timezone is -0000 talos-cp-1: kern: info: [2022-06-30T12:55:09.516716274Z]: IPVS: Registered protocols (TCP, UDP) talos-cp-1: kern: info: [2022-06-30T12:55:09.517365274Z]: IPVS: Connection hash table configured (size=4096, memory=32Kbytes) talos-cp-1: kern: info: [2022-06-30T12:55:09.518331274Z]: IPVS: ipvs loaded. talos-cp-1: kern: info: [2022-06-30T12:55:09.518742274Z]: IPVS: [rr] scheduler registered. talos-cp-1: kern: info: [2022-06-30T12:55:09.519281274Z]: IPVS: [wrr] scheduler registered. talos-cp-1: kern: info: [2022-06-30T12:55:09.519832274Z]: IPVS: [lc] scheduler registered. talos-cp-1: kern: info: [2022-06-30T12:55:09.520347274Z]: IPVS: [sh] scheduler registered. talos-cp-1: kern: info: [2022-06-30T12:55:09.520899274Z]: ipip: IPv4 and MPLS over IPv4 tunneling driver talos-cp-1: kern: info: [2022-06-30T12:55:09.521806274Z]: gre: GRE over IPv4 demultiplexor driver talos-cp-1: kern: info: [2022-06-30T12:55:09.522517274Z]: Initializing XFRM netlink socket talos-cp-1: kern: info: [2022-06-30T12:55:09.523172274Z]: NET: Registered PF_INET6 protocol family talos-cp-1: kern: info: [2022-06-30T12:55:09.524527274Z]: Segment Routing with IPv6 talos-cp-1: kern: info: [2022-06-30T12:55:09.525012274Z]: In-situ OAM (IOAM) with IPv6 talos-cp-1: kern: info: [2022-06-30T12:55:09.525633274Z]: mip6: Mobile IPv6 talos-cp-1: kern: info: [2022-06-30T12:55:09.526189274Z]: sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver talos-cp-1: kern: info: [2022-06-30T12:55:09.527254274Z]: NET: Registered PF_PACKET protocol family talos-cp-1: kern: notice: [2022-06-30T12:55:09.527942274Z]: Bridge firewalling registered talos-cp-1: kern: info: [2022-06-30T12:55:09.528603274Z]: NET: Registered PF_APPLETALK protocol family talos-cp-1: kern: info: [2022-06-30T12:55:09.529297274Z]: NET: Registered PF_X25 protocol family talos-cp-1: kern: info: [2022-06-30T12:55:09.529961274Z]: X25: Linux Version 0.2 talos-cp-1: kern: info: [2022-06-30T12:55:09.530462274Z]: RPC: Registered rdma transport module. talos-cp-1: kern: info: [2022-06-30T12:55:09.531032274Z]: RPC: Registered rdma backchannel transport module. talos-cp-1: kern: info: [2022-06-30T12:55:09.531828274Z]: l2tp_core: L2TP core driver, V2.0 talos-cp-1: kern: info: [2022-06-30T12:55:09.532397274Z]: NET4: DECnet for Linux: V.2.5.68s (C) 1995-2003 Linux DECnet Project Team talos-cp-1: kern: info: [2022-06-30T12:55:09.533419274Z]: DECnet: Routing cache hash table of 1024 buckets, 16Kbytes talos-cp-1: kern: info: [2022-06-30T12:55:09.534223274Z]: NET: Registered PF_DECnet protocol family talos-cp-1: kern: info: [2022-06-30T12:55:09.534821274Z]: NET: Registered PF_PHONET protocol family talos-cp-1: kern: info: [2022-06-30T12:55:09.535445274Z]: 8021q: 802.1Q VLAN Support v1.8 talos-cp-1: kern: info: [2022-06-30T12:55:09.539425274Z]: DCCP: Activated CCID 2 (TCP-like) talos-cp-1: kern: info: [2022-06-30T12:55:09.540021274Z]: DCCP: Activated CCID 3 (TCP-Friendly Rate Control) talos-cp-1: kern: info: [2022-06-30T12:55:09.540928274Z]: sctp: Hash tables configured (bind 512/512) talos-cp-1: kern: info: [2022-06-30T12:55:09.542124274Z]: NET: Registered PF_RDS protocol family talos-cp-1: kern: info: [2022-06-30T12:55:09.880788274Z]: Freeing initrd memory: 56820K talos-cp-1: kern: info: [2022-06-30T12:55:09.884554274Z]: NET: Registered PF_IEEE802154 protocol family talos-cp-1: kern: notice: [2022-06-30T12:55:09.885276274Z]: Key type dns_resolver registered talos-cp-1: kern: notice: [2022-06-30T12:55:09.885866274Z]: Key type ceph registered talos-cp-1: kern: info: [2022-06-30T12:55:09.886520274Z]: libceph: loaded (mon/osd proto 15/24) talos-cp-1: kern: info: [2022-06-30T12:55:09.887155274Z]: openvswitch: Open vSwitch switching datapath talos-cp-1: kern: info: [2022-06-30T12:55:09.889421274Z]: NET: Registered PF_VSOCK protocol family talos-cp-1: kern: info: [2022-06-30T12:55:09.890099274Z]: mpls_gso: MPLS GSO support talos-cp-1: kern: info: [2022-06-30T12:55:09.892390274Z]: IPI shorthand broadcast: enabled talos-cp-1: kern: info: [2022-06-30T12:55:09.893000274Z]: sched_clock: Marking stable (1873998790, 95871581)->(1986316370, -16445999) talos-cp-1: kern: info: [2022-06-30T12:55:09.894289274Z]: registered taskstats version 1 talos-cp-1: kern: notice: [2022-06-30T12:55:09.894846274Z]: Loading compiled-in X.509 certificates talos-cp-1: kern: notice: [2022-06-30T12:55:09.899767274Z]: Loaded X.509 cert 'Sidero Labs, Inc.: Build time throw-away kernel key: 8874cccd056ec92fc5a4eae6fc85296f015f9d2a' talos-cp-1: kern: info: [2022-06-30T12:55:09.904304274Z]: ima: No TPM chip found, activating TPM-bypass! talos-cp-1: kern: info: [2022-06-30T12:55:09.905005274Z]: ima: Allocated hash algorithm: sha512 talos-cp-1: kern: info: [2022-06-30T12:55:09.905666274Z]: ima: No architecture policies found talos-cp-1: kern: info: [2022-06-30T12:55:09.906411274Z]: PM: Magic number: 10:418:937 talos-cp-1: kern: info: [2022-06-30T12:55:09.906997274Z]: platform PNP0103:00: hash matches SUBSYSTEM=platform DEVICE=+platform:PNP0103:00 talos-cp-1: kern: info: [2022-06-30T12:55:09.907530274Z]: acpi PNP0103:00: hash matches SUBSYSTEM=acpi DEVICE=+acpi:PNP0103:00 talos-cp-1: kern: info: [2022-06-30T12:55:09.908057274Z]: printk: console [netcon0] enabled talos-cp-1: kern: info: [2022-06-30T12:55:09.908584274Z]: netconsole: network logging started talos-cp-1: kern: info: [2022-06-30T12:55:09.909949274Z]: rdma_rxe: loaded talos-cp-1: kern: notice: [2022-06-30T12:55:09.910430274Z]: cfg80211: Loading compiled-in X.509 certificates for regulatory database talos-cp-1: kern: notice: [2022-06-30T12:55:09.911872274Z]: cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' talos-cp-1: kern: warning: [2022-06-30T12:55:09.912700274Z]: platform regulatory.0: Direct firmware load for regulatory.db failed with error -2 SUBSYSTEM=platform DEVICE=+platform:regulatory.0 talos-cp-1: kern: info: [2022-06-30T12:55:09.913793274Z]: cfg80211: failed to load regulatory.db talos-cp-1: kern: info: [2022-06-30T12:55:09.915739274Z]: Freeing unused kernel image (initmem) memory: 2252K talos-cp-1: kern: info: [2022-06-30T12:55:09.953335274Z]: Write protecting the kernel read-only data: 43008k talos-cp-1: kern: info: [2022-06-30T12:55:09.955513274Z]: Freeing unused kernel image (text/rodata gap) memory: 2020K talos-cp-1: kern: info: [2022-06-30T12:55:09.957042274Z]: Freeing unused kernel image (rodata/data gap) memory: 1980K talos-cp-1: kern: info: [2022-06-30T12:55:09.960897274Z]: x86/mm: Checked W+X mappings: passed, no W+X pages found. talos-cp-1: kern: info: [2022-06-30T12:55:09.961699274Z]: x86/mm: Checking user space page tables talos-cp-1: kern: info: [2022-06-30T12:55:09.962630274Z]: x86/mm: Checked W+X mappings: passed, no W+X pages found. talos-cp-1: kern: info: [2022-06-30T12:55:09.963523274Z]: Run /init as init process talos-cp-1: kern: debug: [2022-06-30T12:55:09.964049274Z]: with arguments: talos-cp-1: kern: debug: [2022-06-30T12:55:09.964050274Z]: /init talos-cp-1: kern: debug: [2022-06-30T12:55:09.964051274Z]: with environment: talos-cp-1: kern: debug: [2022-06-30T12:55:09.964051274Z]: HOME=/ talos-cp-1: kern: debug: [2022-06-30T12:55:09.964051274Z]: TERM=linux talos-cp-1: kern: debug: [2022-06-30T12:55:09.964052274Z]: BOOT_IMAGE=/A/vmlinuz talos-cp-1: kern: debug: [2022-06-30T12:55:09.964052274Z]: pti=on talos-cp-1: kern: info: [2022-06-30T12:55:10.114548274Z]: input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3 talos-cp-1: kern: notice: [2022-06-30T12:55:11.901193274Z]: random: crng init done talos-cp-1: user: warning: [2022-06-30T12:55:11.905471274Z]: [talos] [initramfs] booting Talos v1.1.0 talos-cp-1: user: warning: [2022-06-30T12:55:11.906144274Z]: [talos] [initramfs] mounting the rootfs talos-cp-1: kern: info: [2022-06-30T12:55:11.906882274Z]: loop0: detected capacity change from 0 to 88600 talos-cp-1: user: warning: [2022-06-30T12:55:11.946098274Z]: [talos] [initramfs] bind mounting /lib/firmware talos-cp-1: user: warning: [2022-06-30T12:55:11.947801274Z]: [talos] [initramfs] entering the rootfs talos-cp-1: user: warning: [2022-06-30T12:55:11.948414274Z]: [talos] [initramfs] moving mounts to the new rootfs talos-cp-1: user: warning: [2022-06-30T12:55:11.949256274Z]: [talos] [initramfs] changing working directory into /root talos-cp-1: user: warning: [2022-06-30T12:55:11.950096274Z]: [talos] [initramfs] moving /root to / talos-cp-1: user: warning: [2022-06-30T12:55:11.950670274Z]: [talos] [initramfs] changing root directory talos-cp-1: user: warning: [2022-06-30T12:55:11.951281274Z]: [talos] [initramfs] cleaning up initramfs talos-cp-1: user: warning: [2022-06-30T12:55:11.952088274Z]: [talos] [initramfs] executing /sbin/init talos-cp-1: user: warning: [2022-06-30T12:55:14.315065274Z]: [talos] task setupLogger (1/1): done, 546.941\xc2\xb5s talos-cp-1: user: warning: [2022-06-30T12:55:14.315815274Z]: [talos] phase logger (1/7): done, 3.067105ms talos-cp-1: user: warning: [2022-06-30T12:55:14.316486274Z]: [talos] phase systemRequirements (2/7): 7 tasks(s) talos-cp-1: user: warning: [2022-06-30T12:55:14.317282274Z]: [talos] task mountPseudoFilesystems (5/7): starting talos-cp-1: user: warning: [2022-06-30T12:55:14.318048274Z]: [talos] task enforceKSPPRequirements (1/7): starting talos-cp-1: user: warning: [2022-06-30T12:55:14.318889274Z]: [talos] task mountCgroups (4/7): starting talos-cp-1: user: warning: [2022-06-30T12:55:14.319653274Z]: [talos] task setupSystemDirectory (2/7): starting talos-cp-1: user: warning: [2022-06-30T12:55:14.320492274Z]: [talos] task setupSystemDirectory (2/7): done, 3.105301ms talos-cp-1: user: warning: [2022-06-30T12:55:14.321345274Z]: [talos] task mountBPFFS (3/7): starting talos-cp-1: user: warning: [2022-06-30T12:55:14.322009274Z]: [talos] task setRLimit (6/7): starting talos-cp-1: user: warning: [2022-06-30T12:55:14.322632274Z]: [talos] task dropCapabilities (7/7): starting talos-cp-1: user: warning: [2022-06-30T12:55:14.323348274Z]: [talos] task mountCgroups (4/7): done, 2.174373ms talos-cp-1: user: warning: [2022-06-30T12:55:14.324263274Z]: [talos] task setRLimit (6/7): done, 6.763854ms talos-cp-1: user: warning: [2022-06-30T12:55:14.325686274Z]: [talos] task dropCapabilities (7/7): done, 8.435381ms talos-cp-1: user: warning: [2022-06-30T12:55:14.326559274Z]: [talos] task mountBPFFS (3/7): done, 8.361572ms talos-cp-1: user: warning: [2022-06-30T12:55:14.327285274Z]: [talos] task mountPseudoFilesystems (5/7): done, 8.674065ms talos-cp-1: user: warning: [2022-06-30T12:55:14.352064274Z]: [talos] setting time servers {"component": "controller-runtime", "controller": "network.TimeServerSpecController", "addresses": ["pool.ntp.org"]} talos-cp-1: user: warning: [2022-06-30T12:55:14.353752274Z]: [talos] setting time servers {"component": "controller-runtime", "controller": "network.TimeServerSpecController", "addresses": ["pool.ntp.org"]} talos-cp-1: user: warning: [2022-06-30T12:55:14.355559274Z]: [talos] setting resolvers {"component": "controller-runtime", "controller": "network.ResolverSpecController", "resolvers": ["1.1.1.1", "8.8.8.8"]} talos-cp-1: user: warning: [2022-06-30T12:55:14.357353274Z]: [talos] assigned address {"component": "controller-runtime", "controller": "network.AddressSpecController", "address": "127.0.0.1/8", "link": "lo"} talos-cp-1: user: warning: [2022-06-30T12:55:14.359150274Z]: [talos] task enforceKSPPRequirements (1/7): done, 38.488904ms talos-cp-1: kern: info: [2022-06-30T12:55:14.360321274Z]: 8021q: adding VLAN 0 to HW filter on device eth0 talos-cp-1: user: warning: [2022-06-30T12:55:14.360670274Z]: [talos] setting resolvers {"component": "controller-runtime", "controller": "network.ResolverSpecController", "resolvers": ["1.1.1.1", "8.8.8.8"]} talos-cp-1: user: warning: [2022-06-30T12:55:14.363484274Z]: [talos] phase systemRequirements (2/7): done, 44.18102ms talos-cp-1: user: warning: [2022-06-30T12:55:14.364275274Z]: [talos] phase integrity (3/7): 1 tasks(s) talos-cp-1: user: warning: [2022-06-30T12:55:14.364979274Z]: [talos] task writeIMAPolicy (1/1): starting talos-cp-1: kern: notice: [2022-06-30T12:55:14.365735274Z]: audit: type=1807 audit(1656593714.319:2): action=dont_measure fsmagic=0x9fa0 res=1 talos-cp-1: kern: info: [2022-06-30T12:55:14.365842274Z]: ima: policy update completed talos-cp-1: kern: notice: [2022-06-30T12:55:14.366830274Z]: audit: type=1807 audit(1656593714.319:3): action=dont_measure fsmagic=0x62656572 res=1 talos-cp-1: kern: notice: [2022-06-30T12:55:14.368444274Z]: audit: type=1807 audit(1656593714.319:4): action=dont_measure fsmagic=0x64626720 res=1 talos-cp-1: kern: notice: [2022-06-30T12:55:14.369563274Z]: audit: type=1807 audit(1656593714.319:5): action=dont_measure fsmagic=0x1021994 res=1 talos-cp-1: kern: notice: [2022-06-30T12:55:14.370717274Z]: audit: type=1807 audit(1656593714.319:6): action=dont_measure fsmagic=0x1cd1 res=1 talos-cp-1: kern: notice: [2022-06-30T12:55:14.371720274Z]: audit: type=1807 audit(1656593714.319:7): action=dont_measure fsmagic=0x42494e4d res=1 talos-cp-1: kern: notice: [2022-06-30T12:55:14.372810274Z]: audit: type=1807 audit(1656593714.319:8): action=dont_measure fsmagic=0x73636673 res=1 talos-cp-1: kern: notice: [2022-06-30T12:55:14.374015274Z]: audit: type=1807 audit(1656593714.319:9): action=dont_measure fsmagic=0xf97cff8c res=1 talos-cp-1: kern: notice: [2022-06-30T12:55:14.375110274Z]: audit: type=1807 audit(1656593714.319:10): action=dont_measure fsmagic=0x43415d53 res=1 talos-cp-1: kern: notice: [2022-06-30T12:55:14.376190274Z]: audit: type=1807 audit(1656593714.319:11): action=dont_measure fsmagic=0x27e0eb res=1 talos-cp-1: user: warning: [2022-06-30T12:55:14.385343274Z]: [talos] task writeIMAPolicy (1/1): done, 20.359476ms talos-cp-1: user: warning: [2022-06-30T12:55:14.386244274Z]: [talos] phase integrity (3/7): done, 21.95224ms talos-cp-1: user: warning: [2022-06-30T12:55:14.386961274Z]: [talos] phase etc (4/7): 2 tasks(s) talos-cp-1: user: warning: [2022-06-30T12:55:14.387542274Z]: [talos] task createOSReleaseFile (2/2): starting talos-cp-1: user: warning: [2022-06-30T12:55:14.388407274Z]: [talos] task CreateSystemCgroups (1/2): starting talos-cp-1: user: warning: [2022-06-30T12:55:14.389273274Z]: [talos] task createOSReleaseFile (2/2): done, 997.475\xc2\xb5s talos-cp-1: user: warning: [2022-06-30T12:55:14.392241274Z]: [talos] task CreateSystemCgroups (1/2): done, 4.647849ms talos-cp-1: user: warning: [2022-06-30T12:55:14.393052274Z]: [talos] phase etc (4/7): done, 6.09085ms talos-cp-1: user: warning: [2022-06-30T12:55:14.393744274Z]: [talos] phase mountSystem (5/7): 1 tasks(s) talos-cp-1: user: warning: [2022-06-30T12:55:14.394433274Z]: [talos] task mountStatePartition (1/1): starting talos-cp-1: user: warning: [2022-06-30T12:55:14.402006274Z]: [talos] task mountStatePartition (1/1): mount skipped talos-cp-1: user: warning: [2022-06-30T12:55:14.402812274Z]: [talos] task mountStatePartition (1/1): done, 8.38385ms talos-cp-1: user: warning: [2022-06-30T12:55:14.403614274Z]: [talos] phase mountSystem (5/7): done, 9.853961ms talos-cp-1: user: warning: [2022-06-30T12:55:14.404298274Z]: [talos] phase config (6/7): 1 tasks(s) talos-cp-1: user: warning: [2022-06-30T12:55:14.404880274Z]: [talos] task loadConfig (1/1): starting talos-cp-1: user: warning: [2022-06-30T12:55:14.405558274Z]: [talos] task loadConfig (1/1): downloading config talos-cp-1: user: warning: [2022-06-30T12:55:14.406269274Z]: [talos] task loadConfig (1/1): machine configuration not found; starting maintenance service talos-cp-1: user: warning: [2022-06-30T12:55:14.407331274Z]: [talos] task loadConfig (1/1): waiting for network address and hostname to be ready talos-cp-1: user: warning: [2022-06-30T12:55:14.417497274Z]: [talos] setting resolvers {"component": "controller-runtime", "controller": "network.ResolverSpecController", "resolvers": ["10.0.0.1"]} talos-cp-1: user: warning: [2022-06-30T12:55:14.419109274Z]: [talos] setting hostname {"component": "controller-runtime", "controller": "network.HostnameSpecController", "hostname": "talos-cp-1", "domainname": ""} talos-cp-1: user: warning: [2022-06-30T12:55:14.421843274Z]: [talos] setting hostname {"component": "controller-runtime", "controller": "network.HostnameSpecController", "hostname": "talos-cp-1", "domainname": ""} talos-cp-1: user: warning: [2022-06-30T12:55:14.424302274Z]: [talos] created route {"component": "controller-runtime", "controller": "network.RouteSpecController", "destination": "default", "gateway": "10.0.0.1", "table": "main", "link": "eth0"} talos-cp-1: user: warning: [2022-06-30T12:55:14.427025274Z]: [talos] assigned address {"component": "controller-runtime", "controller": "network.AddressSpecController", "address": "10.0.0.41/23", "link": "eth0"} talos-cp-1: user: warning: [2022-06-30T12:55:14.428924274Z]: [talos] issued certificate with fingerprint MWBKsedU6BLYsmHXBGkz4T25bv4ksTgM2eURb/E8mas= talos-cp-1: user: warning: [2022-06-30T12:55:14.430221274Z]: [talos] next renewal in 11h59m59.807521249s talos-cp-1: user: warning: [2022-06-30T12:55:14.430861274Z]: [talos] task loadConfig (1/1): this machine is reachable at: talos-cp-1: user: warning: [2022-06-30T12:55:14.431606274Z]: [talos] task loadConfig (1/1): 10.0.0.41 talos-cp-1: user: warning: [2022-06-30T12:55:14.432191274Z]: [talos] task loadConfig (1/1): server certificate fingerprint: talos-cp-1: user: warning: [2022-06-30T12:55:14.432938274Z]: [talos] task loadConfig (1/1): MWBKsedU6BLYsmHXBGkz4T25bv4ksTgM2eURb/E8mas= talos-cp-1: user: warning: [2022-06-30T12:55:14.433968274Z]: [talos] task loadConfig (1/1): talos-cp-1: user: warning: [2022-06-30T12:55:14.434481274Z]: [talos] task loadConfig (1/1): upload configuration using talosctl: talos-cp-1: user: warning: [2022-06-30T12:55:14.435328274Z]: [talos] task loadConfig (1/1): talosctl apply-config --insecure --nodes 10.0.0.41 --file talos-cp-1: user: warning: [2022-06-30T12:55:14.436458274Z]: [talos] task loadConfig (1/1): or apply configuration using talosctl interactive installer: talos-cp-1: user: warning: [2022-06-30T12:55:14.437544274Z]: [talos] task loadConfig (1/1): talosctl apply-config --insecure --nodes 10.0.0.41 --mode=interactive talos-cp-1: user: warning: [2022-06-30T12:55:14.438718274Z]: [talos] task loadConfig (1/1): optionally with node fingerprint check: talos-cp-1: user: warning: [2022-06-30T12:55:14.439589274Z]: [talos] task loadConfig (1/1): talosctl apply-config --insecure --nodes 10.0.0.41 --cert-fingerprint 'MWBKsedU6BLYsmHXBGkz4T25bv4ksTgM2eURb/E8mas=' --file talos-cp-1: user: warning: [2022-06-30T12:55:19.358045274Z]: [talos] failed looking up "pool.ntp.org", ignored {"component": "controller-runtime", "controller": "time.SyncController", "error": "lookup pool.ntp.org on [::1]:53: read udp [::1]:50747->[::1]:53: read: connection refused"} talos-cp-1: user: warning: [2022-06-30T12:55:20.790829274Z]: [talos] adjusting time (slew) by 111.23025ms via 151.236.30.71, state TIME_OK, status STA_PLL | STA_NANO {"component": "controller-runtime", "controller": "time.SyncController"} talos-cp-1: kern: info: [2022-06-30T12:55:29.311787274Z]: nf_conntrack: default automatic helper assignment has been turned off for security reasons and CT-based firewall rule not found. Use the iptables CT target to attach helpers instead. talos-cp-1: user: warning: [2022-06-30T12:55:51.449200274Z]: OK [/machine.MachineService/ApplyConfiguration] 934.957\xc2\xb5s unary Success (:authority=talos-cp-1:50000;content-type=application/grpc;user-agent=grpc-go/1.46.2) talos-cp-1: user: warning: [2022-06-30T12:55:51.452220274Z]: [talos] task loadConfig (1/1): storing config in memory talos-cp-1: user: warning: [2022-06-30T12:55:51.453626274Z]: [talos] task loadConfig (1/1): done, 37.143991009s talos-cp-1: user: warning: [2022-06-30T12:55:51.454405274Z]: [talos] setting time servers {"component": "controller-runtime", "controller": "network.TimeServerSpecController", "addresses": ["0.at.pool.ntp.org", "1.at.pool.ntp.org", "2.at.pool.ntp.org", "3.at.pool.ntp.org"]} talos-cp-1: user: warning: [2022-06-30T12:55:51.456729274Z]: [talos] phase config (6/7): done, 37.145347087s talos-cp-1: user: warning: [2022-06-30T12:55:51.457481274Z]: [talos] phase unmountSystem (7/7): 1 tasks(s) talos-cp-1: user: warning: [2022-06-30T12:55:51.458213274Z]: [talos] setting resolvers {"component": "controller-runtime", "controller": "network.ResolverSpecController", "resolvers": ["10.0.0.1"]} talos-cp-1: user: warning: [2022-06-30T12:55:51.459923274Z]: [talos] setting hostname {"component": "controller-runtime", "controller": "network.HostnameSpecController", "hostname": "talos-10-0-0-41", "domainname": ""} talos-cp-1: user: warning: [2022-06-30T12:55:51.462488274Z]: [talos] task unmountStatePartition (1/1): starting talos-cp-1: user: warning: [2022-06-30T12:55:51.463670274Z]: [talos] removed address 10.0.0.41/23 from "eth0" {"component": "controller-runtime", "controller": "network.AddressSpecController"} talos-cp-1: user: warning: [2022-06-30T12:55:51.468756274Z]: [talos] task unmountStatePartition (1/1): unmount skipped talos-cp-1: user: warning: [2022-06-30T12:55:51.469604274Z]: [talos] task unmountStatePartition (1/1): done, 11.391493ms talos-cp-1: user: warning: [2022-06-30T12:55:51.470435274Z]: [talos] phase unmountSystem (7/7): done, 12.96818ms talos-cp-1: user: warning: [2022-06-30T12:55:51.471192274Z]: [talos] initialize sequence: done: 37.254138615s talos-cp-1: user: warning: [2022-06-30T12:55:51.471955274Z]: [talos] install sequence: 0 phase(s) talos-cp-1: user: warning: [2022-06-30T12:55:51.472626274Z]: [talos] install sequence: done: 669.744\xc2\xb5s talos-cp-1: user: warning: [2022-06-30T12:55:51.473330274Z]: [talos] boot sequence: 19 phase(s) talos-cp-1: user: warning: [2022-06-30T12:55:51.473960274Z]: [talos] service[apid](Waiting): Waiting for service "containerd" to be "up", api certificates talos-cp-1: user: warning: [2022-06-30T12:55:51.475215274Z]: [talos] phase saveStateEncryptionConfig (1/19): 1 tasks(s) talos-cp-1: user: warning: [2022-06-30T12:55:51.476047274Z]: [talos] service[machined](Preparing): Running pre state talos-cp-1: user: warning: [2022-06-30T12:55:51.476845274Z]: [talos] service[machined](Preparing): Creating service runner talos-cp-1: user: warning: [2022-06-30T12:55:51.477704274Z]: [talos] service[machined](Running): Service started as goroutine talos-cp-1: user: warning: [2022-06-30T12:55:51.478596274Z]: [talos] task SaveStateEncryptionConfig (1/1): starting talos-cp-1: user: warning: [2022-06-30T12:55:51.479465274Z]: [talos] task SaveStateEncryptionConfig (1/1): done, 3.414731ms talos-cp-1: user: warning: [2022-06-30T12:55:51.480328274Z]: [talos] phase saveStateEncryptionConfig (1/19): done, 6.357101ms talos-cp-1: user: warning: [2022-06-30T12:55:51.481269274Z]: [talos] phase mountState (2/19): 1 tasks(s) talos-cp-1: user: warning: [2022-06-30T12:55:51.482076274Z]: [talos] task mountStatePartition (1/1): starting talos-cp-1: user: warning: [2022-06-30T12:55:51.489955274Z]: [talos] formatting the partition "/dev/vda5" as "xfs" with label "STATE" talos-cp-1: user: warning: [2022-06-30T12:55:51.492361274Z]: [talos] setting hostname {"component": "controller-runtime", "controller": "network.HostnameSpecController", "hostname": "talos-cp-1", "domainname": ""} talos-cp-1: user: warning: [2022-06-30T12:55:51.494294274Z]: [talos] setting hostname {"component": "controller-runtime", "controller": "network.HostnameSpecController", "hostname": "talos-cp-1", "domainname": ""} talos-cp-1: user: warning: [2022-06-30T12:55:51.496256274Z]: [talos] assigned address {"component": "controller-runtime", "controller": "network.AddressSpecController", "address": "10.0.0.41/23", "link": "eth0"} talos-cp-1: user: warning: [2022-06-30T12:55:51.498048274Z]: [talos] created route {"component": "controller-runtime", "controller": "network.RouteSpecController", "destination": "default", "gateway": "10.0.0.1", "table": "main", "link": "eth0"} talos-cp-1: user: warning: [2022-06-30T12:55:51.839250274Z]: [talos] downloaded manifest "https://raw.githubusercontent.com/alex1989hu/kubelet-serving-cert-approver/main/deploy/standalone-install.yaml" {"component": "controller-runtime", "controller": "k8s.ExtraManifestController"} talos-cp-1: kern: notice: [2022-06-30T12:55:51.886843274Z]: XFS (vda5): Mounting V5 Filesystem talos-cp-1: kern: info: [2022-06-30T12:55:51.891981274Z]: XFS (vda5): Ending clean mount talos-cp-1: user: warning: [2022-06-30T12:55:51.893635274Z]: [talos] task mountStatePartition (1/1): done, 411.964542ms talos-cp-1: user: warning: [2022-06-30T12:55:51.894519274Z]: [talos] node identity established {"component": "controller-runtime", "controller": "cluster.NodeIdentityController", "node_id": "OYxnZhHdMOzsXc5DGTLcRlBk9tObU9QUzszZWJI4RNE"} talos-cp-1: user: warning: [2022-06-30T12:55:51.896463274Z]: [talos] phase mountState (2/19): done, 413.666925ms talos-cp-1: user: warning: [2022-06-30T12:55:51.897221274Z]: [talos] phase validateConfig (3/19): 1 tasks(s) talos-cp-1: user: warning: [2022-06-30T12:55:51.898011274Z]: [talos] task validateConfig (1/1): starting talos-cp-1: user: warning: [2022-06-30T12:55:51.898796274Z]: [talos] task validateConfig (1/1): done, 780.18\xc2\xb5s talos-cp-1: user: warning: [2022-06-30T12:55:51.899560274Z]: [talos] phase validateConfig (3/19): done, 2.326538ms talos-cp-1: user: warning: [2022-06-30T12:55:51.900322274Z]: [talos] phase saveConfig (4/19): 1 tasks(s) talos-cp-1: user: warning: [2022-06-30T12:55:51.900997274Z]: [talos] task saveConfig (1/1): starting talos-cp-1: user: warning: [2022-06-30T12:55:51.901800274Z]: [talos] task saveConfig (1/1): done, 823.526\xc2\xb5s talos-cp-1: user: warning: [2022-06-30T12:55:51.902539274Z]: [talos] phase saveConfig (4/19): done, 2.203935ms talos-cp-1: user: warning: [2022-06-30T12:55:51.903271274Z]: [talos] phase env (5/19): 1 tasks(s) talos-cp-1: user: warning: [2022-06-30T12:55:51.903864274Z]: [talos] task setUserEnvVars (1/1): starting talos-cp-1: user: warning: [2022-06-30T12:55:51.904556274Z]: [talos] task setUserEnvVars (1/1): done, 714.539\xc2\xb5s talos-cp-1: user: warning: [2022-06-30T12:55:51.905342274Z]: [talos] phase env (5/19): done, 2.057246ms talos-cp-1: user: warning: [2022-06-30T12:55:51.906064274Z]: [talos] phase containerd (6/19): 1 tasks(s) talos-cp-1: user: warning: [2022-06-30T12:55:51.906742274Z]: [talos] task startContainerd (1/1): starting talos-cp-1: user: warning: [2022-06-30T12:55:51.907512274Z]: [talos] service[containerd](Preparing): Running pre state talos-cp-1: user: warning: [2022-06-30T12:55:51.908394274Z]: [talos] service[containerd](Preparing): Creating service runner talos-cp-1: user: warning: [2022-06-30T12:55:52.269862274Z]: [talos] downloaded manifest "https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml" {"component": "controller-runtime", "controller": "k8s.ExtraManifestController"} talos-cp-1: user: warning: [2022-06-30T12:55:52.474539274Z]: [talos] service[apid](Waiting): Waiting for service "containerd" to be "up" talos-cp-1: user: warning: [2022-06-30T12:55:53.153168274Z]: [talos] service[containerd](Running): Process Process(["/bin/containerd" "--address" "/system/run/containerd/containerd.sock" "--state" "/system/run/containerd" "--root" "/system/var/lib/containerd"]) started with PID 977 talos-cp-1: user: warning: [2022-06-30T12:55:53.212386274Z]: [talos] service[containerd](Running): Health check successful talos-cp-1: user: warning: [2022-06-30T12:55:53.213301274Z]: [talos] task startContainerd (1/1): done, 1.307783227s talos-cp-1: user: warning: [2022-06-30T12:55:53.214139274Z]: [talos] phase containerd (6/19): done, 1.309290842s talos-cp-1: user: warning: [2022-06-30T12:55:53.214882274Z]: [talos] phase dbus (7/19): 1 tasks(s) talos-cp-1: user: warning: [2022-06-30T12:55:53.215457274Z]: [talos] service[apid](Preparing): Running pre state talos-cp-1: user: warning: [2022-06-30T12:55:53.216238274Z]: [talos] task startDBus (1/1): starting talos-cp-1: user: warning: [2022-06-30T12:55:53.217018274Z]: [talos] service[apid](Preparing): Creating service runner talos-cp-1: user: warning: [2022-06-30T12:55:53.217964274Z]: [talos] task startDBus (1/1): done, 2.052666ms talos-cp-1: user: warning: [2022-06-30T12:55:53.218742274Z]: [talos] phase dbus (7/19): done, 3.863411ms talos-cp-1: user: warning: [2022-06-30T12:55:53.219413274Z]: [talos] phase ephemeral (8/19): 1 tasks(s) talos-cp-1: user: warning: [2022-06-30T12:55:53.220087274Z]: [talos] task mountEphemeralPartition (1/1): starting talos-cp-1: user: warning: [2022-06-30T12:55:53.222441274Z]: [talos] formatting the partition "/dev/vda6" as "xfs" with label "EPHEMERAL" talos-cp-1: kern: notice: [2022-06-30T12:55:53.325834274Z]: XFS (vda6): Mounting V5 Filesystem talos-cp-1: kern: info: [2022-06-30T12:55:53.330383274Z]: XFS (vda6): Ending clean mount talos-cp-1: user: warning: [2022-06-30T12:55:53.349664274Z]: [talos] task mountEphemeralPartition (1/1): done, 129.688429ms talos-cp-1: user: warning: [2022-06-30T12:55:53.350597274Z]: [talos] phase ephemeral (8/19): done, 131.298271ms talos-cp-1: user: warning: [2022-06-30T12:55:53.351366274Z]: [talos] phase var (9/19): 1 tasks(s) talos-cp-1: user: warning: [2022-06-30T12:55:53.351982274Z]: [talos] controller failed {"component": "controller-runtime", "controller": "k8s.KubeletServiceController", "error": "error writing kubelet PKI: open /etc/kubernetes/bootstrap-kubeconfig: read-only file system"} talos-cp-1: user: warning: [2022-06-30T12:55:53.354322274Z]: [talos] task setupVarDirectory (1/1): starting talos-cp-1: user: warning: [2022-06-30T12:55:53.356270274Z]: [talos] task setupVarDirectory (1/1): done, 4.255319ms talos-cp-1: user: warning: [2022-06-30T12:55:53.357055274Z]: [talos] phase var (9/19): done, 5.696719ms talos-cp-1: user: warning: [2022-06-30T12:55:53.357726274Z]: [talos] phase overlay (10/19): 1 tasks(s) talos-cp-1: user: warning: [2022-06-30T12:55:53.358405274Z]: [talos] task mountOverlayFilesystems (1/1): starting talos-cp-1: user: warning: [2022-06-30T12:55:53.360537274Z]: [talos] task mountOverlayFilesystems (1/1): done, 2.145008ms talos-cp-1: user: warning: [2022-06-30T12:55:53.361415274Z]: [talos] phase overlay (10/19): done, 3.692499ms talos-cp-1: user: warning: [2022-06-30T12:55:53.362143274Z]: [talos] phase udevSetup (11/19): 1 tasks(s) talos-cp-1: user: warning: [2022-06-30T12:55:53.362833274Z]: [talos] task writeUdevRules (1/1): starting talos-cp-1: user: warning: [2022-06-30T12:55:53.363631274Z]: [talos] task writeUdevRules (1/1): done, 803.764\xc2\xb5s talos-cp-1: user: warning: [2022-06-30T12:55:53.364426274Z]: [talos] phase udevSetup (11/19): done, 2.28352ms talos-cp-1: user: warning: [2022-06-30T12:55:53.365175274Z]: [talos] phase udevd (12/19): 1 tasks(s) talos-cp-1: user: warning: [2022-06-30T12:55:53.365848274Z]: [talos] task startUdevd (1/1): starting talos-cp-1: user: warning: [2022-06-30T12:55:53.366565274Z]: [talos] service[udevd](Preparing): Running pre state talos-cp-1: user: warning: [2022-06-30T12:55:53.394505274Z]: [talos] service[udevd](Preparing): Creating service runner talos-cp-1: user: warning: [2022-06-30T12:55:53.404568274Z]: [talos] service[udevd](Running): Process Process(["/sbin/udevd" "--resolve-names=never"]) started with PID 1007 talos-cp-1: daemon: info: [2022-06-30T12:55:53.405014274Z]: udevd[1007]: starting version 3.2.11 talos-cp-1: daemon: info: [2022-06-30T12:55:53.413256274Z]: udevd[1007]: starting eudev-3.2.11 talos-cp-1: user: warning: [2022-06-30T12:55:53.596234274Z]: [talos] service[udevd](Running): Health check successful talos-cp-1: user: warning: [2022-06-30T12:55:53.597141274Z]: [talos] task startUdevd (1/1): done, 231.505551ms talos-cp-1: user: warning: [2022-06-30T12:55:53.597910274Z]: [talos] phase udevd (12/19): done, 232.925391ms talos-cp-1: user: warning: [2022-06-30T12:55:53.598610274Z]: [talos] phase userDisks (13/19): 1 tasks(s) talos-cp-1: user: warning: [2022-06-30T12:55:53.599301274Z]: [talos] task mountUserDisks (1/1): starting talos-cp-1: user: warning: [2022-06-30T12:55:53.600019274Z]: [talos] task mountUserDisks (1/1): done, 731.777\xc2\xb5s talos-cp-1: user: warning: [2022-06-30T12:55:53.600781274Z]: [talos] phase userDisks (13/19): done, 2.157541ms talos-cp-1: user: warning: [2022-06-30T12:55:53.601495274Z]: [talos] phase userSetup (14/19): 1 tasks(s) talos-cp-1: user: warning: [2022-06-30T12:55:53.602191274Z]: [talos] task writeUserFiles (1/1): starting talos-cp-1: user: warning: [2022-06-30T12:55:53.602865274Z]: [talos] task writeUserFiles (1/1): done, 696.018\xc2\xb5s talos-cp-1: user: warning: [2022-06-30T12:55:53.603680274Z]: [talos] phase userSetup (14/19): done, 2.170639ms talos-cp-1: user: warning: [2022-06-30T12:55:53.604448274Z]: [talos] phase lvm (15/19): 1 tasks(s) talos-cp-1: user: warning: [2022-06-30T12:55:53.605109274Z]: [talos] task activateLogicalVolumes (1/1): starting talos-cp-1: user: warning: [2022-06-30T12:55:53.736551274Z]: [talos] task activateLogicalVolumes (1/1): done, 131.556469ms talos-cp-1: user: warning: [2022-06-30T12:55:53.737412274Z]: [talos] phase lvm (15/19): done, 133.094857ms talos-cp-1: user: warning: [2022-06-30T12:55:53.739134274Z]: [talos] phase startEverything (16/19): 1 tasks(s) talos-cp-1: user: warning: [2022-06-30T12:55:53.741346274Z]: [talos] task startAllServices (1/1): starting talos-cp-1: user: warning: [2022-06-30T12:55:53.743657274Z]: [talos] task startAllServices (1/1): waiting for 8 services talos-cp-1: user: warning: [2022-06-30T12:55:53.745162274Z]: [talos] service[cri](Waiting): Waiting for network talos-cp-1: user: warning: [2022-06-30T12:55:53.747316274Z]: [talos] service[cri](Preparing): Running pre state talos-cp-1: user: warning: [2022-06-30T12:55:53.748112274Z]: [talos] service[trustd](Waiting): Waiting for service "containerd" to be "up", time sync, network talos-cp-1: user: warning: [2022-06-30T12:55:53.749346274Z]: [talos] service[etcd](Waiting): Waiting for service "cri" to be "up", time sync, network talos-cp-1: user: warning: [2022-06-30T12:55:53.750472274Z]: [talos] service[cri](Preparing): Creating service runner talos-cp-1: user: warning: [2022-06-30T12:55:53.751315274Z]: [talos] service[trustd](Preparing): Running pre state talos-cp-1: user: warning: [2022-06-30T12:55:53.752148274Z]: [talos] service[cri](Running): Process Process(["/bin/containerd" "--address" "/run/containerd/containerd.sock" "--config" "/etc/cri/containerd.toml"]) started with PID 2037 talos-cp-1: user: warning: [2022-06-30T12:55:53.754248274Z]: [talos] task startAllServices (1/1): service "apid" to be "up", service "containerd" to be "up", service "cri" to be "up", service "etcd" to be "up", service "kubelet" to be "up", service "machined" to be "up", service "trustd" to be "up", service "udevd" to be "up" talos-cp-1: user: warning: [2022-06-30T12:55:53.757699274Z]: [talos] service[trustd](Preparing): Creating service runner talos-cp-1: user: warning: [2022-06-30T12:55:53.908428274Z]: [talos] service[kubelet](Waiting): Waiting for service "cri" to be "up", time sync, network talos-cp-1: user: warning: [2022-06-30T12:55:54.232344274Z]: [talos] service[trustd](Running): Started task trustd (PID 2091) for container trustd talos-cp-1: user: warning: [2022-06-30T12:55:54.233689274Z]: [talos] service[apid](Running): Started task apid (PID 2090) for container apid talos-cp-1: user: warning: [2022-06-30T12:55:54.750117274Z]: [talos] service[etcd](Waiting): Waiting for service "cri" to be "up" talos-cp-1: user: warning: [2022-06-30T12:55:54.752425274Z]: [talos] service[cri](Running): Health check successful talos-cp-1: user: warning: [2022-06-30T12:55:54.753257274Z]: [talos] service[etcd](Preparing): Running pre state talos-cp-1: user: warning: [2022-06-30T12:55:54.754112274Z]: [talos] service[kubelet](Preparing): Running pre state talos-cp-1: user: warning: [2022-06-30T12:55:54.761302274Z]: [talos] service[trustd](Running): Health check successful talos-cp-1: user: warning: [2022-06-30T12:55:59.219958274Z]: [talos] service[apid](Running): Health check successful talos-cp-1: user: warning: [2022-06-30T12:56:01.975938274Z]: [talos] bootstrap request received talos-cp-1: user: warning: [2022-06-30T12:56:01.976751274Z]: [talos] bootstrap sequence: 1 phase(s) talos-cp-1: user: warning: [2022-06-30T12:56:01.977390274Z]: [talos] phase etcd (1/1): 1 tasks(s) talos-cp-1: user: warning: [2022-06-30T12:56:01.978032274Z]: [talos] task bootstrapEtcd (1/1): starting talos-cp-1: user: warning: [2022-06-30T12:56:01.978807274Z]: [talos] service[etcd](Failed): Failed to run pre stage: failed to pull image "gcr.io/etcd-development/etcd:v3.5.4": 1 error(s) occurred: talos-cp-1: user: warning: [2022-06-30T12:56:01.980424274Z]: failed to pull image "gcr.io/etcd-development/etcd:v3.5.4": context canceled talos-cp-1: user: warning: [2022-06-30T12:56:01.981490274Z]: [talos] service[etcd](Finished): Bootstrap requested talos-cp-1: user: warning: [2022-06-30T12:56:01.982357274Z]: [talos] service[etcd](Waiting): Waiting for service "cri" to be "up", time sync, network talos-cp-1: user: warning: [2022-06-30T12:56:01.983645274Z]: [talos] service[etcd](Preparing): Running pre state talos-cp-1: user: warning: [2022-06-30T12:56:08.745327274Z]: [talos] task startAllServices (1/1): service "kubelet" to be "up" talos-cp-1: user: warning: [2022-06-30T12:56:23.741299274Z]: [talos] task startAllServices (1/1): service "kubelet" to be "up" talos-cp-1: user: warning: [2022-06-30T12:56:30.962646274Z]: [talos] service[etcd](Preparing): Creating service runner talos-cp-1: user: warning: [2022-06-30T12:56:31.093855274Z]: [talos] service[etcd](Running): Started task etcd (PID 2201) for container etcd talos-cp-1: user: warning: [2022-06-30T12:56:38.738817274Z]: [talos] task startAllServices (1/1): service "kubelet" to be "up" talos-cp-1: user: warning: [2022-06-30T12:56:40.995550274Z]: [talos] service[etcd](Running): Health check failed: error building etcd client: context deadline exceeded talos-cp-1: user: warning: [2022-06-30T12:56:53.737018274Z]: [talos] task startAllServices (1/1): service "kubelet" to be "up" talos-cp-1: user: warning: [2022-06-30T12:56:59.827444274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T12:57:04.931035274Z]: [talos] service[etcd](Running): Started task etcd (PID 2277) for container etcd talos-cp-1: user: warning: [2022-06-30T12:57:08.354341274Z]: [talos] service[kubelet](Preparing): Creating service runner talos-cp-1: user: warning: [2022-06-30T12:57:08.551192274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T12:57:08.692824274Z]: [talos] service[kubelet](Running): Started task kubelet (PID 2351) for container kubelet talos-cp-1: user: warning: [2022-06-30T12:57:08.736081274Z]: [talos] task startAllServices (1/1): service "kubelet" to be "up" talos-cp-1: user: warning: [2022-06-30T12:57:10.591773274Z]: [talos] service[kubelet](Running): Health check successful talos-cp-1: user: warning: [2022-06-30T12:57:10.592782274Z]: [talos] task startAllServices (1/1): done, 1m16.867694457s talos-cp-1: user: warning: [2022-06-30T12:57:10.594502274Z]: [talos] phase startEverything (16/19): done, 1m16.871590459s talos-cp-1: user: warning: [2022-06-30T12:57:10.595372274Z]: [talos] phase labelMaster (17/19): 1 tasks(s) talos-cp-1: user: warning: [2022-06-30T12:57:10.596207274Z]: [talos] task labelNodeAsMaster (1/1): starting talos-cp-1: user: warning: [2022-06-30T12:57:10.651122274Z]: [talos] task labelNodeAsMaster (1/1): done, 54.931463ms talos-cp-1: user: warning: [2022-06-30T12:57:10.652037274Z]: [talos] phase labelMaster (17/19): done, 56.664474ms talos-cp-1: user: warning: [2022-06-30T12:57:10.652862274Z]: [talos] phase uncordon (18/19): 1 tasks(s) talos-cp-1: user: warning: [2022-06-30T12:57:10.653576274Z]: [talos] task uncordonNode (1/1): starting talos-cp-1: user: warning: [2022-06-30T12:57:10.674275274Z]: [talos] retrying error: node not ready talos-cp-1: user: warning: [2022-06-30T12:57:13.642859274Z]: [talos] service[etcd](Running): Started task etcd (PID 2492) for container etcd talos-cp-1: user: warning: [2022-06-30T12:57:14.155736274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T12:57:14.239868274Z]: [talos] controller failed {"component": "controller-runtime", "controller": "k8s.KubeletStaticPodController", "error": "error refreshing pod status: error fetching pod status: Get \x5c"https://127.0.0.1:10250/pods/?timeout=30s\x5c": remote error: tls: internal error"} talos-cp-1: user: warning: [2022-06-30T12:57:19.248286274Z]: [talos] service[etcd](Running): Started task etcd (PID 2563) for container etcd talos-cp-1: user: warning: [2022-06-30T12:57:19.984084274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T12:57:25.069540274Z]: [talos] service[etcd](Running): Started task etcd (PID 2634) for container etcd talos-cp-1: user: warning: [2022-06-30T12:57:25.856081274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T12:57:30.948107274Z]: [talos] service[etcd](Running): Started task etcd (PID 3000) for container etcd talos-cp-1: user: warning: [2022-06-30T12:57:31.268019274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T12:57:36.369517274Z]: [talos] service[etcd](Running): Started task etcd (PID 3154) for container etcd talos-cp-1: user: warning: [2022-06-30T12:57:43.744882274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T12:57:48.843281274Z]: [talos] service[etcd](Running): Started task etcd (PID 3565) for container etcd talos-cp-1: user: warning: [2022-06-30T12:58:06.354927274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T12:58:11.483024274Z]: [talos] service[etcd](Running): Started task etcd (PID 3668) for container etcd talos-cp-1: user: warning: [2022-06-30T12:58:12.835500274Z]: [talos] task uncordonNode (1/1): done, 1m2.183416826s talos-cp-1: user: warning: [2022-06-30T12:58:12.836357274Z]: [talos] phase uncordon (18/19): done, 1m2.184972748s talos-cp-1: user: warning: [2022-06-30T12:58:12.837113274Z]: [talos] phase bootloader (19/19): 1 tasks(s) talos-cp-1: user: warning: [2022-06-30T12:58:12.837838274Z]: [talos] task updateBootloader (1/1): starting talos-cp-1: user: warning: [2022-06-30T12:58:13.066668274Z]: [talos] task updateBootloader (1/1): removing fallback talos-cp-1: user: warning: [2022-06-30T12:58:33.183169274Z]: [talos] task updateBootloader (1/1): done, 20.345874652s talos-cp-1: user: warning: [2022-06-30T12:58:33.184009274Z]: [talos] phase bootloader (19/19): done, 20.347432068s talos-cp-1: user: warning: [2022-06-30T12:58:33.184786274Z]: [talos] boot sequence: done: 2m41.73180355s talos-cp-1: user: warning: [2022-06-30T12:58:33.909951274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T12:58:39.019187274Z]: [talos] service[etcd](Running): Started task etcd (PID 3762) for container etcd talos-cp-1: user: warning: [2022-06-30T12:58:40.260947274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T12:58:45.382466274Z]: [talos] service[etcd](Running): Started task etcd (PID 3911) for container etcd talos-cp-1: user: warning: [2022-06-30T12:59:32.845807274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T12:59:37.935878274Z]: [talos] service[etcd](Running): Started task etcd (PID 4139) for container etcd talos-cp-1: user: warning: [2022-06-30T12:59:44.654339274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T12:59:50.533948274Z]: [talos] service[etcd](Running): Started task etcd (PID 4364) for container etcd talos-cp-1: kern: info: [2022-06-30T12:59:52.926133274Z]: IPv6: ADDRCONF(NETDEV_CHANGE): vethb017c481: link becomes ready talos-cp-1: kern: info: [2022-06-30T12:59:52.927100274Z]: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready talos-cp-1: kern: info: [2022-06-30T12:59:52.928104274Z]: cni0: port 1(vethb017c481) entered blocking state talos-cp-1: kern: info: [2022-06-30T12:59:52.928899274Z]: cni0: port 1(vethb017c481) entered disabled state talos-cp-1: kern: info: [2022-06-30T12:59:52.929711274Z]: device vethb017c481 entered promiscuous mode talos-cp-1: kern: info: [2022-06-30T12:59:52.930451274Z]: cni0: port 1(vethb017c481) entered blocking state talos-cp-1: kern: info: [2022-06-30T12:59:52.931169274Z]: cni0: port 1(vethb017c481) entered forwarding state talos-cp-1: user: warning: [2022-06-30T12:59:58.191293274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:00:03.563271274Z]: [talos] service[etcd](Running): Started task etcd (PID 4710) for container etcd talos-cp-1: user: warning: [2022-06-30T13:00:13.737223274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:00:18.824463274Z]: [talos] service[etcd](Running): Started task etcd (PID 4899) for container etcd talos-cp-1: user: warning: [2022-06-30T13:00:21.108391274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:00:26.240936274Z]: [talos] service[etcd](Running): Started task etcd (PID 5029) for container etcd talos-cp-1: user: warning: [2022-06-30T13:00:26.422735274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:00:31.541687274Z]: [talos] service[etcd](Running): Started task etcd (PID 5429) for container etcd talos-cp-1: user: warning: [2022-06-30T13:00:33.200724274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:00:38.332307274Z]: [talos] service[etcd](Running): Started task etcd (PID 5515) for container etcd talos-cp-1: user: warning: [2022-06-30T13:00:44.680209274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:00:51.005110274Z]: [talos] service[etcd](Running): Started task etcd (PID 5644) for container etcd talos-cp-1: user: warning: [2022-06-30T13:01:00.000975274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:01:05.125772274Z]: [talos] service[etcd](Running): Started task etcd (PID 5797) for container etcd talos-cp-1: user: warning: [2022-06-30T13:01:07.865770274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:01:13.004848274Z]: [talos] service[etcd](Running): Started task etcd (PID 5885) for container etcd talos-cp-1: user: warning: [2022-06-30T13:01:13.385435274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:01:18.511693274Z]: [talos] service[etcd](Running): Started task etcd (PID 5966) for container etcd talos-cp-1: user: warning: [2022-06-30T13:01:20.521069274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:01:25.641687274Z]: [talos] service[etcd](Running): Started task etcd (PID 6043) for container etcd talos-cp-1: user: warning: [2022-06-30T13:01:26.981681274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:01:32.118361274Z]: [talos] service[etcd](Running): Started task etcd (PID 6120) for container etcd talos-cp-1: user: warning: [2022-06-30T13:01:32.211889274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:01:37.309439274Z]: [talos] service[etcd](Running): Started task etcd (PID 6200) for container etcd talos-cp-1: user: warning: [2022-06-30T13:01:37.389627274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:01:42.510170274Z]: [talos] service[etcd](Running): Started task etcd (PID 6275) for container etcd talos-cp-1: user: warning: [2022-06-30T13:01:42.606226274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:01:47.718493274Z]: [talos] service[etcd](Running): Started task etcd (PID 6357) for container etcd talos-cp-1: user: warning: [2022-06-30T13:01:47.808177274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:01:52.930442274Z]: [talos] service[etcd](Running): Started task etcd (PID 6439) for container etcd talos-cp-1: user: warning: [2022-06-30T13:01:53.012827274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:01:58.116869274Z]: [talos] service[etcd](Running): Started task etcd (PID 6535) for container etcd talos-cp-1: user: warning: [2022-06-30T13:01:58.195071274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:02:03.291439274Z]: [talos] service[etcd](Running): Started task etcd (PID 6614) for container etcd talos-cp-1: user: warning: [2022-06-30T13:02:03.366749274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:02:08.454678274Z]: [talos] service[etcd](Running): Started task etcd (PID 6692) for container etcd talos-cp-1: user: warning: [2022-06-30T13:02:08.608847274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:02:13.728031274Z]: [talos] service[etcd](Running): Started task etcd (PID 6772) for container etcd talos-cp-1: user: warning: [2022-06-30T13:02:13.805847274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:02:18.918906274Z]: [talos] service[etcd](Running): Started task etcd (PID 6850) for container etcd talos-cp-1: user: warning: [2022-06-30T13:02:19.003467274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:02:24.086652274Z]: [talos] service[etcd](Running): Started task etcd (PID 6926) for container etcd talos-cp-1: user: warning: [2022-06-30T13:02:24.205295274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:02:29.308828274Z]: [talos] service[etcd](Running): Started task etcd (PID 7015) for container etcd talos-cp-1: user: warning: [2022-06-30T13:02:29.411008274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:02:34.535949274Z]: [talos] service[etcd](Running): Started task etcd (PID 7100) for container etcd talos-cp-1: user: warning: [2022-06-30T13:02:34.609014274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:02:39.720584274Z]: [talos] service[etcd](Running): Started task etcd (PID 7176) for container etcd talos-cp-1: user: warning: [2022-06-30T13:02:39.807013274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:02:44.922711274Z]: [talos] service[etcd](Running): Started task etcd (PID 7254) for container etcd talos-cp-1: user: warning: [2022-06-30T13:02:45.001068274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:02:50.107755274Z]: [talos] service[etcd](Running): Started task etcd (PID 7332) for container etcd talos-cp-1: user: warning: [2022-06-30T13:02:50.205351274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:02:55.312638274Z]: [talos] service[etcd](Running): Started task etcd (PID 7424) for container etcd talos-cp-1: user: warning: [2022-06-30T13:02:55.410400274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:03:00.534452274Z]: [talos] service[etcd](Running): Started task etcd (PID 7499) for container etcd talos-cp-1: user: warning: [2022-06-30T13:03:00.606781274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:03:05.710077274Z]: [talos] service[etcd](Running): Started task etcd (PID 7574) for container etcd talos-cp-1: user: warning: [2022-06-30T13:03:05.807297274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:03:10.925990274Z]: [talos] service[etcd](Running): Started task etcd (PID 7653) for container etcd talos-cp-1: user: warning: [2022-06-30T13:03:11.005936274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 talos-cp-1: user: warning: [2022-06-30T13:03:16.105823274Z]: [talos] service[etcd](Running): Started task etcd (PID 7730) for container etcd talos-cp-1: user: warning: [2022-06-30T13:03:16.204728274Z]: [talos] service[etcd](Waiting): Error running Containerd(etcd), going to restart forever: task "etcd" failed: exit code 2 ```

And here's the etcd (which is now panicking in a restart loop) log output:

``` talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.805Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_CIPHER_SUITES","variable-value":"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.806Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["/usr/local/bin/etcd","--advertise-client-urls=https://10.0.0.41:2379","--auto-tls=false","--cert-file=/system/secrets/etcd/peer.crt","--client-cert-auth=true","--data-dir=/var/lib/etcd","--experimental-initial-corrupt-check=true","--key-file=/system/secrets/etcd/peer.key","--listen-client-urls=https://0.0.0.0:2379","--listen-peer-urls=https://0.0.0.0:2380","--name=talos-cp-1","--peer-auto-tls=false","--peer-cert-file=/system/secrets/etcd/peer.crt","--peer-client-cert-auth=true","--peer-key-file=/system/secrets/etcd/peer.key","--peer-trusted-ca-file=/system/secrets/etcd/ca.crt","--trusted-ca-file=/system/secrets/etcd/ca.crt"]} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.806Z","caller":"etcdmain/etcd.go:94","msg":"detected default host for advertise","host":"10.0.0.41"} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.806Z","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/var/lib/etcd","dir-type":"member"} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.806Z","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["https://0.0.0.0:2380"]} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.806Z","caller":"embed/etcd.go:479","msg":"starting with peer TLS","tls-info":"cert = /system/secrets/etcd/peer.crt, key = /system/secrets/etcd/peer.key, client-cert=, client-key=, trusted-ca = /system/secrets/etcd/ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":["TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"]} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.815Z","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["https://0.0.0.0:2379"]} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.815Z","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.4","git-sha":"08407ff76","go-version":"go1.16.15","go-os":"linux","go-arch":"amd64","max-cpu-set":24,"max-cpu-available":24,"member-initialized":true,"name":"talos-cp-1","data-dir":"/var/lib/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://10.0.0.41:2380"],"listen-peer-urls":["https://0.0.0.0:2380"],"advertise-client-urls":["https://10.0.0.41:2379"],"listen-client-urls":["https://0.0.0.0:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-size-bytes":2147483648,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","auto-compaction-mode":"periodic","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.816Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/etcd/member/snap/db","took":"186.24µs"} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.816Z","caller":"etcdserver/server.go:529","msg":"No snapshot found. Recovering WAL from scratch!"} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.817Z","caller":"etcdserver/raft.go:483","msg":"restarting local member","cluster-id":"ed3ea90b36a924d0","local-member-id":"b3e5838df5f510","commit-index":1} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.817Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"b3e5838df5f510 switched to configuration voters=()"} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.817Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"b3e5838df5f510 became follower at term 1"} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.817Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft b3e5838df5f510 [peers: [], term: 1, commit: 1, applied: 0, lastindex: 1, lastterm: 1]"} talos-cp-1: {"level":"warn","ts":"2022-06-30T13:07:51.820Z","caller":"auth/store.go:1220","msg":"simple token is not cryptographically signed"} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.822Z","caller":"mvcc/kvstore.go:415","msg":"kvstore restored","current-rev":1} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.825Z","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.829Z","caller":"etcdserver/corrupt.go:46","msg":"starting initial corruption check","local-member-id":"b3e5838df5f510","timeout":"7s"} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.829Z","caller":"etcdserver/corrupt.go:116","msg":"initial corruption checking passed; no corruption","local-member-id":"b3e5838df5f510"} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.829Z","caller":"etcdserver/server.go:851","msg":"starting etcd server","local-member-id":"b3e5838df5f510","local-server-version":"3.5.4","cluster-version":"to_be_decided"} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.829Z","caller":"etcdserver/server.go:752","msg":"starting initial election tick advance","election-ticks":10} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.829Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"b3e5838df5f510 switched to configuration voters=(50636374016390416)"} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.830Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"ed3ea90b36a924d0","local-member-id":"b3e5838df5f510","added-peer-id":"b3e5838df5f510","added-peer-peer-urls":["https://10.0.0.41:2380"]} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.831Z","caller":"embed/etcd.go:688","msg":"starting with client TLS","tls-info":"cert = /system/secrets/etcd/peer.crt, key = /system/secrets/etcd/peer.key, client-cert=, client-key=, trusted-ca = /system/secrets/etcd/ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":["TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"]} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.832Z","caller":"embed/etcd.go:581","msg":"serving peer traffic","address":"[::]:2380"} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.832Z","caller":"embed/etcd.go:553","msg":"cmux::serve","address":"[::]:2380"} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.832Z","caller":"embed/etcd.go:277","msg":"now serving peer/client/metrics","local-member-id":"b3e5838df5f510","initial-advertise-peer-urls":["http://10.0.0.41:2380"],"listen-peer-urls":["https://0.0.0.0:2380"],"advertise-client-urls":["https://10.0.0.41:2379"],"listen-client-urls":["https://0.0.0.0:2379"],"listen-metrics-urls":[]} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.832Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"b3e5838df5f510","remote-peer-id":"393e9d1f44127b97"} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.832Z","caller":"rafthttp/transport.go:286","msg":"added new remote peer","local-member-id":"b3e5838df5f510","remote-peer-id":"393e9d1f44127b97","remote-peer-urls":["http://10.0.0.42:2380"]} talos-cp-1: {"level":"warn","ts":"2022-06-30T13:07:51.832Z","caller":"rafthttp/http.go:413","msg":"failed to find remote peer in cluster","local-member-id":"b3e5838df5f510","remote-peer-id-stream-handler":"b3e5838df5f510","remote-peer-id-from":"393e9d1f44127b97","cluster-id":"ed3ea90b36a924d0"} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.832Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"b3e5838df5f510","remote-peer-id":"89a3f3692ae029a8"} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.832Z","caller":"rafthttp/transport.go:286","msg":"added new remote peer","local-member-id":"b3e5838df5f510","remote-peer-id":"89a3f3692ae029a8","remote-peer-urls":["https://10.0.0.43:2380"]} talos-cp-1: {"level":"warn","ts":"2022-06-30T13:07:51.832Z","caller":"rafthttp/http.go:413","msg":"failed to find remote peer in cluster","local-member-id":"b3e5838df5f510","remote-peer-id-stream-handler":"b3e5838df5f510","remote-peer-id-from":"89a3f3692ae029a8","cluster-id":"ed3ea90b36a924d0"} talos-cp-1: {"level":"warn","ts":"2022-06-30T13:07:51.832Z","caller":"rafthttp/http.go:413","msg":"failed to find remote peer in cluster","local-member-id":"b3e5838df5f510","remote-peer-id-stream-handler":"b3e5838df5f510","remote-peer-id-from":"393e9d1f44127b97","cluster-id":"ed3ea90b36a924d0"} talos-cp-1: {"level":"warn","ts":"2022-06-30T13:07:51.834Z","caller":"rafthttp/http.go:413","msg":"failed to find remote peer in cluster","local-member-id":"b3e5838df5f510","remote-peer-id-stream-handler":"b3e5838df5f510","remote-peer-id-from":"89a3f3692ae029a8","cluster-id":"ed3ea90b36a924d0"} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.838Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"b3e5838df5f510 [term: 1] received a MsgHeartbeat message with higher term from 89a3f3692ae029a8 [term: 570]"} talos-cp-1: {"level":"info","ts":"2022-06-30T13:07:51.838Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"b3e5838df5f510 became follower at term 570"} talos-cp-1: {"level":"panic","ts":"2022-06-30T13:07:51.838Z","logger":"raft","caller":"etcdserver/zap_raft.go:101","msg":"tocommit(48045493) is out of range [lastIndex(1)]. Was the raft log corrupted, truncated, or lost?","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver.(*zapRaftLogger).Panicf\n\t/go/src/go.etcd.io/etcd/release/etcd/server/etcdserver/zap_raft.go:101\ngo.etcd.io/etcd/raft/v3.(*raftLog).commitTo\n\t/go/src/go.etcd.io/etcd/release/etcd/raft/log.go:237\ngo.etcd.io/etcd/raft/v3.(*raft).handleHeartbeat\n\t/go/src/go.etcd.io/etcd/release/etcd/raft/raft.go:1508\ngo.etcd.io/etcd/raft/v3.stepFollower\n\t/go/src/go.etcd.io/etcd/release/etcd/raft/raft.go:1434\ngo.etcd.io/etcd/raft/v3.(*raft).Step\n\t/go/src/go.etcd.io/etcd/release/etcd/raft/raft.go:975\ngo.etcd.io/etcd/raft/v3.(*node).run\n\t/go/src/go.etcd.io/etcd/release/etcd/raft/node.go:356"} talos-cp-1: panic: tocommit(48045493) is out of range [lastIndex(1)]. Was the raft log corrupted, truncated, or lost? talos-cp-1: talos-cp-1: goroutine 143 [running]: talos-cp-1: go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0000446c0, 0x0, 0x0, 0x0) talos-cp-1: /go/pkg/mod/go.uber.org/zap@v1.17.0/zapcore/entry.go:234 +0x58d talos-cp-1: go.uber.org/zap.(*SugaredLogger).log(0xc00000e418, 0x4, 0x124ed39, 0x5d, 0xc0002481c0, 0x2, 0x2, 0x0, 0x0, 0x0) talos-cp-1: /go/pkg/mod/go.uber.org/zap@v1.17.0/sugar.go:227 +0x111 talos-cp-1: go.uber.org/zap.(*SugaredLogger).Panicf(...) talos-cp-1: /go/pkg/mod/go.uber.org/zap@v1.17.0/sugar.go:159 talos-cp-1: go.etcd.io/etcd/server/v3/etcdserver.(*zapRaftLogger).Panicf(0xc00011ca40, 0x124ed39, 0x5d, 0xc0002481c0, 0x2, 0x2) talos-cp-1: /go/src/go.etcd.io/etcd/release/etcd/server/etcdserver/zap_raft.go:101 +0x7d talos-cp-1: go.etcd.io/etcd/raft/v3.(*raftLog).commitTo(0xc00043a700, 0x2dd1db5) talos-cp-1: /go/src/go.etcd.io/etcd/release/etcd/raft/log.go:237 +0x135 talos-cp-1: go.etcd.io/etcd/raft/v3.(*raft).handleHeartbeat(0xc00040e2c0, 0x8, 0x615fc079356c731c, 0x89a3f3692ae029a8, 0x23a, 0x0, 0x0, 0x0, 0x0, 0x0, ...) talos-cp-1: /go/src/go.etcd.io/etcd/release/etcd/raft/raft.go:1508 +0x54 talos-cp-1: go.etcd.io/etcd/raft/v3.stepFollower(0xc00040e2c0, 0x8, 0x615fc079356c731c, 0x89a3f3692ae029a8, 0x23a, 0x0, 0x0, 0x0, 0x0, 0x0, ...) talos-cp-1: /go/src/go.etcd.io/etcd/release/etcd/raft/raft.go:1434 +0x478 talos-cp-1: go.etcd.io/etcd/raft/v3.(*raft).Step(0xc00040e2c0, 0x8, 0x615fc079356c731c, 0x89a3f3692ae029a8, 0x23a, 0x0, 0x0, 0x0, 0x0, 0x0, ...) talos-cp-1: /go/src/go.etcd.io/etcd/release/etcd/raft/raft.go:975 +0xa55 talos-cp-1: go.etcd.io/etcd/raft/v3.(*node).run(0xc0000ade00) talos-cp-1: /go/src/go.etcd.io/etcd/release/etcd/raft/node.go:356 +0x798 talos-cp-1: created by go.etcd.io/etcd/raft/v3.RestartNode talos-cp-1: /go/src/go.etcd.io/etcd/release/etcd/raft/node.go:244 +0x330 ```
smira commented 2 years ago

not sure what exactly is wrong, but there's some membership issue with etcd.

you might need to check talosctl etcd members on the healthy cp node

according to the etcd log, it was started with two other peers which seems correct, but after that something went wrong on the etcd side

reitermarkus commented 2 years ago

I manually removed the node from etcd and reset the node again (with --graceful=true, this time). After applying the config again it rejoined correctly this time.

scrayos commented 2 years ago

This issue didn't occur for quite some time for me and I think it was originally caused by a missing port in the cluster.controlPlane.endpoint. After appending the correct port (by default 6443), everything worked fine. So I'll close this.