search

siderolabs / talos

Talos Linux is a modern Linux distribution built for Kubernetes.
https://www.talos.dev
Mozilla Public License 2.0
6.9k stars 555 forks source link

K8s cert getting signed by an IP that is not listed under cluster.apiServer.certSANs #9668

Closed katy-droid closed 2 weeks ago

katy-droid commented 2 weeks ago

Bug Report

Running into an odd problem on a talos baremetal install where an IP that is not listed under my apiServer.certSANs is being signed to the k8s cert.

Description

My setup is as follows: A baremetal system with a public IP and a private IP, my cluster is using the private IP for the kube related activities.

Below is the META config and machine config I am using: META:

addresses:
  - address: [PUBLIC_IP]
    linkName: eno1
    family: inet4
    scope: global
    flags: permanent
    layer: platform
  - address: 192.168.0.2/16
    linkName: eno2
    family: inet4
    scope: global
    flags: permanent
    layer: platform
links:
  - name: eno1
    logical: false
    up: true
    type: ether
    layer: platform
    mtu: 1500
  - name: eno2
    logical: false
    up: true
    type: ether
    layer: platform
    mtu: 9000
routes:
  - family: inet4
    gateway: [REDACTED]
    outLinkName: eno1
    table: main
    scope: global
    type: unicast
    protocol: static
    layer: platform
  - family: inet4
    dst: 192.168.0.0/16
    outLinkName: eno2
    table: main
    scope: global
    type: unicast
    protocol: static
    layer: platform
hostnames:
  - hostname: test-k8s-master-01
    layer: platform
resolvers:
  - dnsServers:
      - 8.8.8.8
      - 8.8.4.4
      - 1.1.1.1
    layer: platform
timeServers: []
operators: []
externalIPs: []

machine config

version: v1alpha1
debug: false
persist: true
machine:
    type: controlplane
    token: [REDACTED]
    ca:
        crt: [REDACTED]
        key: [REDACTED]
    certSANs: {}
    kubelet:
        image: ghcr.io/siderolabs/kubelet:v1.31.1
        defaultRuntimeSeccompProfileEnabled: true
        nodeIP:
            validSubnets:
                - 192.168.0.0/16
        disableManifestsDirectory: true
    network: {}
    install:
        disk: /dev/nvme0n1
        image: [REDACTED]
        wipe: false
    features:
        rbac: true
        stableHostname: true
        apidCheckExtKeyUsage: true
        diskQuotaSupport: true
        kubePrism:
            enabled: true
            port: 7445
        hostDNS:
            enabled: true
            forwardKubeDNSToHost: true
    nodeLabels:
        node.kubernetes.io/exclude-from-external-load-balancers: ""
cluster:
    id: [REDACTED]
    secret: [REDACTED]
    controlPlane:
        endpoint: https://192.168.0.2:6443
    clusterName: test-k8s-cluster
    network:
        cni:
            name: custom
            urls:
                - [REDACTED]
        dnsDomain: cluster.local
        podSubnets:
            - 10.240.0.0/14
        serviceSubnets:
            - 10.96.0.0/12
    token: [REDACTED]
    secretboxEncryptionSecret: [REDACTED]
    ca:
        crt: [REDACTED]
        key: [REDACTED]
    aggregatorCA:
        crt: [REDACTED]
        key: [REDACTED]
    serviceAccount:
        key: [REDACTED]
    apiServer:
        image: registry.k8s.io/kube-apiserver:v1.31.1
        certSANs:
            - 192.168.0.2
        disablePodSecurityPolicy: true
        admissionControl:
            - name: PodSecurity
              configuration:
                apiVersion: pod-security.admission.config.k8s.io/v1alpha1
                defaults:
                    audit: restricted
                    audit-version: latest
                    enforce: baseline
                    enforce-version: latest
                    warn: restricted
                    warn-version: latest
                exemptions:
                    namespaces:
                        - kube-system
                    runtimeClasses: []
                    usernames: []
                kind: PodSecurityConfiguration
        auditPolicy:
            apiVersion: audit.k8s.io/v1
            kind: Policy
            rules:
                - level: Metadata
    controllerManager:
        image: registry.k8s.io/kube-controller-manager:v1.31.1
    proxy:
        disabled: true
        image: registry.k8s.io/kube-proxy:v1.31.1
    scheduler:
        image: registry.k8s.io/kube-scheduler:v1.31.1
    discovery:
        enabled: true
        registries:
            kubernetes:
                disabled: false
            service:
                disabled: true
    etcd:
        ca:
            crt: [REDACTED]
            key: [REDACTED]
        advertisedSubnets:
            - 192.168.0.0/16

Logs

here are the talosctl get links output:

NODE          NAMESPACE   TYPE         ID                VERSION   TYPE       KIND     HW ADDR                                           OPER STATE   LINK STATE
192.168.0.2   network     LinkStatus   bond0             1         ether      bond     [REDACTED]                                 down         false
192.168.0.2   network     LinkStatus   cilium_host       2         ether      veth     [REDACTED]                                 up           true
192.168.0.2   network     LinkStatus   cilium_net        2         ether      veth     [REDACTED]                                 up           true
192.168.0.2   network     LinkStatus   dummy0            1         ether      dummy    [REDACTED]                                 down         false
192.168.0.2   network     LinkStatus   eno1         3         ether               [REDACTED]                                 up           true
192.168.0.2   network     LinkStatus   eno2         6         ether               [REDACTED]                                 up           true
192.168.0.2   network     LinkStatus   lo                2         loopback            00:00:00:00:00:00                                 unknown      true
192.168.0.2   network     LinkStatus   lxc7e32d55b5399   1         ether      netkit   00:00:00:00:00:00                                 up           false
192.168.0.2   network     LinkStatus   lxc8895ced56e45   2         ether      netkit   00:00:00:00:00:00                                 up           false
192.168.0.2   network     LinkStatus   lxc_health        2         ether      netkit   00:00:00:00:00:00                                 up           false
192.168.0.2   network     LinkStatus   lxca7eb07a9551e   1         ether      netkit   00:00:00:00:00:00                                 up           false
192.168.0.2   network     LinkStatus   sit0              1         sit        sit      00:00:00:00                                       down         false
192.168.0.2  network     LinkStatus   teql0             1         void                                                                  down         false
192.168.0.2  network     LinkStatus   tunl0             1         ipip       ipip     00:00:00:00                                       down         false

Here is the output of this command: openssl s_client -connect 192.168.0.2:6443 < /dev/null | openssl x509 -noout -text | grep -C3 -i dns

X509v3 Subject Alternative Name:
                DNS:test-k8s-master-01, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:localhost, IP Address:10.96.0.1, IP Address:[PUBLIC_IP], IP Address:127.0.0.1, IP Address:192.168.0.2

Environment

Please let me know if I'm doing something wrong!

smira commented 2 weeks ago

Talos automatically builds a list of IPs based on all routable machine IPs, .certSANs is only used to inject the IPs/DNS names that Talos is not aware about (e.g. external LB).

As you don't specify which IP you don't recognize, I'd assume it's 10.96.0.1 which is the service cluster IP for kubernetes service (which is used to access api-server internally).

So there seems to be no problem here.

katy-droid commented 2 weeks ago

Hi I was referring to the [PUBLIC_IP] As I don't want this to be included for the api-server I just want the private IP 192.168.x.x and of course the cluster IP.

Is there not a way to exclude the PUBLIC_IP of my system from being included in the list?

smira commented 2 weeks ago

There is no way to do that, what is the reason for excluding? See also https://github.com/siderolabs/talos/discussions/9675

katy-droid commented 2 weeks ago

Ok! I just wanted my internal network to be completely isolated from any access by the public network, for example the kubernetes-api.

smira commented 2 weeks ago

If you want your kube-apiserver not to be available on the public network, certificate SANs is not the way to do it (it won't do that).

A proper way is to use either perimeter firewall, or Talos Ingress Firewall.