ETCD setup: Support private IP assignment

[vmatekole]

Feature Request

Support ability to deploy ETCD over private IP

Environment

OSX 10.11.6

What hardware/cloud provider/hypervisor is being used with Tectonic? Vultr

Desired Feature

Enter private IP's in Tectonic during ETCD configuration.

Other Information

In the meantime how can I do this myself using Ignition? How can I leverage the metadata service to inject private IP's in ignition config: Currently the controller template looks like this:

systemd: units:

• name: etcd-member.service enable: true dropins:
  • name: 40-etcd-cluster.conf contents: | [Service] Environment="ETCD_IMAGE_TAG=v3.1.2" {{- if index . "external_etcd"}} ExecStart= ExecStart=/usr/lib/coreos/etcd-wrapper gateway start \ --listen-addr=0.0.0.0:2379 \ --endpoints={{.etcd_endpoints}} {{- else}} Environment="ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379" Environment="ETCD_INITIAL_CLUSTER={{.etcd_initial_cluster}}" Environment="ETCD_ADVERTISE_CLIENT_URLS=http://{{.domain_name}}:2379" Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=http://{{.domain_name}}:2380" Environment="ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380" Environment="ETCD_NAME={{.etcd_name}}" Environment="ETCD_STRICT_RECONFIG_CHECK=true" {{- end}}
• name: docker.service enable: true
• name: locksmithd.service dropins:
  • name: 40-etcd-lock.conf contents: | [Service] Environment="REBOOT_STRATEGY=etcd-lock"
• name: wait-for-dns.service enable: true contents: | [Unit] Description=Wait for DNS entries Wants=systemd-resolved.service Before=kubelet.service [Service] Type=oneshot RemainAfterExit=true ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done' [Install] RequiredBy=kubelet.service
• name: kubelet.service enable: true contents: | [Unit] Description=Kubelet via Hyperkube ACI [Service] Environment="RKT_OPTS=--uuid-file-save=/var/run/kubelet-pod.uuid \ --volume=resolv,kind=host,source=/etc/resolv.conf \ --mount volume=resolv,target=/etc/resolv.conf \ --volume var-lib-cni,kind=host,source=/var/lib/cni \ --mount volume=var-lib-cni,target=/var/lib/cni \ --volume var-log,kind=host,source=/var/log \ --mount volume=var-log,target=/var/log" EnvironmentFile=/etc/kubernetes/kubelet.env ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests ExecStartPre=/bin/mkdir -p /srv/kubernetes/manifests ExecStartPre=/bin/mkdir -p /etc/kubernetes/checkpoint-secrets ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d ExecStartPre=/bin/mkdir -p /var/lib/cni ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt" ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/run/kubelet-pod.uuid ExecStart=/usr/lib/coreos/kubelet-wrapper \ --kubeconfig=/etc/kubernetes/kubeconfig \ --require-kubeconfig \ --client-ca-file=/etc/kubernetes/ca.crt \ --anonymous-auth=false \ --cni-conf-dir=/etc/kubernetes/cni/net.d \ --network-plugin=cni \ --lock-file=/var/run/lock/kubelet.lock \ --exit-on-lock-contention \ --pod-manifest-path=/etc/kubernetes/manifests \ --allow-privileged \ --hostname-override={{.domain_name}} \ --node-labels=master=true \ --minimum-container-ttl-duration=6m0s \ --cluster_dns={{.k8s_dns_service_ip}} \ --cluster_domain=cluster.local ExecStop=-/usr/bin/rkt stop --uuid-file=/var/run/kubelet-pod.uuid Restart=always RestartSec=10 [Install] WantedBy=multi-user.target
• name: bootkube.service contents: | [Unit] Description=Bootstrap a Kubernetes control plane with a temp api-server [Service] Type=simple WorkingDirectory=/opt/bootkube ExecStartPre=/bin/sh -c '/usr/bin/base64 --decode /opt/bootkube/assets.b64 > /opt/bootkube/assets.zip' ExecStartPre=/usr/bin/unzip /opt/bootkube/assets.zip -d /opt/bootkube ExecStartPre=/bin/sh -c 'chmod +x /opt/bootkube/assets/bootkube-start' ExecStart=/opt/bootkube/assets/bootkube-start

storage: {{ if index . "pxe" }} disks:

• device: /dev/sda wipe_table: true partitions:
  • label: ROOT filesystems:
• name: root mount: device: "/dev/sda1" format: "ext4" create: force: true options:
  • "-LROOT" {{end}} files:
• path: /opt/bootkube/.empty filesystem: root mode: 0644
• path: /etc/kubernetes/kubelet.env filesystem: root mode: 0644 contents: inline: | KUBELET_ACI=quay.io/coreos/hyperkube KUBELET_VERSION=v1.5.4_coreos.0
• path: /etc/kubernetes/kubeconfig filesystem: root mode: 0644 contents: inline: | apiVersion: v1 kind: Config clusters:
  • name: local cluster: server: {{.k8s_controller_endpoint}} certificate-authority-data: {{.k8s_certificate_authority}} users:
  • name: kubelet user: client-certificate-data: {{.k8s_client_certificate}} client-key-data: {{.k8s_client_key}} contexts:
  • context: cluster: local user: kubelet
• path: /etc/hostname filesystem: root mode: 0644 contents: inline: {{.domain_name}}
• path: /etc/sysctl.d/max-user-watches.conf filesystem: root contents: inline: | fs.inotify.max_user_watches=16184
• path: /opt/bootkube/assets.b64 filesystem: root mode: 0600 contents: inline: | {{.zippedAssets}}

{{ if index . "ssh_authorized_keys" }} passwd: users:

• name: core ssh_authorized_keys: {{ range $element := .ssh_authorized_keys }}
  • {{$element}} {{end}} {{end}}

[mfburnett]

Hey @vmatekole, thanks for filing this issue. I'll bring this suggestion to the team and keep you updated.

[dghubble]

You'd write a systemd unit which fetches private IPs and other metadata from the cloud provider (in this case Vultr). Write the data to a file and use it as an EnvironmentFile in units which depend on it.

With that said, dynamically fetching IPs for starting etcd is questionable. What's your real desire? Do you expect the machine's IP to change between reboots? Do you expect to be able to auto-scale etcd nodes? etcd will not tolerate automatic dynamic membership (i.e. how will the other etcd nodes know the IPs of its peers changed).

On bare-metal, the recommendation is to assign static IPs to etcd members. Use convenience DNS names or service records resolving to those IPs when configuring etcd. This allows you to migrate to different static IPs (say you're changing network allocations) later without changing membership or configuration, if needed.

On cloud providers (AWS), a stable DNS name is allocated for each etcd VM instance, which itself can change IPs over time.

You're in an interesting place using the bare-metal installer on a cloud provider, but the advise is the same - configure etcd with stable node identifiers that will not require reconfiguration over time.

Use of private IPs (the original title) is fine and supported already.

[vmatekole]

@dghubble Thanks for your response. My goal here is, one security by binding only to a private interface as opposed to all interfaces, and two having a configuration that can be easily reused by changing IP's in one place. Ideally, the installer would provide fields for this purpose. Being able to deploy a new cluster with minimal changes between the last one is beneficial and less error prone.