Metrics from etcd not properly exposed

[alexwaibel]

Version (k3OS / kernel)

v0.20.7-k3s1r0

Architecture

x86_64

Describe the bug

I've added --etcd-expose-metrics to my k3s_args in my config, but when I try to hit the service I created via the kube-prometheus-stack Helm chart I get "connection refused".

To Reproduce

Add k3s_args section to k3os config to expose metrics:

k3os:
  k3s_args:
  - server
  - "--etcd-expose-metrics"
  - "true"

Create a service to expose the metrics port (below is from the kube-prometheus-stack chart):

    kubeEtcd:
      enabled: true
      endpoints:
        - 192.168.1.10
      service:
        enabled: true
        port: 2381
        targetPort: 2381

Expected behavior

The following should return the etcd metrics:

$ curl "192.168.1.10:2381/metrics"

Actual behavior

$ curl "192.168.1.10:2381/metrics"  
curl: (7) Failed to connect to 192.168.1.10 port 2381: Connection refused

Additional context

Here's my full k3os config as well as my kube-prometheus-stack HelmRelease. Currently running a single node cluster.

I've tried a few variants of the k3os config without success:

k3os:
  k3s_args:
  - server
  - "--etcd-expose-metrics"

k3os:
  k3s_args:
  - server
  - "--etcd-expose-metrics=true"

I've verified the service is up:

$ kubectl -n kube-system get service
NAME                                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                        AGE
kube-dns                             ClusterIP   10.43.0.10     <none>        53/UDP,53/TCP,9153/TCP         4h16m
metrics-server                       ClusterIP   10.43.169.31   <none>        443/TCP                        4h12m
prometheus-kubelet                   ClusterIP   None           <none>        10250/TCP,10255/TCP,4194/TCP   4h12m
prometheus-coredns                   ClusterIP   None           <none>        9153/TCP                       4h12m
prometheus-kube-controller-manager   ClusterIP   None           <none>        10252/TCP                      4h12m
prometheus-kube-etcd                 ClusterIP   None           <none>        2381/TCP                       4h12m
prometheus-kube-proxy                ClusterIP   None           <none>        10249/TCP                      4h12m
prometheus-kube-scheduler            ClusterIP   None           <none>        10251/TCP                      4h12m

[alexwaibel]

Looks like k3s only uses embeded etcd when started with --cluster-init. I'll give that a shot and reopen if I can't get it to work.