dqlite writing huge amounts of data to local drives

canonical / microk8s

MicroK8s is a small, fast, single-package Kubernetes for datacenters and the edge.

https://microk8s.io

Apache License 2.0

8.34k stars 763 forks source link

dqlite writing huge amounts of data to local drives #3064

Open b0n0r opened 2 years ago

b0n0r commented 2 years ago

Running a 3 node microk8s instance on baremetal commodity HW, Ubuntu 20.04.

Started out with 2 nodes, adding a 3rd one later in the process, enabling HA.

What I am seeing is very strange behaviour where dqlite process is tiring out local storage rather quickly.

A particular node has been running for roughly 2 weeks and the dqlite process has written 30TB+ to the drive(s).

My pinpointing process went like this:

uptime for reference 03:47:27 up 13 days, 28 min, 1 user, load average: 2.56, 3.33, 2.74
check all processes in /proc and grep for bytes written, sort numerically: grep ^write_bytes /proc/*/io | awk '{print $2" "$1}'|sort -n the process in my case with most bytes written: 37019017576448 /proc/1092/io:write_bytes: quick conversion amounts this to roughly 33 Terabytes which is exceedingly big number of writes
putting a face to a name (what is this PID) ps auwx|grep 1092 output in my case: root 1092 14.9 4.0 11209648 5295232 ? Ssl Apr01 2810:47 /snap/microk8s/3052/bin/k8s-dqlite --storage-dir=/var/snap/microk8s/3052/var/kubernetes/backend/ --listen=unix:///var/snap/microk8s/3052/var/kubernetes/backend/kine.sock:12379
the storage-dir of the dqlite process du -hs /var/snap/microk8s/3052/var/kubernetes/backend/ 3.5G /var/snap/microk8s/3052/var/kubernetes/backend/

kubectl get nodes seems fine

$ microk8s.kubectl get nodes -o wide
NAME              STATUS   ROLES    AGE    VERSION                    INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
testnode-k8s-n2   Ready    <none>   142d   v1.22.8-3+f833e44163b5b1   xxx.x.x.xxx   <none>        Ubuntu 20.04.3 LTS   5.4.0-91-generic    containerd://1.5.2
testnode-k8s-n1   Ready    <none>   142d   v1.22.8-3+f833e44163b5b1   xxx.x.x.xxx   <none>        Ubuntu 20.04.3 LTS   5.4.0-91-generic    containerd://1.5.2
testnode-k8s-n3   Ready    <none>   16d    v1.23.5-2+c812603a312d2b   xxx.x.x.xxx   <none>        Ubuntu 20.04.4 LTS   5.4.0-107-generic   containerd://1.5.9

microk8s status

$ microk8s status
microk8s is running
high-availability: yes
datastore master nodes: xxx.x.x.xxx:19001 xxx.x.x.xxx:19001 xxx.x.x.xxx:19001
datastore standby nodes: none

just ran this for reference a got a weird permission denied error for dqlite storage-dir though, no idea why (race condition maybe?) all files are owned by root:microk8s with 660 perms:

$ microk8s status
microk8s is running
Error: open servers store: open /var/snap/microk8s/3052/var/kubernetes/backend/cluster.yaml: permission denied
Usage:
  dqlite -s <servers> <database> [command] [flags]

Flags:
  -c, --cert string       public TLS cert
  -f, --format string     output format (tabular, json) (default "tabular")
  -h, --help              help for dqlite
  -k, --key string        private TLS key
  -s, --servers strings   comma-separated list of db servers, or file://<store>

high-availability: yes
  datastore master nodes: xxx.x.x.xxx:19001 xxx.x.x.xxx:19001 xxx.x.x.xxx:19001
  datastore standby nodes: none

What I also see is the dqlite process is only running on 1 of the 3 nodes - is that expected behaviour?

Please run microk8s inspect and attach the generated tarball to this issue.

Would like to, but I went through the generated tarball and seems like it includes a lot of potentially sensitive (internal) data, is there a way to redact this or somehow share it so it does not become public?

Thank you for microk8s :)

MathieuBordere commented 2 years ago

Can you show the result of ls -alh /var/snap/microk8s/3052/var/kubernetes/backend/?

petermihalik commented 2 years ago

Hi Mathieu,

I'm working with @b0n0r on this issue, so here is the requested output:

total 3.5G
drwxrwxr-x 2 root microk8s 4.0K Apr 14 09:30 .
drwxr-xr-x 4 root root     4.0K Mar 28 10:17 ..
-rw-rw---- 1 root microk8s 8.0M Apr 14 09:17 0000000112380813-0000000112381268
-rw-rw---- 1 root microk8s 8.0M Apr 14 09:19 0000000112381269-0000000112381618
-rw-rw---- 1 root microk8s 8.0M Apr 14 09:19 0000000112381619-0000000112382182
-rw-rw---- 1 root microk8s 8.0M Apr 14 09:19 0000000112382183-0000000112382755
-rw-rw---- 1 root microk8s 8.0M Apr 14 09:20 0000000112382756-0000000112383313
-rw-rw---- 1 root microk8s 8.0M Apr 14 09:21 0000000112383314-0000000112383729
-rw-rw---- 1 root microk8s 8.0M Apr 14 09:21 0000000112383730-0000000112384325
-rw-rw---- 1 root microk8s 8.0M Apr 14 09:23 0000000112384326-0000000112384680
-rw-rw---- 1 root microk8s 8.0M Apr 14 09:24 0000000112384681-0000000112385134
-rw-rw---- 1 root microk8s 8.0M Apr 14 09:24 0000000112385135-0000000112385705
-rw-rw---- 1 root microk8s 8.0M Apr 14 09:24 0000000112385706-0000000112386273
-rw-rw---- 1 root microk8s 8.0M Apr 14 09:25 0000000112386274-0000000112386738
-rw-rw---- 1 root microk8s 8.0M Apr 14 09:26 0000000112386739-0000000112387234
-rw-rw---- 1 root microk8s 8.0M Apr 14 09:27 0000000112387235-0000000112387738
-rw-rw---- 1 root microk8s 8.0M Apr 14 09:28 0000000112387739-0000000112388069
-rw-rw---- 1 root microk8s 8.0M Apr 14 09:29 0000000112388070-0000000112388613
-rw-rw---- 1 root microk8s 8.0M Apr 14 09:29 0000000112388614-0000000112389177
-rw-rw---- 1 root microk8s 8.0M Apr 14 09:30 0000000112389178-0000000112389759
-rw-rw---- 1 root microk8s 1.9K Mar 28 10:17 cluster.crt
-rw-rw---- 1 root microk8s 3.2K Mar 28 10:17 cluster.key
-rw-rw---- 1 root microk8s  196 Apr 14 09:30 cluster.yaml
-rw-rw-r-- 1 root microk8s    2 Apr 14 05:55 failure-domain
-rw-rw---- 1 root microk8s   60 Mar 28 10:17 info.yaml
srw-rw---- 1 root microk8s    0 Apr  1 07:19 kine.sock:12379
-rw-rw---- 1 root microk8s   66 Apr  1 07:19 localnode.yaml
-rw-rw---- 1 root microk8s   32 Apr 10 15:09 metadata1
-rw-rw---- 1 root microk8s   32 Apr 10 15:09 metadata2
-rw-rw---- 1 root microk8s 8.0M Apr 14 09:30 open-27604
-rw-rw---- 1 root microk8s 8.0M Apr 14 09:29 open-27605
-rw-rw---- 1 root microk8s 8.0M Apr 14 09:30 open-27606
-rw-rw---- 1 root microk8s 1.7G Apr 14 09:29 snapshot-15-112388421-1131045106
-rw-rw---- 1 root microk8s  128 Apr 14 09:29 snapshot-15-112388421-1131045106.meta
-rw-rw---- 1 root microk8s 1.7G Apr 14 09:30 snapshot-15-112389445-1131085779
-rw-rw---- 1 root microk8s  128 Apr 14 09:30 snapshot-15-112389445-1131085779.meta

MathieuBordere commented 2 years ago

Based on this information, your system writes roughly 112389759 - 112380813 = 8946 (112380813 is from file 0000000112380813-0000000112381268 and 112389759 is from file 0000000112389178-0000000112389759) raft log entries every 12 minutes, which is roughly 45000 entries every hour, about 1 million entries per day. Snapshot are written roughly every 1000 entries, so that means roughly 1000 snapshots per day, with snapshot size 1.7Gb, that is 1.7 TB writes of snapshots per day, times 14 for 2 weeks is roughly 24TB, which is somewhat in line with the 30TB you are seeing. The deviation is probably related to the fact we only see a 12 minute window of the data, it might have been a bit calmer during this period of time.

dqlite has a way to decrease the frequency of the snapshotting, and the microk8s team has been made aware on how to do this. Decreasing the frequency of snapshotting would decrease the total amount of data written to disk.

ktsakalozos commented 2 years ago

Hi @petermihalik,

What could you share about the k8s workload that produces this amount of write operations? I wonder if we could use it as a benchmark for future improvements?

Decreasing the frequency of snapshots would increase the memory footprint used by dqlite. Could you share how much memory dqlite uses right now and how much free memory your system has?

Currently we do not expose the option of manually configuring the raft log threshold size that triggers the snapshot. Would you consider using an external etcd cluster [1] an option?

[1] https://microk8s.io/docs/external-etcd

b0n0r commented 2 years ago

hi @ktsakalozos and the team,

this is a mixed workload running some batch processing deployments as well as some simple microservices for text language detection etc. nothing too exotic i think.

what are the steps for me to pinpoint the source of so many write operations ie. which part of the workload is generating this?

dqlite process (still running on only 1 out of 3 nodes) is currently consuming 5.8% of memory according to ps, while these systems have 128G memory available.

more detailed output for cat /proc/<PID>/status:

$ cat /proc/<PID>/status
Name:   k8s-dqlite
...
VmPeak: 15564948 kB
VmSize: 11507924 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:  11653992 kB
VmRSS:   7764292 kB
RssAnon:     7744276 kB
RssFile:       20016 kB
RssShmem:          0 kB
VmData:  8178700 kB
VmStk:       132 kB
VmExe:     18108 kB
VmLib:      5736 kB
VmPTE:     15756 kB
VmSwap:        0 kB
HugetlbPages:          0 kB
CoreDumping:    0
THP_enabled:    1
Threads:    43
SigQ:   0/514555
...

if external etcd is my only option to save the SSDs, i will have to give it a try i guess? are there any other options apart from this? my concern is that this behaviour is killing the drives and since this is a PoC cluster consumer drives are wearing out very quickly.

nilswieber commented 1 year ago

I'm facing the same problem. dqlite is generating a lot of disk io, which is producing wear on the ssds on which microk8s is running. I reduced the cluster to a single node in order to reduce the impact.

b0n0r commented 1 year ago

I'm facing the same problem. dqlite is generating a lot of disk io, which is producing wear on the ssds on which microk8s is running. I reduced the cluster to a single node in order to reduce the impact.

try disabling the "prometheus" microk8s addon: microk8s disable prometheus and see if that helps?

benben commented 1 year ago

Can confirm this behavior on a a fresh microk8s cluster, not running any workloads. Virtualization is Proxmox. I've attached some debug info including the inspect tarball.

# microk8s version
MicroK8s v1.25.4 revision 4221

# uname -a
Linux hostname 5.15.0-53-generic #59-Ubuntu SMP Mon Oct 17 18:53:30 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

# microk8s status
microk8s is running
high-availability: yes
  datastore master nodes: 10.0.0.1:19001 10.0.0.2:19001 10.0.0.3:19001
  datastore standby nodes: none
addons:
  enabled:
    ha-cluster           # (core) Configure high availability on the current node
    helm                 # (core) Helm - the package manager for Kubernetes
    helm3                # (core) Helm 3 - the package manager for Kubernetes
  disabled:
    cert-manager         # (core) Cloud native certificate management
    community            # (core) The community addons repository
    dashboard            # (core) The Kubernetes dashboard
    dns                  # (core) CoreDNS
    gpu                  # (core) Automatic enablement of Nvidia CUDA
    host-access          # (core) Allow Pods connecting to Host services smoothly
    hostpath-storage     # (core) Storage class; allocates storage from host directory
    ingress              # (core) Ingress controller for external access
    kube-ovn             # (core) An advanced network fabric for Kubernetes
    mayastor             # (core) OpenEBS MayaStor
    metallb              # (core) Loadbalancer for your Kubernetes cluster
    metrics-server       # (core) K8s Metrics Server for API access to service metrics
    observability        # (core) A lightweight observability stack for logs, traces and metrics
    prometheus           # (core) Prometheus operator for monitoring and logging
    rbac                 # (core) Role-Based Access Control for authorisation
    registry             # (core) Private image registry exposed on localhost:32000
    storage              # (core) Alias to hostpath-storage add-on, deprecated

inspection-report-20221201_100122.tar.gz

vinayan3 commented 1 year ago

@ktsakalozos is there any updates? I'm also facing this issue too and would like a way to reduce the number of writes to the SSD to reduce wear just like the other folks on this thread.

microk8s status
microk8s is running
high-availability: no
  datastore master nodes: 127.0.0.1:19001
  datastore standby nodes: none
addons:
  enabled:
    cert-manager         # (core) Cloud native certificate management
    dashboard            # (core) The Kubernetes dashboard
    dns                  # (core) CoreDNS
    gpu                  # (core) Automatic enablement of Nvidia CUDA
    ha-cluster           # (core) Configure high availability on the current node
    helm                 # (core) Helm - the package manager for Kubernetes
    helm3                # (core) Helm 3 - the package manager for Kubernetes
    hostpath-storage     # (core) Storage class; allocates storage from host directory
    metrics-server       # (core) K8s Metrics Server for API access to service metrics
    rbac                 # (core) Role-Based Access Control for authorisation
    registry             # (core) Private image registry exposed on localhost:32000
    storage              # (core) Alias to hostpath-storage add-on, deprecated
  disabled:
    community            # (core) The community addons repository
    host-access          # (core) Allow Pods connecting to Host services smoothly
    ingress              # (core) Ingress controller for external access
    kube-ovn             # (core) An advanced network fabric for Kubernetes
    mayastor             # (core) OpenEBS MayaStor
    metallb              # (core) Loadbalancer for your Kubernetes cluster
    observability        # (core) A lightweight observability stack for logs, traces and metrics
    prometheus           # (core) Prometheus operator for monitoring and logging

duplabe commented 1 year ago

I have the same problem.

My stack: Proxmox with 3 VMs, each running microk8s on Ubuntu 22.04 (1 control-plane, 2 workers). iotop shows constant k8s-dqlite writing on the control-plane (not on the workers).

microk8s version
MicroK8s v1.26.0 revision 4390

microk8s status
microk8s is running
high-availability: no
  datastore master nodes: 192.168.123.50:19001
  datastore standby nodes: none
addons:
  enabled:
    community            # (core) The community addons repository
    dns                  # (core) CoreDNS
    ha-cluster           # (core) Configure high availability on the current node
    helm                 # (core) Helm - the package manager for Kubernetes
    helm3                # (core) Helm 3 - the package manager for Kubernetes
    ingress              # (core) Ingress controller for external access
    metrics-server       # (core) K8s Metrics Server for API access to service metrics
  disabled:
    argocd               # (community) Argo CD is a declarative continuous deployment for Kubernetes.
...

ktsakalozos commented 1 year ago

With the 1.26 release we exposed [1] a few configuration variables you could use to tune dqlite's behavior.

You can create a tuning.yaml file under /var/snap/microk8s/current/var/kubernetes/backend/ that includes:

snapshot:
  trailing: 8192
  threshold: 1024

... and restart microk8s.

This will instruct dqlite to keep more data in memory but take snapshots less frequently (x8 times less).

Note that this is experimental for now and might need some trial and error on your side to make sure in matches your needs/workloads/hardware specs.

[1] https://github.com/canonical/k8s-dqlite/pull/32

selvex commented 1 year ago

With the 1.26 release we exposed [1] a few configuration variables you could use to tune dqlite's behavior.

You can create a tuning.yaml file under /var/snap/microk8s/current/var/kubernetes/backend/ that includes:
snapshot:
  trailing: 8192
  threshold: 1024
... and restart microk8s.

This will instruct dqlite to keep more data in memory but take snapshots less frequently (x8 times less).

Note that this is experimental for now and might need some trial and error on your side to make sure in matches your needs/workloads/hardware specs.

[1] canonical/k8s-dqlite#32

Tried this, but instead of

snapshot:
  trailing: 8192
  threshold: 1024

which are the defaults according to (https://github.com/canonical/k8s-dqlite/pull/32), I chose

snapshot:
  trailing: 8192
  threshold: 8192

which should have the expected effect of reducing the writes by a factor of 8 according to the PR.

However, after setting the values I did not experience any difference in the huge amount of I/O operations. I am running a single node cluster with microk8s and the virtual machine permanently has a write of 5 - 6 mb/s.

davidmonro commented 1 year ago

I've also tried messing with the tuning.yaml, and from the syslog entry it appears to have been accepted:

microk8s.daemon-k8s-dqlite[3041]: I0402 01:43:19.917841 3041 log.go:198] Raft snapshot parameters set to (threshold=8192, trailing=8192)

but I'm not seeing any significant reduction in write traffic.

MathieuBordere commented 1 year ago

@selvex @davidmonro Can you post the contents of your data folder? ls -alh /var/snap/microk8s/3052/var/kubernetes/backend/ 3052 should probably be replaced by some other value.

davidmonro commented 1 year ago

$ ls -alh /var/snap/microk8s/4959/var/kubernetes/backend/
total 244M
drwxrwx--- 2 root microk8s 4.0K Apr  3 19:24 .
drwxr-xr-x 3 root root     4.0K Apr 22  2022 ..
-rw-rw---- 1 root microk8s 8.0M Apr  3 18:47 0000000153727440-0000000153727797
-rw-rw---- 1 root microk8s 8.0M Apr  3 18:48 0000000153727798-0000000153728298
-rw-rw---- 1 root microk8s 8.0M Apr  3 18:48 0000000153728299-0000000153728899
-rw-rw---- 1 root microk8s 8.0M Apr  3 18:50 0000000153728900-0000000153729324
-rw-rw---- 1 root microk8s 8.0M Apr  3 18:53 0000000153729325-0000000153729735
-rw-rw---- 1 root microk8s 8.0M Apr  3 18:53 0000000153729736-0000000153730344
-rw-rw---- 1 root microk8s 8.0M Apr  3 18:55 0000000153730345-0000000153730845
-rw-rw---- 1 root microk8s 8.0M Apr  3 18:58 0000000153730846-0000000153731200
-rw-rw---- 1 root microk8s 8.0M Apr  3 18:58 0000000153731201-0000000153731799
-rw-rw---- 1 root microk8s 8.0M Apr  3 18:59 0000000153731800-0000000153732357
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:02 0000000153732358-0000000153732710
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:03 0000000153732711-0000000153733238
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:03 0000000153733239-0000000153733852
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:06 0000000153733853-0000000153734236
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:08 0000000153734237-0000000153734692
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:08 0000000153734693-0000000153735295
-rw-rw---- 1 root microk8s 5.3M Apr  3 19:09 0000000153735296-0000000153735631
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:12 0000000153735632-0000000153735988
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:13 0000000153735989-0000000153736548
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:13 0000000153736549-0000000153737158
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:16 0000000153737159-0000000153737511
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:18 0000000153737512-0000000153737993
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:18 0000000153737994-0000000153738606
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:20 0000000153738607-0000000153739029
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:23 0000000153739030-0000000153739431
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:23 0000000153739432-0000000153740043
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:24 0000000153740044-0000000153740548
-rw-rw---- 1 root microk8s 1.9K Apr 22  2022 cluster.crt
-rw-rw---- 1 root microk8s 3.2K Apr 22  2022 cluster.key
-rw-rw---- 1 root microk8s   63 Apr  3 19:24 cluster.yaml
-rw-rw-r-- 1 root microk8s    2 Apr  2 01:43 failure-domain
-rw-rw---- 1 root microk8s   57 Apr 22  2022 info.yaml
srw-rw---- 1 root microk8s    0 Apr  2 01:43 kine.sock:12379
-rw-rw---- 1 root microk8s   63 Apr  2 01:43 localnode.yaml
-rw-rw---- 1 root microk8s   32 Apr 22  2022 metadata1
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:25 open-1753
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:23 open-1754
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:24 open-1755
-rw-rw---- 1 root microk8s 3.3M Apr  3 18:43 snapshot-1-153727439-151252375
-rw-rw---- 1 root microk8s   72 Apr  3 18:43 snapshot-1-153727439-151252375.meta
-rw-rw---- 1 root microk8s 3.2M Apr  3 19:09 snapshot-1-153735631-152794636
-rw-rw---- 1 root microk8s   72 Apr  3 19:09 snapshot-1-153735631-152794636.meta
-rw-rw-r-- 1 root microk8s   45 Apr  2 01:08 tuning.yaml

MathieuBordere commented 1 year ago

$ ls -alh /var/snap/microk8s/4959/var/kubernetes/backend/
total 244M
drwxrwx--- 2 root microk8s 4.0K Apr  3 19:24 .
drwxr-xr-x 3 root root     4.0K Apr 22  2022 ..
-rw-rw---- 1 root microk8s 8.0M Apr  3 18:47 0000000153727440-0000000153727797
-rw-rw---- 1 root microk8s 8.0M Apr  3 18:48 0000000153727798-0000000153728298
-rw-rw---- 1 root microk8s 8.0M Apr  3 18:48 0000000153728299-0000000153728899
-rw-rw---- 1 root microk8s 8.0M Apr  3 18:50 0000000153728900-0000000153729324
-rw-rw---- 1 root microk8s 8.0M Apr  3 18:53 0000000153729325-0000000153729735
-rw-rw---- 1 root microk8s 8.0M Apr  3 18:53 0000000153729736-0000000153730344
-rw-rw---- 1 root microk8s 8.0M Apr  3 18:55 0000000153730345-0000000153730845
-rw-rw---- 1 root microk8s 8.0M Apr  3 18:58 0000000153730846-0000000153731200
-rw-rw---- 1 root microk8s 8.0M Apr  3 18:58 0000000153731201-0000000153731799
-rw-rw---- 1 root microk8s 8.0M Apr  3 18:59 0000000153731800-0000000153732357
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:02 0000000153732358-0000000153732710
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:03 0000000153732711-0000000153733238
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:03 0000000153733239-0000000153733852
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:06 0000000153733853-0000000153734236
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:08 0000000153734237-0000000153734692
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:08 0000000153734693-0000000153735295
-rw-rw---- 1 root microk8s 5.3M Apr  3 19:09 0000000153735296-0000000153735631
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:12 0000000153735632-0000000153735988
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:13 0000000153735989-0000000153736548
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:13 0000000153736549-0000000153737158
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:16 0000000153737159-0000000153737511
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:18 0000000153737512-0000000153737993
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:18 0000000153737994-0000000153738606
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:20 0000000153738607-0000000153739029
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:23 0000000153739030-0000000153739431
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:23 0000000153739432-0000000153740043
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:24 0000000153740044-0000000153740548
-rw-rw---- 1 root microk8s 1.9K Apr 22  2022 cluster.crt
-rw-rw---- 1 root microk8s 3.2K Apr 22  2022 cluster.key
-rw-rw---- 1 root microk8s   63 Apr  3 19:24 cluster.yaml
-rw-rw-r-- 1 root microk8s    2 Apr  2 01:43 failure-domain
-rw-rw---- 1 root microk8s   57 Apr 22  2022 info.yaml
srw-rw---- 1 root microk8s    0 Apr  2 01:43 kine.sock:12379
-rw-rw---- 1 root microk8s   63 Apr  2 01:43 localnode.yaml
-rw-rw---- 1 root microk8s   32 Apr 22  2022 metadata1
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:25 open-1753
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:23 open-1754
-rw-rw---- 1 root microk8s 8.0M Apr  3 19:24 open-1755
-rw-rw---- 1 root microk8s 3.3M Apr  3 18:43 snapshot-1-153727439-151252375
-rw-rw---- 1 root microk8s   72 Apr  3 18:43 snapshot-1-153727439-151252375.meta
-rw-rw---- 1 root microk8s 3.2M Apr  3 19:09 snapshot-1-153735631-152794636
-rw-rw---- 1 root microk8s   72 Apr  3 19:09 snapshot-1-153735631-152794636.meta
-rw-rw-r-- 1 root microk8s   45 Apr  2 01:08 tuning.yaml

The impact of the decrease of the snapshot frequency on disk IO will be noticeable when the snapshot files are large, like in the case above where a 1.7GB snapshot is taken with 1 minute between snapshots. In your case, because the snapshot is so small, it doesn't make a lot of difference.

If you see a lot of writes, that's because k8s/microk8s/someone else is performing a lot of writes on the DB for some reason.

davidmonro commented 1 year ago

Interesting - it is a pretty light workload, so not sure what would be doing that. Can dqlite to do query logging?

ktsakalozos commented 1 year ago

@davidmonro what exactly do you see? The I/O load is constant or does it come in spikes? Is this a single node cluster?

You could add a --debug flag in /var/snap/microk8s/current/args/k8s-dqlite and then restart k8s-dqlite with: sudo systemctl restart snap.microk8s.daemon-k8s-dqlite. Then observer the k8s-dqlite logs with journalctl -fu snap.microk8s.daemon-k8s-dqlite.

davidmonro commented 1 year ago

Sorry I left this so long, I hoped upgrading to microk8s 1.27 and completely rebuilding the node would fix it. It did not.

There's a lot of output, and a lot of it seems to relate to leases? I ran it with debugging for about 150 seconds, which generated about 8000 lines of output, 4700 of which containe the word 'lease'.

Is there anything sensitive in that log, or would it be sensible to attach it here?

camille-rodriguez commented 8 months ago

Experienced this with 1.28, local disk filled up with 500GB of data with no much running on it

aalvarado commented 1 month ago

I also have this issue, constantly writing to disk

aalvarado commented 1 month ago

Uninstalling microk8s has fixed to disk every second. It was also filling up the journal with repeated entries