akash node: too much disk i/o

[andy108369]

initially reported by community member SGC (Discord)

RPC node is writing at 10MB/s, which is 864GB written in a day.

My akash RPC node wrote 3.1Ti in the past 4d17h:

# cat /proc/$(pidof -s akash)/io | numfmt --to=iec-i --field 2 | grep -E '^read|^write'
read_bytes:     37Mi
write_bytes:         3.1Ti

# kubectl -n akash-services get pods -l app=akash-node
NAME             READY   STATUS    RESTARTS   AGE
akash-node-1-0   1/1     Running   0          4d17h

A standard consumer SSD of a good kind has an expected life/warranty period of 500TBw.

This would probably mean it will kill most consumer SSDs in 2 years. For instance, an SSD disk with 500 TBw would last 592 days if being written 864Gi a day: (500 TBW *1024)/864 = 592 days.

When comparing to sifchain node v1.1.0-beta (my validator), the process has written only 123Gi within 21d (which makes ~6Gi a day vs. 864Gi with akash)

root@k8s-node-14:~# cat /proc/$(pidof -s sifnoded)/io | numfmt --to=iec-i --field 2 | grep -E '^read|^write'
read_bytes:        44Gi
write_bytes:        123Gi

$ kubectl -n $ns get pods  
NAME     READY   STATUS    RESTARTS       AGE
node-0   1/1     Running   8 (158m ago)   21d

[andy108369]

• akash writes about ~34Gi an hour;
• sifnoded writes about ~26Gi an hour;
• osmosisd writes about ~90Gi an hour;

---

akash

# date ; cat /proc/$(pidof -s akash)/io | numfmt --to=iec-i --field 2 | grep -E '^read|^write' ;date; sleep 1h; date; cat /proc/$(pidof -s akash)/io | numfmt --to=iec-i --field 2 | grep -E '^read|^write' ; date
Tue Jan 24 18:07:41 UTC 2023
read_bytes: 0
write_bytes:         95Gi
Tue Jan 24 18:07:41 UTC 2023
Tue Jan 24 19:07:41 UTC 2023
read_bytes: 0
write_bytes:        129Gi
Tue Jan 24 19:07:41 UTC 2023

sifnoded

# date ; cat /proc/$(pidof -s sifnoded)/io | numfmt --to=iec-i --field 2 | grep -E '^read|^write' ;date; sleep 1h; date; cat /proc/$(pidof -s sifnoded)/io | numfmt --to=iec-i --field 2 | grep -E '^read|^write' ; date
Tue Jan 24 18:09:52 UTC 2023
read_bytes:        59Gi
write_bytes:        194Gi
Tue Jan 24 18:09:52 UTC 2023
Tue Jan 24 19:09:52 UTC 2023
read_bytes:        66Gi
write_bytes:        220Gi
Tue Jan 24 19:09:52 UTC 2023

osmosisd

$ cat /proc/$(pidof -s osmosisd)/io | numfmt --to=iec-i --field 2 | grep -E '^read|^write'
read_bytes:        83Gi
write_bytes:         2.2Ti

$ ps -p $(pidof -s osmosisd) -o pid,cmd,start,etimes,etime
    PID CMD                          STARTED ELAPSED     ELAPSED
3225243 /home/ubuntu/.osmosisd/cosm   Jan 23   92302  1-01:38:22

Thanks polkachu for providing the osmosisd stats