Minio can run out of space

[emosbaugh]

We make a 4GB request here. If you ship too many releases minio can run out of space.

failed to upload to s3: XMinioStorageFull: Storage backend has reached its minimum free disk threshold. Please delete a few objects to proceed. status code: 507, request id: 160A91662FDB6F45, host id:

[emosbaugh]

closing as stale

[andymartin-sch]

is there a way to customize the storage size and clear objects to make space?

[CpuID]

is there a way to customize the storage size and clear objects to make space?

commenting here incase someone finds this issue in future and goes looking for the same answer

https://kurl.sh/docs/add-ons/minio

you can set claimSize in the installer spec to define the size of the PVC space allocation

[andymartin-sch]

I think the problem raised by this issue is actually within the minio pod that's deployed as a part of the KOTS Admin Console. Over time, the artifacts dropped in /exports can add up and result in the error message from the first comment. It looks like this:

$ kubectl -n replicated exec -it kotsadm-minio-0  /bin/bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
bash-4.4$ df -h
Filesystem      Size  Used Avail Use% Mounted on
...
/dev/sdb        3.9G  3.8G   48M  99% /export
...

bash-4.4$ du -sh ./export/*
3.8G    ./export/kotsadm
16K ./export/lost+found

bash-4.4$ du -sh ./export/kotsadm/* 
3.8G    ./export/kotsadm/2CxBmCoL1Ng2wn1rG2aAARvv0RM
7.7M    ./export/kotsadm/supportbundles

bash-4.4$ du -sh ./export/kotsadm/2CxBmCoL1Ng2wn1rG2aAARvv0RM/* | wc -l
728

bash-4.4$ du -sh ./export/kotsadm/2CxBmCoL1Ng2wn1rG2aAARvv0RM/* | head 
3.8M    ./export/kotsadm/2CxBmCoL1Ng2wn1rG2aAARvv0RM/0.tar.gz
3.8M    ./export/kotsadm/2CxBmCoL1Ng2wn1rG2aAARvv0RM/1.tar.gz
3.8M    ./export/kotsadm/2CxBmCoL1Ng2wn1rG2aAARvv0RM/10.tar.gz
3.8M    ./export/kotsadm/2CxBmCoL1Ng2wn1rG2aAARvv0RM/100.tar.gz
4.5M    ./export/kotsadm/2CxBmCoL1Ng2wn1rG2aAARvv0RM/101.tar.gz
4.5M    ./export/kotsadm/2CxBmCoL1Ng2wn1rG2aAARvv0RM/102.tar.gz
4.5M    ./export/kotsadm/2CxBmCoL1Ng2wn1rG2aAARvv0RM/103.tar.gz
4.1M    ./export/kotsadm/2CxBmCoL1Ng2wn1rG2aAARvv0RM/104.tar.gz
4.1M    ./export/kotsadm/2CxBmCoL1Ng2wn1rG2aAARvv0RM/105.tar.gz
3.8M    ./export/kotsadm/2CxBmCoL1Ng2wn1rG2aAARvv0RM/106.tar.gz

I think the remaining questions are:

• Is there documentation for how these files should be cleaned up? (Will anything break if I manually remove these tar.gz files?)
• How can we make it so these are automatically cleaned up going forward? (Asking QA/Eng/SREs to manually exec and rm things will cost time & add confusion)
  • Can we provision extra disk space for this volume at the time we install the admin console?

[banjoh]

Reopening cause its hitting users.