search

uber-archive / cherami-server

Distributed, scalable, durable, and highly available message queue system. This project is deprecated and not maintained.
https://eng.uber.com/cherami/
MIT License
1.42k stars 102 forks source link

Is MinFreeDiskSpaceBytes still relevant? #333

Open doodle-tnw opened 6 years ago

doodle-tnw commented 6 years ago

Following an update a couple of by @kirg days ago I'm confused about this value.

This can be configured by setting: ./cherami-admin --env=prod --hostport=fehost2:4922 cfg set cherami-storehost.*.*.*.minFreeDiskSpaceBytes 10000000 and defaults to 40GB.

I've noticed too that the storagemonitor has now hardcoded thresholds of:

thresholdWarn                = 75GB
thresholdReadOnly         = 50GB
thresholdResumeWrites = 100GB

Why is the placement threshold set below the threshold for storage hosts to become read only, why is it such large values and is there any way we can affect the monitor's thresholds?

I test cherami locally on my machine with Docker, and ony my mac docker's default overlay disk size is 63GB, and theres probably only about 32GB of that free - this was fine as I used to hop on, and change the minFree to 1gb, so I could carry on testing. Now it doesn't seem like I can do that anymore, as the storehosts all become readonly.

In production we use cherami as a distributed message queue and don't expect more than 20minutes of messages to be backed to disk at any one time, which is no way near the 50GB limit.

Cheers, Dan