search

ytsaurus / yt-k8s-operator

Kubernetes operator for YTsaurus.
https://ytsaurus.tech
Other
31 stars 22 forks source link

More options for store locations. #294

Closed sgburtsev closed 5 days ago

sgburtsev commented 1 week ago

I added two additional options for the store location. The first, MaxTrashMilliseconds, is intended to tune the max_trash_ttl parameter of the store_locations config. By default YTsaurus uses 1 hour. The second one, Watermark, is needed to adjust location watermarks. The operator uses this proportions to calculate them:

storeLocation.LowWatermark = min(0.1 * storeLocation.Quota, 5GiB)
storeLocation.HighWatermark = storeLocation.LowWatermark / 2
storeLocation.DisableWritesWatermark = storeLocation.HighWatermark / 2

While keeping this proportions, I wanted to change the initial percentage for LowWatermark, which was hard-coded. I am pretty sure there is a mistake in using min function. Because of min, it is impossible to use more than 5GiB as LowWatermark. However, it makes more sense to use at least 5GiB of space as LowWatermark. So I changed it to max. Also, I added into calculations the missing parameter TrashCleanupWatermark. The final solution:

storeLocation.LowWatermark = max(storeLocation.Watermark * storeLocation.Quota, 5GiB)
storeLocation.HighWatermark = storeLocation.LowWatermark / 2
storeLocation.DisableWritesWatermark = storeLocation.HighWatermark / 2
storeLocation.TrashCleanupWatermark = storeLocation.LowWatermark
sgburtsev commented 6 days ago

After discussion with @psushin I changed the initial proposal in the watermarks calculation part. Now it looks like that:

storeLocation.LowWatermark = min(0.1 * storeLocation.Quota, 25GiB)
storeLocation.HighWatermark = storeLocation.LowWatermark / 2
storeLocation.DisableWritesWatermark = storeLocation.HighWatermark / 2
storeLocation.TrashCleanupWatermark = storeLocation.LowWatermark

with ability to provide your own LowWatermark. Such approach should work well in both small testing environments and production with high-capacity volumes .

l0kix2 commented 5 days ago

regenerate yamls please so linter don't complain

zlobober commented 5 days ago

I'm ok with this change.

sgburtsev commented 5 days ago

regenerate yamls please so linter don't complain

Committed needed changes. Sorry for that.