search

EngineerBetter / concourse-up

Deprecated - used Control Tower instead
https://github.com/EngineerBetter/control-tower
Apache License 2.0
203 stars 28 forks source link

InfluxDB has default retention policy resulting in out of disk on web node #95

Closed sneakybeaky closed 5 years ago

sneakybeaky commented 5 years ago

concourse-up version 0.15.1.

After around 80 days of running our setup the web node ran out of disk. After some investigation we saw that InfluxDB had consumed all disk. Further digging showed that it had just the default retention policy which has an infinite duration.

sneakybeaky commented 5 years ago

After raising this in the concourse-up slack @dark5un commented :

Got it, we have retention-autocreate=true which according to docs:

Enables the automatic creation of the DEFAULT retention policy autogen when a database is created. The retention policy autogen has an infinite duration and is also set as the database's DEFAULT retention policy, which is used when a write or query does not specify a retention policy. Disable this setting to prevent the creation of this retention policy when creating databases.

evadinckel commented 5 years ago

Hello @sneakybeaky,

Thanks for reaching out to us and for raising interesting questions on this matter. This is just to inform you that we have acknowledged your issue by creating a task in our workflow tracker. We will get back to you as soon as our engineers come to a solution.

Thanks, Best regards Eva

evadinckel commented 5 years ago

Hello @sneakybeaky,

Hoping you're well today. This is to let you know that this is now fixed: we have made sure that the retention policy on InfluxDB is now extended to 28 days to prevent out of disk space on the web node. Patch release available here: https://github.com/EngineerBetter/concourse-up/releases/tag/0.20.2.

Thanks!

sneakybeaky commented 5 years ago

Thank you for all your hard work on this great product :-)