search

jaredjennings / helm-thehive

*Unofficial* Helm chart for TheHive
GNU Affero General Public License v3.0
16 stars 8 forks source link

with local storage, use StatefulSet rather than Deployment #15

Closed jaredjennings closed 3 years ago

jaredjennings commented 3 years ago

In particular when berkeleyje database storage is used, an exclusive lock on files in the database directory is sought. For initial installation, this works OK. For upgrades or changes, the Deployment tries to set up a new Pod and make sure it is ready before taking down the old one. But the berkeleyje code in the new Pod fails to get an exclusive lock on its files.

One way to fix this is to tell the deployment to use the "Recreate" strategy rather than "RollingUpdate." But a note there advises that this will only fix the behavior in the case when the Deployment is being upgraded; if instead a Pod is deleted, its replacement will be created while it is still terminating. "If you need an 'at most' guarantee for your Pods, you should consider using a StatefulSet."

And indeed, when using local storage, TheHive is stateful. It would be meaningless, for example, to scale such an arrangement to multiple replicas: first one must move the stored state outside the pods, into an external database and/or index.

(Attachment storage, like Cortex job I/O, is likely most simply expressed as a ReadWriteMany volume, equally for a single Pod and multiples.)

jaredjennings commented 3 years ago

TheHIve documentation says, "Using Cassandra is strongly recommended for production use while Berkeley DB can be prefered for testing and training purpose." Furthermore I couldn't find anything about migrating from one to the other; and the backup/restore directions are only for Cassandra. So it appears that if you were to install with the berkeleyje database, you would have no good upgrade path if you wanted to scale up.

Given that the Docker image was made and is maintained to cover the single-node, testing and development use case, and that this Helm chart automates Cassandra installation and configuration and is made to enable scaling, maybe the right approach is to just document the single-node caveat: you have to scale to 0 before helm upgrading.

jaredjennings commented 3 years ago

Migration between database types is now handled by a tool called cloner that comes with TheHive. However, all the hand-wringing and web-searching I did for the Hive 3-to-4 migration case applies here, too: it's not really possible to ask Helm what the old values were, and it's not very good to try to get the user to specify both old and new values. So, just as with migration, I think calling the cloner script would work better in the context of an operator than in a Helm chart.