Support for hooks and Helm hooks

kvaps commented 4 years ago

It would be nice to expand qbec directives with hooks similar to which helm already has:

https://helm.sh/docs/topics/charts_hooks/#the-available-hooks

Also to translate helm hooks the same way in expandHelmTemplate

kvaps commented 4 years ago

Good news, I just found that ArgoCD support helm hooks natively. Thus it will just run qbec show <env> to generate YAML, and consider the output as the rendered Helm-chart. So all the helm hooks will be working even if chart rendered via expandHelmTemplate

gotwarlost commented 4 years ago

@kvaps I'm curious about what hooks need to be supported in qbec with some examples.

kvaps commented 4 years ago

Sure! Most of the examples can be found in helm charts/stable

61 chart of 282 are using helm hooks:

ambassador                    anchore-engine                ark                           atlantis
burrow                        centrifugo                    cloudserver                   cockroachdb
collabora-code                consul                        contour                       dex
elasticsearch                 elasticsearch-curator         etcd-operator                 falco
fluent-bit                    gocd                          grafana                       hazelcast
hazelcast-jet                 inbucket                      influxdb                      jaeger-operator
jenkins                       keycloak                      kibana                        kong
mariadb                       mattermost-team-edition       memcached                     mercure
metrics-server                minio                         mission-control               mongodb-replicaset
mysql                         neo4j                         nginx-ingress                 openldap
percona-xtradb-cluster        pgadmin                       pomerium                      prestashop
prometheus-mongodb-exporter   prometheus-operator           redis-ha                      sentry
sonarqube                     spark                         spark-history-server          spinnaker
stolon                        sumokube                      sumologic-fluentd             tensorflow-notebook
tensorflow-serving            traefik                       velero                        weave-scope
wordpress

Let's take stolon (high available postgress cluster):

create-cluster-job.yaml
```
annotations:
"helm.sh/hook": post-install
"helm.sh/hook-delete-policy": before-hook-creation
```
this job should be created only after initial chart installation, and if similar job is already existing in the cluster it will be recreated.
init-db-job.yaml
```
annotations:
"helm.sh/hook": post-install
"helm.sh/hook-weight": "2"
"helm.sh/hook-delete-policy": hook-succeeded
```
the same as previous one but this job will be removed after the successful finish. Since it has weight 2, it will be created after the first one. Since it has no specified before-hook-creation deletion policy, then if similar job is already exiting in the cluster helm will try to update it which will cause the error, because jobs are immutable (https://github.com/kubernetes/kubernetes/issues/89657)
update-cluster-spec-job.yaml this job should be created after each update operation, if similar job is already existing in the cluster it will be recreated.
```
annotations:
"helm.sh/hook": post-upgrade
"helm.sh/hook-delete-policy": before-hook-creation
```

Another good example is wire-server charts:

migrate-schema.yaml, galley-migrate-data.yaml, create-index.yaml and migrate-data.yaml are using:
```
annotations:
"helm.sh/hook": post-install,post-upgrade
"helm.sh/hook-delete-policy": "before-hook-creation"
```
this means that hook should be called each time after successfully applying the configuration (initial or upgrade) and remove the job object if another job with similar name already exiting in the cluster before the executing the new one.

kvaps commented 4 years ago

I'm so sad that Kubernetes API does not allow to overwrite the Job manifests and execute the jobs according the changed hash of podTemplate for them, that's why all these hooks are needed for. They are quite useful to provide the database migrations, cluster initialization or uploading initial configuration to the application.

kvaps commented 4 years ago

@gotwarlost it seems all the needed code for hooks is already implemented in gitops-engine project: https://github.com/argoproj/gitops-engine/tree/master/pkg/sync/hook

This project combines the best from Argo CD and Flux CD. Not sure, if we can simple use it, but I would be very glad to see qbec as a platform application in argocd.

The only problem of ArgoCD is no opportunity to apply changes locally (in case if something has gonna wrong). Except these few issues QBEC is fully solves that.

splunk / qbec

Support for hooks and Helm hooks #149