Adding custom annotations?

[laukaichung]

I want to use Linkerd and Bank Vault with Kubegres. But Kubegres doesn't seem to support custom annotations. Would you consider this feature?

if adding custom annotations is supported, I'd use Vault to inject secrets into the environment without relying on creating a secret. But right now it looks like replica pods would fail to start without valueFrom.secretKeyRef. It keeps asking Password in the logs.

spec:
  env:
    - name: POSTGRES_PASSWORD
      value: gffdggfdgfdf (Would use #vault:secret/data/path#DB_PASSWORD if custom annotations can be used)

    - name: POSTGRES_REPLICATION_PASSWORD
      value: fgdgfdgdffd

[alex-arica]

Thank you for your enhancement suggestion.

When you use: annotations: linkerd.io/inject: enabled

Do you set it from a StatefulSet? Or from another resources so that it is applied to your Pod?

Do you have an example?

[laukaichung]

It is set in pods. Some custom resource has a podAnnotations setting so that I can put linkerd and Vault annotations there. The custom annotation feature would allow a few enhancements with third party apps.

Zalando example:

apiVersion: acid.zalan.do/v1
kind: postgresql
metadata:
  name: sssdd
  labels:
    app: dfdfsdsf
    team: dsffddfs
spec:
  podAnnotations:
    linkerd.io/inject: enabled
    vault.security.banzaicloud.io/vault-addr: "https://vault.default.svc.cluster.local:8200"
    vault.security.banzaicloud.io/vault-skip-verify: "false"
    vault.security.banzaicloud.io/vault-tls-secret: "vault-tls"
    vault.security.banzaicloud.io/vault-role: "pg-role"
    vault.security.banzaicloud.io/vault-path: "kubernetes"

By the way, kubegres looks much simpler. I have tried Zalando but it is way over my head when it comes to Patroni.

[laukaichung]

Thank you alex! Looking forward to use this operator. Goodbye Zalando...

[alex-arica]

Thanks for your kind feedback about Kubegres. I agree and we are planning to keep Kubegres simple.

I made a change and manually I tested it locally. I pushed the changes in GIT main so that I can get your feedback.

Could you please test it for me as well?

Once you confirm that it is working, I am going to update the acceptance tests and create a release.

Please reinstall the Kubegres controller as follows:

kubectl apply -f https://raw.githubusercontent.com/reactive-tech/kubegres/main/kubegres.yaml

The above installation YAML is directly from GIT main.

You can add your annotation(s) from the YAML of "kind: Kubegres".

Please see an example below:

kind: Kubegres
metadata:
  name: mypostgres
  namespace: default
  annotations:
    linkerd.io/inject: enabled

[laukaichung]

I just installed it in a cloud server as well as a local k3d. It works perfectly fine! The master and replicas show the custom annotations. I can even use the env variables directly without creating a secret.

To upload dumps to S3, I'm planning to use a custom pg images with AWS cli installed and override the backup script. Is that the right approach?

[alex-arica]

Thank you for confirming that it is working for you. I am writing a new acceptance test for this change. Once it is done, I am going to create a new release, tag shortly and let you know once it is available.

In regards to uploads to S3, yes you can use a custom PG image as you suggested. As long as the image is compatible with the official Postgres docker image, it should be fine.

You can enable backup in your YAML of "kind: Kubegres", as explained here: https://www.kubegres.io/doc/enable-backup.html

And before deploying that YAML, you can override the default backup shell script, as per the page here: https://www.kubegres.io/doc/override-default-configs.html (please see the example in the section "Override backup_database.sh").

I hope that helps.

[alex-arica]

One more thing I thought about, I am not sure if this is possible with S3 ( I am specialised in bare metal clouds rather than Amazon Cloud), there could be a way of doing the same thing without using a custom PG image.

When overriding "backup_database.sh", inside the script, if it is possible to either install S3 binaries or download them using wget then this is what I would do.

[alex-arica]

I created a new tag 1.4 and release here: https://github.com/reactive-tech/kubegres/releases

To install Kubegres 1.4 :

kubectl apply -f https://raw.githubusercontent.com/reactive-tech/kubegres/v1.4/kubegres.yaml

I updated the documentation pages, as follows:

• new version 1.4 added in: Getting started page
• Details about "annotations" added in: All properties explained

Please close this ticket if you are happy with the changes.

Thank you for your suggestions!

[laukaichung]

Again, thank you very much for this feature! And you are right about installing AWS CLI in the backup script, it should work without the hassle of maintaining a custom image. Will get back to you if it works.

[laukaichung]

@alex-arica Is it possible to pass those custom annotations to backup pods as well? I'm working on a backup script. I just found that the secrets from Vault in the cluster manifest are not able to pass to the backup pods because the backup pods do not share those custom annotations.

Here's my helm template:

apiVersion: kubegres.reactive-tech.io/v1
kind: Kubegres
metadata:
  name: mypostgres
  namespace: default
  annotations:
    linkerd.io/inject: enabled
    vault.security.banzaicloud.io/vault-addr: "https://vault.default.svc.cluster.local:8200"
    vault.security.banzaicloud.io/vault-role: "default"
    vault.security.banzaicloud.io/vault-skip-verify: "false"
    vault.security.banzaicloud.io/vault-path: "kubernetes"
    vault.security.banzaicloud.io/vault-tls-secret: "vault-tls"
spec:
  replicas: 3
  image: postgres:13.2
  database:
    size: {{ .Values.size }}
    storageClassName: {{ .Values.storageClassName }}
    volumeMount: /var/lib/postgresql/data
  customConfig: {{ .Values.configMapName }}
  backup:
    schedule: "*/5 * * * *"
    pvcName: {{ .Values.backupPVCName }}
    volumeMount: /var/lib/backup
  env:
    - name: POSTGRES_PASSWORD
      valueFrom:
        secretKeyRef:
          name: mypostgres-secret
          key: superUserPassword
    - name: POSTGRES_REPLICATION_PASSWORD
      valueFrom:
        secretKeyRef:
          name: mypostgres-secret
          key: replicationUserPassword
    - name: POSTGRES_DB
      value: postgres
    - name: POSTGRES_USER
      value: postgres
    - name: DB_NAME
      value: vault:secret/data/ss#DB_NAME
    - name: DB_PASSWORD
      value: vault:secret/data/ss#DB_PASSWORD
    - name: DB_USER
      value: vault:secret/data/ss#DB_USER
    - name: AWS_ROLE_ARN
      value: vault:secret/data/ss#AWS_ROLE
    - name: AWS_ACCESS_KEY_ID
      value: vault:secret/data/ss#AWS_ACCESS
    - name: AWS_SECRET_ACCESS_KEY
      value: vault:secret/data/ss#AWS_SECRET
    - name: S3_BUCKET
      value: vault:secret/data/ss#S3_BUCKET
    - name: S3_BACKUP_DIR
      value: db/logical/

[alex-arica]

It's a good point!

I have to write it in a new version, because 1.4 was tagged.

Could you please close this ticket and create a new one?

I will make the changes against the new ticket.

[vincentserpoul]

@laukaichung having linkerd injected on the backup job makes it hang (known linkerd issue). Right now, if you add the inject annotation on the kubegress yaml, the backup is broken.

Did you encounter that issue?

The only workaround I found for now is to patch the cronjob:

kubectl patch cronjob backup-mypostgres -p '{"spec":{"jobTemplate":{"spec":{"template":{"metadata":{"annotations":{"linkerd.io/inject": "disabled"}}}}}}}'