Deploy the Postgres operator on a dedicated node pool with tolerations since our nodePool has taints

[kseswar]

Following the crunchydata documentation regarding node affinity (https://access.crunchydata.com/documentation/postgres-operator/4.5.1/architecture/high-availability/), we are trying to deploy a PostgresDB operator (with 3 replicas & node affinity) in the following GKE cluster.

As you can see in the command-runner logs, the cluster has been created but not in the desired node pool :

❰royee❙~/code/local-jenkins❙k8s:gke:❱✔≻ kubectl -n sedsac-ea-infra logs -f pgo-command-runner-9-ngq2w Processing pgo-client request at 20201117-130725 Check if namespace "sedsac-ea-infra" exists NAME STATUS AGE sedsac-ea-infra Active 12d Found the running pod pgo-client in sedsac-ea-infra Executing command "create cluster postgres --password-superuser LrtsJPxeqA1JXG1E --password w5K4RQAW1u --pgbouncer --node-label=app=sedsac-ea-infra --replica-count=3" created cluster: postgres workflow id: 7214dd72-bc45-42bc-b464-048d4feeacad database name: db users: username: luminate password: w5K4RQAW1u Done! secret "postgres-operator-token-6f6mb" deleted

Crunchydata documentation states the following:"The Node Affinity only uses the preferred scheduling strategy (similar to what is described in the Pod Anti-Affinity section above), so if a Pod cannot be scheduled to a particular Node matching the label, it will be scheduled to a different Node."

The desired node pool contains taints, therefore, we should somehow configure the operator to add the following tolerations as we did with other operators (i.e. Kafka): tolerations: - effect: NoSchedule key: dedicated operator: Equal value: sedsac-sac- effect: NoExecute key: dedicated operator: Equal value: sedsac-sac How can we add the tolerations to the operator?Unless we add the tolerations, the preferred scheduling strategy will be usedPlease advise

As we want to deploy the Postgres operator on a dedicated node pool since the pg create command seem to only support a nodeSelector (label) without tolerations which are mandatory since our nodePool has taints: create cluster --node-label=app=X The definition of "app=X" is insufficient, we also need to specify tolerations.

How can we specify tolerations?

Please tell us about your environment:

Operating System: Linux based Where is this running ( Local, Cloud Provider) - GCP GKE cluster Storage being used (NFS, Hostpath, Gluster, etc):GCP PVC, Container Image Tag: centos7-4.3.0 / centos7-12.2-4.3.0 PostgreSQL Version:12.2 Platform (Docker, Kubernetes, OpenShift): Kubernetes Platform Version: 1.16.11-gke.5

[jkatz]

Presently the Postgres Operator does not directly support setting tolerations. We already have this on our roadmap for a future release.

However, in the interim you could edit the cluster-deployment.json template that is stored in the pgo-config ConfigMap to set toleration rules.

[kseswar]

Could you please provide a template or an example for reference as where we can add the toleration rules in cluster-deployment.json

[denishpatel]

@jkatz if you have template or example, it will be helpful.

[kseswar]

Hi,

Could you please help us with a template or an example

[kseswar]

Hi Team,

This is important for us to test this workaround. It would be really helpful if you can provide a template or an example as soon as possible. @jkatz

[jkatz]

@kseswar If you are looking for an immediate response, please refer to our support policy and reach out to info@crunchydata.com

[kseswar]

Hi Team, we contacted info@crunchydata.com but it was not a much help. Kindly let me know if you have any example or template to test the workaround

[abrx]

Hi,

Another workaround would be to annotate the namespaces created by the operator with default selector & tolerations :

via yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: <namespace>
  annotations:
    scheduler.alpha.kubernetes.io/defaultTolerations: '[{"operator": "Exists", "effect": "NoSchedule", "key": "dedicated-node"}]'
    scheduler.alpha.kubernetes.io/node-selector: name-of-node-selector

or through CLI :

kubectl annotate namespace <namespace> scheduler.alpha.kubernetes.io/node-selector=name-of-node-selector and so on.

Hope it helps. abrx

[jkatz]

2149 adds support for assigning Pod Tolerations both across all PostgreSQL instances in a PostgreSQL cluster (pgo create cluster --toleration) as well as individual instances (pgo scale --toleration). Multiple tolerations can be assigned at once, and tolerations can be modified directly via the custom resources. This also adds the requisite documentation for doing so. This feature will be available in 4.6.

I did verify that the approach I suggested above (modifying the cluster-deployment.json template) does work. You may need to combine said usage with node affinity to get the assignments that you are looking for.

Providing use cases, examples, and even contributing solutions (both programmatic and "workarounds") are certainly helpful to the process as all of those things leads to a better implementation of a feature, better documentation, or at least a better understanding of the direction in which the development of the Postgres Operator should go.

Repeated asking of the equivalent of "is this done" or "can you provide an example" without bringing additional resources (e.g. examples that have been tried, additional use cases) will lead to limited action being taken. The issue being open is itself a note to active contributors that more work needs to be done, and the reason it is not being done could be a matter of research, priority, or trying to to work towards a better solution (e.g. #2149).

Pending review, support for tolerations will be merged over the next week and, as mentioned, will be available in the 4.6 release in early 2021. Thanks!