search

instana / robot-shop

Sample microservices application for playing with
http://instana.com/
Apache License 2.0
866 stars 3.69k forks source link

Add pod affinity, tolerations and nodeSelector for robot-shop deployments through Helm chart #120

Closed rishinair11 closed 2 years ago

rishinair11 commented 2 years ago

Why

While deploying robot-shop using the Helm chart, there is no way to specify the affinity and tolerations for the deployment pods. In my use case, I have two node pools in my GKE cluster, and I need the robot-shop pods to be scheduled only on one of the 2 pools. This is possible using affinities and tolerations, but there is no support through the Helm chart, out of the box. This commit aims to provide the support to do the same.

What

Using the values.yaml,


######################################
# Affinities for individual workloads
# set in the following way:
# <workload>:
#   affinity: {}
#   nodeSelector: {}
#   tolerations: []
######################################

cart: {}

catalogue: {}

dispatch: {}

mongodb: {}

mysql: {}

payment:
  # Alternative payment gateway URL
  # Default is https://www.paypal.com
  gateway: null
  #gateway: https://www.worldpay.com

rabbitmq: {}

ratings: {}

redis:
  # Storage class to use with redis statefulset.
  storageClassName: standard

shipping: {}

user: {}

web: {}

affinities and tolerations can now be set during Helm chart installation.

I updated the docs as well for examples of how to define affinities for individual workloads.

rishinair11 commented 2 years ago

Testing

I installed the HelmChart with this values.yaml:

# Registry and rpository for Docker images
# Default is docker/robotshop/image:latest
image:
  repo: robotshop
  version: latest
  pullPolicy: IfNotPresent

# EUM configuration
# Provide your key and set the endpoint
eum:
  key: null
  url: https://eum-eu-west-1.instana.io
  #url: https://eum-us-west-2.instana.io

# Pod Security Policy
psp:
  enabled: false

# For the mini ones minikube, minishift set to true
nodeport: true

# "special" Openshift. Set to true when deploying to any openshift flavour
openshift: false

ocCreateRoute: false

payment:
  # Alternative payment gateway URL
  # Default is https://www.paypal.com
  gateway: null
  #gateway: https://www.worldpay.com
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: node-restriction.kubernetes.io/user-apps
                operator: Exists
                values: []
  tolerations:
    - key: "user-apps"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    - key: "user-apps"
      operator: "Equal"
      value: "true"
      effect: "NoExecute"
  nodeSelector: {}

redis:
  # Storage class to use with redis statefulset.
  storageClassName: standard
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: node-restriction.kubernetes.io/user-apps
                operator: Exists
                values: []
  tolerations:
    - key: "user-apps"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    - key: "user-apps"
      operator: "Equal"
      value: "true"
      effect: "NoExecute"
  nodeSelector: {}

cart:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: node-restriction.kubernetes.io/user-apps
                operator: Exists
                values: []
  tolerations:
    - key: "user-apps"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    - key: "user-apps"
      operator: "Equal"
      value: "true"
      effect: "NoExecute"
  nodeSelector: {}

catalogue:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: node-restriction.kubernetes.io/user-apps
                operator: Exists
                values: []
  tolerations:
    - key: "user-apps"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    - key: "user-apps"
      operator: "Equal"
      value: "true"
      effect: "NoExecute"
  nodeSelector: {}

dispatch:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: node-restriction.kubernetes.io/instana-backend
                operator: Exists
                values: []
  nodeSelector: {}
  tolerations: []

mongodb:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: node-restriction.kubernetes.io/instana-backend
                operator: Exists
                values: []
  nodeSelector: {}
  tolerations: []

mysql:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: node-restriction.kubernetes.io/instana-backend
                operator: Exists
                values: []
  nodeSelector: {}
  tolerations: []

rabbitmq:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: node-restriction.kubernetes.io/instana-backend
                operator: Exists
                values: []
  nodeSelector: {}
  tolerations: []

ratings:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: node-restriction.kubernetes.io/instana-backend
                operator: Exists
                values: []
  nodeSelector: {}
  tolerations: []

shipping:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: node-restriction.kubernetes.io/instana-backend
                operator: Exists
                values: []
  nodeSelector: {}
  tolerations: []

user:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: node-restriction.kubernetes.io/instana-backend
                operator: Exists
                values: []
  nodeSelector: {}
  tolerations: []

web:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: node-restriction.kubernetes.io/instana-backend
                operator: Exists
                values: []
  nodeSelector: {}
  tolerations: []

The cart, catalogue, payment and redis workloads were configured with affinities to run on the user-apps node pool and the remaining workloads were configured with affinities to run on the instana-backend node pool.

The Pods are being re-scheduled onto the separate node pools (user-apps which is tainted with the corresponding tolerations above):

% kgp -n robot-shop-2 -o wide                           
NAME                          STATUS     AGE      NODE                                          
cart-74dbbbdf76-w8kp7         Running    2m34s    gke-gke-test-rel-user-apps-2b96fe23-2gqx      
catalogue-64c4f469b5-5s55n    Running    2m34s    gke-gke-test-rel-user-apps-2b96fe23-2gqx      
dispatch-6494c5ff-j9wns       Running    2m33s    gke-gke-test-rel-instana-backend-570ba2f8-tz74
mongodb-787ccdd8f4-6ths2      Running    2m34s    gke-gke-test-rel-instana-backend-570ba2f8-tz74
mysql-5658c88467-8ccc7        Running    2m34s    gke-gke-test-rel-instana-backend-570ba2f8-tz74
payment-749fd8cd65-krrlr      Running    2m33s    gke-gke-test-rel-user-apps-2b96fe23-2gqx      
rabbitmq-686d7c6fb6-6dnl8     Running    2m33s    gke-gke-test-rel-instana-backend-570ba2f8-tz74
ratings-85654f7674-5rvvp      Running    2m32s    gke-gke-test-rel-instana-backend-570ba2f8-tz74
redis-0                       Running    2m33s    gke-gke-test-rel-user-apps-2b96fe23-2gqx      
shipping-77655b8c44-gs94q     Running    2m33s    gke-gke-test-rel-instana-backend-570ba2f8-tz74
user-f9dbccf7-rbmlh           Running    2m33s    gke-gke-test-rel-instana-backend-570ba2f8-tz74
web-589789664c-nlc4p          Running    2m34s    gke-gke-test-rel-instana-backend-570ba2f8-tz74

When I installed Robot shop using the default values.yaml file (without any affinities, tolerations or nodeSelector), the installation succeeded with the following Pod statuses:

% kgp -n robot-shop-2 -o wide
NAME                         STATUS    AGE    NODE                                          
cart-6cbffc4576-pjzmm        Running   98s    gke-gke-test-rel-user-apps-2b96fe23-2gqx      
catalogue-5dd69f5985-gjtkf   Running   100s   gke-gke-test-rel-instana-backend-570ba2f8-z1ph
dispatch-5d65c7cfcf-rks5d    Running   99s    gke-gke-test-rel-user-apps-2b96fe23-2gqx      
mongodb-76457756fb-szbb8     Running   100s   gke-gke-test-rel-instana-backend-570ba2f8-z1ph
mysql-7456d7d8d7-lt6tc       Running   100s   gke-gke-test-rel-user-apps-2b96fe23-2gqx      
payment-67d5dd4dd5-dblrk     Running   100s   gke-gke-test-rel-instana-backend-570ba2f8-z1ph
rabbitmq-75d9cf4484-fmsgp    Running   98s    gke-gke-test-rel-instana-backend-570ba2f8-z1ph
ratings-5d9dff56bd-gzlpw     Running   99s    gke-gke-test-rel-user-apps-2b96fe23-2gqx      
redis-0                      Running   99s    gke-gke-test-rel-user-apps-2b96fe23-2gqx      
shipping-65dd6fdcd6-rlkwb    Running   99s    gke-gke-test-rel-instana-backend-570ba2f8-z1ph
user-77544cc485-kqw6l        Running   100s   gke-gke-test-rel-user-apps-2b96fe23-2gqx      
web-5488d5545f-mrzlz         Running   99s    gke-gke-test-rel-instana-backend-570ba2f8-z1ph

As you can see, the pods are scheduled "randomly". CC: @teresanovielloatinstana for visibility.