search

passbolt / charts-passbolt

Helm charts to run Passbolt on Kubernetes. No strings attached charts to run the open source password manager for teams!
https://passbolt.com
GNU Affero General Public License v3.0
40 stars 27 forks source link

haproxy containers using most of their node's CPU #75

Closed AlexandreRatte closed 6 months ago

AlexandreRatte commented 6 months ago

Hello, I wanted to test the new PostgreSQL database backend and noticed the depl-srv pods using up most of its scheduled node's available CPU resources. The culprit is in fact the haproxy container:

user@kube01:~$ sudo kubectl top pod -n passbolt-postgres passbolt-prod-depl-srv-7ffdf5c686-t6vf5 --containers
POD                                       NAME                                CPU(cores)   MEMORY(bytes)
passbolt-prod-depl-srv-7ffdf5c686-t6vf5   passbolt-prod-depl-srv              40m          80Mi
passbolt-prod-depl-srv-7ffdf5c686-t6vf5   passbolt-prod-depl-srv-redisproxy   7943m        12Mi

user@kube01:~$ sudo kubectl get pod -n passbolt-postgres
NAME                                           READY   STATUS              RESTARTS      AGE
passbolt-prod-cron-proc-email-28414136-8srvb   0/2     Completed           0             3m17s
passbolt-prod-cron-proc-email-28414137-ncx77   0/2     Completed           0             2m17s
passbolt-prod-cron-proc-email-28414138-9gkhx   0/2     Completed           0             77s
passbolt-prod-cron-proc-email-28414139-nmzt2   0/2     ContainerCreating   0             17s
passbolt-prod-depl-srv-7ffdf5c686-dxkrt        2/2     Running             1 (19m ago)   22m
passbolt-prod-depl-srv-7ffdf5c686-t6vf5        2/2     Running             0             22m
passbolt-prod-job-create-gpg-keys-7jbnh        0/1     Completed           0             23m
passbolt-prod-job-create-jwt-keys-rcng8        0/1     Completed           0             23m
passbolt-prod-postgresql-0                     1/1     Running             0             22m
passbolt-prod-redis-node-0                     2/2     Running             0             22m
passbolt-prod-redis-node-1                     2/2     Running             0             21m
passbolt-prod-redis-node-2                     2/2     Running             0             20m

The container's logs is not outputting errors:

user@kube01:~$ sudo kubectl logs -n passbolt-postgres passbolt-prod-depl-srv-7ffdf5c686-t6vf5 --container='passbolt-prod-depl-srv-redisproxy'
[NOTICE]   (1) : New worker (8) forked
[NOTICE]   (1) : Loading success.
Connect from 127.0.0.1:38186 to 127.0.0.1:6379 (ft_redis_master/TCP)
Server bk_redis_master/R1 is DOWN, reason: Layer7 timeout, info: " at step 6 of tcp-check (expect string 'role:master')", check duration: 2002ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING]  (8) : Server bk_redis_master/R1 is DOWN, reason: Layer7 timeout, info: " at step 6 of tcp-check (expect string 'role:master')", check duration: 2002ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Connect from 127.0.0.1:38190 to 127.0.0.1:6379 (ft_redis_master/TCP)
[WARNING]  (8) : Server bk_redis_master/R2 is DOWN, reason: Layer7 timeout, info: " at step 6 of tcp-check (expect string 'role:master')", check duration: 2001ms. 1 active and 0 backup servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
Server bk_redis_master/R2 is DOWN, reason: Layer7 timeout, info: " at step 6 of tcp-check (expect string 'role:master')", check duration: 2001ms. 1 active and 0 backup servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
Connect from 127.0.0.1:38240 to 127.0.0.1:6379 (ft_redis_master/TCP)
Connect from 127.0.0.1:38244 to 127.0.0.1:6379 (ft_redis_master/TCP)

This behavior can be seen with versions 0.7.1 and 0.5.0, but I haven't tried the versions in between.

Here is the values.yaml file I used to deploy the Passbolt chart:

values.yaml contents ```YAML # This is a YAML-formatted file. # Declare variables to be passed into your templates. ## Dependencies configuration parameters ## Redis dependency parameters # -- Install redis as a depending chart redisDependencyEnabled: true # -- Install mariadb as a depending chart mariadbDependencyEnabled: false # -- Install mariadb as a depending chart postgresqlDependencyEnabled: true # Configure postgresql as a dependency chart postgresql: auth: # -- Configure postgresql auth username username: scXGZ6Rh4XXJB5PixR2h9xhOAi1LV2a6 # -- Configure postgresql auth password password: xx2Cfh9iOxkYBRzDiRaVHavamuTLyaXn # -- Configure postgresql auth database database: passbolt global: imageRegistry: "" imagePullSecrets: [] # Configure redis dependency chart redis: auth: # -- Enable redis authentication enabled: true # -- Configure redis password password: "LY0uawmR39L5hCK3aRw2AHgaZdsRYUd" sentinel: # -- Enable redis sentinel enabled: true ## MariaDB dependency parameters # Configure mariadb as a dependency chart mariadb: # -- Configure mariadb architecture architecture: replication auth: # -- Configure mariadb auth root password rootPassword: HY0naOzVXZNPyVkAv0aAkcsLF0TE8jqN # -- Configure mariadb auth username username: scXGZ6Rh4XXJB5PixR2h9xhOAi1LV2a6 # -- Configure mariadb auth password password: xx2Cfh9iOxkYBRzDiRaVHavamuTLyaXn # -- Configure mariadb auth database database: passbolt # -- Configure mariadb auth replicationPassword replicationPassword: BudQvhZilaSZMuTgqdXBmHpjCUNowIKo # -- Configure parameters for the primary instance. primary: # -- Configure persistence options. persistence: # -- Enable persistence on MariaDB primary replicas using a `PersistentVolumeClaim`. If false, use emptyDir enabled: true # -- Name of an existing `PersistentVolumeClaim` for MariaDB primary replicas. When it's set the rest of persistence parameters are ignored. existingClaim: "" # -- Subdirectory of the volume to mount at subPath: "" # -- Primary persistent volume storage Class storageClass: "rook-ceph-block" # -- Labels for the PVC labels: {} # -- Primary persistent volume claim annotations annotations: {} # -- Primary persistent volume access Modes accessModes: - ReadWriteOnce # -- Primary persistent volume size size: 8Gi # -- Selector to match an existing Persistent Volume selector: {} # -- Configure parameters for the secondary instance. secondary: # -- Configure persistence options. persistence: # -- Enable persistence on MariaDB secondary replicas using a `PersistentVolumeClaim`. If false, use emptyDir enabled: true # -- Subdirectory of the volume to mount at subPath: "" # -- Secondary persistent volume storage Class storageClass: "rook-ceph-block" # -- Labels for the PVC labels: {} # -- Secondary persistent volume claim annotations annotations: {} # -- Secondary persistent volume access Modes accessModes: - ReadWriteOnce # -- Secondary persistent volume size size: 8Gi # -- Selector to match an existing Persistent Volume selector: {} ## Passbolt configuration ## Passbolt container and sidecar parameters app: # -- Configure pasbolt deployment init container that waits for database databaseInitContainer: # -- Toggle pasbolt deployment init container that waits for database enabled: true # initImage: # # -- Configure pasbolt deployment init container image client for database # client: mariadb # registry: "" # # -- Configure pasbolt deployment image repsitory # repository: mariadb # # -- Configure pasbolt deployment image pullPolicy # pullPolicy: IfNotPresent # # -- Overrides the image tag whose default is the chart appVersion. # tag: latest image: # -- Configure pasbolt deployment image repsitory registry: "" repository: passbolt/passbolt # -- Configure pasbolt deployment image pullPolicy pullPolicy: IfNotPresent # -- Overrides the image tag whose default is the chart appVersion. tag: 4.4.2-1-ce # Allowed options: mariadb, mysql or postgresql database: kind: postgresql cache: # Use CACHE_CAKE_DEFAULT_* variables to configure the connection to redis instance # on the passboltEnv configuration section redis: # -- By enabling redis the chart will mount a configuration file on /etc/passbolt/app.php # That instructs passbolt to store sessions on redis and to use it as a general cache. enabled: true sentinelProxy: # -- Inject a haproxy sidecar container configured as a proxy to redis sentinel # Make sure that CACHE_CAKE_DEFAULT_SERVER is set to '127.0.0.1' to use the proxy enabled: true # -- Configure redis sentinel proxy image image: registry: "" # -- Configure redis sentinel image repository repository: haproxy # -- Configure redis sentinel image tag tag: "latest" # -- Configure redis sentinel container resources resources: {} # -- Configure the passbolt deployment resources extraPodLabels: {} resources: requests: cpu: "500m" memory: "500Mi" limits: memory: "1500Mi" # -- Enable email cron cronJobEmail: enabled: true schedule: "* * * * *" extraPodLabels: {} ## Passbolt environment parameters # -- Pro subscription key in base64 only if you are using pro version # subscriptionKey: # -- Configure passbolt subscription key path # subscription_keyPath: /etc/passbolt/subscription_key.txt # -- Configure passbolt gpg directory gpgPath: /etc/passbolt/gpg # -- Gpg server private key in base64 gpgServerKeyPrivate: "" # -- Gpg server public key in base64 gpgServerKeyPublic: "" # -- Name of the existing secret for the GPG server keypair. The secret must contain the `serverkey.asc` and `serverkey_private.asc` keys. gpgExistingSecret: "" # -- Name of the existing secret for the JWT server keypair. The secret must contain the `jwt.key` and `jwt.pem` keys. jwtExistingSecret: "" # -- Configure passbolt jwt directory jwtPath: /etc/passbolt/jwt # -- JWT server private key in base64 jwtServerPrivate: "" # -- JWT server public key in base64 jwtServerPublic: "" # -- Forces overwrite JWT keys jwtCreateKeysForced: false jobCreateJwtKeys: extraPodLabels: {} jobCreateGpgKeys: extraPodLabels: {} passboltEnv: plain: # -- Configure passbolt privacy url PASSBOLT_LEGAL_PRIVACYPOLICYURL: https://www.passbolt.com/privacy # -- Configure passbolt fullBaseUrl APP_FULL_BASE_URL: https://passbolt.local # -- Configure passbolt to force ssl PASSBOLT_SSL_FORCE: true # -- Toggle passbolt public registration PASSBOLT_REGISTRATION_PUBLIC: true # -- Configure passbolt cake cache server CACHE_CAKE_DEFAULT_SERVER: 127.0.0.1 # -- Configure passbolt default email service port EMAIL_TRANSPORT_DEFAULT_PORT: 587 # -- Toggle passbolt debug mode DEBUG: false # -- Configure email used on gpg key. This is used when automatically creating a new gpg server key and when automatically calculating the fingerprint. PASSBOLT_KEY_EMAIL: passbolt@yourdomain.com # -- Toggle passbolt selenium mode PASSBOLT_SELENIUM_ACTIVE: false # -- Configure passbolt license path PASSBOLT_PLUGINS_LICENSE_LICENSE: /etc/passbolt/subscription_key.txt # -- Configure passbolt default email from EMAIL_DEFAULT_FROM: no-reply@passbolt.local # -- Configure passbolt default email from name EMAIL_DEFAULT_FROM_NAME: Passbolt # -- Configure passbolt default email host EMAIL_TRANSPORT_DEFAULT_HOST: 127.0.0.1 # -- Configure passbolt default email timeout EMAIL_TRANSPORT_DEFAULT_TIMEOUT: 30 # -- Toggle passbolt tls EMAIL_TRANSPORT_DEFAULT_TLS: true # -- Configure passbolt jwt private key path PASSBOLT_JWT_SERVER_KEY: /var/www/passbolt/config/jwt/jwt.key # -- Configure passbolt jwt public key path PASSBOLT_JWT_SERVER_PEM: /var/www/passbolt/config/jwt/jwt.pem # -- Toggle passbolt jwt authentication PASSBOLT_PLUGINS_JWT_AUTHENTICATION_ENABLED: true # -- Download Command for kubectl KUBECTL_DOWNLOAD_CMD: curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" secret: # -- Configure passbolt cake cache password CACHE_CAKE_DEFAULT_PASSWORD: LY0uawmR39L5hCK3aRw2AHgaZdsRYUd # -- Configure passbolt default database password DATASOURCES_DEFAULT_PASSWORD: xx2Cfh9iOxkYBRzDiRaVHavamuTLyaXn # -- Configure passbolt default database username DATASOURCES_DEFAULT_USERNAME: scXGZ6Rh4XXJB5PixR2h9xhOAi1LV2a6 # -- Configure passbolt default database DATASOURCES_DEFAULT_DATABASE: passbolt # -- Configure passbolt default email service username EMAIL_TRANSPORT_DEFAULT_USERNAME: CHANGEME # -- Configure passbolt default email service password EMAIL_TRANSPORT_DEFAULT_PASSWORD: CHANGEME # -- Configure passbolt server gpg key fingerprint # PASSBOLT_GPG_SERVER_KEY_FINGERPRINT: # -- Configure passbolt security salt. # SECURITY_SALT: # -- Environment variables to add to the passbolt pods extraEnv: [] # -- Environment variables from secrets or configmaps to add to the passbolt pods extraEnvFrom: [] # - secretRef: # name: passbolt-secret ## Passbolt deployment parameters # -- If autoscaling is disabled this will define the number of pods to run replicaCount: 2 # Configure autoscaling on passbolt deployment autoscaling: # -- Enable autoscaling on passbolt deployment enabled: false # -- Configure autoscaling minimum replicas minReplicas: 1 # -- Configure autoscaling maximum replicas maxReplicas: 100 # -- Configure autoscaling target CPU uptilization percentage targetCPUUtilizationPercentage: 80 # targetMemoryUtilizationPercentage: 80 # -- Enable role based access control rbacEnabled: true # -- Configure passbolt container livenessProbe livenessProbe: # @ignore httpGet: port: https scheme: HTTPS path: /healthcheck/status.json httpHeaders: - name: Host value: passbolt.local initialDelaySeconds: 20 periodSeconds: 10 # -- Configure passbolt container RadinessProbe readinessProbe: # @ignore httpGet: port: https scheme: HTTPS httpHeaders: - name: Host value: passbolt.local path: /healthcheck/status.json initialDelaySeconds: 5 periodSeconds: 10 # Configure network policies to allow ingress access passbolt pods # networkPolicy defines which labels are allowed to reach to passbolt # and which namespaces networkPolicy: # -- Enable network policies to allow ingress access passbolt pods enabled: false # -- Configure network policies label for ingress deployment label: app.kubernetes.io/name # -- Configure network policies podLabel for podSelector podLabel: ingress-nginx # -- Configure network policies namespaceLabel for namespaceSelector namespaceLabel: ingress-nginx # -- Configure image pull secrets imagePullSecrets: [] # -- Value to override the chart name on default nameOverride: "" # -- Value to override the whole fullName fullnameOverride: "" serviceAccount: # -- Specifies whether a service account should be created create: true # -- Annotations to add to the service account annotations: {} # -- Map of annotation for passbolt server pod podAnnotations: {} # -- Security Context configuration for passbolt server pod podSecurityContext: {} # fsGroup: 2000 service: # -- Configure passbolt service type type: ClusterIP # -- Configure passbolt service port port: 443 # -- Configure passbolt service targetPort targetPort: 443 # -- Configure passbolt service port name name: https # -- Annotations to add to the service annotations: {} tls: # -- Generates a secret with a self-signed cerfificate that is injected on ingress and passbolt container autogenerate: true # -- Name of an existing kubernetes secret that contains a SSL certificate to inject on ingress and passbolt container #existingSecret: "" ingress: # -- Enable passbolt ingress enabled: false # -- Configure passbolt ingress annotations annotations: {} # -- Configure passbolt ingress hosts hosts: # @ignored - host: passbolt.local paths: - path: / pathType: ImplementationSpecific # -- Configure passbolt ingress tls tls: # If secretname is not empty, the tls entry will use it, otherwise will # have a default name based on the release # @ignored - secretName: "tls" hosts: - passbolt.local # -- Configure passbolt deployment nodeSelector nodeSelector: {} # -- Configure passbolt deployment tolerations tolerations: [] # -- Configure passbolt deployment affinity affinity: {} # -- Add additional volumes, e.g. for overwriting config files extraVolumes: "" # -- Add additional volume mounts, e.g. for overwriting config files extraVolumeMounts: "" ```
dlen commented 6 months ago

Hey @AlexandreRatte thanks for the detailed report I managed to reproduce with your values. Indeed haproxy is consuming a lot of CPU. After a bit of research it looks like a bug that should be patched in 2.9.2 (latest version now is 2.9.1-f72603c).

I managed to workaround it by editing the haproxy.cfg secret and adding the following in the global section: tune.disable-zero-copy-forwarding

More information here: https://github.com/haproxy/haproxy/issues/2402

Note: If you want to edit directly the configuration on the cluster you should edit the -passbolt-sec-redis-proxy

AlexandreRatte commented 6 months ago

Hi @dlen, this additional line did the trick! Thank you.