Readiness/Liveness failing while healthz check on celery-celery

EnterPrayz commented 5 months ago

Describe the issue

I've split helm chart configuration to the as follow: weblate-web.yaml

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: weblate-web
  namespace: weblate
spec:
  interval: 30m
  chart:
    spec:
      chart: weblate
      version: '0.5.9'
      sourceRef:
        kind: HelmRepository
        name: weblate
        namespace: flux-system
      interval: 24h
  values:
    replicaCount: 1
    labels:
      app: "weblate-web"
      flux: "true"
      helm: "true"
    existingSecret: "weblate"
    allowedHosts: "*"
    debug: "1"
    externalSecretName: "weblate-configs"
    extraConfig:
      WEBLATE_LOGLEVEL: "DEBUG"
      WEBLATE_UPDATE_LANGUAGES: "0"
      WEBLATE_ENABLE_HTTPS: "1"
      WEBLATE_REGISTRATION_OPEN: "0"
      WEBLATE_ENABLE_SHARING: "0"
      WEBLATE_SUPPORT_STATUS_CHECK: "0"
      WEBLATE_REQUIRE_LOGIN: "0"
      WEBLATE_NO_EMAIL_AUTH: "1"
      WEBLATE_REGISTRATION_ALLOW_BACKENDS: "google-oauth2"
      WEBLATE_SOCIAL_AUTH_GOOGLE_OAUTH2_WHITELISTED_DOMAINS: "some.host"
      WEBLATE_GITLAB_HOST: "gitlab.some.host"
      WEBLATE_AUTO_UPDATE: "1"
      WEBLATE_SERVICE: "web"
    siteTitle: "My Awesome Weblate"
    siteDomain: "weblate.some.host"
    serviceAccount:
      create: false
    podSecurityContext:
      enabled: true
      fsGroup: 1000
    podAnnotations: {}
    containerSecurityContext:
      enabled: false
    service:
      type: ClusterIP
      port: 80
    ingress:
      enabled: true
      annotations:
        kubernetes.io/ingress.class: "nginx"
      hosts:
        - host: celery.some.host
          paths:
            - path: /
              pathType: Prefix
    persistence:
      enabled: true
      accessMode: ReadWriteOnce
      size: 10Gi
      filestore_dir: /app/data
    resources:
      limits:
        cpu: 500m
        memory: 1024Mi
      requests:
        cpu: 500m
        memory: 1024Mi
    nodeSelector: {}
    tolerations: []
    affinity: {}
    livenessProbe:
      initialDelaySeconds: 300
      periodSeconds: 30
      failureThreshold: 10
    readinessProbe:
      initialDelaySeconds: 60
      periodSeconds: 30
      failureThreshold: 2
    postgresql:
      enabled: false
      postgresqlHost: postgres.some.host
    redis:
      auth:
        enabled: false
      enabled: false
      redisHost: redis.some.host

weblate-celery.yaml

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: weblate-celery
  namespace: weblate
spec:
  interval: 30m
  chart:
    spec:
      chart: weblate
      version: '0.5.9'
      sourceRef:
        kind: HelmRepository
        name: weblate
        namespace: flux-system
      interval: 24h
  values:
    replicaCount: 1
    labels:
      app: "weblate-celery"
      flux: "true"
      helm: "true"
    existingSecret: "weblate"
    allowedHosts: "*"
    debug: "1"
    externalSecretName: "weblate-configs"
    extraConfig:
      WEBLATE_LOGLEVEL: "DEBUG"
      WEBLATE_UPDATE_LANGUAGES: "0"
      WEBLATE_ENABLE_HTTPS: "1"
      WEBLATE_REGISTRATION_OPEN: "0"
      WEBLATE_ENABLE_SHARING: "0"
      WEBLATE_SUPPORT_STATUS_CHECK: "0"
      WEBLATE_REQUIRE_LOGIN: "0"
      WEBLATE_NO_EMAIL_AUTH: "1"
      WEBLATE_REGISTRATION_ALLOW_BACKENDS: "google-oauth2"
      WEBLATE_SOCIAL_AUTH_GOOGLE_OAUTH2_WHITELISTED_DOMAINS: "some.host"
      WEBLATE_GITLAB_HOST: "gitlab.some.host"
      WEBLATE_AUTO_UPDATE: "1"
      WEBLATE_SERVICE: "celery-celery"
    siteTitle: "My Awesome Weblate"
    siteDomain: "weblate.some.host"
    serviceAccount:
      create: false
    podSecurityContext:
      enabled: true
      fsGroup: 1000
    podAnnotations: {}
    containerSecurityContext:
      enabled: false
    service:
      type: ClusterIP
      port: 80
    ingress:
      enabled: true
      annotations:
        kubernetes.io/ingress.class: "nginx"
      hosts:
        - host: celery.some.host
          paths:
            - path: /
              pathType: Prefix
    persistence:
      enabled: true
      existingClaim: "weblate-web"
      accessMode: ReadWriteOnce
      size: 10Gi
      filestore_dir: /app/data
    resources:
      limits:
        cpu: 700m
        memory: 1024Mi
      requests:
        cpu: 700m
        memory: 1024Mi
    nodeSelector: {}
    tolerations: []
    affinity: {}
    livenessProbe:
      initialDelaySeconds: 300
      periodSeconds: 30
      failureThreshold: 10
    readinessProbe:
      initialDelaySeconds: 60
      periodSeconds: 30
      failureThreshold: 2
    postgresql:
      enabled: false
      postgresqlHost: postgres.some.host
    redis:
      auth:
        enabled: false
      enabled: false
      redisHost: redis.some.host

But while kybernates try to check Readiness/Liveness probe healthz pod with WEBLATE_SERVICE=celery-celery , it always receive

Liveness probe failed: Get "http://{my_pod_ip}:8080/healthz/": dial tcp {my_pod_ip}:8080: connect: connection refused

In this time while checking WEBLATE_SERVICE=web- response code 200.

I know that i can use some like this in part of Probe configuration, instead of httpGet:

...
exec:
  command: ["celery", "--app=weblate.utils", "inspect", "ping"]
...

Unfortunately my flux server not support custom probe setup - only default httpGet

Also i tried to use my own http server for pod with WEBLATE_SERVICE=celery-celery, that i run inside settings-override.py - but it's not clear way.

I already tried

[X] I've read and searched the documentation.
[X] I've searched for similar filed issues in this repository.

Steps to reproduce the behavior

Setup weblate helm release chart like i described upper
Try to check pod with WEBLATE_SERVICE: "celery-celery" url {my_pod_ip}:8080/healthz/

Always getting

Liveness probe failed: Get "http://{my_pod_ip}:8080/healthz/": dial tcp {my_pod_ip}:8080: connect: connection refused

Expected behavior

No response

Screenshots

No response

Exception traceback

No response

Additional context

No response

nijel commented 5 months ago

Indeed, the liveness probe only works for the web service container. Can Kubernetes use the health check defined in the Docker container? That would allow us better flexibility in performing it. Presently it does mostly the same, see https://github.com/WeblateOrg/docker/issues/2392.

EnterPrayz commented 5 months ago

Kubernetes does not directly use the health checks defined in a Docker container image (like those specified in a Dockerfile using HEALTHCHECK). Instead, Kubernetes provides its own mechanisms for defining and managing health checks through the use of - Liveness/Readiness/Startup probes

st-myk commented 2 months ago

@nijel , Hi! What do you think about such a solution to this problem — https://github.com/WeblateOrg/helm/pull/466

nijel commented 2 months ago

That is not really a solution, it removes the health check completely. But it's good as a workaround.

WeblateOrg / helm