Liveness probe failed: HTTP probe failed with statuscode: 401

[Awsmsniper]

Describe the bug When I deploy the Pulsar cluster on the Kubernet cluster using Apache/pulsar-helm-chart: 3.2.0 (with JWT authentication enabled), the pulsar proxy service still cannot run properly. The error message is as follows： Liveness probe failed: HTTP probe failed with statuscode: 401 Readiness probe failed: HTTP probe failed with statuscode: 401

To Reproduce Steps to reproduce the behavior:

1. Installation command helm install pulsar --values charts/pulsar/values.yaml --set initialize=true --namespace pulsar pulsar-3.2.0.tgz

Screenshots kubectl get pod -n pulsar output

NAME                                     READY   STATUS      RESTARTS        AGE
pulsar-bookie-0                          1/1     Running     0               11m
pulsar-bookie-1                          1/1     Running     0               11m
pulsar-bookie-2                          1/1     Running     0               11m
pulsar-broker-0                          1/1     Running     2 (4m30s ago)   11m
pulsar-broker-1                          1/1     Running     3 (3m42s ago)   11m
pulsar-broker-2                          1/1     Running     3 (3m42s ago)   11m
pulsar-proxy-0                           0/1     Running     1 (1s ago)      11m
pulsar-proxy-1                           0/1     Running     1 (8s ago)      11m
pulsar-proxy-2                           0/1     Running     1 (9s ago)      11m
pulsar-pulsar-init-7n2jz                 0/1     Completed   0               11m
pulsar-pulsar-manager-7887f99f77-cc9q2   1/1     Running     0               11m
pulsar-recovery-0                        1/1     Running     0               11m
pulsar-toolset-0                         1/1     Running     0               11m
pulsar-zookeeper-0                       1/1     Running     0               11m
pulsar-zookeeper-1                       1/1     Running     0               11m
pulsar-zookeeper-2                       1/1     Running     0               11m

kubectl describe outpu：

Warning  Unhealthy         3s (x7 over 63s)  kubelet            Liveness probe failed: HTTP probe failed with statuscode: 401
Warning  Unhealthy         3s (x8 over 63s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 401

Desktop (please complete the following information):

• kubernetes 1.28.4
• pulsar-all 3.1.1

Modified on default basis values.yaml

namespace: "pulsar"
persistence: true
volumes:
  persistence: true
  local_storage: true

components:
  # zookeeper
  zookeeper: true
  # bookkeeper
  bookkeeper: true
  # bookkeeper - autorecovery
  autorecovery: true
  # broker
  broker: true
  # functions
  functions: true
  # proxy
  proxy: true
  # toolset
  toolset: true
  # pulsar manager
  pulsar_manager: true

# uses chart's appVersion when unspecified
defaultPulsarImageTag: 3.1.2

auth:
  authentication:
    enabled: true
    provider: "jwt"
    jwt:
      # Enable JWT authentication
      # If the token is generated by a secret key, set the usingSecretKey as true.
      # If the token is generated by a private key, set the usingSecretKey as false.
      usingSecretKey: false
  authorization:
    enabled: true
  superUsers:
    # broker to broker communication
    broker: "broker-admin"
    # proxy to broker communication
    proxy: "proxy-admin"
    # pulsar-admin client to broker/proxy communication
    client: "admin"
  # omits the above proxy role from superusers on the proxy
  # and configures it as a proxy role on the broker in addition to the superusers
  useProxyRoles: true

zookeeper:
  podMonitor: 
    enabled: false
  nodeSelector:
    node-role.kubernetes.io/pulsar: pulsar
  volumes: 
    persistence: true
    data:
      name: data
      size: 100Gi
      local_storage: true

bookkeeper:
  podMonitor: 
    enabled: false
  nodeSelector:
    node-role.kubernetes.io/pulsar: pulsar
  volumes:
    persistence: true
    journal:
      name: journal
      size: 100Gi
      local_storage: true
    ledgers:
      name: ledgers
      size: 100Gi
      local_storage: true

autorecovery:
  podMonitor: 
    enabled: false
  nodeSelector:
    node-role.kubernetes.io/pulsar: pulsar

broker:
  podMonitor: 
    enabled: false
  nodeSelector:
    node-role.kubernetes.io/pulsar: pulsar

functions:
  component: functions-worker
  useBookieAsStateStore: true

proxy:
  podMonitor: 
    enabled: false
  nodeSelector:
    node-role.kubernetes.io/pulsar: pulsar

dashboard:
  nodeSelector:
    node-role.kubernetes.io/pulsar: pulsar

toolset:
  nodeSelector:
    node-role.kubernetes.io/pulsar: pulsar

# 关闭监控组件  
kube-prometheus-stack:
  enabled: false

pulsar_manager:
  nodeSelector:
    node-role.kubernetes.io/pulsar: pulsar
  admin:
    user: pulsar_manager
    password: zR7yF5fH
  configData:
    REDIRECT_HOST: "http://127.0.0.1"
    REDIRECT_PORT: "9527"
    DRIVER_CLASS_NAME: org.postgresql.Driver
    URL: jdbc:postgresql://postgresql-ha-pgpool:5432/pulsar_manager
    LOG_LEVEL: DEBUG
    JWT_TOKEN: <token>

[lhotari]

have you tested with the default image which is 3.0.2? (remove defaultPulsarImageTag)

[Awsmsniper]

have you tested with the default image which is 3.0.2? (remove defaultPulsarImageTag)

No, I am using version 3.1.1, which defaults to version 3.1.1

[lhotari]

have you tested with the default image which is 3.0.2? (remove defaultPulsarImageTag)

No, I am using version 3.1.1, which defaults to version 3.1.1

There's a bug in 3.1.1 version of Pulsar. It is fixed in 3.0.2 and 3.1.2 versions of Pulsar. It's this fix: https://github.com/apache/pulsar/pull/21428 . Commit in branch-3.0: https://github.com/apache/pulsar/commit/df4b0d8f (included in 3.0.2) and branch-3.1: https://github.com/apache/pulsar/commit/1496925f (included in 3.1.2).

The CI tests run with Pulsar 3.0.2 images. I have also ran the tests with defaultPulsarImageTag: 3.1.2 and the problem doesn't occur. https://github.com/lhotari/pulsar-helm-chart/actions/runs/7670897358?pr=7

Please use Pulsar 3.0.2 or 3.1.2 to get the fix.

[lhotari]

Duplicates #403

[Awsmsniper]

have you tested with the default image which is 3.0.2? (remove defaultPulsarImageTag)

No, I am using version 3.1.1, which defaults to version 3.1.1

Pulsar 3.1.1 版本中存在一个错误。它在 Pulsar 的 3.0.2 和 3.1.2 版本中已修复。这是这个修复程序： apache/pulsar#21428 。在 branch-3.0 中提交：apache/pulsar@df4b0d8f（包含在 3.0.2 中）和 branch-3.1：apache/pulsar@1496925f（包含在 3.1.2 中）。

CI 测试使用 Pulsar 3.0.2 镜像运行。我也运行了测试，没有出现问题。https://github.com/lhotari/pulsar-helm-chart/actions/runs/7670897358?pr=7`defaultPulsarImageTag: 3.1.2`

请使用 Pulsar 3.0.2 或 3.1.2 来获取修复程序。

Thank you for your reply. Could you please take a look at the configuration of jwt in your values. yaml? Here is my configuration, is there any error?

# uses chart's appVersion when unspecified
defaultPulsarImageTag: 3.1.2

auth:
  authentication:
    enabled: true
    provider: "jwt"
    jwt:
      # Enable JWT authentication
      # If the token is generated by a secret key, set the usingSecretKey as true.
      # If the token is generated by a private key, set the usingSecretKey as false.
      usingSecretKey: false
  authorization:
    enabled: true
  superUsers:
    # broker to broker communication
    broker: "broker-admin"
    # proxy to broker communication
    proxy: "proxy-admin"
    # pulsar-admin client to broker/proxy communication
    client: "admin"
  # omits the above proxy role from superusers on the proxy
  # and configures it as a proxy role on the broker in addition to the superusers
  useProxyRoles: true

[lhotari]

Thank you for your reply. Could you please take a look at the configuration of jwt in your values. yaml? Here is my configuration, is there any error?

Have you run the prepare step? https://pulsar.apache.org/docs/3.1.x/helm-deploy/#prepare-helm-release It's this script https://github.com/apache/pulsar-helm-chart/blob/master/scripts/pulsar/prepare_helm_release.sh . It creates the required JWT key and tokens as secrets.

[Awsmsniper]

Thank you for your reply. Could you please take a look at the configuration of jwt in your values. yaml? Here is my configuration, is there any error?

Have you run the prepare step? https://pulsar.apache.org/docs/3.1.x/helm-deploy/#prepare-helm-release It's this script https://github.com/apache/pulsar-helm-chart/blob/master/scripts/pulsar/prepare_helm_release.sh . It creates the required JWT key and tokens as secrets.

I installed the pulsar cluster in an offline environment, so I manually installed the secret. kubectl apply -f pulsar-token-admin.yaml -f pulsar-token-asymmetric-key.yaml -f pulsar-token-broker-admin.yaml -f pulsar-token-proxy-admin.yaml

pulsar-token-admin            Opaque   2      14s
pulsar-token-asymmetric-key   Opaque   2      14s
pulsar-token-broker-admin     Opaque   2      14s
pulsar-token-proxy-admin      Opaque   2      14s

[lhotari]

Please use Pulsar 3.0.2 or 3.1.2 to get the fix. Did you do that? If you did, please share the logs of the failing pods. Please also share kubectl describe for one of the pods.

[Awsmsniper]

Please use Pulsar 3.0.2 or 3.1.2 to get the fix. Did you do that? If you did, please share the logs of the failing pods. Please also share kubectl describe for one of the pods.

Thank you very much! This problem has been bothering me for a while! When I switch to 3.1.2, it can run normally! Also, I would like to enable TLS. Where can I find detailed reference documents?