search

apache / pulsar-manager

Apache Pulsar Manager
https://pulsar.apache.org/
Apache License 2.0
521 stars 246 forks source link

pulsar manager init Failed to connect to pulsar #577

Closed erictarrence closed 1 month ago

erictarrence commented 1 month ago

Version

pulsar version: 3.0.6 kubernetes version:1.30.0 pulsar auth mode:JWT

Minimal reproduce step

git clone https://github.com/apache/pulsar-helm-chart cd pulsar-helm-chart kubectl create ns pulsar ./scripts/pulsar/prepare_helm_release.sh -n pulsar -k pulsar --symmetric

helm repo add apache https://pulsar.apache.org/charts helm repo update Install pulsar using the attached Helm values.yaml to reproduce the failure pulsar-3.0.6-values.txt

What did you expect to see?

I hope that the pulsar manager can access the pulsar cluster normally

What did you see instead?

pulsar-pulsar-manager-init pod is still CrashLoopBackOff status, manager log display :

2024-10-14 02:54:23.223 ERROR 2662 --- [http-nio-7750-exec-11] o.a.p.m.c.EnvironmentsController         : Failed to get clusters list.

org.apache.pulsar.client.admin.PulsarAdminException$NotAuthorizedException: HTTP 401 Unauthorized
# kubectl -n pulsar get job
NAME                         STATUS     COMPLETIONS   DURATION   AGE
pulsar-bookie-init           Complete   1/1           38s        71m
pulsar-pulsar-init           Complete   1/1           46s        71m
pulsar-pulsar-manager-init   Failed     0/1           71m        71m
kubectl  -n pulsar logs pulsar-pulsar-manager-init-lb2wk
Defaulted container "pulsar-pulsar-manager-init" out of: pulsar-pulsar-manager-init, wait-pulsar-manager-ready (init), wait-broker-ready (init)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    36  100    36    0     0    990      0 --:--:-- --:--:-- --:--:--  1000
*   Trying 172.23.5.35:9527...
* Connected to pulsar-pulsar-manager (172.23.5.35) port 9527 (#0)
> POST /pulsar-manager/login HTTP/1.1
> Host: pulsar-pulsar-manager:9527
> User-Agent: curl/7.81.0
> Accept: application/json, text/plain, */*
> Content-Type: application/json
> X-XSRF-TOKEN: 072e42c8-0024-4943-8397-d0d6c4c0bd60
> Cookie: XSRF-TOKEN=072e42c8-0024-4943-8397-d0d6c4c0bd60
> Content-Length: 49
> 
} [49 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 
< Server: nginx/1.18.0
< Date: Mon, 14 Oct 2024 02:38:09 GMT
< Content-Type: application/json;charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-Frame-Options: DENY
< 
{ [44 bytes data]
* Connection #0 to host pulsar-pulsar-manager left intact
{"error":"The user is not exist"}
creating account
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 172.23.6.198:7750...
* Connected to pulsar-pulsar-manager-admin (172.23.6.198) port 7750 (#0)
> PUT /pulsar-manager/users/superuser HTTP/1.1
> Host: pulsar-pulsar-manager-admin:7750
> User-Agent: curl/7.81.0
> Accept: */*
> X-XSRF-TOKEN: 072e42c8-0024-4943-8397-d0d6c4c0bd60
> Cookie: XSRF-TOKEN=072e42c8-0024-4943-8397-d0d6c4c0bd60;
> Content-Type: application/json
> Content-Length: 120
> 
} [120 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-Frame-Options: DENY
< Content-Type: application/json;charset=UTF-8
< Transfer-Encoding: chunked
< Date: Mon, 14 Oct 2024 02:38:08 GMT
< 
{ [56 bytes data]
100   170    0    50  100   120    639   1533 --:--:-- --:--:-- --:--:--  2179
* Connection #0 to host pulsar-pulsar-manager-admin left intact
*   Trying 172.23.5.35:9527...
* Connected to pulsar-pulsar-manager (172.23.5.35) port 9527 (#0)
> POST /pulsar-manager/login HTTP/1.1
> Host: pulsar-pulsar-manager:9527
> User-Agent: curl/7.81.0
> Accept: application/json, text/plain, */*
> Content-Type: application/json
> X-XSRF-TOKEN: 072e42c8-0024-4943-8397-d0d6c4c0bd60
> Cookie: XSRF-TOKEN=072e42c8-0024-4943-8397-d0d6c4c0bd60
> Content-Length: 49
> 
} [49 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 
< Server: nginx/1.18.0
< Date: Mon, 14 Oct 2024 02:38:09 GMT
< Content-Type: application/json;charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-Frame-Options: DENY
< 
{ [44 bytes data]
* Connection #0 to host pulsar-pulsar-manager left intact
{"message":"Add super user success, please login"}{"error":"The user is not exist"}
{ "name": "pulsar", "broker": "http://pulsar-broker:8080", "bookie": "http://pulsar-bookie:8000"}
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 172.23.5.35:9527...
* Connected to pulsar-pulsar-manager (172.23.5.35) port 9527 (#0)
> PUT /pulsar-manager/environments/environment HTTP/1.1
> Host: pulsar-pulsar-manager:9527
> User-Agent: curl/7.81.0
> Accept: */*
> Content-Type: application/json
> X-XSRF-TOKEN: 072e42c8-0024-4943-8397-d0d6c4c0bd60
> username: pulsar
> Cookie: XSRF-TOKEN=072e42c8-0024-4943-8397-d0d6c4c0bd60; JSESSIONID=;
> Content-Length: 97
> 
} [97 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 
< Server: nginx/1.18.0
< Date: Mon, 14 Oct 2024 02:38:09 GMT
< Content-Length: 27
< Connection: keep-alive
< Set-Cookie: JSESSIONID=255AC3E75D5F3A9CE079A9C91FD0BE80; Path=/; HttpOnly
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-Frame-Options: DENY
< 
{ [27 bytes data]
100   124  100    27  100    97    992   3567 --:--:-- --:--:-- --:--:--  4592
* Connection #0 to host pulsar-pulsar-manager left intact
{"message":"Please login."}
Error creating environment

pulsar_manager configuration is as follows:

pulsar_manager: # 

  replicaCount: 1
  nodeSelector:
    ceph-rbd-plug: enabled
  tolerations:
  - effect: NoExecute
    key: ceph-taint
    operator: Equal
    value: osd
  volumes:
    # use a persistent volume or emptyDir
    persistence: true
    data:
      name: data
      size: 128Mi
      storageClassName: rook-ceph-block
  configData:
    REDIRECT_HOST: "http://127.0.0.1"
    REDIRECT_PORT: "9527"
    LOG_LEVEL: "INFO"
    # DB
    URL: "jdbc:postgresql://bitnami-postgresql-postgresql-ha-pgpool.postgresql.svc.cluster.local:5432/pulsar_manager"
    DRIVER_CLASS_NAME: "org.postgresql.Driver"
    # enables the "message peeking" feature
    PULSAR_PEEK_MESSAGE: "false"
  admin:
    ## Setting a value at existingSecret disables automatic creation of the secret for pulsar_manager admin credentials and instead uses an existing secret to initialize pulsar-manager
    ## The existing secret should have the following keys:
    ## DB_PASSWORD: <database password>
    ## DB_USERNAME: <database username>
    ## UI_PASSWORD: <UI password>
    ## UI_USERNAME: <UI username>
    #existingSecret: ""
    ui_username: "pulsar"
    ui_password: "pulsaradmin"  # leave empty for random password
    db_username: "pulsar"
    db_password: "pulsar"  # leave empty for random password
alpreu commented 1 month ago 
{ [44 bytes data]
* Connection #0 to host pulsar-pulsar-manager left intact
{"message":"Add super user success, please login"}{"error":"The user is not exist"}

Looks strange, it seems the user creation is still successful but then the login does not work. What happens if you manually retry the login command using curl?

erictarrence commented 1 month ago 
{ [44 bytes data]
* Connection #0 to host pulsar-pulsar-manager left intact
{"message":"Add super user success, please login"}{"error":"The user is not exist"}

Looks strange, it seems the user creation is still successful but then the login does not work. What happens if you manually retry the login command using curl?

I failed to retry manually

It worked after manually adding secrets pulsar-token-manager-admin, thank you