search

helm / charts

⚠️(OBSOLETE) Curated applications for Kubernetes
Apache License 2.0
15.49k stars 16.79k forks source link

[stable/rabbitmq] endpoints does not exist #14893

Closed rnkhouse closed 5 years ago

rnkhouse commented 5 years ago

Describe the bug I installed helm chart by using this command:

helm install stable/rabbitmq --name rabbitmq --set rabbitmq.username="admin" --set rabbitmq.password="admin" --set rabbitmq.erlangCookie=secretcookie --set rbacEnabled=true --set service.type=ClusterIP --set ingress.enabled=true --set ingress.hostName="rabbitmq.example.com" --set ingress.tls=true --set ingress.tlsSecret="rabbitmq-tls" --set ingress.annotations."kubernetes.io/ingress.class"="nginx" --set resources.limits.memory="256Mi" --set resources.limits.cpu="100m"

When I tried to open hostname it's giving me 503 (service temporarily unavailable) error.

In kubectl get ep, there is no ip address for rabbitmq service.

In kubectl describe pods/rabbitmq-0 => Readiness probe failed: curl: (22) The requested URL returned error: 401, Liveness probe failed: curl: (22) The requested URL returned error: 401, Readiness probe failed: curl: (7) Failed to connect to 127.0.0.1 port 15672: Connection refused

Please advise, what went wrong here? Service should run on given hostname with https. But, it's not working.

Version of Helm and Kubernetes: helm version Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"} Server: &version.Version{SemVer:"v2.14.1", GitCommit:"5270352a09c7e8b6e8c9593002a73535276507c0", GitTreeState:"clean"}

kubectl version Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:13:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"windows/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.8", GitCommit:"a89f8c11a5f4f132503edbc4918c98518fd504e3", GitTreeState:"clean", BuildDate:"2019-04-23T04:41:47Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

Which chart: stable/rabbitmq

miguelaeh commented 5 years ago

Hi @rnkhouse , I perform some test with your command and the problem seems to be the resources limit. With you command the readinessProbe fails but without the resource limits it works. Probably that resources are not enought. Try with this: 

helm install stable/rabbitmq --name rabbitmq --set rabbitmq.username="admin" --set rabbitmq.password="admin" --set rabbitmq.erlangCookie=secretcookie --set rbacEnabled=true --set service.type=ClusterIP --set ingress.enabled=true --set ingress.hostName="rabbitmq.example.com" --set ingress.tls=true --set ingress.tlsSecret="rabbitmq-tls" --set ingress.annotations."kubernetes.io/ingress.class"="nginx"
rnkhouse commented 5 years ago

@miguelaeh : I am still getting the same error.

miguelaeh commented 5 years ago

Hi @rnkhouse , What cluster are you using? I tested it on a GKE cluster. You are getting a 503 error because the readiness probe failed. In my case the command kubectl get ep returns this:

rabbitmq                                                10.30.9.90:25672,10.30.9.90:4369,10.30.9.90:5672 + 1 more...   5m21s

I can reproduce the issue if I change the resources in the install command:

rabbitmq                                                                                                           36s
rabbitmq-headless                                                                                                  36s

As you can see there are no IPs there.

Does the Chart work on your environment installing it with:

helm install stable/rabbitmq

?

rnkhouse commented 5 years ago

@miguelaeh I also tried with helm install stable/rabbitmq endpoints are blank and 503 error. I am using AKS. It was working fine before, suddenly it stopped working I don't know why!

miguelaeh commented 5 years ago

Could it be possible you changed some configuration on your cluster? Also, check the status of all the components:

$ kubectl get pods
$ kubectl describe pod <pod-name>
$ kubectl get services
$ kubectl describe service <service-name>
$ kubectl get deployments 
$ kubectl describe deployment <deployment-name>
$ kubectl get replicasets
$ kubectl describe replicaset <replicaset-name>

Going component by component it should give more info about what is happening.

rnkhouse commented 5 years ago

@miguelaeh It's a default cluster deployed from Azure Portal.

miguelaeh commented 5 years ago

Could you paste the output of the commands I wrote in my last reply please?

rnkhouse commented 5 years ago

@miguelaeh I ran this command: helm install stable/rabbitmq --name rabbitmq

kubectl get pods
rabbitmq-0                                               0/1       Running            0          1m

kubectl describe pods rabbitmq-0
Name:               rabbitmq-0
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               aks-agentpool-17141372-0/10.240.0.4
Start Time:         Wed, 26 Jun 2019 10:11:57 -0400
Labels:             app=rabbitmq
            chart=rabbitmq-5.5.0
            controller-revision-hash=rabbitmq-5759b675c9
            release=rabbitmq
            statefulset.kubernetes.io/pod-name=rabbitmq-0
Annotations:        <none>
Status:             Running
IP:                 10.240.0.31
Controlled By:      StatefulSet/rabbitmq
Containers:
  rabbitmq:
    Container ID:  docker://9556b28eb12145ca946bad148b128681f6e09580c2cc4bd6181857a84834a3ce
    Image:         docker.io/bitnami/rabbitmq:3.7.14
    Image ID:      docker-pullable://bitnami/rabbitmq@sha256:dac5e88f7d0f5887ffc578f4356cbbd82852cd5435c8fc9edf11b070e6fab8e9
    Ports:         4369/TCP, 5672/TCP, 25672/TCP, 15672/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP
    Command:
      bash
      -ec
      mkdir -p /opt/bitnami/rabbitmq/.rabbitmq/
mkdir -p /opt/bitnami/rabbitmq/etc/rabbitmq/
#persist the erlang cookie in both places for server and cli tools
echo $RABBITMQ_ERL_COOKIE > /opt/bitnami/rabbitmq/var/lib/rabbitmq/.erlang.cookie
cp /opt/bitnami/rabbitmq/var/lib/rabbitmq/.erlang.cookie /opt/bitnami/rabbitmq/.rabbitmq/
#change permission so only the user has access to the cookie file
chmod 600 /opt/bitnami/rabbitmq/.rabbitmq/.erlang.cookie /opt/bitnami/rabbitmq/var/lib/rabbitmq/.erlang.cookie
#copy the mounted configuration to both places
cp  /opt/bitnami/rabbitmq/conf/* /opt/bitnami/rabbitmq/etc/rabbitmq
# Apply resources limits
ulimit -n "${RABBITMQ_ULIMIT_NOFILES}"
#replace the default password that is generated
sed -i "s/CHANGEME/$RABBITMQ_PASSWORD/g" /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq.conf
#api check for probes
cat > /opt/bitnami/rabbitmq/sbin/rabbitmq-api-check <<EOF
#!/bin/sh
set -e
URL=\$1
EXPECTED=\$2
ACTUAL=\$(curl --silent --show-error --fail "\${URL}")
echo "\${ACTUAL}"
test "\${EXPECTED}" = "\${ACTUAL}"
EOF
chmod a+x /opt/bitnami/rabbitmq/sbin/rabbitmq-api-check
exec rabbitmq-server

    State:          Running
      Started:      Wed, 26 Jun 2019 10:13:26 -0400
    Ready:          False
    Restart Count:  0
    Liveness:       exec [sh -c rabbitmq-api-check "http://user:$RABBITMQ_PASSWORD@127.0.0.1:15672/api/healthchecks/node" '{"status":"ok"}'] delay=120s timeout=20s period=30s #success=1 #failure=6
    Readiness:      exec [sh -c rabbitmq-api-check "http://user:$RABBITMQ_PASSWORD@127.0.0.1:15672/api/healthchecks/node" '{"status":"ok"}'] delay=10s timeout=20s period=30s #success=1 #failure=3
    Environment:
      MY_POD_IP:                             (v1:status.podIP)
      MY_POD_NAME:                          rabbitmq-0 (v1:metadata.name)
      MY_POD_NAMESPACE:                     default (v1:metadata.namespace)
      K8S_SERVICE_NAME:                     rabbitmq-headless
      K8S_ADDRESS_TYPE:                     hostname
      RABBITMQ_NODENAME:                    rabbit@$(MY_POD_NAME).$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE).svc.cluster.local
      K8S_HOSTNAME_SUFFIX:                  .$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE).svc.cluster.local
      RABBITMQ_LOGS:                        -
      RABBITMQ_ULIMIT_NOFILES:              65536
      RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS:  +S 2:1
      RABBITMQ_USE_LONGNAME:                true
      RABBITMQ_ERL_COOKIE:                  <set to the key 'rabbitmq-erlang-cookie' in secret 'rabbitmq'>  Optional: false
      RABBITMQ_PASSWORD:                    <set to the key 'rabbitmq-password' in secret 'rabbitmq'>       Optional: false
    Mounts:
      /opt/bitnami/rabbitmq/conf from config-volume (rw)
      /opt/bitnami/rabbitmq/var/lib/rabbitmq from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from rabbitmq-token-xv2bf (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-rabbitmq-0
    ReadOnly:   false
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      rabbitmq-config
    Optional:  false
  rabbitmq-token-xv2bf:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  rabbitmq-token-xv2bf
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
         node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                  Age   From                               Message
  ----     ------                  ----  ----                               -------
  Normal   Scheduled               2m    default-scheduler                  Successfully assigned default/rabbitmq-0 to aks-agentpool-17141372-0
  Normal   SuccessfulAttachVolume  1m    attachdetach-controller            AttachVolume.Attach succeeded for volume "pvc-1415e42b-9776-11e9-9ce0-6e8bcd55bf4b"
  Normal   Pulling                 52s   kubelet, aks-agentpool-17141372-0  pulling image "docker.io/bitnami/rabbitmq:3.7.14"
  Normal   Pulled                  44s   kubelet, aks-agentpool-17141372-0  Successfully pulled image "docker.io/bitnami/rabbitmq:3.7.14"
  Normal   Created                 36s   kubelet, aks-agentpool-17141372-0  Created container
  Normal   Started                 35s   kubelet, aks-agentpool-17141372-0  Started container
  Warning  Unhealthy               25s   kubelet, aks-agentpool-17141372-0  Readiness probe failed: curl: (22) The requested URL returned error: 401

kubectl get services
rabbitmq                                       ClusterIP      10.242.165.106   <none>          4369/TCP,5672/TCP,25672/TCP,15672/TCP   2m
rabbitmq-headless                              ClusterIP      None             <none>          4369/TCP,5672/TCP,25672/TCP,15672/TCP   2m

kubectl describe svc/rabbitmq
Name:              rabbitmq
Namespace:         default
Labels:            app=rabbitmq
           chart=rabbitmq-5.5.0
           heritage=Tiller
           release=rabbitmq
Annotations:       <none>
Selector:          app=rabbitmq,release=rabbitmq
Type:              ClusterIP
IP:                10.242.165.106
Port:              epmd  4369/TCP
TargetPort:        epmd/TCP
Endpoints:
Port:              amqp  5672/TCP
TargetPort:        amqp/TCP
Endpoints:
Port:              dist  25672/TCP
TargetPort:        dist/TCP
Endpoints:
Port:              stats  15672/TCP
TargetPort:        stats/TCP
Endpoints:
Session Affinity:  None
Events:            <none>

There is no deployments and replicaset for rabbitmq.

miguelaeh commented 5 years ago

Hi @rnkhouse , I am not sure about what is happening. Could you paste the logs of the container? Also, not sure if this could be related but, does your cluster have RBAC enabled?

rnkhouse commented 5 years ago

@miguelaeh Cluster is RBAC enabled. Logs:

2019-06-27 14:13:14.010 [info] <0.282.0>
 Starting RabbitMQ 3.7.14 on Erlang 21.3
 Copyright (C) 2007-2019 Pivotal Software, Inc.
 Licensed under the MPL.  See https://www.rabbitmq.com/

  ##  ##
  ##  ##      RabbitMQ 3.7.14. Copyright (C) 2007-2019 Pivotal Software, Inc.
  ##########  Licensed under the MPL.  See https://www.rabbitmq.com/
  ######  ##
  ##########  Logs: <stdout>

          Starting broker...
2019-06-27 14:13:14.011 [info] <0.282.0>
 node           : rabbit@rabbitmq-0.rabbitmq-headless.default.svc.cluster.local
 home dir       : /opt/bitnami/rabbitmq/.rabbitmq
 config file(s) : /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq.conf
 cookie hash    : 3xVCnnyttkChiXyWcNjn9w==
 log(s)         : <stdout>
 database dir   : /opt/bitnami/rabbitmq/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-0.rabbitmq-headless.default.svc.cluster.local
2019-06-27 14:13:14.284 [info] <0.282.0> Running boot step pre_boot defined by app rabbit
2019-06-27 14:13:14.284 [info] <0.282.0> Running boot step rabbit_core_metrics defined by app rabbit
2019-06-27 14:13:14.284 [info] <0.282.0> Running boot step rabbit_alarm defined by app rabbit
2019-06-27 14:13:14.289 [info] <0.288.0> Memory high watermark set to 2778 MiB (2913550336 bytes) of 6946 MiB (7283875840 bytes) total
2019-06-27 14:13:14.295 [info] <0.290.0> Enabling free disk space monitoring
2019-06-27 14:13:14.295 [info] <0.290.0> Disk free limit set to 50MB
2019-06-27 14:13:14.300 [info] <0.282.0> Running boot step code_server_cache defined by app rabbit
2019-06-27 14:13:14.300 [info] <0.282.0> Running boot step file_handle_cache defined by app rabbit
2019-06-27 14:13:14.300 [info] <0.293.0> Limiting to approx 65436 file handles (58890 sockets)
2019-06-27 14:13:14.300 [info] <0.294.0> FHC read buffering:  OFF
2019-06-27 14:13:14.300 [info] <0.294.0> FHC write buffering: ON
2019-06-27 14:13:14.301 [info] <0.282.0> Running boot step worker_pool defined by app rabbit
2019-06-27 14:13:14.301 [info] <0.282.0> Running boot step database defined by app rabbit
2019-06-27 14:13:14.302 [info] <0.282.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2019-06-27 14:13:14.394 [info] <0.282.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2019-06-27 14:13:14.395 [info] <0.282.0> Peer discovery backend rabbit_peer_discovery_k8s does not support registration, skipping registration.
2019-06-27 14:13:14.395 [info] <0.282.0> Running boot step database_sync defined by app rabbit
2019-06-27 14:13:14.396 [info] <0.282.0> Running boot step codec_correctness_check defined by app rabbit
2019-06-27 14:13:14.396 [info] <0.282.0> Running boot step external_infrastructure defined by app rabbit
2019-06-27 14:13:14.396 [info] <0.282.0> Running boot step rabbit_registry defined by app rabbit
2019-06-27 14:13:14.396 [info] <0.282.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2019-06-27 14:13:14.396 [info] <0.282.0> Running boot step rabbit_queue_location_random defined by app rabbit
2019-06-27 14:13:14.396 [info] <0.282.0> Running boot step rabbit_event defined by app rabbit
2019-06-27 14:13:14.396 [info] <0.282.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2019-06-27 14:13:14.397 [info] <0.282.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2019-06-27 14:13:14.397 [info] <0.282.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2019-06-27 14:13:14.397 [info] <0.282.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2019-06-27 14:13:14.397 [info] <0.282.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2019-06-27 14:13:14.398 [info] <0.282.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2019-06-27 14:13:14.398 [info] <0.282.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2019-06-27 14:13:14.398 [info] <0.282.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2019-06-27 14:13:14.398 [info] <0.282.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2019-06-27 14:13:14.398 [info] <0.282.0> Running boot step rabbit_priority_queue defined by app rabbit
2019-06-27 14:13:14.398 [info] <0.282.0> Priority queues enabled, real BQ is rabbit_variable_queue
2019-06-27 14:13:14.398 [info] <0.282.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2019-06-27 14:13:14.398 [info] <0.282.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2019-06-27 14:13:14.399 [info] <0.282.0> Running boot step kernel_ready defined by app rabbit
2019-06-27 14:13:14.399 [info] <0.282.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2019-06-27 14:13:14.399 [info] <0.282.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2019-06-27 14:13:14.405 [info] <0.282.0> Running boot step guid_generator defined by app rabbit
2019-06-27 14:13:14.419 [info] <0.282.0> Running boot step rabbit_node_monitor defined by app rabbit
2019-06-27 14:13:14.420 [info] <0.318.0> Starting rabbit_node_monitor
2019-06-27 14:13:14.422 [info] <0.282.0> Running boot step delegate_sup defined by app rabbit
2019-06-27 14:13:14.424 [info] <0.282.0> Running boot step rabbit_memory_monitor defined by app rabbit
2019-06-27 14:13:14.425 [info] <0.282.0> Running boot step core_initialized defined by app rabbit
2019-06-27 14:13:14.425 [info] <0.282.0> Running boot step upgrade_queues defined by app rabbit
2019-06-27 14:13:14.450 [info] <0.282.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2019-06-27 14:13:14.450 [info] <0.282.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2019-06-27 14:13:14.450 [info] <0.282.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2019-06-27 14:13:14.451 [info] <0.282.0> Running boot step rabbit_policies defined by app rabbit
2019-06-27 14:13:14.451 [info] <0.282.0> Running boot step rabbit_policy defined by app rabbit
2019-06-27 14:13:14.451 [info] <0.282.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2019-06-27 14:13:14.452 [info] <0.282.0> Running boot step rabbit_vhost_limit defined by app rabbit
2019-06-27 14:13:14.452 [info] <0.282.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2019-06-27 14:13:14.452 [info] <0.282.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2019-06-27 14:13:14.452 [info] <0.282.0> Management plugin: using rates mode 'basic'
2019-06-27 14:13:14.452 [info] <0.282.0> Running boot step recovery defined by app rabbit
2019-06-27 14:13:14.453 [info] <0.349.0> Making sure data directory '/opt/bitnami/rabbitmq/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-0.rabbitmq-headless.default.svc.cluster.local/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2019-06-27 14:13:14.475 [info] <0.349.0> Starting message stores for vhost '/'
2019-06-27 14:13:14.476 [info] <0.353.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2019-06-27 14:13:14.478 [info] <0.349.0> Started message store of type transient for vhost '/'
2019-06-27 14:13:14.478 [info] <0.356.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2019-06-27 14:13:14.483 [info] <0.349.0> Started message store of type persistent for vhost '/'
2019-06-27 14:13:14.485 [info] <0.282.0> Running boot step load_definitions defined by app rabbitmq_management
2019-06-27 14:13:14.485 [info] <0.282.0> Running boot step empty_db_check defined by app rabbit
2019-06-27 14:13:14.485 [info] <0.282.0> Running boot step rabbit_looking_glass defined by app rabbit
2019-06-27 14:13:14.485 [info] <0.282.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2019-06-27 14:13:14.485 [info] <0.282.0> Running boot step background_gc defined by app rabbit
2019-06-27 14:13:14.486 [info] <0.282.0> Running boot step connection_tracking defined by app rabbit
2019-06-27 14:13:14.486 [info] <0.282.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_rabbit@rabbitmq-0.rabbitmq-headless.default.svc.cluster.local'
2019-06-27 14:13:14.486 [info] <0.282.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_rabbit@rabbitmq-0.rabbitmq-headless.default.svc.cluster.local'
2019-06-27 14:13:14.486 [info] <0.282.0> Running boot step routing_ready defined by app rabbit
2019-06-27 14:13:14.487 [info] <0.282.0> Running boot step pre_flight defined by app rabbit
2019-06-27 14:13:14.487 [info] <0.282.0> Running boot step notify_cluster defined by app rabbit
2019-06-27 14:13:14.487 [info] <0.282.0> Running boot step networking defined by app rabbit
2019-06-27 14:13:14.489 [warning] <0.379.0> Setting Ranch options together with socket options is deprecated. Please use the new map syntax that allows specifying socket options separately from other options.
2019-06-27 14:13:14.490 [info] <0.393.0> started TCP listener on [::]:5672
2019-06-27 14:13:14.490 [info] <0.282.0> Running boot step direct_client defined by app rabbit
2019-06-27 14:13:14.493 [info] <0.439.0> Peer discovery: enabling node cleanup (will only log warnings). Check interval: 10 seconds.
2019-06-27 14:13:14.526 [info] <0.449.0> Management plugin: HTTP (non-TLS) listener started on port 15672
2019-06-27 14:13:14.526 [info] <0.555.0> Statistics database started.
 completed with 5 plugins.
2019-06-27 14:13:14.860 [info] <0.8.0> Server startup complete; 5 plugins started.
 * rabbitmq_management
 * rabbitmq_web_dispatch
 * rabbitmq_peer_discovery_k8s
 * rabbitmq_peer_discovery_common
 * rabbitmq_management_agent
2019-06-27 14:13:36.598 [warning] <0.568.0> HTTP access denied: user 'user' - invalid credentials
2019-06-27 14:14:06.599 [warning] <0.576.0> HTTP access denied: user 'user' - invalid credentials
miguelaeh commented 5 years ago

Hi @rnkhouse , The problem here seems to be that the readiness probe is obtaining an unauthorized code from the HTTP request, as you can see on the log the credentials are invalid. As you can see on the template, the readinessProbe use a password.

 valueFrom:
              secretKeyRef:
                name: {{ template "rabbitmq.secretPasswordName" . }}
key: rabbitmq-password

That password is taken from a secret witch name is taken from a template:

{{- define "rabbitmq.secretPasswordName" -}}
    {{- if .Values.rabbitmq.existingPasswordSecret -}}
        {{- printf "%s" .Values.rabbitmq.existingPasswordSecret -}}
    {{- else -}}
        {{- printf "%s" (include "rabbitmq.fullname" .) -}}
    {{- end -}}
{{- end -}}

If you do not set a password manually it is created a randomly, so probably you have a secret from an older deployment that is being taken from the new deployment and it doesn't match with the password of the new deployment, so the readiness probe get an unauthorized error.

rnkhouse commented 5 years ago

@miguelaeh I do not have secret from older deployment. I tried deleting chart by helm delete rabbitmq --purge and checked there is no secret left related to rabbitmq and re-deployed the chart. Still getting the same error.

I also tried with assigning password manually as you can see the command in the original question. I am not sure what are you trying to say.

These are the logs from that command:
2019-06-28 14:40:29.945 [info] <0.248.0> Running boot step rabbit_policy defined by app rabbit
2019-06-28 14:40:29.945 [info] <0.248.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2019-06-28 14:40:29.945 [info] <0.248.0> Running boot step rabbit_vhost_limit defined by app rabbit
2019-06-28 14:40:29.945 [info] <0.248.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2019-06-28 14:40:29.945 [info] <0.248.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2019-06-28 14:40:29.945 [info] <0.248.0> Management plugin: using rates mode 'basic'
2019-06-28 14:40:29.945 [info] <0.248.0> Running boot step recovery defined by app rabbit
2019-06-28 14:40:29.946 [info] <0.349.0> Making sure data directory '/opt/bitnami/rabbitmq/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-0.rabbitmq-headless.default.svc.cluster.local/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2019-06-28 14:40:29.957 [info] <0.349.0> Starting message stores for vhost '/'
2019-06-28 14:40:29.958 [info] <0.353.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2019-06-28 14:40:29.959 [info] <0.349.0> Started message store of type transient for vhost '/'
2019-06-28 14:40:29.960 [info] <0.356.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2019-06-28 14:40:29.962 [info] <0.349.0> Started message store of type persistent for vhost '/'
2019-06-28 14:40:29.963 [info] <0.248.0> Running boot step load_definitions defined by app rabbitmq_management
2019-06-28 14:40:29.963 [info] <0.248.0> Running boot step empty_db_check defined by app rabbit
2019-06-28 14:40:29.963 [info] <0.248.0> Running boot step rabbit_looking_glass defined by app rabbit
2019-06-28 14:40:29.963 [info] <0.248.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2019-06-28 14:40:29.964 [info] <0.248.0> Running boot step background_gc defined by app rabbit
2019-06-28 14:40:29.964 [info] <0.248.0> Running boot step connection_tracking defined by app rabbit
2019-06-28 14:40:29.964 [info] <0.248.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_rabbit@rabbitmq-0.rabbitmq-headless.default.svc.cluster.local'
2019-06-28 14:40:29.964 [info] <0.248.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_rabbit@rabbitmq-0.rabbitmq-headless.default.svc.cluster.local'
2019-06-28 14:40:29.965 [info] <0.248.0> Running boot step routing_ready defined by app rabbit
2019-06-28 14:40:29.965 [info] <0.248.0> Running boot step pre_flight defined by app rabbit
2019-06-28 14:40:29.965 [info] <0.248.0> Running boot step notify_cluster defined by app rabbit
2019-06-28 14:40:29.965 [info] <0.248.0> Running boot step networking defined by app rabbit
2019-06-28 14:40:29.967 [warning] <0.379.0> Setting Ranch options together with socket options is deprecated. Please use the new map syntax that allows specifying socket options separately from other options.
2019-06-28 14:40:29.968 [info] <0.393.0> started TCP listener on [::]:5672
2019-06-28 14:40:29.968 [info] <0.248.0> Running boot step direct_client defined by app rabbit
2019-06-28 14:40:29.971 [info] <0.439.0> Peer discovery: enabling node cleanup (will only log warnings). Check interval: 10 seconds.
2019-06-28 14:40:29.999 [info] <0.449.0> Management plugin: HTTP (non-TLS) listener started on port 15672
2019-06-28 14:40:29.999 [info] <0.555.0> Statistics database started.
 completed with 5 plugins.
2019-06-28 14:40:30.201 [info] <0.8.0> Server startup complete; 5 plugins started.
 * rabbitmq_management
 * rabbitmq_web_dispatch
 * rabbitmq_peer_discovery_k8s
 * rabbitmq_peer_discovery_common
 * rabbitmq_management_agent
    2019-06-28 15:00:52.699 [warning] <0.568.0> HTTP access denied: user 'admin' - invalid credentials
    2019-06-28 15:01:22.687 [warning] <0.576.0> HTTP access denied: user 'admin' - invalid credentials
    2019-06-28 15:01:52.709 [warning] <0.584.0> HTTP access denied: user 'admin' - invalid credentials

As you can see even with given username and password getting this error.

kubectl get secret/rabbitmq -o yaml
apiVersion: v1
data:
  rabbitmq-erlang-cookie: c2VjcmV0Y29va2ll
  rabbitmq-password: YWRtaW4=
kind: Secret
metadata:
  creationTimestamp: 2019-06-28T14:38:42Z
  labels:
    app: rabbitmq
    chart: rabbitmq-5.5.0
    heritage: Tiller
    release: rabbitmq
  name: rabbitmq
  namespace: default
  resourceVersion: "377908"
  selfLink: /api/v1/namespaces/default/secrets/rabbitmq
  uid: 6d13a815-99b2-11e9-9ce0-6e8bcd55bf4b
type: Opaque

rabbitmq-password: YWRtaW4= which is admin

miguelaeh commented 5 years ago

The helm delete --purge does not delete the PVCs so probably you are using always the same password on the configuration, that is because even if you give a password it is incorrect. Delete manually you PVCs after purge and redeploy. That should solve the problem.

rnkhouse commented 5 years ago

@miguelaeh What is PVC? How to delete previous password from there?

miguelaeh commented 5 years ago

Hi @rnkhouse , PVC is Persistent Volume Claim. To delete them, first obtain the name with:

kubectl get pvc

Copy the name and delete with:

kubectl delete pvc <name>
rnkhouse commented 5 years ago

That worked! Thanks for your support.