search

Azure / open-service-broker-azure

The Open Service Broker API Server for Azure Services
https://osba.sh
MIT License
248 stars 100 forks source link

Service-broker pods goes into crashloopbackoff #701

Closed mallikarjun-b-r closed 5 years ago

mallikarjun-b-r commented 5 years ago

I am installing the azure broker with helm after installing service broker goes into crashloopbackoff and when i do describe on that pod it gives me the following Warning FailedScheduling 22s (x8 over 5m48s) default-scheduler pod has unbound immediate PersistentVolumeClaims (repeated 6 times) also if i do

kubectl get pvc -n osba osba osba-redis Pending 22m

if i describe that it gives me no persistent volumes available for this claim and no storage class is set

krancour commented 5 years ago

I wish there were more detail here, so I am reading between the lines a bit and can take a guess at what is going on...

after installing service broker goes into crashloopbackoff

I assume this is referring to the broker pod itself.

when i do describe on that pod it gives me the following

This, however, sounds like you're talking about the Redis pod; not the broker pod.

I believe this is what is happening...

The broker pod will fail until the Redis it talks to is available. So, it's normal for the broker to error out and restart a few times while waiting for Redis to come up. This is the name of the game in k8s-- eventual consistency.

The problem you're encountering is that the Redis pod isn't ever coming up. So after several errors, the broker pod goes into the crash loop backoff.

The question then is why the Redis pod isn't coming up. The describe output you provided indicates it cannot be scheduled to any node because of unbound PersistentVolumeClaims. A pod must be collocated on the same node as the volume(s) it uses. Here, the volume isn't bound to any node, so no node is a suitable home for the pod yet.

So what's up with the volume?

The describe you did on the PVC shows what's wrong:

no persistent volumes available for this claim and no storage class is set

You do not typically need to specify what storage class the PVC will use to dynamically provision a new volume. When not specified, it will default to using the cluster's default storage class. I'm going to take a guess that for one reason or another, your cluster doesn't have a default storage class. You can confirm this like so:

$ kubectl get storageclasses
NAME                PROVISIONER                                AGE
default (default)   kubernetes.io/azure-disk                   38h
local-nfs           cluster.local/nfs-nfs-server-provisioner   56d
managed-premium     kubernetes.io/azure-disk                   38h

As you can see, my cluster has one named default, which is clearly designated as the default (by the (default) in parens). I am thinking you do not have this.

Sound right?

Possible paths forward for you:

  1. Disable persistence on Redis. (There are consequences to this. Certainly do not do that in production.)
  2. Define a default storage class for your cluster.
  3. Explicitly specify which (non-default) storage class you'd like Redis to use.
mallikarjun-b-r commented 5 years ago

I tried fetching helm charts manually and change the pvc to use pv instead of storage class but ran into a lot of issues after disabling persistance its working

Disable persistence on Redis. (There are consequences to this. Certainly do not do that in production.) Did the job for now thanks

krancour commented 5 years ago

I'm curious why you wouldn't just straighten out the storage class issue, but I am glad to hear it is working.

mallikarjun-b-r commented 5 years ago

The kubernetes that we use for our product is a little bit modified and it has a lot of things disabled like rbac etc for some reasons hence i cannot use storageclass