search

rosskukulinski / kubernetes-rethinkdb-cluster

RethinkDB cluster on top of Kubernetes made easy.
MIT License
139 stars 40 forks source link

CrashLoopBackOff on rethinkdb-admin and rethinkdb-proxy #40

Closed cogmeta closed 5 years ago

cogmeta commented 5 years ago

rethinkdb-admin-64b7789fc-59hvw 0/1 CrashLoopBackOff 2 62s rethinkdb-proxy-b5b8f5888-mm7jb 0/1 CrashLoopBackOff 2 62s rethinkdb-replica-6847df5d5f-v82ns 1/1 Running 0 62s

The admin and proxy enter into infinite CrashLoopBackOff. Replica seems fine. Tried both 2.3.4 and 2.3.5. No luck!

describe pod does not tell much

sudo kubectl describe po rethinkdb-proxy-b5b8f5888-mm7jb . . . vents: Type Reason Age From Message

Normal Scheduled 80s default-scheduler Successfully assigned default/rethinkdb-proxy-b5b8f5888-mm7jb to olelo Normal Pulling 12s (x4 over 77s) kubelet, olelo pulling image "rosskukulinski/rethinkdb-kubernetes:2.3.5" Normal Pulled 9s (x4 over 71s) kubelet, olelo Successfully pulled image "rosskukulinski/rethinkdb-kubernetes:2.3.5" Normal Created 9s (x4 over 71s) kubelet, olelo Created container Normal Started 8s (x4 over 70s) kubelet, olelo Started container Warning BackOff 7s (x9 over 62s) kubelet, olelo Back-off restarting failed container

cogmeta commented 5 years ago

sudo kubectl log rethinkdb-proxy-86784f57f6-8v2v8
log is DEPRECATED and will be removed in a future version. Use logs instead. Using additional CLI flags: Pod IP: 192.0.0.8 Pod namespace: default Using service name: rethinkdb Using server name: rethinkdb_proxy_86784f57f6_8v2v8 Checking for other nodes... Using endpoints to lookup other nodes... Endpoint url: https://10.96.0.1:443/api/v1/namespaces/default/endpoints/rethinkdb Looking for IPs... jq: error: Cannot iterate over null No other nodes detected, will be a single instance. Cannot start in proxy mode without endpoints.

sudo kubectl get endpoints NAME ENDPOINTS AGE kubernetes 10.9.47.63:6443 2m41s rethinkdb 192.0.0.6:29015 2m8s rethinkdb-admin 2m8s rethinkdb-proxy 2m7s

rosskukulinski commented 5 years ago

Hi @cogmeta - I'm not really maintaining this repo anymore and haven't tested in a long time. I won't be able to help much.

My assumption is that you have RBAC enabled on your cluster, so the request to Endpoint url: https://10.96.0.1:443/api/v1/namespaces/default/endpoints/rethinkdb is being rejected.

You may need to modify the init script in the Rethinkdb docker image to be able to communicate to the k8s api server with a service account.

initcron commented 5 years ago

@rosskukulinski you are right. This was due to rbac. @cogmeta and I have fixed it, and added a rethinkdb-rbac.yaml file (https://github.com/initcron/kubernetes-rethinkdb-cluster/blob/master/rethinkdb-rbac.yaml). That needs to be applied before running quickstart. Pull req on the way.

timothybean commented 5 years ago

This fixed this issue for me as well