Can't run server with "seldon-up.sh" - mysql status is forever Pending

[ukrbublik]

I run Kubernetes locally via Minikube 0.12.1. My OS: Ubuntu 16.04. RAM: 16G (and 16GB swap). Problem: when I run "seldon-up.sh" (Seldon v1.3.11), I see mysql node status is forever Pending. When I run "kubectl get events", I see this: 24m 35m 40 mysql Pod Warning FailedScheduling {default-scheduler } pod (mysql) failed to fit in any node fit failure on node (minikube): Insufficient memory

But I have 8.4GB RAM free.

[ukclivecox]

Did you start minikube with enough memory?, e.g. minikube start --memory=8000 --cpus=4

[ukrbublik]

Did you start minikube with enough memory?, e.g. minikube start --memory=8000 --cpus=4

Yes, tried now with this flags, no changes.

[ukclivecox]

Did you to do a minikube delete before doing a minikube start? If you just do minikube stop then the new memory setting will not be used.

[ukrbublik]

Did you to do a minikube delete before doing a minikube start? If you just do minikube stop then the new memory setting will not be used.

Thanks, now mysql is running and seldon server starts ok.

But I have another problem with trying Reuters Newswire Recommendation example, can you help? When I run "seldon-cli --quiet api --client-name reuters --endpoint /js/recommendations --item 6020 --limit 3", I get: WARNING:kazoo.client:Connection dropped: socket connection error: No address associated with hostname WARNING:kazoo.client:Connection dropped: socket connection error: No address associated with hostname ... WARNING:kazoo.client:Failed connecting to Zookeeper within the connection retry policy. Traceback (most recent call last): File "/usr/local/bin/seldon-cli", line 5, in seldon.cli.start_seldoncli() File "/usr/local/lib/python2.7/dist-packages/seldon/cli/init.py", line 3, in start_seldoncli cli_main.main() File "/usr/local/lib/python2.7/dist-packages/seldon/cli/cli_main.py", line 346, in main start_zk_client(opts) File "/usr/local/lib/python2.7/dist-packages/seldon/cli/cli_main.py", line 301, in start_zk_client gdata["zk_client"].start() File "/usr/local/lib/python2.7/dist-packages/kazoo/client.py", line 546, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.threading.KazooTimeoutError: Connection time-out

Maybe it's related to fact that I tried to follow this guide "Manual install outside Kubernetes" (http://docs.seldon.io/seldon-cli.html#manual) when I had problem with minikube. But now I returned "zk_hosts" value to "zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181" in ~/.seldon/seldon.conf

[ukclivecox]

This looks like either a DNS issue, but if you are running inside minikube this seems unlikely. Can you recreate the conf - "make clean conf" from kubernetes/conf Suggest you also ssh into seldon-control and check you can find a zookeeper instance by name, e.g. try telnet zookeeper-1 2181

[ukrbublik]

Can you recreate the conf - "make clean conf" from kubernetes/conf

Recreated. Did not help.

Suggest you also ssh into seldon-control and check you can find a zookeeper instance by name, e.g. try telnet zookeeper-1 2181

Yes, seems I can.

root@seldon-control:/home/seldon# telnet zookeeper-1 2181 Trying 10.0.0.198... Connected to zookeeper-1.default.svc.cluster.local. Escape character is '^]'.

I also tried to add this to /etc/hosts: 172.17.0.6 zookeeper-1 172.17.0.5 zookeeper-2 172.17.0.7 zookeeper-3 (IPs taken from kubernetes' web ui pods section) Then I get: WARNING:kazoo.client:Connection dropped: socket connection error: No route to host

[ukclivecox]

You shouldn't need to anything to /etc/hosts Are you running inside minikube?

[ukrbublik]

No. I just did these commands at local pc terminal:

minikube start
seldon-up.sh
... run import data job, recommendation microservice
seldon-cli --quiet api --client-name reuters --endpoint  /js/recommendations --item 6020 --limit 3

Sorry for my noob questions. I should first run kubectl exec -ti seldon-control -- /bin/bash Then at root@seldon-control I should run seldon-cli commands. Right?

[ukclivecox]

No. You can add seldon-cli which is in kubernetes/bin to your PATH.

(internally it will call seldon-cli running inside seldon-control)

[ukrbublik]

Now it works. Thanks. seldon-cli was also in /usr/local/bin/ , this caused problem

[ukclivecox]

Great. We'll update out docs regards minikube to cover your situation.