search

SeldonIO / seldon-server

Machine Learning Platform and Recommendation Engine built on Kubernetes
https://www.seldon.io/
Apache License 2.0
1.47k stars 300 forks source link

Python support #53

Closed kination closed 6 years ago

kination commented 7 years ago

Hello! Is it support python 3.x? Because when I started with start-microservice --type prediction --client test -i iris-xgboost seldonio/iris_xgboost:2.1 rest 1.0 after running kubernetes, it shows bug like

$ start-microservice --type prediction --client test -i iris-xgboost seldonio/iris_xgboost:2.1 rest 1.0
  File "/Users/kwangin/workspace/seldon-server/kubernetes/bin/start-microservice", line 144
    print "Replicas is ",self.replicas
                       ^
SyntaxError: Missing parentheses in call to 'print'

Also, it shows an error at python 2.x like below.

$ start-microservice --type prediction --client test -i iris-xgboost seldonio/iris_xgboost:2.1 rest 1.0
[Microservice(iris-xgboost,seldonio/iris_xgboost:2.1,rest,1.000000)]
Replicas is  1
kubectl apply -f /Users/kwangin/workspace/seldon-server/kubernetes/bin/../conf/microservices/microservice-iris-xgboost.json
deployment "iris-xgboost" created
service "iris-xgboost" created
error: expected 'exec POD_NAME COMMAND [ARG1] [ARG2] ... [ARGN]'.
POD_NAME and COMMAND are required arguments for the exec command
See 'kubectl exec -h' for help and examples.
Traceback (most recent call last):
  File "/Users/kwangin/workspace/seldon-server/kubernetes/bin/start-microservice", line 313, in <module>
    runner.run(args.type,args.client,args.i + args.p)
  File "/Users/kwangin/workspace/seldon-server/kubernetes/bin/start-microservice", line 214, in run
    self.predictCreate(client,services)
  File "/Users/kwangin/workspace/seldon-server/kubernetes/bin/start-microservice", line 203, in predictCreate
    raise MicroserviceError("failed to run seldon cli to create conf"+name)
NameError: global name 'name' is not defined

Is there something missing?

Thanks.

ukclivecox commented 7 years ago

Yes that script is python 2 script. We will look to see if it can be made python 3 compatible.

kination commented 7 years ago

@cliveseldon Thanks. Is python 2 working? Because I still have an error NameError: global name 'name' is not defined with python 2 as you see in first comment.

I used command in sample. Am I missing something? start-microservice --type prediction --client test -i iris-xgboost seldonio/iris_xgboost:2.1 rest 1.0

ukclivecox commented 7 years ago

I think that is a simple typo bug. So the underlying issue is why it failed to run against the seldon-cli on your cluster. It looks like an issue calling kubectl:

error: expected 'exec POD_NAME COMMAND [ARG1] [ARG2] ... [ARGN]'. POD_NAME and COMMAND are required arguments for the exec command See 'kubectl exec -h' for help and examples.

ukclivecox commented 7 years ago

It could be the following issue is the underlying reason: #52

In that case it was that there was insufficient memory to start all pods. Can you do kubectl get all and check if everything is running ok?

kination commented 7 years ago

@cliveseldon Sorry for late review. Typo bug has been solved, but seems some other problem. When I startup service with seldon-up, it keeps showing status below and it seems service is not fully running.

...
Sleeping for 5 seconds...
NAME                               READY     STATUS              RESTARTS   AGE
influxdb-grafana-842592602-qr533   0/2       ContainerCreating   0          1h
influxdb-grafana-842592602-w1k46   2/2       Terminating         0          1h
iris-xgboost-1750015301-sdjnm      0/1       ContainerCreating   0          12m
memcached1-2136693305-1qf6g        0/1       ContainerCreating   0          1h
memcached1-2136693305-43h89        1/1       Terminating         0          1h
memcached2-2533120572-19t9n        1/1       Terminating         0          1h
memcached2-2533120572-h03xq        0/1       ContainerCreating   0          1h
mysql-2529449154-s0qr9             0/1       Pending             0          1h
redis-1963070708-1q6gc             1/1       Terminating         0          1h
redis-1963070708-4sr9k             0/1       ContainerCreating   0          1h
reuters-example-327623259-84gn1    0/1       ContainerCreating   0          36m
reuters-import-data-cgx2j          0/1       ContainerCreating   0          45m
seldon-control-2832234452-c2wz0    0/1       ContainerCreating   0          1h
zookeeper1-467704625-gvwrp         1/1       Terminating         0          1h
zookeeper1-467704625-mmkjq         0/1       ContainerCreating   0          1h
zookeeper2-1006738229-j4fgk        1/1       Terminating         0          1h
zookeeper2-1006738229-qc1q0        0/1       ContainerCreating   0          1h
zookeeper3-1545771833-4r1g1        0/1       ContainerCreating   0          1h
zookeeper3-1545771833-hf948        1/1       Terminating         1          1h
Waiting for pods to be running as found 19 in non-running state
Sleeping for 5 seconds...
...

This is when I run kubectl get all.

$ kubectl get all
NAME                                  READY     STATUS              RESTARTS   AGE
po/influxdb-grafana-842592602-qr533   0/2       ContainerCreating   0          1h
po/influxdb-grafana-842592602-w1k46   2/2       Terminating         0          1h
po/iris-xgboost-1750015301-sdjnm      0/1       ContainerCreating   0          14m
po/memcached1-2136693305-1qf6g        0/1       ContainerCreating   0          1h
po/memcached1-2136693305-43h89        1/1       Terminating         0          1h
po/memcached2-2533120572-19t9n        1/1       Terminating         0          1h
po/memcached2-2533120572-h03xq        0/1       ContainerCreating   0          1h
po/mysql-2529449154-s0qr9             0/1       Pending             0          1h
po/redis-1963070708-1q6gc             1/1       Terminating         0          1h
po/redis-1963070708-4sr9k             0/1       ContainerCreating   0          1h
po/reuters-example-327623259-84gn1    0/1       ContainerCreating   0          38m
po/reuters-import-data-cgx2j          0/1       ContainerCreating   0          47m
po/seldon-control-2832234452-c2wz0    0/1       ContainerCreating   0          1h
po/zookeeper1-467704625-gvwrp         1/1       Terminating         0          1h
po/zookeeper1-467704625-mmkjq         0/1       ContainerCreating   0          1h
po/zookeeper2-1006738229-j4fgk        1/1       Terminating         0          1h
po/zookeeper2-1006738229-qc1q0        0/1       ContainerCreating   0          1h
po/zookeeper3-1545771833-4r1g1        0/1       ContainerCreating   0          1h
po/zookeeper3-1545771833-hf948        1/1       Terminating         1          1h

NAME                      CLUSTER-IP   EXTERNAL-IP   PORT(S)                      AGE
svc/iris-xgboost          10.0.0.249   <none>        5000/TCP                     14m
svc/kubernetes            10.0.0.1     <none>        443/TCP                      108d
svc/memcached1            10.0.0.141   <none>        11211/TCP                    1h
svc/memcached2            10.0.0.170   <none>        11211/TCP                    1h
svc/monitoring-grafana    10.0.0.61    <pending>     80:30002/TCP                 1h
svc/monitoring-influxdb   10.0.0.46    <none>        8083/TCP,8086/TCP            1h
svc/mysql                 10.0.0.223   <none>        3306/TCP                     1h
svc/redis                 10.0.0.124   <none>        6379/TCP                     1h
svc/reuters-example       10.0.0.247   <none>        5000/TCP                     38m
svc/zookeeper-1           10.0.0.107   <none>        2181/TCP,2888/TCP,3888/TCP   1h
svc/zookeeper-2           10.0.0.84    <none>        2181/TCP,2888/TCP,3888/TCP   1h
svc/zookeeper-3           10.0.0.18    <none>        2181/TCP,2888/TCP,3888/TCP   1h

NAME                       DESIRED   SUCCESSFUL   AGE
jobs/reuters-import-data   1         0            47m

NAME                      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/influxdb-grafana   1         1         1            0           1h
deploy/iris-xgboost       1         1         1            0           14m
deploy/memcached1         1         1         1            0           1h
deploy/memcached2         1         1         1            0           1h
deploy/mysql              1         1         1            0           1h
deploy/redis              1         1         1            0           1h
deploy/reuters-example    1         1         1            0           38m
deploy/seldon-control     1         1         1            0           1h
deploy/zookeeper1         1         1         1            0           1h
deploy/zookeeper2         1         1         1            0           1h
deploy/zookeeper3         1         1         1            0           1h

NAME                            DESIRED   CURRENT   READY     AGE
rs/influxdb-grafana-842592602   1         1         0         1h
rs/iris-xgboost-1750015301      1         1         0         14m
rs/memcached1-2136693305        1         1         0         1h
rs/memcached2-2533120572        1         1         0         1h
rs/mysql-2529449154             1         1         0         1h
rs/redis-1963070708             1         1         0         1h
rs/reuters-example-327623259    1         1         0         38m
rs/seldon-control-2832234452    1         1         0         1h
rs/zookeeper1-467704625         1         1         0         1h
rs/zookeeper2-1006738229        1         1         0         1h
rs/zookeeper3-1545771833        1         1         0         1h

I started microservice after running seldon, but status is still in 'creating'.

$ start-microservice --type prediction --client test -i iris-xgboost seldonio/iris_xgboost:2.1 rest 1.0
[Microservice(iris-xgboost,seldonio/iris_xgboost:2.1,rest,1.000000)]
Replicas is  1
kubectl apply -f /Users/kwangin/workspace/seldon-server/kubernetes/bin/../conf/microservices/microservice-iris-xgboost.json
deployment "iris-xgboost" created
service "iris-xgboost" created
$ kubectl get pods -l name=iris-xgboost
NAME                            READY     STATUS              RESTARTS   AGE
iris-xgboost-1750015301-sdjnm   0/1       ContainerCreating   0          15m

Could you find out something? Thanks.

ukclivecox commented 7 years ago

You would need to diagnose the issue by doing a "kubectl describe" on the problem pods and seeing what it says is the reason they are not starting.

kination commented 7 years ago

@cliveseldon Thanks. Which point do I have to look on kubectl describe? There are several options. Also, sometimes seldon-up ends like this.

...
NAME                               READY     STATUS              RESTARTS   AGE
influxdb-grafana-842592602-2q098   2/2       Running             0          10m
memcached1-2136693305-f164w        1/1       Running             0          10m
memcached2-2533120572-9h397        1/1       Running             0          10m
mysql-2529449154-pzvm4             0/1       Pending             0          10m
redis-1963070708-p593s             0/1       ContainerCreating   0          10m
reuters-import-data-cgx2j          0/1       ContainerCreating   0          22h
seldon-control-2832234452-t73hc    0/1       ContainerCreating   0          10m
zookeeper1-467704625-977z3         1/1       Running             0          10m
zookeeper2-1006738229-vtb8d        1/1       Running             0          10m
zookeeper3-1545771833-hp95t        1/1       Running             0          10m
Waiting for pods to be running as found 4 in non-running state
Sleeping for 5 seconds...
Unable to connect to the server: read tcp 192.168.99.1:51937->192.168.99.100:8443: read: connection reset by peer
The connection to the server 192.168.99.100:8443 was refused - did you specify the right host or port?
error: expected 'exec POD_NAME COMMAND [ARG1] [ARG2] ... [ARGN]'.
POD_NAME and COMMAND are required arguments for the exec command
See 'kubectl exec -h' for help and examples.

Could you find out? Thanks.