search

vmware-archive / kubeless

Kubernetes Native Serverless Framework
https://kubeless.io
Apache License 2.0
6.86k stars 754 forks source link

OS X DEV environment #426

Closed andresmgot closed 6 years ago

andresmgot commented 7 years ago

It is difficult to set up the Linux environment to build new features and run the integration test in a OS X environment.

We should provide the documentation and tools needed to build and test kubeless in OS X.

cc/ @phsiao

phsiao commented 7 years ago

Some details of the issue:

$ make integration-tests
./script/integration-tests
/usr/local/bin/kubectl
NAME       STATUS     ROLES     AGE       VERSION
minikube   NotReady   <none>    7d        v1.8.0
Error from server (AlreadyExists): namespaces "kubeless" already exists
 ✓ Verify TEST_CONTEXT envvar
 ✓ Verify needed kubernetes tools installed
 ✓ Verify minikube running (if context=='minikube')
 ✓ Verify k8s RBAC mode
 ✗ Test simple function failure without RBAC rules
   (from function `kubecfg' in file tests/../script/libtest.bash, line 39,
    from function `kubeless_recreate' in file tests/../script/libtest.bash, line 129,
    from function `test_must_fail_without_rbac_roles' in file tests/../script/libtest.bash, line 233,
    in test file tests/integration-tests.bats, line 32)
     `test_must_fail_without_rbac_roles' failed
   get-python   default     helloget.foo    python2.7   HTTP                            0/1 NOT READY
   No ksonnet application found
 ✗ Test simple function success with proper RBAC rules
   (from function `kubecfg' in file tests/../script/libtest.bash, line 39,
    from function `kubeless_recreate' in file tests/../script/libtest.bash, line 129,
    from function `test_must_pass_with_rbac_roles' in file tests/../script/libtest.bash, line 242,
    in test file tests/integration-tests.bats, line 35)
     `test_must_pass_with_rbac_roles' failed
   No ksonnet application found
   Test function: get-python

It did spin up several components in minikube, but looks like it can't get the actual runtime to be running.

NAMESPACE     NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
kube-system   deploy/kube-dns              1         1         1            1           7d
kubeless      deploy/kubeless-controller   1         1         1            1           1d

NAMESPACE     NAME                               DESIRED   CURRENT   READY     AGE
kube-system   rs/kube-dns-6fc954457d             1         1         1         7d
kubeless      rs/kubeless-controller-6f667955c   1         1         1         1d

NAMESPACE     NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
kube-system   deploy/kube-dns              1         1         1            1           7d
kubeless      deploy/kubeless-controller   1         1         1            1           1d

NAMESPACE   NAME                 DESIRED   CURRENT   AGE
kubeless    statefulsets/kafka   1         1         1d
kubeless    statefulsets/zoo     1         1         1d

NAMESPACE     NAME                               DESIRED   CURRENT   READY     AGE
kube-system   rs/kube-dns-6fc954457d             1         1         1         7d
kubeless      rs/kubeless-controller-6f667955c   1         1         1         1d

NAMESPACE     NAME                                     READY     STATUS    RESTARTS   AGE
kube-system   po/kube-addon-manager-minikube           1/1       Running   5          7d
kube-system   po/kube-dns-6fc954457d-l4dbk             3/3       Running   15         7d
kube-system   po/kubernetes-dashboard-866sq            1/1       Running   5          7d
kubeless      po/kafka-0                               1/1       Running   3          1d
kubeless      po/kubeless-controller-6f667955c-mrxfb   1/1       Running   1          1d
kubeless      po/zoo-0                                 1/1       Running   2          1d

NAMESPACE     NAME                      DESIRED   CURRENT   READY     AGE
kube-system   rc/kubernetes-dashboard   1         1         1         7d

NAMESPACE     NAME                       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)             AGE
default       svc/kubernetes             ClusterIP   10.0.0.1     <none>        443/TCP             7d
kube-system   svc/kube-dns               ClusterIP   10.0.0.10    <none>        53/UDP,53/TCP       7d
kube-system   svc/kubernetes-dashboard   NodePort    10.0.0.102   <none>        80:30000/TCP        7d
kubeless      svc/broker                 ClusterIP   None         <none>        9092/TCP            1d
kubeless      svc/kafka                  ClusterIP   10.0.0.51    <none>        9092/TCP            1d
kubeless      svc/zoo                    ClusterIP   None         <none>        9092/TCP,3888/TCP   1d
kubeless      svc/zookeeper              ClusterIP   10.0.0.70    <none>        2181/TCP            1d

This is my kubecfg version

$ kubecfg version
kubecfg version: (dev build)
jsonnet version: v0.9.4
client-go version: v1.6.8-beta.0+$Format:%h$
phsiao commented 7 years ago

Also, I can build controller, and minikube ssh into the VM to setup/update new controller docker image, as well as runtime images for testing, just could not run the integration-tests.

phsiao commented 7 years ago

Actually, later tests might be succeeding, I just was not patient enough to wait and let them run to its completion.

 ✗ Test simple function failure without RBAC rules
   (from function `kubecfg' in file tests/../script/libtest.bash, line 39,
    from function `kubeless_recreate' in file tests/../script/libtest.bash, line 129,
    from function `test_must_fail_without_rbac_roles' in file tests/../script/libtest.bash, line 233,
    in test file tests/integration-tests.bats, line 32)
     `test_must_fail_without_rbac_roles' failed
   get-python   default     helloget.foo    python2.7   HTTP                            0/1 NOT READY
   No ksonnet application found
 ✗ Test simple function success with proper RBAC rules
   (from function `kubecfg' in file tests/../script/libtest.bash, line 39,
    from function `kubeless_recreate' in file tests/../script/libtest.bash, line 129,
    from function `test_must_pass_with_rbac_roles' in file tests/../script/libtest.bash, line 242,
    in test file tests/integration-tests.bats, line 35)
     `test_must_pass_with_rbac_roles' failed
   No ksonnet application found
 ✓ Test function: get-python
 ✓ Test function: get-python-deps

Will report back on the rest when it completes.

phsiao commented 7 years ago

Here is after running for more than an hour.

$ make integration-tests
./script/integration-tests
/usr/local/bin/kubectl
NAME       STATUS    ROLES     AGE       VERSION
minikube   Ready     <none>    7d        v1.8.0
Error from server (AlreadyExists): namespaces "kubeless" already exists
 ✓ Verify TEST_CONTEXT envvar
 ✓ Verify needed kubernetes tools installed
 ✓ Verify minikube running (if context=='minikube')
 ✓ Verify k8s RBAC mode
 ✗ Test simple function failure without RBAC rules
   (from function `kubecfg' in file tests/../script/libtest.bash, line 39,
    from function `kubeless_recreate' in file tests/../script/libtest.bash, line 129,
    from function `test_must_fail_without_rbac_roles' in file tests/../script/libtest.bash, line 233,
    in test file tests/integration-tests.bats, line 32)
     `test_must_fail_without_rbac_roles' failed
   get-python   default     helloget.foo    python2.7   HTTP                            0/1 NOT READY
   No ksonnet application found
 ✗ Test simple function success with proper RBAC rules
   (from function `kubecfg' in file tests/../script/libtest.bash, line 39,
    from function `kubeless_recreate' in file tests/../script/libtest.bash, line 129,
    from function `test_must_pass_with_rbac_roles' in file tests/../script/libtest.bash, line 242,
    in test file tests/integration-tests.bats, line 35)
     `test_must_pass_with_rbac_roles' failed
   No ksonnet application found
 ✓ Test function: get-python
 ✓ Test function: get-python-deps
 ✓ Test function update: get-python
 ✓ Test function ingress: get-python
 ✗ Test function autoscale: get-python
   (from function `test_kubeless_autoscale' in file tests/../script/libtest.bash, line 306,
    in test file tests/integration-tests.bats, line 51)
     `test_kubeless_autoscale get-python' failed
   time="2017-11-13T13:06:35-05:00" level=fatal msg="Can't create autoscale: the server could not find the requested resource" 
 ✓ Test function: get-nodejs
 ✓ Test function: get-nodejs-deps
 ✓ Test function: get-nodejs-multi
 ✓ Test function: get-ruby
 ✓ Test function: get-ruby-deps
 ✗ Test function: get-dotnetcore
   (from function `verify_function' in file tests/../script/libtest.bash, line 268,
    from function `test_kubeless_function' in file tests/../script/libtest.bash, line 273,
    in test file tests/integration-tests.bats, line 69)
     `test_kubeless_function get-dotnetcore' failed with status 2
   time="2017-11-13T13:08:24-05:00" level=info msg="Deploying function..." 
   time="2017-11-13T13:08:24-05:00" level=info msg="Function get-dotnetcore submitted for deployment" 
   time="2017-11-13T13:08:24-05:00" level=info msg="Check the deployment status executing 'kubeless function ls get-dotnetcore'" 
   curl localhost:8080/api/v1/proxy/namespaces/default/services/get-dotnetcore/
   make[1]: *** [get-dotnetcore-verify] Error 1
 ✗ Test function: post-python
   (from function `verify_function' in file tests/../script/libtest.bash, line 268,
    from function `test_kubeless_function' in file tests/../script/libtest.bash, line 273,
    in test file tests/integration-tests.bats, line 72)
     `test_kubeless_function post-python' failed with status 2
   time="2017-11-13T13:08:45-05:00" level=info msg="Deploying function..." 
   time="2017-11-13T13:08:45-05:00" level=info msg="Function post-python submitted for deployment" 
   time="2017-11-13T13:08:45-05:00" level=info msg="Check the deployment status executing 'kubeless function ls post-python'" 
   curl --data '{"hello":"world"}' localhost:8080/api/v1/proxy/namespaces/default/services/post-python/ --header "Content-Type:application/json"
   make[1]: *** [post-python-verify] Error 1
 ✓ Test function: post-nodejs
 ✓ Test function: post-ruby
 ✗ Test function: post-dotnetcore
   (from function `verify_function' in file tests/../script/libtest.bash, line 268,
    from function `test_kubeless_function' in file tests/../script/libtest.bash, line 273,
    in test file tests/integration-tests.bats, line 81)
     `test_kubeless_function post-dotnetcore' failed with status 2
   time="2017-11-13T13:08:57-05:00" level=info msg="Deploying function..." 
   time="2017-11-13T13:08:57-05:00" level=info msg="Function post-dotnetcore submitted for deployment" 
   time="2017-11-13T13:08:57-05:00" level=info msg="Check the deployment status executing 'kubeless function ls post-dotnetcore'" 
   make[1]: *** [post-dotnetcore-verify] Error 1
 ✓ Test function: get-python-metadata
 ✓ Test function: pubsub-python
 ✓ Test function update: pubsub-python
 ✗ Test function: pubsub-nodejs
   (from function `verify_function' in file tests/../script/libtest.bash, line 268,
    from function `test_kubeless_function' in file tests/../script/libtest.bash, line 273,
    in test file tests/integration-tests.bats, line 93)
     `test_kubeless_function pubsub-nodejs' failed with status 2
   Created topic "s3-nodejs".
   time="2017-11-13T13:11:41-05:00" level=info msg="Deploying function..." 
   time="2017-11-13T13:11:41-05:00" level=info msg="Function pubsub-nodejs submitted for deployment" 
   time="2017-11-13T13:11:41-05:00" level=info msg="Check the deployment status executing 'kubeless function ls pubsub-nodejs'" 
   mktemp: illegal option -- p
   usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
          mktemp [-d] [-q] [-u] -t prefix 
   bash: timeout: command not found
   make[1]: *** [pubsub-nodejs-verify] Error 1
 ✓ Test function: pubsub-ruby
 ✓ Test topic list
 - Test topic delete (skipped: topic delete does nothing with kafka delete.topic.enable=false)
   Test custom runtime image

And it has been stuck in the last test, "Test custom runtime image " for a long time and I don't have any reason to believe that it will make progress.

phsiao commented 6 years ago

I think my problem might be due to incompatible kubecfg (binary timestamp in late Oct). Re-running with the current dev version of kubecfg.

phsiao commented 6 years ago

Better, but stil have a few issues.

$ make integration-tests
./script/integration-tests
/usr/local/bin/kubectl
NAME       STATUS    ROLES     AGE       VERSION
minikube   Ready     <none>    12d       v1.8.0
Error from server (AlreadyExists): namespaces "kubeless" already exists
 ✓ Verify TEST_CONTEXT envvar
 ✓ Verify needed kubernetes tools installed
 ✓ Verify minikube running (if context=='minikube')
 ✓ Verify k8s RBAC mode
 ✓ Test simple function failure without RBAC rules
 ✓ Test simple function success with proper RBAC rules
 ✓ Test function: get-python
 ✓ Test function: get-python-deps
 ✓ Test function update: get-python
 ✓ Test function ingress: get-python
 ✗ Test function autoscale: get-python
   (from function `test_kubeless_autoscale' in file tests/../script/libtest.bash, line 306,
    in test file tests/integration-tests.bats, line 51)
     `test_kubeless_autoscale get-python' failed
   time="2017-11-19T10:09:40-05:00" level=fatal msg="Can't create autoscale: the server could not find the requested resource"
 ✓ Test function: get-nodejs
 ✓ Test function: get-nodejs-deps
 ✓ Test function: get-nodejs-multi
 ✓ Test function: get-ruby
 ✓ Test function: get-ruby-deps
 ✗ Test function: get-dotnetcore
   (from function `verify_function' in file tests/../script/libtest.bash, line 268,
    from function `test_kubeless_function' in file tests/../script/libtest.bash, line 273,
    in test file tests/integration-tests.bats, line 69)
     `test_kubeless_function get-dotnetcore' failed with status 2
   time="2017-11-19T10:10:11-05:00" level=info msg="Deploying function..."
   time="2017-11-19T10:10:11-05:00" level=info msg="Function get-dotnetcore submitted for deployment"
   time="2017-11-19T10:10:11-05:00" level=info msg="Check the deployment status executing 'kubeless function ls get-dotnetcore'"
   curl localhost:8080/api/v1/proxy/namespaces/default/services/get-dotnetcore/
   time="2017-11-19T10:10:14-05:00" level=error
   time="2017-11-19T10:10:14-05:00" level=fatal msg="an error on the server (\"Error: 'dial tcp 172.17.0.14:8080: getsockopt: connection refused'\nTrying to reach: 'http://172.17.0.14:8080/'\") has prevented the request from succeeding (get services get-dotnetcore:function-port)"
   make[1]: *** [get-dotnetcore-verify] Error 1
 ✓ Test function: post-python
 ✓ Test function: post-nodejs
 ✓ Test function: post-ruby
 ✗ Test function: post-dotnetcore
   (from function `verify_function' in file tests/../script/libtest.bash, line 268,
    from function `test_kubeless_function' in file tests/../script/libtest.bash, line 273,
    in test file tests/integration-tests.bats, line 81)
     `test_kubeless_function post-dotnetcore' failed with status 2
   time="2017-11-19T10:10:27-05:00" level=info msg="Deploying function..."
   time="2017-11-19T10:10:27-05:00" level=info msg="Function post-dotnetcore submitted for deployment"
   time="2017-11-19T10:10:27-05:00" level=info msg="Check the deployment status executing 'kubeless function ls post-dotnetcore'"
   time="2017-11-19T10:10:31-05:00" level=error
   time="2017-11-19T10:10:31-05:00" level=fatal msg="an error on the server (\"Error: 'dial tcp 172.17.0.18:8080: getsockopt: connection refused'\nTrying to reach: 'http://172.17.0.18:8080/'\") has prevented the request from succeeding"
   make[1]: *** [post-dotnetcore-verify] Error 1
 ✓ Test function: get-python-metadata
 ✓ Test function: pubsub-python
 ✓ Test function update: pubsub-python
 ✗ Test function: pubsub-nodejs
   (from function `verify_function' in file tests/../script/libtest.bash, line 268,
    from function `test_kubeless_function' in file tests/../script/libtest.bash, line 273,
    in test file tests/integration-tests.bats, line 93)
     `test_kubeless_function pubsub-nodejs' failed with status 2
   Created topic "s3-nodejs".
   time="2017-11-19T10:12:48-05:00" level=info msg="Deploying function..."
   time="2017-11-19T10:12:48-05:00" level=info msg="Function pubsub-nodejs submitted for deployment"
   time="2017-11-19T10:12:48-05:00" level=info msg="Check the deployment status executing 'kubeless function ls pubsub-nodejs'"
   mktemp: illegal option -- p
   usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
          mktemp [-d] [-q] [-u] -t prefix 
   bash: timeout: command not found
   make[1]: *** [pubsub-nodejs-verify] Error 1
 ✓ Test function: pubsub-ruby
 ✓ Test topic list
 ✓ Test topic deletion
 ^CTest custom runtime image

I had to abort the Test custom runtime image because it just took too long.

andresmgot commented 6 years ago

We probably want to provide a docker environment in which you can mount your GOPATH and execute the unit and integration tests.

andresmgot commented 6 years ago

@phsiao can you try the above PR? In particular the script start-test-environment.sh. It should start a minikube environment and give you a bash shell in which you can run make binary && make controller-image CONTROLLER_IMAGE=bitnami/kubeless-controller:latest && make integration-tests.

Let us know if that worked for you.

phsiao commented 6 years ago

Commented my test in #458.

andresmgot commented 6 years ago

Closed with #458