Open rushins opened 6 years ago
Did you install kubeless itself in your k8s cluster ?
yes, i did.
Can you tell us a which version of kubeless and which version of k8s you are running ?
Also if you deploy a function with kubeless deploy
does it work ?
hi @rushins
there are two options here: either the function is deployed but the client is not able to find it (which can be similar to #77) or the function failed to be deployed.
If you can see the pod when running kubectl get pods
it is probably the first option (you can check the linked issue to try to debug it). If no pod is shown you can use the controller logs to debug what's happening. You can obtain the logs executing kubectl logs -n kubeless $(kubectl get pods -n kubeless -l kubeless=controller -o=name)
.
@sebgoa
my kubless version: 0.0.20 my K8S version: 1.7.3
$ kubectl version $ brew install kubeless/tap/kubeless $ KUBELESS_VERSION=0.2.4 $ kubectl create ns kubeless $ kubectl create -f https://github.com/kubeless/kubeless/releases/download/v$KUBELESS_VERSION/kubeless-v$KUBELESS_VERSION.yaml Then install serverless
$ npm install serverless -g Try out the example
Clone this repo and check the example function
$ git clone https://github.com/serverless/serverless-kubeless $ cd examples/get-python $ cat serverless.yml service: hello
provider: name: kubeless runtime: python2.7
plugins:
functions: hello: description: 'Hello function' handler: handler.hello Download dependencies
i get this error while deploying (serverless deploy)
======================================= Giving up, unable to retrieve the status of the hello deployment. /Users/i811144/rushi/devops/serverless/kubeless/serverless-kubeless/examples/get-python/node_modules/bluebird/js/release/async.js:61 fn = function () { throw arg; }; ^
@rushins Probably something went wrong with the deployment. Do you have RBAC enabled in your cluster? Can you send us the output of executing kubectl logs -n kubeless $(kubectl get pods -n kubeless -l kubeless=controller -o=name)
?
yes, its RBAC enabled in cluster.
i get this output from the command you requested. (kubectl logs -n kubeless $(kubectl get pods -n kubeless -l kubeless=controller -o=name)
1 reflector.go:201] github.com/kubeless/kubeless/pkg/controller/controller.go:118: Failed to list spec.Function: User "system:serviceaccount:kubeless:controller-acct" cannot list functions.k8s.io at the cluster scope. (get functions.k8s.io) E1121 10:03:21.480131 1 reflector.go:201] github.com/kubeless/kubeless/pkg/controller/controller.go:118: Failed to list spec.Function: User "system:serviceaccount:kubeless:controller-acct" cannot list functions.k8s.io at the cluster scope. (get functions.k8s.io) E1121 10:03:22.489364 1 reflector.go:201] github.com/kubeless/kubeless/pkg/controller/controller.go:118: Failed to list spec.Function: User "system:serviceaccount:kubeless:controller-acct" cannot list functions.k8s.io at the cluster scope. (get functions.k8s.io) E1121 10:03:23.498692 1 reflector.go:201] github.com/kubeless/kubeless/pkg/controller/controller.go:118: Failed to list spec.Function: User "system:servi^C
If that is the case you need to deploy the RBAC version of kubeless:
kubectl create -f https://github.com/kubeless/kubeless/releases/download/v$KUBELESS_VERSION/kubeless-rbac-v$KUBELESS_VERSION.yaml
ok i tried but it has all old ones which i wasnt able to remove.
Error from server (AlreadyExists): error when creating "https://github.com/kubeless/kubeless/releases/download/v0.2.4/kubeless-rbac-v0.2.4.yaml": statefulsets.apps "kafka" already exists Error from server (AlreadyExists): error when creating "https://github.com/kubeless/kubeless/releases/download/v0.2.4/kubeless-rbac-v0.2.4.yaml": services "kafka" already exists Error from server (AlreadyExists): error when creating "https://github.com/kubeless/kubeless/releases/download/v0.2.4/kubeless-rbac-v0.2.4.yaml": deployments.extensions "kubeless-controller" already exists Error from server (AlreadyExists): error when creating "https://github.com/kubeless/kubeless/releases/download/v0.2.4/kubeless-rbac-v0.2.4.yaml": serviceaccounts "controller-acct" already exists Error from server (AlreadyExists): error when creating "https://github.com/kubeless/kubeless/releases/download/v0.2.4/kubeless-rbac-v0.2.4.yaml": clusterroles.rbac.authorization.k8s.io "kubeless-controller-deployer" already exists Error from server (AlreadyExists): error when creating "https://github.com/kubeless/kubeless/releases/download/v0.2.4/kubeless-rbac-v0.2.4.yaml": clusterrolebindings.rbac.authorization.k8s.io "kubeless-controller-deployer" already exists Error from server (AlreadyExists): error when creating "https://github.com/kubeless/kubeless/releases/download/v0.2.4/kubeless-rbac-v0.2.4.yaml": customresourcedefinitions.apiextensions.k8s.io "functions.k8s.io" already exists Error from server (AlreadyExists): error when creating "https://github.com/kubeless/kubeless/releases/download/v0.2.4/kubeless-rbac-v0.2.4.yaml": services "broker" already exists Error from server (AlreadyExists): error when creating "https://github.com/kubeless/kubeless/releases/download/v0.2.4/kubeless-rbac-v0.2.4.yaml": services "zoo" already exists Error from server (AlreadyExists): error when creating "https://github.com/kubeless/kubeless/releases/download/v0.2.4/kubeless-rbac-v0.2.4.yaml": statefulsets.apps "zoo" already exists Error from server (AlreadyExists): error when creating "https://github.com/kubeless/kubeless/releases/download/v0.2.4/kubeless-rbac-v0.2.4.yaml": services "zookeeper" already exists
i tried to clean up and deleted majority of the old ones like deployments, services, pods but i wasn't able to delete the pod "zoo-0" .
Sorry, you can clean up the installation using kubectl delete ns kubeless
but in any case if the elements has been already created your issue should be solved. Can you try to deploy the function again?
@andresmgot : i did cleanup and tried. this time worked nice but one particular thing is still not working (sererless logs -f hello). here are my ouptuts.
kubectl get function NAME AGE hello 6d
hello world
$ serverless logs -f hello /Users/i811144/rushi/devops/serverless/kubeless/serverless-kubeless/examples/get-python/node_modules/serverless-kubeless/lib/get-logs.js:101 podInfo.metadata.labels.function === func ^
TypeError: Cannot read property 'function' of undefined
at _.filter (/Users/i811144/rushi/devops/serverless/kubeless/serverless-kubeless/examples/get-python/node_modules/serverless-kubeless/lib/get-logs.js:101:44)
at arrayFilter (/Users/i811144/rushi/devops/serverless/kubeless/serverless-kubeless/examples/get-python/node_modules/lodash/lodash.js:603:11)
at Function.filter (/Users/i811144/rushi/devops/serverless/kubeless/serverless-kubeless/examples/get-python/node_modules/lodash/lodash.js:9190:14)
at core.ns.pods.get (/Users/i811144/rushi/devops/serverless/kubeless/serverless-kubeless/examples/get-python/node_modules/serverless-kubeless/lib/get-logs.js:98:30)
at /Users/i811144/rushi/devops/serverless/kubeless/serverless-kubeless/examples/get-python/node_modules/kubernetes-client/lib/base.js:15:5
at Request.request [as _callback] (/Users/i811144/rushi/devops/serverless/kubeless/serverless-kubeless/examples/get-python/node_modules/kubernetes-client/lib/api-group.js:127:7)
at Request.self.callback (/Users/i811144/rushi/devops/serverless/kubeless/serverless-kubeless/examples/get-python/node_modules/request/request.js:186:22)
at emitTwo (events.js:125:13)
at Request.emit (events.js:213:7)
at Request.
Any idea why this "serverless logs -f hello" typeerror: cannot read property 'function" of undefined.
Hi @rushins, indeed that's a bug on our side. The command is finding a pod (that is not relevant to serverless) and since it doesn't have labels it causes the crash. The above PR should fix it.
The fix is now landed, @rushins can you try with the master version of the plugin and check if it fix your issue?
i will try and let you know . thanks for quick fix.
hi @andresmgot : no change i still see the error while running the logs command
serverless logs -f hello /Users/i811144/rushi/devops/kubernetes/sapinternalprojects/sdnpal/kubeless/serverless-kubeless/examples/get-python/node_modules/serverless-kubeless/lib/get-logs.js:101 podInfo.metadata.labels.function === func ^
TypeError: Cannot read property 'function' of undefined
at _.filter (/Users/i811144/rushi/devops/kubernetes/sapinternalprojects/sdnpal/kubeless/serverless-kubeless/examples/get-python/node_modules/serverless-kubeless/lib/get-logs.js:101:44)
at arrayFilter (/Users/i811144/rushi/devops/kubernetes/sapinternalprojects/sdnpal/kubeless/serverless-kubeless/examples/get-python/node_modules/lodash/lodash.js:603:11)
at Function.filter (/Users/i811144/rushi/devops/kubernetes/sapinternalprojects/sdnpal/kubeless/serverless-kubeless/examples/get-python/node_modules/lodash/lodash.js:9190:14)
at core.ns.pods.get (/Users/i811144/rushi/devops/kubernetes/sapinternalprojects/sdnpal/kubeless/serverless-kubeless/examples/get-python/node_modules/serverless-kubeless/lib/get-logs.js:98:30)
at /Users/i811144/rushi/devops/kubernetes/sapinternalprojects/sdnpal/kubeless/serverless-kubeless/examples/get-python/node_modules/kubernetes-client/lib/base.js:15:5
at Request.request [as _callback] (/Users/i811144/rushi/devops/kubernetes/sapinternalprojects/sdnpal/kubeless/serverless-kubeless/examples/get-python/node_modules/kubernetes-client/lib/api-group.js:127:7)
at Request.self.callback (/Users/i811144/rushi/devops/kubernetes/sapinternalprojects/sdnpal/kubeless/serverless-kubeless/examples/get-python/node_modules/request/request.js:186:22)
at emitTwo (events.js:125:13)
at Request.emit (events.js:213:7)
at Request.
hi @rushins, it seems that you are still using the old code (serverless-kubeless/lib/get-logs.js:101 is not podInfo.metadata.labels.function === func
anymore).
Let me release a new patch version, that will make it easier to see the changes.
you should be able to see the fix with the latest version 0.2.4
same error here
consent sls deploy -v
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Deploying function capitalize...
Serverless: Unable to find any running pod for capitalize. Retrying...
Serverless: Unable to find any running pod for capitalize. Retrying...
Serverless: Unable to find any running pod for capitalize. Retrying...
Serverless: Unable to find any running pod for capitalize. Retrying...
Serverless: Giving up, unable to retrieve the status of the capitalize deployment.
/Users/gabrielmancini/Developer/Watson/test/kubeless/consent/node_modules/bluebird/js/release/async.js:61
fn = function () { throw arg; };
^
Error: Unable to retrieve the status of the capitalize deployment
at module.exports.logError.errorHandlingError (/Users/gabrielmancini/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/Error.js:94:11)
at process.on (/Users/gabrielmancini/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/bin/serverless:21:3)
at emitTwo (events.js:106:13)
at process.emit (events.js:191:7)
at methodName (/Users/gabrielmancini/Developer/Watson/test/kubeless/consent/node_modules/bluebird/js/release/debuggability.js:189:33)
at activeFireEvent (/Users/gabrielmancini/Developer/Watson/test/kubeless/consent/node_modules/bluebird/js/release/debuggability.js:232:44)
at fireRejectionEvent (/Users/gabrielmancini/Developer/Watson/test/kubeless/consent/node_modules/bluebird/js/release/debuggability.js:617:14)
at Promise._notifyUnhandledRejection (/Users/gabrielmancini/Developer/Watson/test/kubeless/consent/node_modules/bluebird/js/release/debuggability.js:64:9)
at Timeout._onTimeout (/Users/gabrielmancini/Developer/Watson/test/kubeless/consent/node_modules/bluebird/js/release/debuggability.js:43:14)
at ontimeout (timers.js:386:14)
at tryOnTimeout (timers.js:250:5)
at Timer.listOnTimeout (timers.js:214:5)
when try deploy again:
consent sls deploy -v
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Error --------------------------------------------------
Found errors while deploying the given functions:
Error: Unable to update the function capitalize. Received:
Code: 422
Message: functions.k8s.io "capitalize" is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update
For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.
Get Support --------------------------------------------
Docs: docs.serverless.com
Bugs: github.com/serverless/serverless/issues
Forums: forum.serverless.com
Chat: gitter.im/serverless/serverless
Your Environment Information -----------------------------
OS: darwin
Node Version: 6.10.3
Serverless Version: 1.24.1
hi @gabrielmancini,
can you send the logs of the controller? You can retrieve them executing:
kubectl logs -n kubeless $(kubectl get pods -n kubeless -l kubeless=controller -o=name)
I had the same issue and I was using v0.3.2 for kubeless.
So I checked if there was a new version for that and I found 0.3.3
And it said it fixed a backward incompatibility bug so I figured that it might be the issue.
So I updated kubeless
binary to 0.3.3 and still had the same issue.
And I updated kubeless-controller
's image inside k8s cluster also to 0.3.3 and tried again and finally it worked!
@ryuheechul Thanks for your confirmation. Yes v0.3.2 was sent out with a backward incompatibility issue (we apologize for that) and v0.3.3 fixed that. This serverless plugin was tested with v0.3.3 so anyone got trouble with v0.3.2 please consider to upgrade
@ngtuna You are very welcome!
@andresmgot
using kubeless v0.3.3
serverless -v 1.25.0
now on linux, when deploy
serverless deploy -v
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Deploying function capitalize...
Serverless: Unable to find any running pod for capitalize. Retrying...
Serverless: Unable to find any running pod for capitalize. Retrying...
Serverless: Unable to find any running pod for capitalize. Retrying...
Serverless: Unable to find any running pod for capitalize. Retrying...
Serverless: Giving up, unable to retrieve the status of the capitalize deployment.
/home/gabriel/Developer/kubernetes/test-serverless/node_modules/bluebird/js/release/async.js:61
fn = function () { throw arg; };
^
Error: Unable to retrieve the status of the capitalize deployment
at module.exports.logError (/home/gabriel/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/lib/classes/Error.js:94:11)
at process.on (/home/gabriel/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/bin/serverless:21:3)
at emitTwo (events.js:126:13)
at process.emit (events.js:214:7)
at methodName (/home/gabriel/Developer/kubernetes/test-serverless/node_modules/bluebird/js/release/debuggability.js:189:33)
at activeFireEvent (/home/gabriel/Developer/kubernetes/test-serverless/node_modules/bluebird/js/release/debuggability.js:232:44)
at fireRejectionEvent (/home/gabriel/Developer/kubernetes/test-serverless/node_modules/bluebird/js/release/debuggability.js:617:14)
at Promise._notifyUnhandledRejection (/home/gabriel/Developer/kubernetes/test-serverless/node_modules/bluebird/js/release/debuggability.js:64:9)
at Timeout._onTimeout (/home/gabriel/Developer/kubernetes/test-serverless/node_modules/bluebird/js/release/debuggability.js:43:14)
at ontimeout (timers.js:475:11)
at tryOnTimeout (timers.js:310:5)
at Timer.listOnTimeout (timers.js:270:5)
when try deploy again
serverless deploy -vServerless: Packaging service...
Serverless: Excluding development dependencies...
Error --------------------------------------------------
Found errors while deploying the given functions:
Error: Unable to update the function capitalize. Received:
Code: 422
Message: functions.k8s.io "capitalize" is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update
For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.
Get Support --------------------------------------------
Docs: docs.serverless.com
Bugs: github.com/serverless/serverless/issues
Forums: forum.serverless.com
Chat: gitter.im/serverless/serverless
Your Environment Information -----------------------------
OS: linux
Node Version: 8.9.4
Serverless Version: 1.25.0
the log u ask
kubectl logs -n kubeless $(kubectl get pods -n kubeless -l kubeless=controller -o=name)
time="2018-01-09T20:12:30Z" level=info msg="Starting kubeless controller" pkg=controller
time="2018-01-09T20:12:30Z" level=info msg="Kubeless controller synced and ready" pkg=controller
time="2018-01-09T20:19:14Z" level=info msg="Processing change to Function default/capitalize" pkg=controller
time="2018-01-09T20:19:14Z" level=info msg="Updated Function default/capitalize" pkg=controller
versions
kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T21:07:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4", GitTreeState:"clean", BuildDate:"2017-11-29T22:43:34Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"}
@gabrielmancini thank you for the logs. It seems that the function gets successfully created:
time="2018-01-09T20:19:14Z" level=info msg="Processing change to Function default/capitalize" pkg=controller
time="2018-01-09T20:19:14Z" level=info msg="Updated Function default/capitalize" pkg=controller
but the client is not able to retrieve the deployment information. Does the deployment/pod gets created? To try to reproduce it, in which platform are you running Kubernetes? Does it have RBAC enabled?
cat serverless.yml
# Welcome to Serverless!
#
# For full config options, check the kubeless plugin docs:
# https://github.com/serverless/serverless-kubeless
#
# For documentation on kubeless itself:
# http://kubeless.io
# Update the service name below with your own service name
service: test-serverless
# Please ensure the serverless-kubeless provider plugin is installed globally.
# $ npm install -g serverless-kubeless
#
# ...before installing project dependencies to register this provider.
# $ npm install
provider:
name: kubeless
runtime: nodejs8
plugins:
- serverless-kubeless
functions:
capitalize:
handler: handler.capitalize
cat handler.js
'use strict'
const _ = require('lodash')
module.exports = {
capitalize (req, res) {
let body = []
req.on('error', (err) => {
console.error(err)
}).on('data', (chunk) => {
body.push(chunk)
}).on('end', () => {
body = Buffer.concat(body).toString()
res.end(_.capitalize(body))
})
}
}
cat package.json
{
"name": "kubeless-nodejs",
"version": "1.0.0",
"description": "Example function for serverless kubeless",
"dependencies": {
"serverless-kubeless": "^0.1.8",
"lodash": "^4.1.0"
},
"devDependencies": {},
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"keywords": [
"serverless",
"kubeless"
],
"author": "The Kubeless Authors",
"license": "Apache-2.0"
}
@gabrielmancini those files seem correct, does serverless invoke -f capitalize --data "hello world" -l
works for you?
If that is the case, the issue about the client not able to retrieve the function could be caused due to a bad synchronization between the client clock and the Kubernetes cluster. When you deploy a function and then you execute kubectl get pods
does the function pod get created with an AGE
of 0 seconds?
If there are bad values in the labels
part of your serverless.yml
for your functions you will encounter this issue, where the functions exist in k8s but the pods cannot be started due to a ReadString
error. See: https://github.com/serverless/serverless/pull/4993
hi @ianserlin, I have just released the version v0.4.2 for fixing that issue. Could you check if it is now resolved?
I can confirm this now converts to string and does not cause an issue, I'll close the PR in the serverless repo. Thanks @andresmgot!
hi @rushins I've just encountered the same issue as you
[root@localhost ~]# kubectl create -f kubeless-v1.0.0-alpha.7.yaml deployment.apps/kubeless-controller-manager created serviceaccount/controller-acct created Error from server (AlreadyExists): error when creating "kubeless-v1.0.0-alpha.7.yaml": clusterroles.rbac.authorization.k8s.io "kubeless-controller-deployer" already exists Error from server (AlreadyExists): error when creating "kubeless-v1.0.0-alpha.7.yaml": clusterrolebindings.rbac.authorization.k8s.io "kubeless-controller-deployer" already exists error validating "kubeless-v1.0.0-alpha.7.yaml": error validating data: ValidationError(CustomResourceDefinition): unknown field "description" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.CustomResourceDefinition; if you choose to ignore these errors, turn validation off with --validate=false
how did you clean this already exists ones ?
Just deleted the old namespace completely and reinstall as stated “kubectl delete ns kubeless” first and create again “kubectl create ns kubeless” and install .
i got this error "Unable to find any running pod for hello. Retrying.." while doing serverless deploy ? i have cloned all the files and i have kubernetes cluster running on-premise ? what i am missing here