Starting kubeless | kubeless-controller-manager in CrashLoopBackOff

iamshreeram commented 4 years ago

Is this a BUG REPORT or FEATURE REQUEST?: BUG

What happened: Installed kubeless with below

kube-shell> kubectl create ns kubeless
namespace/kubeless created
kube-shell> kubectl create -f https://github.com/kubeless/kubeless/releases/download/v1.0.7/kubeless-non-rbac-v1.0.7.yaml
configmap/kubeless-config created
deployment.apps/kubeless-controller-manager created
serviceaccount/controller-acct created
customresourcedefinition.apiextensions.k8s.io/functions.kubeless.io created
customresourcedefinition.apiextensions.k8s.io/httptriggers.kubeless.io created
customresourcedefinition.apiextensions.k8s.io/cronjobtriggers.kubeless.io created
kube-shell> kubectl --namespace kubeless get pods
NAME                                          READY   STATUS             RESTARTS   AGE
kubeless-controller-manager-67d484f5d-xf2mj   0/3     CrashLoopBackOff   9          107s

What you expected to happen: Kubeless Pods being created

How to reproduce it (as minimally and precisely as possible):

kubectl create ns kubeless
kubectl create -f https://github.com/kubeless/kubeless/releases/download/v1.0.7/kubeless-non-rbac-v1.0.7.yaml

Anything else we need to know?:

Environment:

Kubernetes version (use kubectl version):

Client Version: v1.16.6-beta.0
Server Version: v1.18.3

Kubeless version (use kubeless version): v1.0.7-dirty
Cloud provider or physical cluster: Local Mac

Pod description :

Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  2m25s                  default-scheduler  Successfully assigned kubeless/kubeless-controller-manager-67d484f5d-xf2mj  to minikube
  Normal   Created    2m22s (x2 over 2m23s)  kubelet, minikube  Created container http-trigger-controller
  Normal   Created    2m22s (x2 over 2m23s)  kubelet, minikube  Created container kubeless-function-controller
  Normal   Started    2m22s (x2 over 2m23s)  kubelet, minikube  Started container kubeless-function-controller
  Normal   Pulled     2m22s (x2 over 2m23s)  kubelet, minikube  Container image "kubeless/http-trigger-controller:v1.0.1" already present on machine
  Normal   Started    2m21s (x2 over 2m23s)  kubelet, minikube  Started container http-trigger-controller
  Normal   Started    2m21s (x2 over 2m22s)  kubelet, minikube  Started container cronjob-trigger-controller
  Normal   Pulled     2m21s (x2 over 2m23s)  kubelet, minikube  Container image "kubeless/cronjob-trigger-controller:v1.0.1" already present on machine
  Normal   Created    2m21s (x2 over 2m23s)  kubelet, minikube  Created container cronjob-trigger-controller
  Warning  BackOff    2m20s (x2 over 2m21s)  kubelet, minikube  Back-off restarting failed container
  Warning  BackOff    2m20s (x2 over 2m21s)  kubelet, minikube  Back-off restarting failed container
  Warning  BackOff    2m20s (x2 over 2m21s)  kubelet, minikube  Back-off restarting failed container
  Normal   Pulled     2m8s (x3 over 2m23s)   kubelet, minikube  Container image "kubeless/function-controller:v1.0.7" already present on machine

andresmgot commented 4 years ago

I @iamshreeram, can you get the logs of the container failing? First you need to get which one of the 3 containers is failing (you can that info from the pod description), then get the logs with kubectl logs -n kubeless kubeless-controller-manager-67d484f5d-xf2mj -c <container>. That should give more info about what's gone wrong.

iamshreeram commented 4 years ago

@andresmgot , Thanks for looking at the issue.

Below is the error for all 3 containers [kubeless-function-controller, http-trigger-controller, cronjob-trigger-controller] in all 3 pods :

time="2020-07-20T12:17:04Z" level=info msg="Running Kubeless controller manager version: v1.0.7"
time="2020-07-20T12:17:04Z" level=fatal msg="Unable to read the configmap: Error while fetching config location: customresourcedefinitions.apiextensions.k8s.io \"functions.kubeless.io\" is forbidden: User \"system:serviceaccount:kubeless:controller-acct\" cannot get resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope"

andresmgot commented 4 years ago

ah, I see the problem. You are using the manifest that doesn't include the RBAC configuration (and RBAC is now always enabled by default). Install Kubeless like this:

kubectl create -f https://github.com/kubeless/kubeless/releases/download/v1.0.7/kubeless-v1.0.7.yaml

(rather than using the non-rbac version)

iamshreeram commented 4 years ago

@andresmgot , Thanks for response.

Does this mean users can never use non-rbac - kubectl create -f https://github.com/kubeless/kubeless/releases/download/v1.0.7/kubeless-non-rbac-v1.0.7.yaml? If so, can this be removed from documentation?

andresmgot commented 4 years ago

While it's not usual, people can disable RBAC in their clusters. If they do so, the manifest with RBAC rules will fail, that's why we offer the two options. In the documentation, we point to the other manifest but if you think of another way of making it more prominent please send a PR to help others:

https://github.com/kubeless/kubeless/blob/master/docs/quick-start.md

iamshreeram commented 4 years ago

@andresmgot , I think we can just remove - kubeless-non-rbac-$RELEASE.yaml is used for non-RBAC Kubernetes cluster. from the doc and provide documentation to disable RBAC.

Let me know. thanks!

andresmgot commented 4 years ago

Yes, we can do that. I don't think that file is used very often.

fc7 commented 3 years ago

I had the same issue, and it took me a while until I found this page. I also vote for the above proposal.

shubhomoy commented 3 years ago

I am having the same issue but not due to RBAC

kubeless-controller-manager-5d9bf7f5c6-hs7s5   2/3     CrashLoopBackOff   5          3m30s

The cronjob-trigger-controller container is crashing. Following with following the logs

time="2021-05-02T20:30:09Z" level=info msg="Running Kubeless cronjob trigger controller version: v1.0.3"
time="2021-05-02T20:30:09Z" level=fatal msg="Unable to read the configmap: Error while fetching config location: v1beta1.CustomResourceDefinition: ObjectMeta: v1.ObjectMeta: readObjectFieldAsBytes: expect : after object field, parsing 535 ...s\":{\"k:{\\\"... at {\"kind\":\"CustomResourceDefinition\",\"apiVersion\":\"apiextensions.k8s.io/v1beta1\",\"metadata\":{\"name\":\"functions.kubeless.io\",\"uid\":\"1fb634d2-032e-472b-a8c7-10edd69e96f8\",\"resourceVersion\":\"582946\",\"generation\":1,\"creationTimestamp\":\"2021-05-02T20:29:48Z\",\"managedFields\":[{\"manager\":\"kube-apiserver\",\"operation\":\"Update\",\"apiVersion\":\"apiextensions.k8s.io/v1\",\"time\":\"2021-05-02T20:29:48Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:status\":{\"f:acceptedNames\":{\"f:kind\":{},\"f:listKind\":{},\"f:plural\":{},\"f:singular\":{}},\"f:conditions\":{\"k:{\\\"type\\\":\\\"Established\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}},\"k:{\\\"type\\\":\\\"NamesAccepted\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}},\"k:{\\\"type\\\":\\\"NonStructuralSchema\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}}}}}},{\"manager\":\"kubectl-create\",\"operation\":\"Update\",\"apiVersion\":\"apiextensions.k8s.io/v1beta1\",\"time\":\"2021-05-02T20:29:48Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:spec\":{\"f:conversion\":{\".\":{},\"f:strategy\":{}},\"f:group\":{},\"f:names\":{\"f:kind\":{},\"f:listKind\":{},\"f:plural\":{},\"f:singular\":{}},\"f:preserveUnknownFields\":{},\"f:scope\":{},\"f:version\":{},\"f:versions\":{}}}}]},\"spec\":{\"group\":\"kubeless.io\",\"version\":\"v1beta1\",\"names\":{\"plural\":\"functions\",\"singular\":\"function\",\"kind\":\"Function\",\"listKind\":\"FunctionList\"},\"scope\":\"Namespaced\",\"versions\":[{\"name\":\"v1beta1\",\"served\":true,\"storage\":true}],\"conversion\":{\"strategy\":\"None\"},\"preserveUnknownFields\":true},\"status\":{\"conditions\":[{\"type\":\"NamesAccepted\",\"status\":\"True\",\"lastTransitionTime\":\"2021-05-02T20:29:48Z\",\"reason\":\"NoConflicts\",\"message\":\"no conflicts found\"},{\"type\":\"Established\",\"status\":\"True\",\"lastTransitionTime\":\"2021-05-02T20:29:48Z\",\"reason\":\"InitialNamesAccepted\",\"message\":\"the initial names have been accepted\"},{\"type\":\"NonStructuralSchema\",\"status\":\"True\",\"lastTransitionTime\":\"2021-05-02T20:29:48Z\",\"reason\":\"Violations\",\"message\":\"spec.preserveUnknownFields: Invalid value: true: must be false\"}],\"acceptedNames\":{\"plural\":\"functions\",\"singular\":\"function\",\"kind\":\"Function\",\"listKind\":\"FunctionList\"},\"storedVersions\":[\"v1beta1\"]}}"

andresmgot commented 3 years ago

@shubhomoy Kubernetes +1.18 is not supported for the cronjob trigger so that may be the reason. See #1130

eduard93 commented 3 years ago

I got the same error as @shubhomoy on:

kubeless version
Kubeless version: v1.0.8

kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-12T14:18:45Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-12T14:12:29Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}

The error (parsing position is slightly different, but the error is the same):

kubectl logs kubeless-controller-manager-5d9bf7f5c6-sqhvg -n kubeless -c cronjob-trigger-controller
time="2021-07-11T19:53:44Z" level=info msg="Running Kubeless cronjob trigger controller version: v1.0.3"
time="2021-07-11T19:53:44Z" level=fatal msg="Unable to read the configmap: Error while fetching config location: v1beta1.CustomResourceDefinition: ObjectMeta: v1.ObjectMeta: readObjectFieldAsBytes: expect : after object field, parsing 901 ...s\":{\"k:{\\\"... at {\"kind\":\"CustomResourceDefinition\",\"apiVersion\":\"apiextensions.k8s.io/v1beta1\",\"metadata\":{\"name\":\"functions.kubeless.io\",\"uid\":\"f0b89e5a-0ca7-4db2-b133-f65a50fc2566\",\"resourceVersion\":\"2620\",\"generation\":1,\"creationTimestamp\":\"2021-07-11T19:36:54Z\",\"managedFields\":[{\"manager\":\"kubectl-create\",\"operation\":\"Update\",\"apiVersion\":\"apiextensions.k8s.io/v1beta1\",\"time\":\"2021-07-11T19:36:54Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:spec\":{\"f:conversion\":{\".\":{},\"f:strategy\":{}},\"f:group\":{},\"f:names\":{\"f:kind\":{},\"f:listKind\":{},\"f:plural\":{},\"f:singular\":{}},\"f:preserveUnknownFields\":{},\"f:scope\":{},\"f:version\":{},\"f:versions\":{}}}},{\"manager\":\"kube-apiserver\",\"operation\":\"Update\",\"apiVersion\":\"apiextensions.k8s.io/v1\",\"time\":\"2021-07-11T19:36:55Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:status\":{\"f:acceptedNames\":{\"f:kind\":{},\"f:listKind\":{},\"f:plural\":{},\"f:singular\":{}},\"f:conditions\":{\"k:{\\\"type\\\":\\\"Established\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}},\"k:{\\\"type\\\":\\\"NamesAccepted\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}},\"k:{\\\"type\\\":\\\"NonStructuralSchema\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}}}}}}]},\"spec\":{\"group\":\"kubeless.io\",\"version\":\"v1beta1\",\"names\":{\"plural\":\"functions\",\"singular\":\"function\",\"kind\":\"Function\",\"listKind\":\"FunctionList\"},\"scope\":\"Namespaced\",\"versions\":[{\"name\":\"v1beta1\",\"served\":true,\"storage\":true}],\"conversion\":{\"strategy\":\"None\"},\"preserveUnknownFields\":true},\"status\":{\"conditions\":[{\"type\":\"NonStructuralSchema\",\"status\":\"True\",\"lastTransitionTime\":\"2021-07-11T19:36:54Z\",\"reason\":\"Violations\",\"message\":\"spec.preserveUnknownFields: Invalid value: true: must be false\"},{\"type\":\"NamesAccepted\",\"status\":\"True\",\"lastTransitionTime\":\"2021-07-11T19:36:54Z\",\"reason\":\"NoConflicts\",\"message\":\"no conflicts found\"},{\"type\":\"Established\",\"status\":\"True\",\"lastTransitionTime\":\"2021-07-11T19:36:55Z\",\"reason\":\"InitialNamesAccepted\",\"message\":\"the initial names have been accepted\"}],\"acceptedNames\":{\"plural\":\"functions\",\"singular\":\"function\",\"kind\":\"Function\",\"listKind\":\"FunctionList\"},\"storedVersions\":[\"v1beta1\"]}}"

Error pretty print

```json { "kind":"CustomResourceDefinition", "apiVersion":"apiextensions.k8s.io/v1beta1", "metadata":{ "name":"functions.kubeless.io", "uid":"f0b89e5a-0ca7-4db2-b133-f65a50fc2566", "resourceVersion":"2620", "generation":1, "creationTimestamp":"2021-07-11T19:36:54Z", "managedFields":[ { "manager":"kubectl-create", "operation":"Update", "apiVersion":"apiextensions.k8s.io/v1beta1", "time":"2021-07-11T19:36:54Z", "fieldsType":"FieldsV1", "fieldsV1":{ "f:spec":{ "f:conversion":{ ".":{ }, "f:strategy":{ } }, "f:group":{ }, "f:names":{ "f:kind":{ }, "f:listKind":{ }, "f:plural":{ }, "f:singular":{ } }, "f:preserveUnknownFields":{ }, "f:scope":{ }, "f:version":{ }, "f:versions":{ } } } }, { "manager":"kube-apiserver", "operation":"Update", "apiVersion":"apiextensions.k8s.io/v1", "time":"2021-07-11T19:36:55Z", "fieldsType":"FieldsV1", "fieldsV1":{ "f:status":{ "f:acceptedNames":{ "f:kind":{ }, "f:listKind":{ }, "f:plural":{ }, "f:singular":{ } }, "f:conditions":{ "k:{""type"":""Established""}":{ ".":{ }, "f:lastTransitionTime":{ }, "f:message":{ }, "f:reason":{ }, "f:status":{ }, "f:type":{ } }, "k:{""type"":""NamesAccepted""}":{ ".":{ }, "f:lastTransitionTime":{ }, "f:message":{ }, "f:reason":{ }, "f:status":{ }, "f:type":{ } }, "k:{""type"":""NonStructuralSchema""}":{ ".":{ }, "f:lastTransitionTime":{ }, "f:message":{ }, "f:reason":{ }, "f:status":{ }, "f:type":{ } } } } } } ] }, "spec":{ "group":"kubeless.io", "version":"v1beta1", "names":{ "plural":"functions", "singular":"function", "kind":"Function", "listKind":"FunctionList" }, "scope":"Namespaced", "versions":[ { "name":"v1beta1", "served":true, "storage":true } ], "conversion":{ "strategy":"None" }, "preserveUnknownFields":true }, "status":{ "conditions":[ { "type":"NonStructuralSchema", "status":"True", "lastTransitionTime":"2021-07-11T19:36:54Z", "reason":"Violations", "message":"spec.preserveUnknownFields: Invalid value: true: must be false" }, { "type":"NamesAccepted", "status":"True", "lastTransitionTime":"2021-07-11T19:36:54Z", "reason":"NoConflicts", "message":"no conflicts found" }, { "type":"Established", "status":"True", "lastTransitionTime":"2021-07-11T19:36:55Z", "reason":"InitialNamesAccepted", "message":"the initial names have been accepted" } ], "acceptedNames":{ "plural":"functions", "singular":"function", "kind":"Function", "listKind":"FunctionList" }, "storedVersions":[ "v1beta1" ] } } ```

I have tried to add preserveUnknownFields: false at the end of kubeless-v1.0.8.yaml:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: cronjobtriggers.kubeless.io
spec:
  group: kubeless.io
  names:
    kind: CronJobTrigger
    plural: cronjobtriggers
    singular: cronjobtrigger
  scope: Namespaced
  version: v1beta1
  preserveUnknownFields: false

But it causes:

The CustomResourceDefinition "cronjobtriggers.kubeless.io" is invalid: spec.versions[0].schema.openAPIV3Schema: Required value: because otherwise all fields are pruned

Which makes sense. Can someone please point me, where are the object which openAPIV3Schema must describe is defined/located? I can write a schema from that and try to run kubeless with that.

Another question - can cronjobtriggers be safely removed? I'm only interested in http and cli triggers.

Links:

eduard93 commented 3 years ago

Trivial fix (preserveUnknownFields: false) didn't work, the same error persists.

New yaml for cronjobtriggers.kubeless.io

```yaml apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: cronjobtriggers.kubeless.io spec: group: kubeless.io names: kind: CronJobTrigger plural: cronjobtriggers singular: cronjobtrigger scope: Namespaced version: v1beta1 versions: - name: v1beta1 served: true storage: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: schedule: type: string jobTemplate: type: object - name: v1 served: false storage: false schema: openAPIV3Schema: type: object properties: spec: type: object properties: schedule: type: string jobTemplateA: type: object preserveUnknownFields: false ```

huangjiasingle commented 3 years ago

@eduard93 @eduard93 @shubhomoy @iamshreeram the reason of the crashbackoff is cronjob-trigger-controller version v1.0.3 dependence kubeless version define as github.com/kubeless/kubeless v1.0.0-alpha.6, the kubeless v1.0.0-alpha.6 dependece client-go version was very low (k8s.io/client-go v5.0.0+incompatible), the decode interface not compatible the high version kubernetes crd resource.

how to solve this problem？

rebuild the cronjob-trigger-controller: git clone https://github.com/kubeless/cronjob-trigger.git && cd cronjob-trigger, and the modify the go.mod, change the github.com/kubeless/kubeless v1.0.0-alpha.6 to github.com/kubeless/kubeless v1.0.8, the v1.0.8 is the version which you to use kubeless’s version , and the go mod tidy , rebuild the cronjob-trigger-controller image, that will be succedded.

SCLogo commented 2 years ago

@eduard93 @eduard93 @shubhomoy @iamshreeram the reason of the crashbackoff is cronjob-trigger-controller version v1.0.3 dependence kubeless version define as github.com/kubeless/kubeless v1.0.0-alpha.6, the kubeless v1.0.0-alpha.6 dependece client-go version was very low (k8s.io/client-go v5.0.0+incompatible), the decode interface not compatible the high version kubernetes crd resource.

how to solve this problem？

rebuild the cronjob-trigger-controller: git clone https://github.com/kubeless/cronjob-trigger.git && cd cronjob-trigger, and the modify the go.mod, change the github.com/kubeless/kubeless v1.0.0-alpha.6 to github.com/kubeless/kubeless v1.0.8, the v1.0.8 is the version which you to use kubeless’s version , and the go mod tidy , rebuild the cronjob-trigger-controller image, that will be succedded.

Did you manage to work kubeless with this modification on k8s > 1.18? I am trying to move our clusters to 1.22

vmware-archive / kubeless

Starting kubeless | kubeless-controller-manager in CrashLoopBackOff #1152