search

serverless / serverless-kubeless

This plugin enables support for Kubeless within the Serverless Framework.
Apache License 2.0
303 stars 81 forks source link

Function Pod doesn't come up when requirements.txt is used for dependencies #97

Closed vishweshwarp closed 6 years ago

vishweshwarp commented 6 years ago

If we use requirements.txt with dependencies defined in it, 'serverless deploy' gets stuck with pod at Init state.

Dependency mentioned in requirements.txt is not even used in my handler.py. And if I just delete requirements.txt file from service folder, everything starts working, function gets deployed without any issues.

mongo-connect-7dfb4868f-crt96 0/1 Init:CrashLoopBackOff 4 5m

handler.py

def mongoConnect():
    return 'Connected to MongoDB'

requirements.txt

pymongo==2.7

serverless.yml

service: mongo-utils

provider:
  name: kubeless
  runtime: python2.7

plugins:
  - serverless-python-requirements
  - serverless-kubeless
functions:
  mongo-connect:
    handler: handler.mongoConnect

custom:
  pythonRequirements:
    pythonBin: C:\Python27\python.exe
    invalidateCaches: true

Serverless Commands:

C:\Users\myusername\workspace\devopsinsight\functions\mongo-utils>sls deploy
Serverless: Installing required Python packages with C:\Python27\python.exe...
Serverless: Linking required Python packages...
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Unlinking required Python packages...
Serverless: Deploying function mongo-connect...

Pod logs:

C:\>kubectl logs mongo-connect-7dfb4868f-crt96
Error from server (BadRequest): container "mongo-connect" in pod "mongo-connect-7dfb4868f-crt96" is waiting to start: PodInitializing

Function pod describe:

C:\>kubectl describe po mongo-connect-7dfb4868f-crt96
Name:           mongo-connect-7dfb4868f-crt96
Namespace:      default
Node:           cmbu-devops-services-kube-node2/10.197.10.228
Start Time:     Thu, 28 Dec 2017 21:38:38 +0530
Labels:         function=mongo-connect
                pod-template-hash=389604249
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"mongo-connect-7dfb4868f","uid":"96ba80eb-ebe9-11e7-b07f-005056ac...
                prometheus.io/path=/metrics
                prometheus.io/port=8080
                prometheus.io/scrape=true
Status:         Pending
IP:             10.42.0.9
Created By:     ReplicaSet/mongo-connect-7dfb4868f
Controlled By:  ReplicaSet/mongo-connect-7dfb4868f
Init Containers:
  prepare:
    Container ID:  docker://947ac94729615804397bc6bd0d707262e9d166a799be6c78dfe424bd56dd8950
    Image:         kubeless/unzip@sha256:f162c062973cca05459834de6ed14c039d45df8cdb76097f50b028a1621b3697
    Image ID:      docker-pullable://kubeless/unzip@sha256:f162c062973cca05459834de6ed14c039d45df8cdb76097f50b028a1621b3697
    Port:          <none>
    Command:
      sh
      -c
    Args:
      cp /src/handler.py /kubeless/handler.py && cp /src/requirements.txt /kubeless
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 28 Dec 2017 21:38:39 +0530
      Finished:     Thu, 28 Dec 2017 21:38:39 +0530
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /kubeless from mongo-connect (rw)
      /src from mongo-connect-deps (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-qkfws (ro)
  install:
    Container ID:  docker://6f70b3837f8ae4379c8ce7bc63f2f0590759dc038818d56e595d25a61d18bd11
    Image:         tuna/python-pillow:2.7.11-alpine
    Image ID:      docker-pullable://tuna/python-pillow@sha256:6e39c3a1382e0666eac1e435b8dd78e202d1cf791166ba2a77ad52a6584e04ac
    Port:          <none>
    Command:
      sh
      -c
    Args:
      pip install --prefix=/kubeless -r /kubeless/requirements.txt
    State:          Running
      Started:      Thu, 28 Dec 2017 21:42:27 +0530
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 28 Dec 2017 21:40:56 +0530
      Finished:     Thu, 28 Dec 2017 21:42:01 +0530
    Ready:          False
    Restart Count:  3
    Environment:
      FUNC_HANDLER:  mongoConnect
      MOD_NAME:      handler
      FUNC_TIMEOUT:  180
      TOPIC_NAME:
    Mounts:
      /kubeless from mongo-connect (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-qkfws (ro)
Containers:
  mongo-connect:
    Container ID:
    Image:          kubeless/python@sha256:ba948a6783b93d75037b7b1806a3925d441401ae6fba18282f712a1b1a786899
    Image ID:
    Port:           8080/TCP
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://:8080/healthz delay=3s timeout=1s period=30s #success=1 #failure=3
    Environment:
      FUNC_HANDLER:  mongoConnect
      MOD_NAME:      handler
      FUNC_TIMEOUT:  180
      TOPIC_NAME:
      PYTHONPATH:    /kubeless/lib/python2.7/site-packages
    Mounts:
      /kubeless from mongo-connect (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-qkfws (ro)
Conditions:
  Type           Status
  Initialized    False
  Ready          False
  PodScheduled   True
Volumes:
  mongo-connect:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  mongo-connect-deps:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      mongo-connect
    Optional:  false
  default-token-qkfws:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-qkfws
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.alpha.kubernetes.io/notReady:NoExecute for 300s
                 node.alpha.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                 Age                     From                                      Message
  ----     ------                 ----                    ----                                      -------
  Normal   SuccessfulMountVolume  3m                      kubelet, cmbu-devops-services-kube-node2  MountVolume.SetUp succeeded for volume "mongo-connect"
  Normal   SuccessfulMountVolume  3m                      kubelet, cmbu-devops-services-kube-node2  MountVolume.SetUp succeeded for volume "mongo-connect-deps"
  Normal   SuccessfulMountVolume  3m                      kubelet, cmbu-devops-services-kube-node2  MountVolume.SetUp succeeded for volume "default-token-qkfws"
  Normal   Pulled                 3m                      kubelet, cmbu-devops-services-kube-node2  Container image "kubeless/unzip@sha256:f162c062973cca05459834de6ed14c039d45df8cdb76097f50b028a1621b3697" already present on machine
  Normal   Created                3m                      kubelet, cmbu-devops-services-kube-node2  Created container
  Normal   Started                3m                      kubelet, cmbu-devops-services-kube-node2  Started container
  Normal   Scheduled              2m                      default-scheduler                         Successfully assigned mongo-connect-7dfb4868f-crt96 to cmbu-devops-services-kube-node2
  Warning  BackOff                6s (x3 over 1m)         kubelet, cmbu-devops-services-kube-node2  Back-off restarting failed container
  Warning  FailedSync             6s (x3 over 1m)         kubelet, cmbu-devops-services-kube-node2  Error syncing pod
  Normal   Pulled                 <invalid> (x4 over 3m)  kubelet, cmbu-devops-services-kube-node2  Container image "tuna/python-pillow:2.7.11-alpine" already present on machine
  Normal   Created                <invalid> (x4 over 3m)  kubelet, cmbu-devops-services-kube-node2  Created container
  Normal   Started                <invalid> (x4 over 3m)  kubelet, cmbu-devops-services-kube-node2  Started container
vishweshwarp commented 6 years ago

This issue is blocking us from deploying actual application Functions where multiple dependencies are defined.

chipironcin commented 6 years ago

Hello @vishweshwarp Unfortunately, some of our team are taking a couple of days off and I am unable to correctly answer to your issue.

Our team will take a look at it shortly. Please bear with us

ngtuna commented 6 years ago

@vishweshwarp Sorry for the delayed response. We just got back to work on this week and catch up a bit after new year holiday.

I am not seeing any error in your logs. May I know the kubeless version you are using and kubernetes environment you are running it on ? Are you sure to wait long enough and the pod still stuck at Init state. The problem might came from network issue or resource insufficiency

vishweshwarp commented 6 years ago

I am now using Kubeless 0.3.3, Kubernetes 1.8.4 with RBAC. I installed Kubeless using RBAC yaml https://github.com/kubeless/kubeless/releases/download/v0.3.3/kubeless-rbac-v0.3.3.yaml

Yes I waited more than 2 hours.

Here are the kube controller logs

root@vishu-dev-vm:/workspace/devops-insight/functions/mongo-utils# kubectl logs kubeless-controller-5957f7b87d-jqz6q -n kubeless
time="2018-01-04T10:56:57Z" level=info msg="Starting kubeless controller" pkg=controller
time="2018-01-04T10:56:57Z" level=info msg="Kubeless controller synced and ready" pkg=controller
time="2018-01-04T11:07:17Z" level=info msg="Processing change to Function default/mongo-connect" pkg=controller
time="2018-01-04T11:07:17Z" level=error msg="Function can not be created/updated: servicemonitors.monitoring.coreos.com \"mongo-connect\" is forbidden: User \"system:serviceaccount:kubeless:controller-acct\" cannot delete servicemonitors.monitoring.coreos.com in the namespace \"default\"" pkg=controller
time="2018-01-04T11:07:17Z" level=error msg="Error processing default/mongo-connect (will retry): servicemonitors.monitoring.coreos.com \"mongo-connect\" is forbidden: User \"system:serviceaccount:kubeless:controller-acct\" cannot delete servicemonitors.monitoring.coreos.com in the namespace \"default\"" pkg=controller
time="2018-01-04T11:07:17Z" level=info msg="Processing change to Function default/mongo-connect" pkg=controller
time="2018-01-04T11:07:17Z" level=error msg="Function can not be created/updated: servicemonitors.monitoring.coreos.com \"mongo-connect\" is forbidden: User \"system:serviceaccount:kubeless:controller-acct\" cannot delete servicemonitors.monitoring.coreos.com in the namespace \"default\"" pkg=controller
time="2018-01-04T11:07:17Z" level=error msg="Error processing default/mongo-connect (will retry): servicemonitors.monitoring.coreos.com \"mongo-connect\" is forbidden: User \"system:serviceaccount:kubeless:controller-acct\" cannot delete servicemonitors.monitoring.coreos.com in the namespace \"default\"" pkg=controller
time="2018-01-04T11:07:17Z" level=info msg="Processing change to Function default/mongo-connect" pkg=controller
time="2018-01-04T11:07:19Z" level=error msg="Function can not be created/updated: servicemonitors.monitoring.coreos.com \"mongo-connect\" is forbidden: User \"system:serviceaccount:kubeless:controller-acct\" cannot delete servicemonitors.monitoring.coreos.com in the namespace \"default\"" pkg=controller
time="2018-01-04T11:07:19Z" level=error msg="Error processing default/mongo-connect (will retry): servicemonitors.monitoring.coreos.com \"mongo-connect\" is forbidden: User \"system:serviceaccount:kubeless:controller-acct\" cannot delete servicemonitors.monitoring.coreos.com in the namespace \"default\"" pkg=controller
time="2018-01-04T11:07:19Z" level=info msg="Processing change to Function default/mongo-connect" pkg=controller
time="2018-01-04T11:07:21Z" level=error msg="Function can not be created/updated: servicemonitors.monitoring.coreos.com \"mongo-connect\" is forbidden: User \"system:serviceaccount:kubeless:controller-acct\" cannot delete servicemonitors.monitoring.coreos.com in the namespace \"default\"" pkg=controller
time="2018-01-04T11:07:21Z" level=error msg="Error processing default/mongo-connect (will retry): servicemonitors.monitoring.coreos.com \"mongo-connect\" is forbidden: User \"system:serviceaccount:kubeless:controller-acct\" cannot delete servicemonitors.monitoring.coreos.com in the namespace \"default\"" pkg=controller
time="2018-01-04T11:07:21Z" level=info msg="Processing change to Function default/mongo-connect" pkg=controller
time="2018-01-04T11:07:23Z" level=error msg="Function can not be created/updated: servicemonitors.monitoring.coreos.com \"mongo-connect\" is forbidden: User \"system:serviceaccount:kubeless:controller-acct\" cannot delete servicemonitors.monitoring.coreos.com in the namespace \"default\"" pkg=controller
time="2018-01-04T11:07:23Z" level=error msg="Error processing default/mongo-connect (will retry): servicemonitors.monitoring.coreos.com \"mongo-connect\" is forbidden: User \"system:serviceaccount:kubeless:controller-acct\" cannot delete servicemonitors.monitoring.coreos.com in the namespace \"default\"" pkg=controller
time="2018-01-04T11:07:23Z" level=info msg="Processing change to Function default/mongo-connect" pkg=controller
time="2018-01-04T11:07:25Z" level=error msg="Function can not be created/updated: servicemonitors.monitoring.coreos.com \"mongo-connect\" is forbidden: User \"system:serviceaccount:kubeless:controller-acct\" cannot delete servicemonitors.monitoring.coreos.com in the namespace \"default\"" pkg=controller
time="2018-01-04T11:07:25Z" level=error msg="Error processing default/mongo-connect (giving up): servicemonitors.monitoring.coreos.com \"mongo-connect\" is forbidden: User \"system:serviceaccount:kubeless:controller-acct\" cannot delete servicemonitors.monitoring.coreos.com in the namespace \"default\"" pkg=controller
E0104 11:07:25.214721       1 controller.go:171] servicemonitors.monitoring.coreos.com "mongo-connect" is forbidden: User "system:serviceaccount:kubeless:controller-acct" cannot delete servicemonitors.monitoring.coreos.com in the namespace "default"
ngtuna commented 6 years ago

I guess you are running prometheus-operator. From your info, I can see the dependency doesn't cause problem and in fact it run success. The controller log shows that function pod can't start up because it creates servicemonitor mongo-connect which is not relevant to kubeless btw.

If you look into our manifest, you will see the kubeless-controller is running under service account controller-acct which is assigning the clusterrole kubeless-controller-deployer. So can you work around this issue by adding to the clusterrole kubeless-controller-deployer the proper actions for servicemonitors resource ?

andresmgot commented 6 years ago

Hi @vishweshwarp,

Actually the error entry in the controller should not affect the deployment (since monitoring is handled at the very end).

My guess is that the initContainer does not have access to the internet or pypi.python.org so the connection hangs. In which platform/cloud provider are you running your example? What are the logs of the initContainer? You can retrieve them executing:

kubectl logs mongo-connect-7dfb4868f-crt96 -c prepare

Also note that you don't need the serverless-python-requirements plugin, that one is not used with Kubeless, we handle the installation so your serverless.yaml should look something like:

service: mongo-utils

provider:
  name: kubeless
  runtime: python2.7

plugins:
  - serverless-kubeless
functions:
  mongo-connect:
    handler: handler.mongoConnect
vishweshwarp commented 6 years ago

@andresmgot init container has access to internet but I noticed a problem with pod network which is causing kube-dns restart each time function pod tries to resolve python depenencies. After I solved network issue, function pod came up successfully.