Deploying Failed: ConfigMap too long?

[centerorbit]

Describe the problem:

I get a failed serverless deploy with this error in the controller logs:

Function can not be created/updated: ConfigMap \"get-asset\" is invalid: data: Too long: must have at most 1048576 characters" pkg=function-controller

The build that's failing is much larger and complex, and uses typescript compiled down to js using webpack. In the end, a handler.js is created and a zip formed.

I don't know how to retrieve the ConfigMap to inspect the contents, or how one might go about modifying the data that's put in there.

I may be tracking down the wrong log error, there are other things that show up in the config logs, like:

time="2018-11-15T18:54:39Z" level=error msg="Error processing default/get-asset (will retry): Operation cannot be fulfilled on functions.kubeless.io \"get-asset\": the object has been modified; please apply your changes to the latest version and try again" pkg=function-controller

But, the ConfigMap one seemed the most likely? But I'm not sure where else to look.

Steps to reproduce:

kubeless function ls

NAME    NAMESPACE       HANDLER RUNTIME DEPENDENCIES    STATUS

serverless deploy -v

Serverless: Zip service: /mnt/c/Users/andy/docker-scaffolding/applications/code/lambda/storage/.webpack/service [145 ms]
Serverless: Packaging service...
Serverless: Remove /mnt/c/Users/andy/docker-scaffolding/applications/code/lambda/storage/.webpack
Serverless: Deploying function get-asset...
Serverless: Unable to find any running pod for get-asset. Retrying...
Serverless: Unable to find any running pod for get-asset. Retrying...
Serverless: Unable to find any running pod for get-asset. Retrying...
Serverless: Unable to find any running pod for get-asset. Retrying...
Serverless: Giving up, unable to retrieve the status of the get-asset deployment.

  Error --------------------------------------------------

  Found errors while processing the given functions:
Unable to retrieve the status of the get-asset deployment

kubeless function ls

NAME            NAMESPACE       HANDLER                 RUNTIME DEPENDENCIES    STATUS
get-asset       default         handler.getAsset        nodejs8                 MISSING: Check controller logs

kubectl get pods -n kubeless

NAME                                           READY     STATUS    RESTARTS   AGE
kubeless-controller-manager-598dd94559-2kq8c   3/3       Running   0          23h

kubectl logs kubeless-controller-manager-598dd94559-2kq8c --container=kubeless-function-controller --namespace=kubeless

time="2018-11-15T18:34:41Z" level=info msg="Processing change to Function default/get-asset" pkg=function-controller
time="2018-11-15T18:34:42Z" level=error msg="Function can not be created/updated: ConfigMap \"get-asset\" is invalid: data: Too long: must have at most 1048576 characters" pkg=function-controller
time="2018-11-15T18:34:42Z" level=error msg="Error processing default/get-asset (will retry): ConfigMap \"get-asset\" is invalid: data: Too long: must have at most 1048576 characters" pkg=function-controller
time="2018-11-15T18:34:42Z" level=info msg="Processing change to Function default/get-asset" pkg=function-controller
ERROR: logging before flag.Parse: E1115 18:34:42.239692       1 streamwatcher.go:109] Unable to decode an event from the watch stream: object to decode was longer than maximum allowed size

"Tricking" it to work:

1. Delete the failing function with: kubeless function delete get-asset
2. Use the kubeless example, and rename the function to match get-asset
3. Deploy that

   Serverless: Packaging service...
   Serverless: Excluding development dependencies...
   Serverless: Function get-asset successfully deployed

NAME            NAMESPACE       HANDLER                 RUNTIME DEPENDENCIES                    STATUS
get-asset       default         handler.capitalize      nodejs8 serverless-offline: ^3.31.0     1/1 READY
                                                                lodash: ^4.1.0
                                                                serverless-kubeless: ^0.5.1

4. now run failing deploy (in the other project):

   Serverless: Zip service: /mnt/c/Users/andy/docker-scaffolding/applications/code/lambda/storage/.webpack/service [140 ms]
   Serverless: Packaging service...
   Serverless: Remove /mnt/c/Users/andy/docker-scaffolding/applications/code/lambda/storage/.webpack
   Serverless: Function get-asset successfully deployed
   Serverless: Deployed Ingress rule storage

   NAME            NAMESPACE       HANDLER                 RUNTIME DEPENDENCIES    STATUS
   get-asset       default         handler.getAsset        nodejs8                 1/1 READY

Configs:

Failing:

service: storage

provider:
  name: kubeless
  runtime: nodejs8

plugins:
  - serverless-webpack
  - serverless-kubeless

functions:
  get-asset:
    handler: handler.getAsset
    events:
      - http:
          method: get
          path: file/{s3Folder}/{s3Filename}
          cors: true

Working config (slightly modified from example template):

service: storage

provider:
  name: kubeless
  runtime: nodejs8

plugins:
  - serverless-kubeless

functions:
  get-asset:
    handler: handler.capitalize

[andresmgot]

Hi @centerorbit,

Thank you for the detailed notes. It seems that you are hitting the limit for the package size. See https://kubeless.io/docs/advanced-function-deployment/#deploying-large-functions.

If you are using web pack you can read the section regarding that here: http://kubeless.io/docs/runtimes/#nodejs. Basically you'll need to skip including dependencies in the zip file to avoid the 1.5MB limit. Unfortunately I don't have a serverless examples for you using that.

Let us know if you manage to make that work.

[centerorbit]

Hey @andresmgot,

Thank you so much for that tip! Your suggestion got it working!

Interestingly enough though, the resulting zip file was only 924KB. I figured, perhaps it was also base64 encoded, but that brings it up to 1,248 KB. Perhaps the remaining 252 KB was consumed by additional config wrapped around the function in the etcd?

What I did realize was that the handler.js.map file was being included in the zip, and as soon as I excluded that, it dropped the zip size to about 511 KB, and deployed right away!

It'd be nice to understand if it was the inclusion of the map, or nearing of the size limit, but I'm just happy it's working!

[andresmgot]

I am glad you got it working!

Regarding the limit, only newer versions of etcd allow fields of 1.5MB. In previous versions, the limit was 1MB. Depending on the version that your Kubernetes cluster includes that may be the reason. The other possibility is indeed that the rest of elements in the ConfigMap causes the size to be over the limit.