Request too large when create action

[Farrrrland]

Hi, I've found that OpenWhisk keep telling me the request size is too large when I try creating an action with zip file ~40MB large (base64 format will be ~52MB).

wsk action create gmm --kind python:3 --main media_meta get-media-meta.zip -i -v

The error looks like:

REQUEST:
[PUT]   https://127.0.0.1:31001/api/v1/namespaces/_/actions/gmm?overwrite=false
Req Headers
{
  "Authorization": [
    "Basic MjNiYzQ2YjEtNzFmNi00ZWQ1LThjNTQtODE2YWE0ZjhjNTAyOjEyM3pPM3haQ0xyTU42djJCS0sxZFhZRnBYbFBrY2NPRnFtMTJDZEFzTWdSVTRWck5aOWx5R1ZDR3VNREdJd1A="
  ],
  "Content-Type": [
    "application/json"
  ],
  "User-Agent": [
    "OpenWhisk-CLI/1.0 (2021-04-01T23:49:54.523+0000) linux amd64"
  ]
}
Req Body
Body exceeds 1000 bytes and will be truncated
{"namespace":"_","name":"gmm","exec":{"kind":"python:3","code":"UEsDBBQAAAAIAJo1aFZyvYV4WwIAALgEAAAfABwAdmlydHVhbGVudi9iaW4vYWN0aXZhdGVfdGhpcy5weVVUCQAD5C4IZOQuCGR1eAsAAQToAwAABOgDAACdU02L2zAQvetXDNrDWjRxWlraYthDDgstlGUpuwslBKPY41itLRlJziaU/veOZDvxtvTSHCJpPt+8eeacrwuvDtIjHJT1vWxQH6AyForeWtQelPZoO4v0nzH26BDwiEViOtSJr5XLK9WgSC3KMhEL+HmdR0ueX2dw9v8SKWMP9IJCatgh9A5LeK5Rw8n00PbOBxOQE4/KeaX3cH/ytdHz/gvQxlPRF1h3Sq+6GJoyzjlTbWesB+Omm1Mez/eTY8zbU8aAfnI3oIMbCk876euUTOFMpiEEw2OBnYc72eKttcYOqVYqgrt2Dq1XRkdPwr/NZ/kvmgQXjNFIeansDBa9NAFIJsSC7aQLuMfQTQZLYiPh9OYClvBmC3AFzlvVgXyWp8gaOaGTRENlTRstE4gFdE3voin0A4cUJ72xjF0BkU9TlDHdG7hfP3yCOBNE8uhU2qkSzz0IEBaUfBKMBqAlKWv0hodEvr1MRU3S70bpZDNOsYVXcElI9+iTIWkBnIvUdY3yySVZvCz/9Pnrw+P6S3579xS7RIaIhFErEMRCWK3a155Wbw5/gA2jyrKc6wvG4i19CA4atbPSKnSBhRBVG1p1N8p0EFiLRS21ci0j1g457WRPdN5AWA6JL0IXLHxfVI14A56mK7q+fzeK+G36YRUUu+xk8UPu0WVDwL/c/G9aBoXGPV4URMJrorInQ2R+JH4R0BCdIS9UT4mHcJIrGfSHhSkx4b2vlh9JYKqijQA2btCLYNNsmyxQf37NSMjCds+O19nMtWUxP0DMyVyp41QjPtjlOq6V/QZQSwMEFAAAAAgA...
RESPONSE:Got response with code 500
Resp Headers
{
  "Connection": [
    "keep-alive"
  ],
  "Content-Length": [
    "17"
  ],
  "Content-Type": [
    "text/plain; charset=UTF-8"
  ],
  "Date": [
    "Wed, 08 Mar 2023 13:20:08 GMT"
  ],
  "Server": [
    "nginx/1.21.1"
  ]
}
Response body size is 17 bytes
Response body received:
Request too large
error: Unable to create action 'gmm': The connection failed, or timed out. (HTTP status code 500)
Run 'wsk --help' for usage.

The file size is as follows:

$ ls -lh
total 46M
-rw-rw-r-- 1 ec2-user ec2-user  39M Mar  8 11:43 get-media-meta.zip
-rw-rw-r-- 1 ec2-user ec2-user  898 Mar  8 11:42 __main__.py
-rw-rw-r-- 1 ec2-user ec2-user 7.3M Mar 18  2019 mp4_test_yewen.mp4
-rw-rw-r-- 1 ec2-user ec2-user   19 Mar  7 08:40 requirements.txt
drwxrwxr-x 5 ec2-user ec2-user   77 Mar  8 06:44 virtualenv

$ base64 get-media-meta.zip > temp
$ ls -lh
total 98M
-rw-rw-r-- 1 ec2-user ec2-user  39M Mar  8 11:43 get-media-meta.zip
-rw-rw-r-- 1 ec2-user ec2-user  898 Mar  8 11:42 __main__.py
-rw-rw-r-- 1 ec2-user ec2-user 7.3M Mar 18  2019 mp4_test_yewen.mp4
-rw-rw-r-- 1 ec2-user ec2-user   19 Mar  7 08:40 requirements.txt
-rw-rw-r-- 1 ec2-user ec2-user  52M Mar  8 13:29 temp
drwxrwxr-x 5 ec2-user ec2-user   77 Mar  8 06:44 virtualenv

I've changed nginx.ingress.kubernetes.io/proxy-body-size: "100m" when deploying but it seems not solving the problem. Smaller size zip files are working fine, but I'm not so sure how can I use a larger file to create my actions?

[Farrrrland]

I'm running on AWS EC2, will it be a problem?

[style95]

You need to increase the nginx config and akka-http configuration. https://github.com/apache/openwhisk/blob/master/common/scala/src/main/resources/application.conf#L29 https://github.com/apache/openwhisk/pull/5279/files

[Farrrrland]

You need to increase the nginx config and akka-http configuration. https://github.com/apache/openwhisk/blob/master/common/scala/src/main/resources/application.conf#L29 https://github.com/apache/openwhisk/pull/5279/files

@style95 Sorry about the late reply, but after changing those two files and set the limitition to ~100M, it still has the same output of request too large problem. I've recreated a KinD cluster and redeployed whisk but seems the problem still exists.

Shall I clean up anything else after changing the configurations?

[style95]

Did you rebuild the images with those configurations?

[Farrrrland]

@style95 Yes, I've restarted the whole process. But I was using helm install owdev ~/openwhisk-deploy-kube/helm/openwhisk -n openwhisk --create-namespace -f ~/openwhisk-deploy-kube/deploy/kind/mycluster.yaml to install packages, which I believe modifiying configurations in openwhisk folder does not changing the actual configuratins. In this case if I'm running with helm and with openwhisk-deploy-kube, how can I change those configurations?

[bdoyle0182]

openwhisk has the hard 48mb attachment limit and i'm pretty sure that includes base64 encoding so the true limit is actually like a third lower

[Farrrrland]

@bdoyle0182 Yes, actually this is the case I'm facing, I'm just wondering since deploying from source code can relax this limits by editing configurations, how can I do the same when deploying helm charts?

[bdoyle0182]

the 48mb limit is actually hard coded in code right now I'm pretty sure, how are you modifying it? it's the one thing that can't really be configured besides the 1mb payload limit

[bdoyle0182]

You can use the whisk.exec-size-limit configuration to configure the max file size to be 0 to 48mb, but not larger.

From Exec.scala:

  val maxSize: ByteSize = 48.MB
  val sizeLimit = loadConfigOrThrow[ByteSize](ConfigKeys.execSizeLimit)

  require(
    sizeLimit <= maxSize,
    s"Executable code size limit $sizeLimit specified by '${ConfigKeys.execSizeLimit}' should not be more than max size of $maxSize")

[Farrrrland]

@bdoyle0182 So I guess it is not actually a problem of nginx size limit but a size limit of packaged uploaded in openwhisk it self right? Let me see if decreasing the size settings in mycluster.yaml will work.

[bdoyle0182]

Dom might be right here you are hitting a higher level limit first based on your error logs either with nginx configuration or an akka http configuration before actually entering the openwhisk create action route. Though once past that I think you may still hit the error I posted above for the package in question due to the base64 encoding putting it over 48mb, not 100% sure.

[Farrrrland]

@style95 Things got weird here actually, I've tried to minimize nginx size limit in mycluster.yaml and eventually found a package size of ~23M can still be successfully uploaded, I guess there might be some problems of the actual usage of mycluster.yaml file following the instructions in customize-the-deployment?

This is how my configuration looks like:

whisk:
  ingress:
    type: NodePort
    apiHostName: localhost
    apiHostPort: 31001
    useInternally: false
    annotations:
      nginx.ingress.kubernetes.io/proxy-body-size: "2m"

nginx:
  httpsNodePort: 31001

# disable affinity
affinity:
  enabled: false
toleration:
  enabled: false
invoker:
  options: "-Dwhisk.kubernetes.user-pod-node-affinity.enabled=false"
  # must use KCF as kind uses containerd as its container runtime
  containerFactory:
    impl: "kubernetes"

[Farrrrland]

@bdoyle0182 I guess there might be multiple limits here, however, I suspect there might be something wrong when customizing the deployment following the instructions, hopefully I did not miss any steps but I think mycluster.yaml is not working in my case.

[QWQyyy]

I think you should revised the openwhisk scala source code

[singhsegv]

@Farrrrland Did you try rebuilding the OpenWhisk source code after updating the Exec.scala? If yes, then any issues faced while doing so?