search

cloudfoundry / cf-for-k8s

The open source deployment manifest for Cloud Foundry on Kubernetes
Apache License 2.0
301 stars 115 forks source link

cf-for-k8s deployed locally - buildpack creation fails 413 Payload Too Large/Request Entity Too Large; Error unmarshalling the following into a cloud controller error #634

Closed krismarc closed 3 years ago

krismarc commented 3 years ago

Hi CF community!

Describe the bug

I came up with an idea to deploy cf-for-k8s using minikube on WSL2 - Ubuntu 20.04.1 LTS (Focal Fossa). https://github.com/cloudfoundry/cf-for-k8s/blob/develop/docs/deploy-local.md#steps-to-deploy-on-minikube

1) First problem which I've faced was a cf-domain. It's proposed in the instruction to use $(minikube ip) which results with a local IP address.

DESKTOP-Q8TBEVJ:~/$ minikube ip
192.168.49.2

After deployment, any cf command resulted in a timeout. So I decided to use 127.0.0.1 instead and this worked fine.

2) once it started to work, another thing which I wanted to do was a buildpack (https://github.com/cloudfoundry/ibm-websphere-liberty-buildpack) creation and it failed with:

Uploading buildpack liberty11 as admin...
 0 B / 153.10 MiB [-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------]   0.00%REQUEST: [2021-03-07T22:11:29+01:00]
POST /v3/buildpacks/f23df4c1-bb27-4d9f-a2ee-fb33644dbf67/upload HTTP/1.1
Host: api.127.0.0.1.nip.io
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Content-Type: multipart/form-data; boundary=14da644ce650df7630783c65847511fa0ef7045602903bae206dd6bf89de
User-Agent: cf/7.2.0+be4a5ce2b.2020-12-10 (go1.13.15; amd64 linux)
[multipart/form-data Content Hidden]

RESPONSE: [2021-03-07T22:11:29+01:00]
HTTP/1.1 413 Payload Too Large
Content-Length: 17
Content-Type: text/plain
Date: Sun, 07 Mar 2021 21:11:29 GMT
Server: istio-envoy
Payload Too Large

 14.19 MiB / 153.10 MiB [==================>-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------]   9.27% 1s
io: read/write on closed pipe

TIP: A buildpack with name 'liberty11' and nil stack has been created. Use 'cf delete-buildpack' to delete it or 'cf update-buildpack' to try again.
FAILED

so I did some research around and found out a solution for this by applying envoy filter for istio ingress: https://stackoverflow.com/questions/61439893/istio-how-to-use-envoyfilter-to-change-max-request-bytes-of-envoy-sidecar

DESKTOP-Q8TBEVJ:~/buildpack$ kubectl get envoyfilter -n istio-system request-size-limit
NAME                 AGE
request-size-limit   40m

since then, it started to fail with another error:

Uploading buildpack liberty13 as admin...
 0 B / 153.10 MiB [-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------]   0.00%REQUEST: [2021-03-07T22:23:58+01:00]
POST /v3/buildpacks/920042da-5e16-4284-84d3-2c0f314bbbab/upload HTTP/1.1
Host: api.127.0.0.1.nip.io
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Content-Type: multipart/form-data; boundary=76729c3712fcf4bbd626ad25cfd3361e7444f0a1a0a13b3f17d923420d5d
User-Agent: cf/7.2.0+be4a5ce2b.2020-12-10 (go1.13.15; amd64 linux)
[multipart/form-data Content Hidden]

 153.10 MiB / 153.10 MiB [==========================================================================================================================================================================================================] 100.00%RESPONSE: [2021-03-07T22:23:59+01:00]
HTTP/1.1 413 Payload Too Large
Content-Length: 176
Content-Type: text/html
Date: Sun, 07 Mar 2021 21:23:59 GMT
Server: istio-envoy
X-Envoy-Upstream-Service-Time: 1
<html>
<head><title>413 Request Entity Too Large</title></head>
<body>
<center><h1>413 Request Entity Too Large</h1></center>
<hr><center>nginx</center>
</body>
</html>

 153.10 MiB / 153.10 MiB [=======================================================================================================================================================================================================] 100.00% 2s
Error unmarshalling the following into a cloud controller error: <html>
<head><title>413 Request Entity Too Large</title></head>
<body>
<center><h1>413 Request Entity Too Large</h1></center>
<hr><center>nginx</center>
</body>
</html>

TIP: A buildpack with name 'liberty13' and nil stack has been created. Use 'cf delete-buildpack' to delete it or 'cf update-buildpack' to try again.
FAILED

..so far I am stuck. Anyone knows if this can be handled by any other filter?

Something's also wrong with local buildpacks "store". After all those failures, it leaves some metadata which can't be deleted nor recreated. It says the buildpack already exists but can't be deleted or listed using cf buildpacks. /// update: seems like buildpack record gets created and it's in 'AWAITING' status for the file upload

s8me5s@DESKTOP-Q8TBEVJ:~/buildpack$ cf curl /v3/buildpacks/e8b8f1a8-693b-4160-bccf-f5d5ed4bee27 | jq -r .state
AWAITING_UPLOAD

..disappears if not provided in a short period of time. ///

s8me5s@DESKTOP-Q8TBEVJ:~/buildpack$ cf create-buildpack liberty13 ibm-websphere-liberty-buildpack-v3.21-9-gfaf82f7.zip 1
Creating buildpack liberty13 as admin...
Buildpack with name 'liberty13' and an unassigned stack already exists
FAILED
s8me5s@DESKTOP-Q8TBEVJ:~/buildpack$ cf delete-buildpack liberty13
Really delete the buildpack liberty13? [yN]: y
Deleting buildpack liberty13...
Multiple buildpacks named liberty13 found. Specify a stack name by using a '-s' flag.
FAILED
s8me5s@DESKTOP-Q8TBEVJ:~/buildpack$ cf buildpacks
Getting buildpacks as admin...

position   name                            stack                         enabled   locked   filename
0          paketo-buildpacks/ruby          io.buildpacks.stacks.bionic   true      false    paketo-buildpacks/ruby@0.4.0
0          paketo-buildpacks/dotnet-core   io.buildpacks.stacks.bionic   true      false    paketo-buildpacks/dotnet-core@0.1.3
0          paketo-buildpacks/nodejs        io.buildpacks.stacks.bionic   true      false    paketo-buildpacks/nodejs@0.1.1
0          paketo-buildpacks/go            io.buildpacks.stacks.bionic   true      false    paketo-buildpacks/go@0.4.0
0          paketo-community/python         io.buildpacks.stacks.bionic   true      false    paketo-community/python@0.0.4
0          paketo-buildpacks/php           io.buildpacks.stacks.bionic   true      false    paketo-buildpacks/php@0.1.1
0          paketo-buildpacks/nginx         io.buildpacks.stacks.bionic   true      false    paketo-buildpacks/nginx@0.0.199
0          paketo-buildpacks/httpd         io.buildpacks.stacks.bionic   true      false    paketo-buildpacks/httpd@0.0.168
0          paketo-buildpacks/java          io.buildpacks.stacks.bionic   true      false    paketo-buildpacks/java@5.2.0
0          paketo-buildpacks/procfile      io.buildpacks.stacks.bionic   true      false    paketo-buildpacks/procfile@4.0.0

What's also weird. Buildpack creation says 'OK' and error appears afterwards. This part of comment also contains the command I use to create the buildpack.

s8me5s@DESKTOP-Q8TBEVJ:~/buildpack$ cf create-buildpack liberty14 ibm-websphere-liberty-buildpack-v3.21-9-gfaf82f7.zip 1
Creating buildpack liberty14 as admin...
OK

Uploading buildpack liberty14 as admin...
 153.10 MiB / 153.10 MiB [=======================================================================================================================================================================================================] 100.00% 2s
Error unmarshalling the following into a cloud controller error: <html>
<head><title>413 Request Entity Too Large</title></head>
<body>
<center><h1>413 Request Entity Too Large</h1></center>
<hr><center>nginx</center>
</body>
</html>

TIP: A buildpack with name 'liberty14' and nil stack has been created. Use 'cf delete-buildpack' to delete it or 'cf update-buildpack' to try again.
FAILED

Expected behavior

My current goal is to create a buildpack and push IBM's test app. However, buildpack creation fails. buildpack: https://github.com/cloudfoundry/ibm-websphere-liberty-buildpack app: https://github.com/IBM-Cloud/get-started-java

Cluster information

local minikube using WSL2

s8me5s@DESKTOP-Q8TBEVJ:~/liberty-test-app$ kubectl cluster-info
Kubernetes master is running at https://127.0.0.1:49179
KubeDNS is running at https://127.0.0.1:49179/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
s8me5s@DESKTOP-Q8TBEVJ:~/liberty-test-app$ cf t
API endpoint:   https://api.127.0.0.1.nip.io
API version:    3.98.0
user:           admin
org:            test-org
space:          test-space

CLI versions

paste output of the following commands

  1. ytt --version: ytt version 0.31.0
  2. kapp --version: kapp version 0.35.0
  3. kubectl version: v1.19.3
  4. cf version: cf version 7.2.0+be4a5ce2b.2020-12-10
cf-gitbot commented 3 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/177238080

The labels on this github issue will be updated when the story is started.

matt-royal commented 3 years ago

Thank you, @KrzMar, for your detailed issue report. Unfortunately, cf-for-k8s does not currently support adding or updating buildpacks through the same method as cf-for-vms, so I'm not surprised you're running into errors. Though the related API endpoints have not been disabled, they are not expected to work. cf-for-k8s uses kpack for staging, and the kpack configuration is what controls the list of available buildpacks. If the ibm-websphere-liberty-buildpack were a Cloud Native Buildpack like the paketo buildpacks, then you could likely integrate it with kpack and add it to your deployment. At first glance it seems that it is an older, Cloud Foundry-specific buildpack that likely won't work with kpack.

krismarc commented 3 years ago

Hi @matt-royal, thx for your reply. Ok, so it's more or less expected behavior. As far as I know IBM plans to release another buildpack based on buildpacks.io.

Meanwhile, I gave a try against another app example (python) and status of the app can't checked, an error occurs details in this issue -> https://github.com/cloudfoundry/cf-for-k8s/issues/639

Best regards, K.M.

Birdrock commented 3 years ago

Since it appears all the relevant information has been shared, I'm closing this issue.