search

kubernetes / ingress-nginx

Ingress-NGINX Controller for Kubernetes
https://kubernetes.github.io/ingress-nginx/
Apache License 2.0
17.28k stars 8.21k forks source link

uploading file via ingress increases size of file #11026

Open kishorpacefin opened 6 months ago

kishorpacefin commented 6 months ago

Description

--->I have deployed a application in AKS cluster and access this same application through ingress host. While uploading multipart file using ingress host("https://abc.com/swagger/dao/index.html#/dao%20V1/post_api_dao_v1_check_uploadImageDigioCheck") for this application api my file size got increased. but same multipart file i have deployed without ingress host using application load balancer("http://21.264.74.91/swagger/dao/index.html#/dao%20V1/post_api_dao_v1_check_uploadImageDigioCheck") i am getting exact file size.

--->And also same docker image which i have used in my kuberntes cluster same docker image i am pulling in to my local and ran that particular image in my localhost without ingress , their when i am uploading same multipart file their also i am getting exact file size.

--->This is when i am uploading file using application load balancer or using same docker image in locally without any ingress i am getting exact file size. time="26-02-2024 12:04:09" level=info msg="File size:1270296 fileName: pan_image.jpg" file="kyc.go:241 "time="26-02-2024 12:04:09" level=info msg="File size:1270296 fileName: pan_image.jpg" file="kyc.go:241"

--->Tihs is when i am uploading file using ingress nginx controller host name time="26-02-2024 11:38:26" level=info msg="File size:2305390 fileName: pan_image.jpg" file="kyc.go:241 "time="26-02-2024 11:38:26" level=info msg="File size:2305390 fileName: pan_image.jpg" file="kyc.go:241"

here if can see the file size is File size:2305390 which is wrong the actual file size is 1270296.

So the issue is from ingress nginx controller.

Actual output

--->Here i am expecting the the exact file size i should get .

ingress and kubernetes Descriptions

kubectl -n ingress-nginx describe po ingress-nginx-controller-85b7678f89-sssh4 Name: ingress-nginx-controller-85b7678f89-sssh4 Namespace: ingress-nginx Priority: 0 Service Account: ingress-nginx Node: aks-agentpool-33655406-vmss000000/10.224.0.4 Start Time: Fri, 17 Nov 2023 21:32:19 +0530 Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/name=ingress-nginx pod-template-hash=85b7678f89 Annotations: cni.projectcalico.org/containerID: a074c8e1b32581e45bdaa98839545e8d59e8459a23f8c7d8a9c26c8cf2f5c0d5 cni.projectcalico.org/podIP: 10.298.1.28/32 cni.projectcalico.org/podIPs: 10.298.1.28/32 Status: Running IP: 10.298.1.28 IPs: IP: 10.298.1.28 Controlled By: ReplicaSet/ingress-nginx-controller-85b7678f89 Containers: controller: Container ID: containerd://5415e7cb70a153545316598fcf532b6f016b1ce92941d89cdab68798fd67ec86 Image: registry.k8s.io/ingress-nginx/controller:v1.3.0@sha256:d1707ca76d3b044ab8a28277a2466a02100ee9f58a86af1535a3edf9323ea1b5 Image ID: registry.k8s.io/ingress-nginx/controller@sha256:d1707ca76d3b044ab8a28277a2466a02100ee9f58a86af1535a3edf9323ea1b5 Ports: 80/TCP, 443/TCP, 8443/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP Args: /nginx-ingress-controller --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller --election-id=ingress-controller-leader --controller-class=k8s.io/ingress-nginx --ingress-class=nginx --configmap=$(POD_NAMESPACE)/ingress-nginx-controller --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key State: Running Started: Fri, 17 Nov 2023 21:35:02 +0530 Ready: True Restart Count: 0 Requests: cpu: 100m memory: 90Mi Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5 Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: POD_NAME: ingress-nginx-controller-85b7678f89-sssh4 (v1:metadata.name) POD_NAMESPACE: ingress-nginx (v1:metadata.namespace) LD_PRELOAD: /usr/local/lib/libmimalloc.so Mounts: /usr/local/certificates/ from webhook-cert (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8jtp5 (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: webhook-cert: Type: Secret (a volume populated by a Secret) SecretName: ingress-nginx-admission Optional: false kube-api-access-8jtp5: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: kubernetes.io/os=linux Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events:

Here how you will Reproduce issue

Deploy a application in kuberntes cluster and access this application using the hostname then add a functionality to upload a file . here when you upload the file i am observing in my logs my file size gets increased, but same file i am uploading using application load balancer (no ingress) here in logs i am getting exact same file .

in both case file get uploaded successfully. but the only difference is in case of ingress size got increased.

Here is my ingress file.

apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: dev-router-master annotations: nginx.ingress.kubernetes.io/proxy-body-size: "50m" spec: ingressClassName: nginx rules:

here in ingress i have also mentioned body size up to 50 mb . so up to 50 mb it can uploded, but my actual file size is 1.3 mb.

---> This is my application deployment

apiVersion: apps/v1 kind: Deployment metadata: name: dao-app labels: app: dao-app spec: replicas: 1 selector: matchLabels: app: dao-app template: metadata: labels: app: dao-app spec: imagePullSecrets:

--->Here is the screeenshot of file uploaded sucessfully using ingress hostname but with increment size

Screenshot from 2024-02-27 11-28-59

here if can see the file size is File size:2305390 which is wrong the actual file size is 1270296 time="26-02-2024 11:38:26" level=info msg="File size:2305390 fileName: pan_image.jpg" file="kyc.go:241 "time="26-02-2024 11:38:26" level=info msg="File size:2305390 fileName: pan_image.jpg" file="kyc.go:241"

Here is the screeenshot of file uploaded sucessfully using deployment load balancer but with exact size

image

here actual file size is uploaded which is 1270296 time="26-02-2024 12:04:09" level=info msg="File size:1270296 fileName: pan_image.jpg" file="kyc.go:241"time="26-02-2024 12:04:09" level=info msg="File size:1270296 fileName: pan_image.jpg" file="kyc.go:241"

k8s-ci-robot commented 6 months ago

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
longwuyuan commented 6 months ago

/remove-kind bug /retitle download file-size problem when using ingress

longwuyuan commented 6 months ago

/triage needs-information

kishorpacefin commented 6 months ago

Please go through the full description i have mentioned everythings . why you are just closed the bugs without go through properly.

longwuyuan commented 6 months ago

image

% k describe ing httpd                                  
Name:             httpd
Labels:           <none>
Namespace:        default
Address:          192.168.59.101
Ingress Class:    nginx
Default backend:  <default>
Rules:
  Host                       Path  Backends
  ----                       ----  --------
  httpd.dev.enjoydevops.com  
                             /   httpd:80 (10.244.0.79:80)
Annotations:                 <none>
Events:
  Type    Reason  Age                   From                      Message
  ----    ------  ----                  ----                      -------
  Normal  Sync    104s (x2 over 5d23h)  nginx-ingress-controller  Scheduled for sync
[~] 
% k get po httpd-764dbd796-ggkgj -o yaml | grep -i image:
  - image: httpd:alpine
    image: docker.io/library/httpd:alpine

/retitle download file-size problem when using ingress

kishorpacefin commented 6 months ago

I think you have not getting my question , i am describing my issue

1) i have deploy my application code in to a docker image. 2)Then same image i am deployed into kuberntes cluster with a load balancer service type. 3)And provide this loadbalancer service to my ingress backend. 4) So one of the functionality of the application is to upload multipart file which i am provided a image in the (https://github.com/kubernetes/ingress-nginx/issues/11026#issue-2155742193) description you can varify , their while uploading it is uploding successfully, but the size of files gets incremented(so my actual size is around 1270296 mb and the size i am gettiing is 2305390mb which i am observing in my logs . here is the logs i am getting inside pod time="26-02-2024 11:38:26" level=info msg="File size:2305390 fileName: pan_image.jpg" file="kyc.go:241 "time="26-02-2024 11:38:26" level=info msg="File size:2305390 fileName: pan_image.jpg" file="kyc.go:241"

5)Here to varify if this issue is from ingress or anything else i have used the application load balancer ip which i have created . so while uploading using this loadbalncer ip , i am also getting same exact file size which is 1270296.

here i am replacing the ingress hostaname to load balancer ip http://21.264.74.91/swagger/dao/index.html#/dao%20V1/post_api_dao_v1_check_uploadImageDigioCheck here is the logs inside my pod time="26-02-2024 12:04:09" level=info msg="File size:1270296 fileName: pan_image.jpg" file="kyc.go:241" time="26-02-2024 12:04:09" level=info msg="File size:1270296 fileName: pan_image.jpg" file="kyc.go:241"

6)the same image i am deploying which i have used in kuberntes cluster i am deploying in to my local system without any ingress while uploading image using api call which is a functionality of my application . here i am getting exact same size.

http://localhost:8080/swagger/dao/index.html#/dao%20V1/post_api_dao_v1_check_uploadImageDigioCheck

logs inside container time="26-02-2024 12:04:09" level=info msg="File size:1270296 fileName: pan_image.jpg" file="kyc.go:241" time="26-02-2024 12:04:09" level=info msg="File size:1270296 fileName: pan_image.jpg" file="kyc.go:241"

so , here is my conclusion issue is related to ingress nginx controller, beacause using load balancer ip and using same docker image without ingress when i am uploading multipart file through my application i am getting exact size of file which is 1270296.

kishorpacefin commented 6 months ago
  • i have tested and I can see that I can get the exact same file size that is on the server behind the ingress

image

% k describe ing httpd                                  
Name:             httpd
Labels:           <none>
Namespace:        default
Address:          192.168.59.101
Ingress Class:    nginx
Default backend:  <default>
Rules:
  Host                       Path  Backends
  ----                       ----  --------
  httpd.dev.enjoydevops.com  
                             /   httpd:80 (10.244.0.79:80)
Annotations:                 <none>
Events:
  Type    Reason  Age                   From                      Message
  ----    ------  ----                  ----                      -------
  Normal  Sync    104s (x2 over 5d23h)  nginx-ingress-controller  Scheduled for sync
[~] 
% k get po httpd-764dbd796-ggkgj -o yaml | grep -i image:
  - image: httpd:alpine
    image: docker.io/library/httpd:alpine
  • so the label bug can be applied after there is acceptable data that proves the existence of a bug. wait for someone to reproduce the bug to see the bug label. wrong labels causes problems in tracking the issues so you can help there
  • the title of the issue gets reported in many places so title needs to be short and informative. long sentences are not acceptable as title of a issue because the reports and tracking information gets messed up if the title is actually too long. so do not change the title to a long sentence as a help to others

/retitle download file-size problem when using ingress

  • you can reproduce the problem using a kind cluster or a minikube cluster (while creating a file using dd under the webroot and curling with -o for the file) then post the entire config related to the app and the file and the ingress with -v etc etc
  • the new issue template asks questions that help get the details of the cluster, the controller, the ingress and so on. Edit your issue description so that that information is visible to a reader here in markdown format and also answer any questions that have not been ansewered previously
  • providing the details of your config and internals helps a reader understand your use case and so more chances to help solve your problem. so help out by showing kubectl and curl command outputs as much as possible

Mistake you did

Here what you did to download the file from the host using curl command , but i am not downloading file from the host i am uploading a file using the host. without host i am getting exact file size.

longwuyuan commented 6 months ago

@kishorpacefin , the info you have posted is clear to understand. Your problem is file size increase when using ingres

kishorpacefin commented 6 months ago

@kishorpacefin , the info you have posted is clear to understand. Your problem is file size increase when using ingres

  • What is not clear is the details
  • What is required is the details in the form of outputs of kubectl command and hopefully curl command as well as `linux ls -lth1 etc etc
  • I showed you an example of a test using download because it is easy. So you can do a similar test for upload and write the reproduce steps here so a reader can reproduce the problem on their own cluster
  • In a new browser, click the button to create new bug report for this project and look at the qestions asked there. Then look at the issue description you have provided. You will find that neither all questions are answered and whatever info provided is very very hard to read or understand because it not formatted in markdown. Fix that and improve the issue description so readers can make helpful comments

okay , sorry for my mistake, later i will try to provide you with every details

here i am uploading multipart file using curl with load balancer ip using ingress hostname

1)Here is my ingress hostname curl command and it's output (here i have given wrong url name so just ignore the url ) curl -X 'POST' 'https://abc.com/api/dao/v1/check/uploadImageDigioCheck' -H 'accept: application/json' -H 'P-DeviceType: web' -H 'Content-Type: multipart/form-data' -F 'file1=@/home/pace/Downloads/pan_image.jpg;type=image/jpeg' -F 'file2=@/home/pace/Downloads/pan_image.jpg;type=image/jpeg' File uploaded successfullyFile uploaded successfully

2)here is the response in my application ui if you want

access-control-allow-credentials: true access-control-allow-headers: Content-Type,Content-Length,Accept-Encoding,X-CSRF-Token,Authorization,X-Max access-control-allow-methods: POST,GET,PUT,DELETE,UPDATE access-control-allow-origin: * access-control-max-age: 86400 content-length: 52 content-type: application/json date: Tue,27 Feb 2024 08:58:55 GMT via: 1.1 93882c19d869a346304c.cloudfront.net (CloudFront) x-amz-apigw-id: TyacfHbzQ= x-amz-cf-id: BRQaBrq9GYJqZZ_QB8Oun7CbW9_Z8u_86AtA== x-amz-cf-pop: DEL54-P4 x-amzn-remapped-connection: keep-alive x-amzn-remapped-content-length: 52 x-amzn-remapped-date: Tue,27 Feb 2024 08:58:55 GMT x-amzn-requestid: f98db4ba-499a-4cf8-b3f-15897c60 x-cache: Miss from cloudfront

here it will go to like this first go to dns name ----> route 53----> api gateway----> then ingress----> backend service

3) here you can see my file size inside pod -rw-r--r-- 1 root root 2305390 Feb 27 09:52 pan_image.jpg

here i am uploading multipart file using curl with load balancer ip

4)same i am uploading with using loadbalancer service ip my file got uploaded successfully and also i am getting exact size.(here igone the ip i am giving wrong load balancer ip address)

curl -X 'POST' 'http://21.264.74.91/api/dao/v1/check/uploadImageDigioCheck' -H 'accept: application/json' -H 'P-DeviceType: web' -H 'Content-Type: multipart/form-data' -F 'file1=@/home/pace/Downloads/pan_image.jpg;type=image/jpeg' -F 'file2=@/home/pace/Downloads/pan_image.jpg;type=image/jpeg' File uploaded successfullyFile uploaded successfully

5) here is the response in my application ui if you want

access-control-allow-credentials: true access-control-allow-headers: Content-Type,Content-Length,Accept-Encoding,X-CSRF-Token,Authorization,X-Max access-control-allow-methods: POST,GET,PUT,DELETE,UPDATE access-control-allow-origin: * access-control-max-age: 86400 content-length: 52 content-type: application/json date: Tue,27 Feb 2024 10:14:01 GMT

here is the difference is it is going like load balancer ip ------> then application deployment

6) here you can my file size inside pod rw-r--r-- 1 root root 1270296 Feb 27 10:14 pan_image.jpg

longwuyuan commented 6 months ago

/retitle uploading file via ingress increases size of file

kishorpacefin commented 6 months ago

@longwuyuan Any update or finding regarding this issue ?

longwuyuan commented 6 months ago

@kishorpacefin no. My reasons are

mloskot commented 6 months ago

@kishorpacefin

Please go through the full description i have mentioned everythings, why you are just closed the bugs without go through properly.

Because you haven't put any effort to make your issue readable?

Because you haven't followed suggestion posted in https://github.com/kubernetes/ingress-nginx/issues/11026#issuecomment-1965867431 to re-format your issue according to https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax, so it does not look like a pile of mess and a human being has a chance to actually make sense of it?

Garbage in, garbage out, simple as that.

github-actions[bot] commented 5 months ago

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.