Wrong content-type header when calling function through gateway

tlemarchand commented 4 years ago

My actions before raising this issue

[x] Followed the troubleshooting guide
[x] Read/searched the docs
[x] Searched past issues

Expected Behaviour

When a function is called through the gateway, it should return the same headers (and specifically the same content-type header) than when it's called directly.

Current Behaviour

When a function is called through gateway, if direct_functions parameter is set to false, content-type return header is equal to the content-type header sent by the client.

As an example, I have called a function directly (port 8080, through openfaas-watchdog), returned content-type is correct :

> POST / HTTP/1.1
> Host: splitter-permis.axa-fr-dev-int.teal.westeurope.azure.openpaas.axa-cloud.com
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Length: 118212
> Content-Type: multipart/form-data; boundary=------------------------3a7a50e10eb396f9
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< content-length: 471
< content-type: application/json
< date: Mon, 31 Aug 2020 12:51:43 GMT
< x-duration-seconds: 2.179032
< set-cookie: 0c76b3b7e3633de0795f8b70331200d1=1778425f7f0222fcd72a67a69b8b2d83; path=/; HttpOnly; Secure
<
{"files":[{"filename":"saltstack-enterprise-release-notes-6_2_0.pdf-1.png","height":2200,"id":"f5879bfa-e0e3-46ba-9c5a-7a3bfba694c5","page_number":1,"width":1700},{"filename":"saltstack-enterprise-release-notes-6_2_0.pdf-2.png","height":2200,"id":"48093010-31fa-464c-a400-6cd1bf98116e","page_number":2,"width":1700},{"filename":"saltstack-enterprise-release-notes-6_2_0.pdf-3.png","height":2200,"id":"0f933a3d-31de-43cf-9131-fbc94c955afc","page_number":3,"width":1700}]}
* Connection #0 to host splitter-permis.axa-fr-dev-int.teal.westeurope.azure.openpaas.axa-cloud.com left intact

Same function, this time through openfaas gateway, returned content-type is wrong :

> POST /function/license-splitter HTTP/1.1
> Host: permis-openfaas.axa-fr-dev-int.teal.westeurope.azure.openpaas.axa-cloud.com
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Length: 118212
> Content-Type: multipart/form-data; boundary=------------------------f01cc375500cc272
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< content-length: 471
< content-type: multipart/form-data; boundary=------------------------f01cc375500cc272
< date: Mon, 31 Aug 2020 12:33:32 GMT
< x-duration-seconds: 2.156178
< set-cookie: 9757cb406d295d7baa9a8e457d584d0a=44c1eb50977a4a0a264409ad1f7f2217; path=/; HttpOnly; Secure
<
{"files":[{"filename":"saltstack-enterprise-release-notes-6_2_0.pdf-1.png","height":2200,"id":"39110274-f359-4d10-a835-a7a1e9663343","page_number":1,"width":1700},{"filename":"saltstack-enterprise-release-notes-6_2_0.pdf-2.png","height":2200,"id":"499ed6fe-8c48-4bbd-8689-550f38399c8f","page_number":2,"width":1700},{"filename":"saltstack-enterprise-release-notes-6_2_0.pdf-3.png","height":2200,"id":"d607cf27-77f5-4962-99b2-6bc416e73133","page_number":3,"width":1700}]}
* Connection #0 to host permis-openfaas.axa-fr-dev-int.teal.westeurope.azure.openpaas.axa-cloud.com left intact

Possible Solution

I have found another issue, describing the exact same behavior, but closed in 2017, so maybe same problem but different root cause : https://github.com/openfaas/faas/issues/312

Steps to Reproduce (for bugs)

Deploy openfaas with direct_functions set to false.
Deploy a function asking for a multipart upload and answering with a json (for example).
Call this function with curl -v through gateway, and check returned content-type header.

Context

Your Environment

FaaS-CLI version ( Full output from: faas-cli version ): 0.12.4
Docker version docker version (e.g. Docker 17.0.05 ): 19.03.13-beta2
Are you using Docker Swarm or Kubernetes (FaaS-netes)? Kubernetes
Operating System and version (e.g. Linux, Windows, MacOS): MacOs 10.15.4
Code example or link to GitHub repo or gist to reproduce problem:
Other diagnostic information / logs from troubleshooting guide

If using Swarm, run the following and include the output:

curl -sLS https://raw.githubusercontent.com/moby/moby/master/contrib/check-config.sh | bash

Next steps

You may join Slack for community support.

alexellis commented 4 years ago

Please help us to help you by providing a Github repository that we can use to reproduce your issue among with detailed instructions.

alexellis commented 4 years ago

@tlemarchand hope to hear back from you soon? Do you still need this issue open?

tlemarchand commented 4 years ago

@alexellis Thanks for your answer. I still have the issue. Unfortunately the functions that are impacted are not public, and I have not found any public function working with an multipart upload and returning json or something else. So I have to put up a mock to make it easily reproducible, it might take a few days ...

alexellis commented 4 years ago

Ok, but this shouldn't really take days to reproduce. Perhaps find a simple upload example for express.js or whatever you're using and use that? I'll keep the issue open another week.

tlemarchand commented 4 years ago

@alexellis you're right, it didn't took days (a nice colleague built a mock function for me) !

Steps to reproduce, on OpenFaas with operator over Kubernetes and direct_functions set to false :

Deploy this function :

apiVersion: openfaas.com/v1
kind: Function
metadata:
name: mock
spec:
environment:
combine_output: "false"
log_level: ERROR
read_debug: "true"
read_timeout: 10s
write_debug: "true"
write_timeout: 10s
image: tlemarchand/mock-function:v0
labels:
com.openfaas.scale.min: "1"
limits:
cpu: 50m
memory: 256Mi
name: mock
requests:
cpu: 25m
memory: 128Mi

Upload a PDF to the function through the gateway ( : curl -v -X POST -F file=@"/home/thomas/test.pdf" https://gateway.openfaas/function/mock And check the return :

> POST /function/mock HTTP/1.1
> Host: gateway.openfaas
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Length: 118212
> Content-Type: multipart/form-data; boundary=------------------------acc50ded887785be
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< content-length: 129
< content-type: multipart/form-data; boundary=------------------------acc50ded887785be
< date: Fri, 04 Sep 2020 12:58:04 GMT
< x-duration-seconds: 0.865100
< set-cookie: 9757cb406d295d7baa9a8e457d584d0a=66cfc9bb9377472353f808963f5d90fc; path=/; HttpOnly; Secure
<
[{"confidence":0.9834659099578857,"coordinates":{"xmax":411,"xmin":-9,"ymax":1104,"ymin":-53},"document_id":"0","label":"mock"}]

Returned content-type header is wrong.

Expose the function directly to bypass the gateway (did it with a route, but you can do the same with an ingress) :

kind: Route
apiVersion: route.openshift.io/v1
metadata:
name: mock
spec:
host: mock.direct
to:
kind: Service
name: mock
weight: 100
port:
targetPort: http
tls:
termination: edge
insecureEdgeTerminationPolicy: None
wildcardPolicy: None

Upload same PDF to the function directly : curl -v -X POST -F file=@"/home/thomas/test.pdf" https://mock.direct/ And check the return :

> POST / HTTP/1.1
> Host: mock.direct
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Length: 118212
> Content-Type: multipart/form-data; boundary=------------------------4140fb8cc6f9a44a
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< content-length: 129
< content-type: application/json
< date: Fri, 04 Sep 2020 13:09:50 GMT
< x-duration-seconds: 0.356345
< set-cookie: 8ac40157395f389a681c21b2265cd916=383e2a4e214d222b7af35b5c53542117; path=/; HttpOnly; Secure
<
[{"confidence":0.9834659099578857,"coordinates":{"xmax":411,"xmin":-9,"ymax":1104,"ymin":-53},"document_id":"0","label":"mock"}]

Returned content-type header is correct.

alexellis commented 3 years ago

/lock: closing due to inactivity

openfaas / faas