Add gateway_invoke feature

alexellis commented 5 years ago

Signed-off-by: Alex Ellis alexellis2@gmail.com

Add gateway_invoke

Description

This change introduces the gateway_invoke env-var which is used to have the queue-worker invoke functions via the gateway. When set to false the functions are invoked directly over HTTP.

Motivation and Context

[x] I have raised an issue to propose this change (required)

Fixes: #32

How Has This Been Tested?

Tested on Swarm with the flag on/off. Unit tests added for parsing new configuration option.

Options documented in README.md

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.
[x] I have updated the documentation accordingly.
[x] I've read the CONTRIBUTION guide
[x] I have signed-off my commits with git commit -s
[x] I have added tests to cover my changes.
[x] All new and existing tests passed.

Additional configuration and options will be needed in docker-compose.yml and the helm chart.

alexellis commented 5 years ago

An image is available as openfaas/queue-worker:0.7.2-rc1. The faas_gateway_address address must be set on Kubernetes i.e. gateway.openfaas

matipan commented 5 years ago

Looks good. I'm deploying that image now and trying it out on GKE 👍

matipan commented 5 years ago

@alexellis After deploying the new queue worker I see that git-tar is responding with a 500 when building an image, and it's due to the hmac validation:

2019/05/05 03:10:17 Out=2019/05/05 03:10:17 invalid message digest or secret

In the logs of the queue worker we can see that it failed:

[#1] Received on [faas-request]: 'sequence:135 subject:"faas-request" data:"{\"Header\":{\"Accept-Encoding\":[\"gzip\"],\"Content-Length\":[\"404\"],\"User-Agent\":[\"Go-http-client/1.1\"],\"X-Call-Id\":[\"8ae0d8e9-923d-4f6d-8a6e-808f7f41b071\"],\"X-Cloud-Signature\":[\"sha1=Redacted\"],\"X-Start-Time\":[\"1557025635768610414\"]},\"Host\":\"gateway.openfaas:8080\",\"Body\":\"eyJyZWYiOiJyZWZzL2hlYWRzL2RldmVsb3AiLCJSZXBvc2l0b3J5Ijp7Im5hbWUiOiJmYWFzLWRlbW8iLCJmdWxsX25hbWUiOiJtYXRpcGFuL2ZhYXMtZGVtbyIsImNsb25lX3VybCI6Imh0dHBzOi8vZ2l0aHViLmNvbS9tYXRpcGFuL2ZhYXMtZGVtby5naXQiLCJwcml2YXRlIjpmYWxzZSwiaWQiOjE4MjA5MjkzNywidXJsIjoiaHR0cHM6Ly9naXRodWIuY29tL21hdGlwYW4vZmFhcy1kZW1vIiwib3duZXIiOnsibG9naW4iOiJtYXRpcGFuIiwiZW1haWwiOiJtYXRpcGFuQHVzZXJzLm5vcmVwbHkuZ2l0aHViLmNvbSIsImlkIjo4MTI2ODkxfX0sImFmdGVyIjoiN2U4OWZmNDZjNjJlYTU0ZWZlODllY2Q2ZjM0Nzk0MjBmMTkwNDdmOCIsIkluc3RhbGxhdGlvbiI6eyJpZCI6NzQwNzY3fSwiU0NNIjoiZ2l0aHViIn0=\",\"Method\":\"POST\",\"Path\":\"\",\"QueryString\":\"\",\"Function\":\"git-tar\",\"CallbackUrl\":null}" timestamp:1557025635769354722 '
Invoking: git-tar.
Invoked: git-tar [500] in 0.032390s
Wrote 67 Bytes
500 Internal Server Error

I redacted the value of the X-Cloud-Signature but I saw that it's valid, is it possible that the gateway is dropping that header?

If I go back to the other queue worker git-tar works ok. Any ideas why this might be? The configs I added to the environment of the new image are:

        - name: gateway_invoke
          value: "true"
        - name: faas_gateway_address
          value: gateway.openfaas

alexellis commented 5 years ago

@matipan you're right. I haven't been able to find out what is different or wrong as of yet.

I am using this to reproduce the issue https://github.com/openfaas/workshop/blob/master/lab11.md

I get 0 bytes in the function when invoking via the new queue worker whatever the gateway_invoke value is.

matipan commented 5 years ago

I think I found the issue. For some reason(which I have not analyzed yet), the forwarding proxy handler of the gateway is receiving the request as a GET. However the queue worker is sending the request as a POST. Here are the logs of the queue worker:

Invoking: git-tar, 404 bytes. Headers: map[Accept-Encoding:[gzip] Content-Length:[404] User-Agent:[Go-http-client/1.1] X-Call-Id:[c0fd2ddd-c9b2-4ef2-bf7d-c1a821295d02] X-Cloud-Signature:[sha1=Redacted] X-Start-Time:[1557153201790353402]]
Outgoing request: &{Method:POST URL:http://gateway.openfaas:8080/function/git-tar// Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[X-Call-Id:[c0fd2ddd-c9b2-4ef2-bf7d-c1a821295d02] X-Cloud-Signature:[sha1=968e399d7d3f96b67edee3b3dc0fd9c733aa8988] X-Start-Time:[1557153201790353402] Accept-Encoding:[gzip] Content-Length:[404] User-Agent:[Go-http-client/1.1]] Body:{Reader:0xc4201dc570} GetBody:0x5ed590 ContentLength:404 TransferEncoding:[] Close:false Host:gateway.openfaas:8080 Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil> Response:<nil> ctx:<nil>}

And here is the request that the gateway handler is receiving:

Incoming request: &{Method:GET URL:/function/git-tar/ Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[User-Agent:[Go-http-client/1.1] Accept-Encoding:[gzip] Referer:[http://gateway.openfaas:8080/function/git-tar//] X-Call-Id:[c0fd2ddd-c9b2-4ef2-bf7d-c1a821295d02] X-Cloud-Signature:[sha1=Redacted] X-Start-Time:[1557153201790353402 1557153201804283535] Connection:[close]] Body:{} GetBody:<nil> ContentLength:0 TransferEncoding:[] Close:true Host:gateway.openfaas:8080 Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr:10.28.4.118:57450 RequestURI:/function/git-tar/ TLS:<nil> Cancel:<nil> Response:<nil> ctx:0xc4203c1410}

I'll look into why this is happening. Looks like an issue with the gateway and not the queue worker

matipan commented 5 years ago

If the nats queue worker makes a call to http://gateway.openfaas:8080, does it go through some other http server before reaching the gateway's server? I can confirm that the nats queue worker is successfully sending a POST request, however the http server of the gateway is receiving a GET request. This means something in the middle is tampering with the original request the worker is sending.

matipan commented 5 years ago

@alexellis Forget my previous comment, I found the issue. If we look closely into how the queue worker is building the function URL we can see that if the path of the request is empty we default to a /. But then when we format the request we see the following:

functionURL = fmt.Sprintf("http://%s:8080/function/%s/%s%s",
        config.GatewayAddress,
        req.Function,
        path,
        queryString)

Notice that we are placing yet another / in the format of the URL. If we send a simple request to /async-function/git-tar, the resulting function URL will be:

http://gateway.openfaas:8080/function/git-tar//

Notice the two //. Due to the second trailing slash something(I still do not know what) is redirecting that request to GET /function/git-tar/ which is weird in my opinion.

Fixing the formatting of the function url to not append that second / fixed the issue :tada:

matipan commented 5 years ago

It seems that the net/http client is doing that. I created a simple http server:

package main

import (
    "fmt"
    "net/http"
)

func main() {
    http.HandleFunc("/test", func(w http.ResponseWriter, r *http.Request) {
        fmt.Printf("Receiving: %+v\n", r)
    })
    http.HandleFunc("/test/", func(w http.ResponseWriter, r *http.Request) {
        fmt.Printf("Receiving: %+v\n", r)
    })
    http.ListenAndServe(":2626", nil)
}

And made 3 different http requests to it:

package main

import "net/http"

func main() {
    r1, _ := http.NewRequest(http.MethodPost, "http://localhost:2626/test/", nil)
    r2, _ := http.NewRequest(http.MethodPost, "http://localhost:2626/test//", nil)
    r3, _ := http.NewRequest(http.MethodPost, "http://localhost:2626/test", nil)
    http.DefaultClient.Do(r1)
    http.DefaultClient.Do(r2)
    http.DefaultClient.Do(r3)
}

The HTTP server logged the following http requests:

Receiving: &{Method:POST URL:/test/ Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[User-Agent:[Go-http-client/1.1] Content-Length:[0] Accept-Encoding:[gzip]] Body:{} GetBody:<nil> ContentLength:0 TransferEncoding:[] Close:false Host:localhost:2626 Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr:127.0.0.1:59228 RequestURI:/test/ TLS:<nil> Cancel:<nil> Response:<nil> ctx:0xc0000fc180}
Receiving: &{Method:GET URL:/test/ Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1] Referer:[http://localhost:2626/test//]] Body:{} GetBody:<nil> ContentLength:0 TransferEncoding:[] Close:false Host:localhost:2626 Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr:127.0.0.1:59228 RequestURI:/test/ TLS:<nil> Cancel:<nil> Response:<nil> ctx:0xc000120080}
Receiving: &{Method:POST URL:/test Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[User-Agent:[Go-http-client/1.1] Content-Length:[0] Accept-Encoding:[gzip]] Body:{} GetBody:<nil> ContentLength:0 TransferEncoding:[] Close:false Host:localhost:2626 Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr:127.0.0.1:59228 RequestURI:/test TLS:<nil> Cancel:<nil> Response:<nil> ctx:0xc0000fc300}

Notice that the one in the middle(which originally was to /test//) is being received as a GET request

openfaas / nats-queue-worker