fn server behind TLS

[prologic]

It isn't clear to me from any docs I've read how to configure fn server behind a TLS LB.

I can create apps, functions, build, etc; but when I fn invoke I get an error like:

$ fn invoke myapp hello

Fn: Error invoking function: Post http://fn.mydomain.com:443/invoke/01DGSKRPEG18004KGZJ0000002: net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x01\x00\x02\x02\x16"

See 'fn <command> --help' for more information. Client version: 0.5.84

NB: the http:// in the URI? That's wrong. How do I convince fn invoke (and this fails similarly in the UI too) to use TLS and https://... URIs?

[denismakogon]

Hi, thanks for reporting. Well, there is no point in bringing TLS into all-in-one Fn (fn start). TLS is possible only when you run Fn in a distributed mode. More you can read in my blogpost: https://medium.com/fnproject/distributed-fn-beyond-fn-start-2ee0baf6fb77

[prologic]

Sorry no I don't understand. I have the fn server behind a load balancer that terminates TLS. I'm not using fn start. The problem is I can't figure out a way to convince the fn CLI tool or the FN Web UI that all the endpoints of functions are https:// and not http://

[denismakogon]

Would you mind to tell how you deployed Fn? If it's not fn start and not a distributed Fn, then what?

Fn CLI is not the place to configure TLS support, it's the server-side thing and TLS is only available in distributed Fn (read the post, take a look at Fn helm chart where we use ingress proxy with TLS enabled).

[prologic]

I'm sorry we have a communication failure here.

I am not using "Distributed Fn".

I just simply deploy the Fn Server behind a very typical Reverse Proxy / Load Balancer that does TLS termination.

How can I explain more clearly that the problem is the Client(s) -- Both the CLI and the Web UI do not understand that they should be talking to the Fn API over TLS using https://... instead of http://...?

Would a PR that fixes this help? For example I do something like export FN_URL=https://fn.mydomain.com but the fn CLI doesn't care about this for vn invoke -- it only uses it for other things (which is weirdly inconsistent but understandable since it looks like once functions are registered there is metadata stored about them in a database somewhere including their trigger URIs -- this is what's wrong).

[denismakogon]

All-in-one Fn is not capable to handle TLS and meant to be a dev tool only. A single instance of the Fn running behind a reverse proxy still would be HTTP instead of HTTPS. So, if you'd like to make Fn handle TLS you need to switch to distributed Fn because this is the ONLY mode that allows HTTPS. There are no other options.

The concept of a single instance of Fn server is all about local development, on your laptop, on the go, where there's no internet, etc. So, the thing you're trying to accomplish is running a dev tool like it was a production setup which is not correct and this is why we have distributed Fn which is meant to be the only option to running Fn in a context of production setup.

[denismakogon]

Fn CLI doesn't care about your endpoint configuration because an invoke annotation comes from a server (dynamically configured at the runtime), so, as I stated above, there's only one option when you can configure Fn server to handle HTTPS triggers and invocation is to go distributed mode.

In details, there's env opt: FN_PUBLIC_LB_URL that stands for annotation base URL, but you can't apply it when your run Fn as a single instance, that env opt will be omitted. So, you need to deploy at least 1 API node, 3 runners and 1 runners LB (where LB node does all heavy lifting and processes all invocations, see https://github.com/fnproject/docs/blob/master/fn/operate/runner_pools.md). So, in a distributed setup, you need to let your API node to be aware of LB runner URL, in this case, API node will provide you that endpoint + path to your function, the same thing applies to triggers). That's how it works.

[prologic]

So you are saying the following is not supported?

This doesn't work:

$ fn invoke myapp hello

Fn: Error invoking function: Post http://fn.mydomain.com:443/invoke/01DGSP2TVG18004KGZJ0000007: net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x01\x00\x02\x02\x16"

See 'fn <command> --help' for more information. Client version: 0.5.84

But this works:

$ curl -q -s -o - -X POST https://fn.mydomain.com:443/invoke/01DGSP2TVG18004KGZJ0000007
{"message":"Failed to pull image 'r.mydomain.com/hello:0.0.6': Get https://r.mydomain.com/v2/hello/manifests/0.0.6: no basic auth credentials"}

Without doing a full LB Node + Runners setup?

[prologic]

Also just to clarify your architecture here... Are we really talking about "Distributed" here or "Load Balancing"? Distributed has a very different meaning and usually requires some notion of "concensus" and "distribution of state".

[denismakogon]

This doesn't work

correct, Fn will return your an HTTP endpoint that wouldn't work because Fn must be aware of what it runs being off.

But this works

because you have explicitly used HTTPS schema which works by default.

Are we really talking about "Distributed"

yes, we're talking about the "distributed mode" even taking into account that there's an LB node (gRCP custom-made runner load balancer).

distribution of state

correct, you'd need additionally run shared storage among nodes of clusters, networking isolation,, etc.

[denismakogon]

Without doing a full LB Node + Runners setup?

I guess you're missing the key point. When you do fn invoke ... Fn CLI doesn't take FN_API_URL (or whatever is in your default context, see fn context) but picks up an annotation from a function object (see fn inspect fn) and uses that URL (which is generated by Fn server) for further HTTP POST.

Base URL that Fn use for Invoke API is not configurable unless you'll do the distributed Fn because Invoke API is a responsibility of runner LB node.

A good example of the reverse proxy configuration here is K8s ingress controller. In Fn helm chart, we tell API node that LB runner will be running behind a reverse proxy by specifying an endpoint of reverse proxy in FN_PUBLIC_LB_URL pod env var. With that, every time a user will attempt to call a function, API node will provide an endpoint of the reverse proxy endpoint that routes to LB runner node.

[prologic]

Okay... Fair enough. Whilst I understand more about your "original design" which is fine; I have to admit I'm a little frustrated now :) Because the barrier to entry for FnProject is much higher. Sadly your project however is one of the most well made out there AFAICT :)

Are we sure we can't have it both ways and reduce the adoption curve slightly by allowing my kind of setup?

[denismakogon]

I'd say the contribution is highly appreciated. Most of us (from Oracle) use distributed Fn, this design is applied in Oracle Functions. Most of our OSS users made their way to k8s deployment. So, if you're willing to help to make your own use case to work - I'd be more than happy to help you to get things done.

[prologic]

Cool! Then please keep this issue open. I'll try to find some time to figure out enough internals to ake this work -- In a much larger scale deployment I would of course go with the multi-runner - lb-noe deployment. But thank you for supporting a potential contribution to make a much simpler deployment work!

[arieltorti]

Hey, I tried to reproduce this issue, and I came to the conclusion that maybe the function you're trying to invoke wasn't deployed behind tls and as such the created invoke endpoint used http instead of https

The full setup I used is in https://github.com/maks500/fndemo/tree/master/tls-issue, I deployed it using docker without k8s and Caddy as the lb/rp