[sdk-v2]: ability to create a custom HttpClient with custom TLS certificates/connect via proxy

[bacongobbler]

Currently, Spin's v2 SDK doesn't support self-signed server certificates which are used by almost all Kubernetes deployments, including AWS and Google. This is because we're using the web platform's fetch() API, which does not allow programs to load custom TLS certificates due to the security implications of the browser APIs.

It would be great if there was a way to create a custom HttpClient to use with fetch(). This would be an extension of the web platform Fetch API which would allow applications to load custom TLS certificates and connect via a proxy while using fetch().

See also https://deno.land/api@v1.44.4?s=fetch&unstable=

[bacongobbler]

@lann mentioned an idea in https://github.com/fermyon/spin/pull/2596#discussion_r1651672082:

I think ideally we could define a standard API in terms of wasi:http, possibly using proposed https://github.com/WebAssembly/wasi-http/issues/4 (TLS config was one of my use cases for that proposal).

[karthik2804]

I believe this will need changes upstream before the SDK can support it. @tschneidereit can you confirm that?

[tschneidereit]

It would be great if there was a way to create a custom HttpClient to use with fetch(). This would be an extension of the web platform Fetch API which would allow applications to load custom TLS certificates and connect via a proxy while using fetch().

@bacongobbler do you think it's strictly necessary to do this in content? A potential alternative could be to add a way to specify certificates to use for specific connections in the Spin.toml manifest.

That way, content would never see the certificate, and we'd not have the overhead of loading it for every instance.

We could do the same thing for proxies, in addition or alternatively to building something on top of WASI extensions, along the lines of a sketch I did a while ago.

[bacongobbler]

@tschneidereit it's not strictly necessary, no. fermyon/spin#2596 adds it to runtime-config.toml. In that PR, it would result in a runtime-config.toml file that looks like this:

[[client_tls]]
hosts = ["kubernetes"]
component_ids = ["app"]
custom_root_ca_file = "certs/ca.crt"
cert_chain_file = "certs/client.pem"
private_key_file = "certs/client.key"

Adding it to guest code would allow things like kubernetes client libraries to continue to work as expected, though. It expects to load TLS certificates from the filesystem. To work around this, we're loading certificates into the host via runtime-config and making raw fetch calls to the API.

Have you read @endocrimes's comment in that thread? Might be worth considering that use case and seeing if we may need to implement it on the guest side.