[Serve] Expose load balancer and services via TLS.

skypilot-org / skypilot

SkyPilot: Run AI and batch jobs on any infra (Kubernetes or 12+ clouds). Get unified execution, cost savings, and high GPU availability via a simple interface.

https://skypilot.readthedocs.io

Apache License 2.0

6.62k stars 481 forks source link

[Serve] Expose load balancer and services via TLS. #3198

Open maxmele opened 7 months ago

maxmele commented 7 months ago

Recently, I've been testing SkyPilot to expose a LLM API using sky serve. Everything is working great and I really like the project, thank you for the great work!

However, I couldn't find any reference in the documentation on how to expose the services and load-balancers via TLS.

The prompts we send to the API contain sensitive data that we prefer not to transmit in plain text over the internet. So it would be really nice to be able to configure TLS for both the load balancer and services.

Do you have any plans to implement this feature in the future?

concretevitamin commented 7 months ago

Thanks for this report and glad to see SkyPilot is working @maxmele!

We're actively looking into the security aspects of SkyServe. A few questions:

Is it possible for the serving app to handle encryption + decryption? Or is it too much of a burden?
Is any of these options good enough (they are not directly about traffic encryption)?
- (Supported in main already) Launch an entire serve deployment (controller + replicas) in a private VPC, exposing private IPs only
- Use a VPN service (e.g., Tailscale) to put an entire serve deployment (controller + replicas) in
- Some other options we're brainstorming

maxmele commented 7 months ago

Hi, thanks for the response!

Absolutely! We were tinking about implementing encryption/decryption between ends, but we're hoping to find an alternative solution to avoid that.

Regarding the option of using private IPs within a VPC... doesn't it limit the ability to be multicloud and even multi-region? Unless you establish connections between different VPCs beforehand.

For me, it would be absolutely amazing if skypilot could connect nodes to a Tailscale network, and even better if it could use a custom control server like Headscale.

concretevitamin commented 7 months ago

Regarding the option of using private IPs within a VPC... doesn't it limit the ability to be multicloud and even multi-region? Unless you establish connections between different VPCs beforehand.

Yep, that's the tradeoff of that approach. Peering between VPCs needed for multi-region, and it doesn't support multi-cloud out of the box.

The team actually has been brainstorming quite a few options (cc @Michaelvll @cblmemo). Feel free to join https://slack.skypilot.co/ Slack as we'd love to learn more about your deployment requirements!

github-actions[bot] commented 3 months ago

This issue is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 10 days.

Michaelvll commented 1 week ago

This is being added by #3380