search

kubernetes-sigs / aws-load-balancer-controller

A Kubernetes controller for Elastic Load Balancers
https://kubernetes-sigs.github.io/aws-load-balancer-controller/
Apache License 2.0
3.9k stars 1.45k forks source link

[Feature Request] PrivateLink Support #1859

Open rifelpet opened 3 years ago

rifelpet commented 3 years ago

We have a need to create PrivateLink Endpoint Services with our NLBs. A Service resource would correspond to one NLB as well as one Endpoint Service configured to use that NLB.

I imagine the annotation design would be something like:

service.beta.kubernetes.io/aws-load-balancer-endpoint-service-acceptance-required: 'true'
service.beta.kubernetes.io/aws-load-balancer-endpoint-service-allowed-principals: '["arn:aws:iam::123456789012:root"]'
service.beta.kubernetes.io/aws-load-balancer-endpoint-service-private-dns-name: 'foobar.example.com'

I don't have a good idea for how to conditionally add the DNS verification record... typically external-dns would handle that but given that both the name and value are computed, it doesn't align as well with the external-dns pattern of reading the name from a spec or annotation and the value from a resource's status or annotation.

A stretch goal would be to support creating Endpoints in a specified set of VPCs but given that (in our case) many of the VPCs would be in different AWS accounts and aws-load-balancer-controller doesn't support using multiple credentials for multiple accounts, I understand if we don't want to incorporate that complexity directly into aws-load-balancer-controller.

Would a PR for supporting Endpoint Services be accepted in the load balancer controller? Or are there any existing plans to add something similar?

fandujar commented 3 years ago

@rifelpet I need something similar. I got some issues when deleting a load balancer associated with a VPC Endpoint Service.

Maybe it would worth adding an annotation to your request to reuse a VPC Endpoint Service: service.beta.kubernetes.io/aws-load-balancer-endpoint-service-id: 'vpce-svc-xxxxxx'

That way you can manage your VPC Endpoint Service without add complexity to aws-load-balancer-controller and still manage this association lifecycle of a load balancer with private link.

M00nF1sh commented 3 years ago

@fandujar @rifelpet

We don't have plans internally for this. PRs are always welcome, however given this is a little complicated, i think we need a placeholder PR finalize the interfaces first before any code implementations, which we can review internally along with community review.

Personally i'd favor the aws-load-balancer-endpoint-service-id options if we can create a endpointService without NLB. (however seems an NLB is required when create a endpointService. I'm not familiar with privateLink so i might be wrong).

For the DNS validation, i don't think we can do much from the controller as it don't have control over the DomainNames. (we could modify it if it's owned by route53 like coreDNS, but seems it's over complicated for this controller's purpose). how about expose the validation records via annotation?

rifelpet commented 3 years ago

You are correct that ec2.CreateVpcEndpointServiceConfiguration requires that either an NLB or GWLB is provided, though you can later disassociate them from the endpoint service, resulting in an endpoint service with no load balancers behind it.

I think an endpoint-service-id annotation would only be useful for the following workflows:

  1. Create Service resource in k8s (load-balancer-controller creates NLB)
  2. Create VPC Endpoint Service externally, providing the new NLB's ARN
  3. Add endpoint-service-id annotation to existing Service resource
  4. When Service resource is deleted, aws-load-balancer-controller first detaches it from the endpoint service

or

  1. Create VPC Endpoint Service externally attaching to some other NLB
  2. Create Service resource in k8s with endpoint-service-id annotation for the new NLB to be attached to the existing endpoint service
  3. When Service resource is deleted, aws-load-balancer-controller first detaches the NLB from the endpoint service

The first use case would likely be made easier if the aws-load-balancer-controller could manage the entire lifecycle of the endpoint service, which is what my original feature request addresses. The second use-case would be useful for endpoint services with many NLBs attached. If aws-load-balancer-controller were to manage the endpoint services for this use-case it could likely use an equivalent of the IngressGroup CRD but for NLBs behind an endpoint service (EndpointServiceGroup?), I'm not planning on adding support for this.

I'm going to plan on moving forward with my proposal for having aws-load-balancer-controller create and manage the endpoint service resources, though much of the work could be reused in supporting an endpoint-service-id annotation. I'll have an interface PR draft open soon for review. I agree that aws-load-balancer-controller shouldn't handle the DNS validation but could expose that information via annotations.

M00nF1sh commented 3 years ago

just thought about this again, i agree with you. personally i think we can expose the created service-id via annotation instead of validation informations. users can have automation tools to read the validation from the service-id

fejta-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

rifelpet commented 3 years ago

/remove-lifecycle stale

@M00nF1sh any update on this or reviewing the interface proposed in the linked PR?

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

rifelpet commented 2 years ago

/remove-lifecycle stale

Cyanopus commented 2 years ago

This is a very important business case. Adding a private link as a single annotation is a great way to expose your service B2B without network-level trust.

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

hintofbasil commented 2 years ago

/remove-lifecycle stale

nijave commented 2 years ago

It'd also be useful if removing load balancers/deleting PrivateLinks on delete is possible. Right now, if you add a PrivateLink and try to delete the load balancer in Kubernetes, it gets stuck deleting the load balancer due to the associated PrivateLink

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

Cyanopus commented 2 years ago

/remove-lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

seh commented 2 years ago

/remove-lifecycle stale

tariq1890 commented 1 year ago

You are correct that ec2.CreateVpcEndpointServiceConfiguration requires that either an NLB or GWLB is provided, though you can later disassociate them from the endpoint service, resulting in an endpoint service with no load balancers behind it.

@rifelpet @M00nF1sh I think it's no longer required to have an NLB to create a VPC endpoint service.

Source: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc_endpoint_service#argument-reference

rifelpet commented 1 year ago

What in that link leads you to believe an NLB or GWLB are not required? the latest AWS CLI still requires it:

$ aws ec2 create-vpc-endpoint-service-configuration

An error occurred (InvalidParameter) when calling the CreateVpcEndpointServiceConfiguration operation: exactly one of network_load_balancer_arn or gateway_load_balancer_arn is a required member
tariq1890 commented 1 year ago

I was going based off the terraform documentation. acceptance_required seems to be the only required field for the aws_vpc_endpoint_service resource. Seems like the TF docs may be misleading

rifelpet commented 1 year ago

Yea it looks like each individual ARN field is optional but the docs don't mention that one of the two ARN fields must be provided

valeriodelsarto commented 1 year ago

Is the PR https://github.com/kubernetes-sigs/aws-load-balancer-controller/pull/2636 that aim to add this feature still WIP? Seems stale since a few months... having this feature will help a lot managing PrivateLink Endpoint Services from NLBs using only K8s resources

eminugurkenar commented 1 year ago

yes , having this is very useful for us too, would be good to have it on next releases.

gosunilgo commented 1 year ago

bumping this request - What tasks are left to release-it / get-over-hump? reminder pr #2636 from @valeriodelsarto

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

kjenney commented 1 year ago

/remove-lifecycle stale

kjenney commented 1 year ago

This issue is blocking me at the moment. I see that the last comment in the PR was 2 days ago. Is there any way to get this PR merged ASAP?

k8s-triage-robot commented 8 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

kjenney commented 8 months ago

/remove-lifecycle stale

k8s-triage-robot commented 4 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

kjenney commented 4 months ago

/remove-lifecycle stale

m17kea commented 2 months ago

Any update on this?