kubernetes-sigs / cloud-provider-equinix-metal

Kubernetes Cloud Provider for Equinix Metal (formerly Packet Cloud Controller Manager)
https://deploy.equinix.com/labs/cloud-provider-equinix-metal
Apache License 2.0
74 stars 27 forks source link

CCM can easily break BGP peering #457

Open TimJones opened 1 year ago

TimJones commented 1 year ago

We recently ran into an issue in one of our clusters in that we had more than 10 LoadBalancer Services which tried to peer via MetalLB BGP. Due to a default limit of 10 prefixes, this caused all subsequent BGP peering to fail until we had Equinix support reset the BGP session after removing the excess Services (switched to using shared IPs).

Since this is a limit in Equinix side, I think it would be of value for the Equinix CCM to also limit number of IPs/prefixes that can be requested/ordered. Ideally an arg that can be set if the customer has arranged a higher session limit.

See https://deploy.equinix.com/developers/docs/metal/bgp/bgp-on-equinix-metal/#bgp-prefix-limit

k8s-triage-robot commented 9 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 8 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 7 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 7 months ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes-sigs/cloud-provider-equinix-metal/issues/457#issuecomment-2024499467): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
cprivitere commented 6 months ago

/reopen

cprivitere commented 6 months ago

/remove-lifecycle rotten

k8s-ci-robot commented 6 months ago

@cprivitere: Reopened this issue.

In response to [this](https://github.com/kubernetes-sigs/cloud-provider-equinix-metal/issues/457#issuecomment-2110597830): >/reopen Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
cprivitere commented 6 months ago

/lifecycle frozen

cprivitere commented 6 months ago

/triage accepted

cprivitere commented 6 months ago

/remove-lifecycle frozen

cprivitere commented 6 months ago

@TimJones Hey did you get an error message from the API when this occurred? Was CPEM continuing to do bad things that caused it to get quickly back into the error state?

Also, does CPEM cause all the BGP to fail by asking for 11 prefixes? I'm trying to understand the value of CPEM throwing the error instead of just passing along the API error.

If we add that setting, we're now making a customer change TWO configs (one via support and one in CPEM) so I really want to understand the value before we do that.

TimJones commented 6 months ago

Apologies but I don't have any of the logs from CPEM for incident any more, only from the MetalLB side, which was:

{"caller":"level.go:63","error":"read OPEN from \"169.254.255.1:179\": got BGP notification code 0x0601 (Maximum Number of Prefixes Reached)","level":"error","localASN":65000,"msg":"failed to connect to peer","op":"connect","peer":"169.254.255.1:179","peerASN":65530,"ts":"2023-09-08T14:10:04.171904229Z"}