search

loft-sh / vcluster

vCluster - Create fully functional virtual Kubernetes clusters - Each vcluster runs inside a namespace of the underlying k8s cluster. It's cheaper than creating separate full-blown clusters and it offers better multi-tenancy and isolation than regular namespaces.
https://www.vcluster.com
Apache License 2.0
6.38k stars 410 forks source link

Generated Ingress annotation too long #1385

Open frank-bee opened 11 months ago

frank-bee commented 11 months ago

What happened?

This error is thrown when creating an Ingress annotation.

Error syncing to physical cluster: Ingress.extensions "seldon-ingress-api-tenant-a-x-istio-system-x-tenant-a-tenant-a" is invalid: metadata.annotations: Invalid value: "alb.ingress.kubernetes.io/conditions.istio-ingressgateway-x-istio-system-x-tenant-a-tenant-a": name part must be no more than 63 characters

The annotation is as follows

alb.ingress.kubernetes.io/conditions.istio-ingressgateway: >
      [{"field":"http-header","httpHeaderConfig":{"httpHeaderName": "KeyId",
      "values":["..."]}},
      {"field":"http-request-method","httpRequestMethodConfig":{"Values":["GET",
      "POST"]}}]

What did you expect to happen?

The Ingress should be created in the host cluster incl. this annotation

How can we reproduce it (as minimally and precisely as possible)?

Create an ingress with an annotation key longer than allowed

Anything else we need to know?

See also https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set

Host cluster Kubernetes version

```console $ kubectl version Client Version: v1.28.4 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.25.15-eks-4f4795d WARNING: version difference between client (1.28) and server (1.25) exceeds the supported minor version skew of +/-1 ```

Host cluster Kubernetes distribution

``` AWS EKS ```

vlcuster version

```console $ vcluster --version vcluster version 0.17.1 ```

Vcluster Kubernetes distribution(k3s(default)), k8s, k0s)

default

OS and Arch

locally I work with a MAc

FabianKramm commented 10 months ago

@frank-bee thanks a lot for creating this issue! I guess what we can do is to add logic to skip annotations that would exceed the limit so that the ingress is able to sync, but the problem that the condition doesn't apply to the service would remain, so I'm unsure how we would fix this as there seems to be a general Kubernetes limitation on this / bad design by AWS ingress-controller that has problems with long service names

frank-bee commented 10 months ago

Fully agree to what you say, @FabianKramm . Better do not skip these annotations, because if they are there, probably they are not "optionally" (like in our case). I think we will try to reduce the characters in these 2 substrings (service , release):

alb.ingress.kubernetes.io/conditions.service-name-x-istio-system-x-vcluster-release-name

FabianKramm commented 10 months ago

@frank-bee yeah that sounds like a short term workaround at least until we find an actual solution