Invalid Ingress Definition With IngressRuleValue

[dmizelle]

Hey all,

I'm trying to make a few functions to let developers have an easier time defining Kubernetes resources rather than using YAML, but I'm running into a very strange issue with regards to writing a function for Ingresses:

def ingress(
    name,
    service=None,
    cloudflare=False,
    ingress_class="private",
    additional_annotations={},
    additional_labels={},
    services=[],  # { "name": "graph", "port": 3000 }
):
    i = extensionsv1beta1.Ingress()
    i.metadata.name = name

    annotations = i.metadata.annotations

    if cloudflare:
        annotations.update(
            {
                "certmanager.k8s.io/acme-challenge-type": "http01",
                "external-dns.alpha.kubernetes.io/cloudflare-proxied": "true",
            }
        )
        domains = ["example.com"]
    else:
        annotations.update(
            {"certmanager.k8s.io/acme-challenge-type": "dns01",}
        )
        domains = ["example.com", "example.co.uk", "example.de", "example.ca"]

    annotations.update(
        {
            "certmanager.k8s.io/cluster-issuer": "letsencrypt",
            "ingress.kubernetes.io/ssl-redirect": "true",
            "kubernetes.io/ingress.class": ingress_class,
            "kubernetes.io/tls-acme": "true",
        }
    )

    rules = i.spec.rules
    for domain in domains:
        rules.append(
            extensionsv1beta1.IngressRule(
                # issue starts here
                ingressRuleValue=extensionsv1beta1.IngressRuleValue(
                    http=extensionsv1beta1.HTTPIngressRuleValue(
                        paths=[
                            extensionsv1beta1.HTTPIngressPath(
                                path="/",
                                backend=extensionsv1beta1.IngressBackend(
                                    serviceName=service.metadata.name,
                                    servicePort=intstr.IntOrString(
                                        intVal=service.spec.ports[0].port
                                    ),
                                ),
                            )
                        ]
                    )
                ),
                host="{}.{}.k8s.{}".format(name, "dev", domain),
            )
        )

    annotations.update(additional_annotations)

    return i

I've marked above where I'm running into the issue.

The above starlark generates yaml like the following:

  - host: nginx.dev.k8s.example.de
    ingressRuleValue:
      http:
        paths:
        - path: /
          backend:
            serviceName: nginx
            servicePort: 8081

As you can see if you are familiar with Ingress objects, ingressRuleValue isnt supposed to be there, and generates the following error if you try and dry-run apply this with kubectl:

error: error validating "STDIN": error validating data: [ValidationError(Ingress.spec.rules[0]): unknown field "ingressRuleValue" in io.k8s.api.extensions.v1beta1.IngressRule, ValidationError(Ingress.spec.rules[1]): unknown field "ingressRuleValue" in io.k8s.api.extensions.v1beta1.IngressRule, ValidationError(Ingress.spec.rules[2]): unknown field "ingressRuleValue" in io.k8s.api.extensions.v1beta1.IngressRule, ValidationError(Ingress.spec.rules[3]): unknown field "ingressRuleValue" in io.k8s.api.extensions.v1beta1.IngressRule];

The path should have a definition of:

  - host: nginx.dev.k8s.example.de
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx
          servicePort: 8081

By taking a look at the documentation of IngressRule, it looks like the field of IngressRuleValue doesn't actually have a name:

https://godoc.org/k8s.io/api/extensions/v1beta1#IngressRule

Is this what could be causing this? Is there a workaround I can use?

[jmillikin-stripe]

Kubernetes has different schemas for inputs in YAML and Protobuf format. Skycfg generates Protobuf, so you'll want to submit the input to the Kubernetes API server as application/vnd.kubernetes.protobuf. The upstream documentation at https://kubernetes.io/docs/reference/using-api/api-concepts/#protobuf-encoding has additional details about the expected encoding.

I'm not sure whether kubectl supports sending requests in Protobuf format -- https://github.com/kubernetes/kubernetes/issues/50403 suggests it doesn't. This may be a blocker if you want to use kubectl inputs as an intermediate format between Skycfg and Kubernetes.

[dmizelle]

Hey John!

Thanks for the super quick response.

I was writing this tool to actually provide YAML to a CD application (ArgoCD) in this case. Mainly trying to abstract YAML away from developers while avoiding the nightmare that is helm templating.

Based on your response, it looks like that taking the protobuf object returned from skycfg.Load() and marshalling it to YAML (as in the skycfg example) isn't enough here to generate kubectl-friendly YAML.

I'll investigate if there is a k8s golang function to do something like this for me (I have no clue) and comment back soon.

On Mon, Jan 6, 2020, 12:05 AM John Millikin notifications@github.com wrote:

Kubernetes has different schemas for inputs in YAML and Protobuf format. Skycfg generates Protobuf, so you'll want to submit the input to the Kubernetes API server as application/vnd.kubernetes.protobuf. The upstream documentation at https://kubernetes.io/docs/reference/using-api/api-concepts/#protobuf-encoding has additional details about the expected encoding.

I'm not sure whether kubectl supports sending requests in Protobuf format -- kubernetes/kubernetes#50403 https://github.com/kubernetes/kubernetes/issues/50403 suggests it doesn't. This may be a blocker if you want to use kubectl inputs as an intermediate format between Skycfg and Kubernetes.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/stripe/skycfg/issues/71?email_source=notifications&email_token=AACIK2SFG7725AE5WA6ODVTQ4K3YRA5CNFSM4KC7PX32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIENQBA#issuecomment-571004932, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACIK2SGMYKVDTDMF77A5S3Q4K3YRANCNFSM4KC7PX3Q .

[jmillikin-stripe]

You may be interested in Isopod (https://github.com/cruise-automation/isopod), which uses Skycfg as a base language and layers on Kubernetes-specific behaviors. It may already have functionality to construct kubectl-compliant YAML, or if not, it would be a good place to add that functionality.

[dmizelle]

:wave:

I was originally using the example from here: https://github.com/stripe/skycfg/blob/b7cd9cad87cd6b5f634138936b72f46c7aa4dd65/_examples/k8s/main.go#L321-L338

Instead, I changed it to look something like this (which is marshalling the proto/struct to JSON, then to YAML) instead of using jsonpb and I ended up getting valid kubectl-friendly YAML!

        for _, msg := range protos {
                group, version, kind, err := gvkFromProto(msg)
                if err != nil {
                        die("unable to parse group/version/kind from protobuf message", err)
                }
                marshaled, err := json.Marshal(msg)
                if err != nil {
                        die("unable to marshal struct to json", err)
                }
                var yamlMap yaml.MapSlice
                err = yaml.Unmarshal(
                        []byte(marshaled),
                        &yamlMap,
                )
                if err != nil {
                        die("unable to convert json to yaml mapslice", err)
                }
                yamlMarshaled, err := yaml.Marshal(yamlMap)
                if err != nil {
                        die("unable to generate yaml document from yaml mapslice", err)
                }
                fmt.Println("---")
                apiVersion := ""
                if group != "" {
                        apiVersion = fmt.Sprintf("%s/%s", group, version)
                } else {
                        apiVersion = version
                }
                fmt.Printf("apiVersion: %s\n", apiVersion)
                fmt.Printf("kind: %s\n", kind)
                fmt.Println(string(yamlMarshaled))
        }

It works for now, so I'll close out this issue. Thanks for a great project.