FrontEndConfig does not create HTTPS redirect Load Balancers, only one HTTP LB

[Gageperrin]

Posted this originally here, per @boredabdel 's request I am opening as an issue here.

Expected behavior: Create Load Balancing with HTTPS redirect without manual changes using manifest file annotations.

Actual behavior: Only a HTTP front-end Load Balancer is created.

---

This is a relatively straightforward configuration. One cluster, one node, one Ingress with a FQDN.

The FrontEnd Config annotation in the Ingress YAML creates what appears in the console menu as a HTTP(S) load balancer but only has a single HTTP front-end with no certificate or SSL policy attached, despite these being annotated in the Ingress YAML. the SSL cert is stuck in provisioning as FAILED_NOT_VISIBLE because it has no Target Proxy to attach to.

If I understand correctly from the docs here, two load balancers should be provisioned, one HTTP and one HTTPS. The HTTP should redirect to the HTTPS. Instead, one HTTP load balancer is created with the original Ingress backends.

The HTTP Load Balancer reports the original HTTP backends as healthy while the Ingress reports its backend services as "Unknown".

Is there something wrong with syntax or spec that is preventing the HTTPS Load Balancer from generating correctly?

The HTTP LB worked fine before the frontend config was added, but I want to force HTTPS.

Note my deployment was written through Terraform, so there may be typos in my "translation" back into YAML.

Screenshots will be attached below.

FrontEnd Config YAML:

apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
  name: api-ingress-frontend-config
  namespace: default
spec:
  sslPolicy: ssl-policy <--- externally created
  redirectToHttps:
    enabled: true

BackEnd Config YAML:

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: api-ingress-backend-config
  namespace: default
spec:
  healthCheck:
    checkIntervalSec: 90
    port: 8000
    type: HTTP
    requestPath: "/health/"

Ingress YAML: (The Terraform resource has wait_for_load_balancer set to true.)

apiVersion: v1
kind: Ingress
metadata:
  annotations:
    "kubernetes.io/ingress.class": "gce"
 # "kubernetes.io/ingress.allow-http": "false" <--- commented out because this creates a Load Balancer sync error.
    "kubernetes.io/ingress.global-static-ip-name": [pre-defined static IP]
    "networking.gke.io/frontend-config": "api-ingress-frontend-config"
    "networking.gke.io/managed-certificates": [pre-defined SSL cert for ingress subdomain]
spec:
 rule:
   host: api.example.com
   http:
     path:
       backend:
         serviceName: api-service
         servicePort: 8000

Service YAML:

apiVersion: v1
kind: Service
metadata:
  name: api-service
  labels:
    app: backend
  annotations:
    "cloud.google.com/backend-config": {"default": "api-ingress-backend-config"}
 # "cloud.google.com/neg": {ingress: true}
spec:
  selector:
    app: backend
  sessionAffinity: None
  ports:
    port: 8000
    targetPort: 8000
  type: NodePort

kubectl describe crd frontendconfigs.networking.gke.io returns the following:

Name:         frontendconfigs.networking.gke.io
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  apiextensions.k8s.io/v1
Kind:         CustomResourceDefinition
Metadata:
  Creation Timestamp:  2021-10-29T12:39:10Z
  Generation:          1
  Managed Fields:
    API Version:  apiextensions.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:conversion:
          .:
          f:strategy:
        f:group:
        f:names:
          f:kind:
          f:listKind:
          f:plural:
          f:singular:
        f:scope:
        f:versions:
      f:status:
        f:acceptedNames:
          f:kind:
          f:plural:
        f:conditions:
        f:storedVersions:
    Manager:         glbc
    Operation:       Update
    Time:            2021-10-29T12:48:18Z
  Resource Version:  2865
  UID:               ae694fde-68cf-4cea-ad7f-6c2f1490f8ae
Spec:
  Conversion:
    Strategy:  None
  Group:       networking.gke.io
  Names:
    Kind:       FrontendConfig
    List Kind:  FrontendConfigList
    Plural:     frontendconfigs
    Singular:   frontendconfig
  Scope:        Namespaced
  Versions:
    Name:  v1beta1
    Schema:
      openAPIV3Schema:
        Properties:
          API Version:
            Description:  APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
            Type:         string
          Kind:
            Description:  Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
            Type:         string
          Metadata:
            Type:  object
          Spec:
            Description:  FrontendConfigSpec is the spec for a FrontendConfig resource
            Properties:
              Redirect To Https:
                Description:  HttpsRedirectConfig representing the configuration of Https redirects
                Properties:
                  Enabled:
                    Type:  boolean
                  Response Code Name:
                    Description:  String representing the HTTP response code Options are MOVED_PERMANENTLY_DEFAULT, FOUND, TEMPORARY_REDIRECT, or PERMANENT_REDIRECT
                    Type:         string
                Required:
                  enabled
                Type:  object
              Ssl Policy:
                Type:  string
            Type:      object
          Status:
            Type:  object
        Type:      object
    Served:        true
    Storage:       true
Status:
  Accepted Names:
    Kind:       FrontendConfig
    List Kind:  FrontendConfigList
    Plural:     frontendconfigs
    Singular:   frontendconfig
  Conditions:
    Last Transition Time:  2021-10-29T12:39:10Z
    Message:               no conflicts found
    Reason:                NoConflicts
    Status:                True
    Type:                  NamesAccepted
    Last Transition Time:  2021-10-29T12:39:10Z
    Message:               the initial names have been accepted
    Reason:                InitialNamesAccepted
    Status:                True
    Type:                  Established
  Stored Versions:
    v1beta1
Events:  <none>

[Gageperrin]

Ingress when it first starts creating:

Ingress after a few minutes:

Target Proxy is provisioned as only HTTP:

[Gageperrin]

Load Balancer final state:

[boredabdel]

You are missing the ManagedCertificate object. Did you created it in the same namespace ?

https://cloud.google.com/kubernetes-engine/docs/how-to/managed-certs#setting_up_a_google-managed_certificate

Depending on your GKE cluster version. The ManagedCertificate CRD have changed. Have a look here https://cloud.google.com/kubernetes-engine/docs/how-to/managed-certs#api_versions

[Gageperrin]

Yes, I provisioned a Google-managed SSL certificate via Terraform's google_compute_managed_ssl_certificate resource.

I attempted it with the annotation ingress.kubernetes.io/pre-shared-cert but this did not work so I switched to trying with networking.gke.io/managed-certificates.

Your comment made me check out Terraform's provider for Google SSL certs and I found this.

It looks like I had the annotation syntax wrong. It's supposed to be ingress.gcp.kubernetes.io/pre-shared-cert not ingress.kubernetes.io/pre-shared-cert.

WIth this change, a HTTPS Load Balancer is provisioning. I am waiting on the SSL certificate to verify and will close the issue if all goes well.

Is there a open feature request for documentation for all GKE annotations (particularly networking)? This is deeply needed because the annotation syntax varies a lot is hard to locate.

Thank you for your help though, I will close if everything provisions smoothly.

[Gageperrin]

@boredabdel I spoke too soon. It looks like the FrontEndConfig is generating both HTTPS and HTTP frontends alongside each other in the same Load Balancer. No other LB is being created.

It should be creating two LBs: a HTTP frontend LB that redirects to a HTTPS only frontend LB with the same IP, right? Right now it is accepting both HTTP and HTTPS traffic instead of redirecting.

Console screenshot:

Here is the kubectl output for describing frontend config:

Name:         api-ingress-frontend-config
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  networking.gke.io/v1beta1
Kind:         FrontendConfig
Metadata:
  Creation Timestamp:  2021-10-29T16:37:37Z
  Generation:          2
  Managed Fields:
    API Version:  networking.gke.io/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:redirectToHttps:
          f:enabled:
        f:sslPolicy:
    Manager:         Terraform
    Operation:       Apply
    Time:            2021-11-01T16:09:59Z
  Resource Version:  1386500
  UID:               cdb54ace-59a9-4cf3-9ff4-4ec3331eef6b
Spec:
  Redirect To Https:
    Enabled:   true
  Ssl Policy:  GCP default
Events:        <none>

[boredabdel]

Hey.

So yeah the TF google_compute_managed_ssl_certificate module creates Google Managed Certs which you can use in Ingress via the ingress.gcp.kubernetes.io/pre-shared-cert annotation

GKE managed Certificates are GKE provisioned certs (via YAML files) and you can use them with the networking.gke.io/managed-certificates Annotation.

So they are different.

To your last point that's expected. There are two scenarios

You either use FrontendConfig with http to https redirect enabled. In which case your LoadBalancer will have two rules for HTTP and HTTP. Calls to the http URL will be redirected.

Or you can NOT enable the redirect and use the "kubernetes.io/ingress.allow-http": true to disable HTTP all together

You cannot do both (allow-http set to true and http to https redirect enabled).

So what you see is working as expected. From the previous updated you seem to have set the allow-http to false and your frontendconfig has http to https set to true.

Hope this makes sense

[Gageperrin]

Hi, yeah so I have the allow-http false annotation commented out because it had conflicts with the redirect. It was not enabled in the above. When I do include the annotation allow-http = false then it only creates one HTTPS LB without HTTP redirect but has unhealthy backends.

From what I understand, allow HTTP is by default set to true when not explicitly mentioned in annotations. In my scenario I am just looking to create a forced HTTPS ingress with or without redirect.

So when I try to set up the redirect with allow-http set to default, then I have the above error.

The alternative when I set allow-http to false creates unhealthy backends as seen below.

Ideally we can fix the problem I have in my previous comment so that there is a HTTP -> HTTPS redirect, but I am open to a HTTPS only configuration if that is not possible. Currently neither is functioning for me.

[boredabdel]

It seems like you are editing your YAML file and applying on an existing Ingress. From the last screenshot it seems like you have the allow-http set to false.

Can you delete the Ingress Object from the cluster. wait few minutes and apply it again ?

[boredabdel]

Also your Kubernetes Service should have the "cloud.google.com/neg": {ingress: true} set in order to use NEG (network endpoint groups). It looks like it's commented out in the yaml files you posted.

[Gageperrin]

The allow-http annotation was set to false in the previous screenshot to demonstrate it does not work, the post above that had allow-http set to true.

"cloud.google.com/neg": {ingress: true} is the default for my cluster version, but I will uncomment it again.

These do not resolve the issue that the Load Balancer generated is both HTTP and HTTPS instead of two LBs that redirect HTTP to HTTPS.

Here is a fresh configuration. I deleted the Ingress, Service, and Load Balancer and re-created them. allow-http is by default true here and the "cloud.google.com/neg": {ingress: true} is included.

The HTTP forwarding rule, target proxy, and URL map should be directed toward the HTTPS load balancing resources instead of the backend, but it is not doing so right now.

[boredabdel]

I don't understand what you mean by "the Load Balancer generated is both HTTP and HTTPS instead of two LBs that redirect HTTP to HTTPS." ?

That's what should happen when you have http to https redirect enabled via FrontendConfig. You will have two LoadBalancers one HTTP and one HTTPS. Your screenshot seems good to me with how you described it.

Last thing to check is your firewall rules. You need to make sure you have a firewall rule that allows HealthCheck on the pods. These are created automatically if the GKE cluster and VPC are in the same project and if the IAM permissions for the robot Service Account are set properly.If not you will have to create them manually. Check this doc https://cloud.google.com/kubernetes-engine/docs/concepts/firewall-rules#ingress-fws

and look in your firewall rule config. If they are not there. create them with the IP ranges from the link above

[Gageperrin]

Okay, I thought it would show up as two separate Load Balancers like in the docs here rather than one combined one. The problem is that the verification command is not returning expected results, so it does not seem like the HTTP -> HTTPS redirect is working. It is passing HTTP to the backend and HTTPS to the backend.

In the docs I linked ^, it says you verify the redirect through curl http://IP_ADDRESS which should return a 301 code. Instead I receive response 404 (backend NotFound), service rules for the path non-existent. This happens also with

• curl IP_ADDRESS
• curl http://IP_ADDRESS:80

but curl IP_ADDRESS:443 hits the backend as expected.

This seems to indicate the HTTP load balancer is not redirecting to HTTPS because it is sending calls directly to the backend. Am I interpreting this incorrectly? Is there another way to verify if the redirect is set up? The monitoring also shows calls going directly to the backend.

The HealthChecks on the pods all return successful after a few minutes, so this does not seem related to how the Load Balancer sets up the front-end redirect. The Firewall Rules seem good and the Health Checks are good.

[boredabdel]

If you want the redirect to work with curl you need two things.

Set the responseCodeName to FOUND in the FrontendConfig object. Otherwise the default is MOVED_PERMANENTLY_DEFAULT which will only return a 301. This will force the LoadBalancer to respond with the https link for the client to follow

add -L to curl. Like curl -L http://IP_ADDRESS. This flag tells curl to follow the returned link

Give it a try. You might need to re-create the Ingress but try without first

[Gageperrin]

Okay, I tried curl -L http://IP_ADDRESS with FrontendConfig ResponseCode set to MOVED_PERMANENTLY_DEFAULT and then deleted the Ingress and reapplied the manifests and FrontendConfig ResponseCode set to FOUND. Both return the response 404 (backend NotFound), service rules for the path non-existent. I also tried the docs' instructions here, but it is not returning the expected message:

In my browser when I access the Load Balancer subdomain with http://api.[domain].com it allows me to access the backend instead of refreshing the page with https://api.[domain].com like how http://www.google.com sends the user to https://www.google.com . Is this expected? Just need to be 100% sure this ingress is secure and does not pass HTTP to the backend.

[boredabdel]

Please have a look at this recipe https://github.com/GoogleCloudPlatform/gke-networking-recipes/tree/master/ingress/multi-cluster/mci-frontend-config

It's if for MultiClusterIngress but should work for Ingress (replace MCS with a Service and MCI with Ingress)

I have no idea what's the issue you are facing and it's hard to help you figure things out over github.

[Gageperrin]

Yeah, I copied your recipe to make the front-end CRD and switched values to make it work for single cluster.

I'll leave this issue open in case someone else discovers the problem. Thank you for your responsiveness and trying to help me sort this out.

To summarize for anyone who finds this, I tried to create a single cluster Ingress from this FrontEnd Config recipe which should create a Load Balancing Configuration like the one documented here.

Instead of creating Load Balancing with HTTP to HTTPS redirect, the FrontEnd Config CRD creates a Load Balancer that accepts both HTTP and HTTPS traffic. We attempted toggling various GKE annotations, but without success. What adds to the difficulty, is that GKE annotations are not properly documented at the time of this issue, hopefully this will be resolved in the future.

[boredabdel]

Hi @Gageperrin

I just realised we have this recipe which covers this use case already https://github.com/GoogleCloudPlatform/gke-networking-recipes/tree/master/ingress/single-cluster/ingress-https

Could you give it a try and let me know. I'm happy to work with you on improving the recipe to make sure it works

[ghost]

@Gageperrin , did you ever solve the issue? I am currently running into this exact problem.

[bowei]

/assign @spencerhance

[spencerhance]

Hi Folks, has this been resolved? Just from scanning through the thread, I would recommend verifying that the FrontendConfig is getting attached to your Ingress.

GKE Ingress: networking.gke.io/v1beta1.FrontendConfig: "FRONTENDCONFIG_NAME"

MultiClusterIngress: networking.gke.io/frontend-config: "FRONTENDCONFIG_NAME"

Annotations documented here: https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#associating_frontendconfig_with_your_ingress

[boredabdel]

@Gageperrin did you manage to solve this ?

[boredabdel]

Closing for inactivity