search

cloudfoundry / cf-for-k8s

The open source deployment manifest for Cloud Foundry on Kubernetes
Apache License 2.0
300 stars 115 forks source link

Cannot use top level domain as app route #558

Open jimmykarily opened 3 years ago

jimmykarily commented 3 years ago

Describe the bug

When defining app_domains in values, there is no way to add the root (top level) domain.

E.g.

One can specify this in values:

app_domain:
  - example.com

but this results in an Istio Gateway that looks like this:

spec:
  servers:
  - hosts:
    - cf-workloads/*.example.com

(relevant code)

Doing this:

cf map-route myapp example.com

won't work because the domain is mapped to the app but Istio gateway is not setup correctly for the top level domain.

Workaround

Manually editing the Istio Gateway and adding the top level domain makes it work:

spec:
  servers:
  - hosts:
    - cf-workloads/example.com

To Reproduce*

Steps to reproduce the behavior:

  1. Define a top level domain in app_domains in your values
  2. Deploy cf-for-k8s
  3. map the top level domain to an app
  4. Try to reach the app with that domain

It's helpful to include snippets of the error response or logs output

Expected behavior

A clear and concise description of what you expected to happen.

Reaching the app with the top level domain should work but it doesn't

Additional context

cf-for-k8s SHA

Please paste cf-for-k8s SHA hyperlink v1.0.0

Deploy instructions

Please include the kapp deploy... command, including all config-optional files As per the README

Cluster information

Please provide IaaS name(s) here (GKE, AKS, EKS, minikube, kind, etc.) GKE

cc @cloudfoundry/eirini

cf-gitbot commented 3 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/175556559

The labels on this github issue will be updated when the story is started.

acosta11 commented 3 years ago

Hi @jimmykarily ,

It looks like this functionality is supported through private domains.

I was able to reproduce the functionality by:

  1. Creating a DNS A record for the app domain E.G. top-level-domain.my-env.k8s-dev.relint.rocks
  2. Do not include the domain as an apps_domain passed to cloud controller
  3. Deploy normally and log in
  4. cf create-org my-org && cf create-space -o my-org my-space && cf target -o my-org -s my-space
  5. cf push my-app -p tests/smoke/assets/test-node-app
  6. cf create-private-domain my-org top-level-domain.my-env.k8s-dev.relint.rocks
  7. cf map-route my-app top-level-domain.my-env.k8s-dev.relint.rocks

Closing this issue, but feel free to reopen if it seems like this warrants additional documentation directly on the cf-for-k8s repo.

Thanks, Andrew and @Birdrock

gcapizzi commented 3 years ago

Hey @acosta11! It looks like this is exactly what we are doing, just using CLI v6:

cf create-org eirinidotcf
cf create-space -o eirinidotcf eirinidotcf
cf target -o eirinidotcf -s eirinidotcf

if ! cf domains | grep "^eirini.cf" >/dev/null 2>&1; then
  cf create-domain eirinidotcf eirini.cf
fi
cf map-route eirinidotcf-web eirini.cf

We don't see any changes to the Gateway though. We do see a VirtualService for eirini.cf but that's it. Is this a different bug, or are we missing something?

Thanks!

/cc @mnitchev

acosta11 commented 3 years ago

Hi @gcapizzi and @jimmykarily ,

Oh, I was using cli v7. We've seen a few instances already of cli v6 diverging from v7 on cf-for-k8s functionality so I wouldn't be surprised if that's what we're observing here. But is the cf create-domain command creating a shared domain for you by chance?

For instance when I deployed my cf-for-k8s with the desired top level domain in values like you described in the original issue body, it showed up as a shared domain, which cannot be used directly without specifying a host subdomain. I instead had to make sure the domain was private so that it was possible to not specify the host on map route. Did cli v6 not throw the same error I was seeing below? (My actual examples here are named a slightly differently than my example above)

$ cf map-route test top-level-domain.adventurous-spacesuit.k8s-dev.relint.rocks
Creating route top-level-domain.adventurous-spacesuit.k8s-dev.relint.rocks for org o / space s as admin...
Missing host. Routes in shared domains must have a host defined.

$ cf domains
Getting domains in org o as admin...
name                                                          availability   internal   protocols
adventurous-spacesuit.k8s-dev.relint.rocks                    shared                    http
apps.adventurous-spacesuit.k8s-dev.relint.rocks               shared                    http
top-level-domain.adventurous-spacesuit.k8s-dev.relint.rocks   shared                    http

So I thought another requirement in this flow was to use cf create-private-domain instead of cf create-domain. And when map route succeeds I see that lack of host show up in the output of cf routes as well.

$ cf routes
Getting routes for org o / space s as admin...
space   host   domain                                                        port   path   protocol   apps
s       test   apps.adventurous-spacesuit.k8s-dev.relint.rocks                             http       test
s              top-level-domain.adventurous-spacesuit.k8s-dev.relint.rocks                 http       test

But if this a case of cli v6 limitations, unfortunately the current support stance from capi and the cli folks for api v2 and cli v6 in general is to upgrade to v3 and v7. We've mirrored that stance in the debugging docs: https://github.com/cloudfoundry/cf-for-k8s/blob/f461c53843738c429d5518ca5447eeb30adebed9/docs/debugging.md#1-you-may-be-using-the-v6-cf-cli-which-is-not-supported.

We've reached out to the capi and cli folks to get more transparent messaging around the version support as a next step.

Thanks, Andrew

cf-gitbot commented 3 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/175580730

The labels on this github issue will be updated when the story is started.

acosta11 commented 3 years ago

Hi @gcapizzi and @jimmykarily ,

Circling back here, did y'all happen to resolve this issue with the approach detailed above? I wanted to make sure this time before closing out the issue.

Thanks, Andrew

jimmykarily commented 3 years ago

Hi @acosta11 , we used cf cli 7.1.0+4c3168f9a.2020-09-09 and deployed without specifying the top level domain in app_domains. Then we created a private domain with cf create-private-domain and we mapped that to an app with cf map-route. cf routes looks like the output from your command but curling the application doesn't work. Our investigation shows that it's Istio's gateway that is not configured properly.

Our workaround is this: https://github.com/cloudfoundry-incubator/eirini-ci/blob/master/cert-manager/custom-app-domain.yml

because the cf-for-k8s code doesn't allow us to add a top level domains (here: https://github.com/cloudfoundry/cf-for-k8s/blob/1c2307c6adcd08e4e278ad9f7f55e05bea0c8d9a/config/istio/gateway.lib.yml#L32-L33)

There should be a way to achieve this without the workaround.

Birdrock commented 3 years ago

@jimmykarily Thanks for sharing the details. We'll work with our PM to prioritize the issue.

jamespollard8 commented 3 years ago

We recognize this as a "known issue" and are not yet able to prioritize this debugging/fix work