search

jenkins-x / jx

Jenkins X provides automated CI+CD for Kubernetes with Preview Environments on Pull Requests using Cloud Native pipelines from Tekton
https://jenkins-x.io/
Apache License 2.0
4.57k stars 787 forks source link

setting up AWS wildcard DNS for an ELB? #1256

Closed jstrachan closed 5 years ago

jstrachan commented 6 years ago

We're super close to great EKS support in Jenkins X but we're stuck right now on setting up wildcard DNS on an AWS ELB so that:

myapp.myns.elbid.us-west-2.elb.amazonaws.com can map to the IP address of dualstack.elbid.us-west-2.elb.amazonaws.com so that it can be served up by an Ingress that points to the ELB along these lines:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: 500m
  name: jenkins
  namespace: jx
spec:
  rules:
  - host: jenkins.jx.elbid.us-west-2.elb.amazonaws.com
    http:
      paths:
      - backend:
          serviceName: jenkins
          servicePort: 8080

So far I've tried creating an A or CNAME of * on the ELB host name using Alias Yes - then when that failed, with No and could never nslookup / dig any wildcard DNS names (e.g. looking up foo.elbid.us-west-2.elb.amazonaws.com or foo.bar.elbid.us-west-2.elb.amazonaws.com)

There must be some way to do this - anyone know how?

As a fallback we could switch to using path based ingress; so using host elbid.us-west-2.elb.amazonaws.com and then specifying a path in each Ingress resource - the downside is this then means every microservice deployed in kubernetes typically needs custom configuration to specify the path its deployed within - which adds lots of complexity and generally breaks most apps OOTB without lots of manual tweaks which would make AWS a much worse place to develop on k8s than AKS / GKE

gabeduke commented 6 years ago

Just a note: we've noticed that it can take up to 20 minutes for a new elb to register with route 53.

jstrachan commented 6 years ago

@gabeduke ah thanks!

Have you ever managed to setup a wildcard sub domains on an ELB host via route 53? Just wondering if its even possible really - maybe I just need to wait an hour or two ;). If so which permutation works - CNAME/A record and Alias Yes/No?

gabeduke commented 6 years ago

Yes a wild card subdomain with either alias or cname will work as long as it's not the Apex record. We generally use the dns name that's generated by the elb until the route53 dns starts resolving. Also our TTL is quite short (around 5 minutes) and it still takes a while to register. The other thing to note is this only is an issue on elb creation.

jstrachan commented 6 years ago

@gabeduke many thanks. I wonder what I'm doing wrong. I tried creating a CNAME alias for "*" and hours later it still doesn't resolve to a sub domain:

$ nslookup foo.ab3b4ed5d80e411e8875b06439f0e641-1796460012.us-west-2.elb.amazonaws.com
Server:     8.8.8.8
Address:    8.8.8.8#53

** server can't find foo.ab3b4ed5d80e411e8875b06439f0e641-1796460012.us-west-2.elb.amazonaws.com: NXDOMAIN

$ dig ns foo.ab3b4ed5d80e411e8875b06439f0e641-1796460012.us-west-2.elb.amazonaws.com

; <<>> DiG 9.10.6 <<>> ns foo.ab3b4ed5d80e411e8875b06439f0e641-1796460012.us-west-2.elb.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 63233
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;foo.ab3b4ed5d80e411e8875b06439f0e641-1796460012.us-west-2.elb.amazonaws.com. IN    NS

;; AUTHORITY SECTION:
us-west-2.elb.amazonaws.com. 59 IN  SOA ns-332.awsdns-41.com. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 60

;; Query time: 72 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Jul 11 13:46:37 BST 2018
;; MSG SIZE  rcvd: 182
jstrachan commented 6 years ago

@gabeduke you are using a Hosted zone to setup the route 53 CNAME alias right?

gabeduke commented 6 years ago

@jstrachan I think I know what the issue is. I don't know that it's possible to use the ELB DNS name as an actual route53 record, it would have to be the target. So in your Hosted zone you would point *.[zone name].com at the target ab3b4ed5d80e411e8875b06439f0e641-1796460012.us-west-2.elb.amazonaws.com. Let me know if that helps!

jstrachan commented 6 years ago

not sure what the [zone name] is as right now the zone domain name is the ELB domain name. Or do you mean we'd need to create a new hosted zone for a new DNS name? Basically am trying to do the smallest possible change to get Ingress working on AWS/EKS without requiring folks have a custom DNS name pre-registered with a registrar that they can use for ingress; was hoping to setup some kind of wildcard DNS with the existing ELB domain name.

gabeduke commented 6 years ago

I was misunderstanding what you are trying to do. As far as I know [zone name] would have to be a public DNS zone in order to access externally. Unfortunately, I don't think you would be able to modify records in the AWS provided zone.

fabiodouek commented 6 years ago

Hi @gabeduke , that's right, there is no way to add records for the out of the box ELB public DNS names. I agree with you, these names will need to be registered in Hosted Zone associated to the VPC. Would be great if there was an option to specify also a Route 53 Private Hosted Zone, as in some cases you might want to have a private cluster, with an internal Load Balancer rather than internet-facing.

jstrachan commented 6 years ago

OK another option to look at is can we use an NLB with the nginx-ingress controller https://aws.amazon.com/blogs/opensource/network-load-balancer-support-in-kubernetes-1-9/ then we'd get a static IP address for the NLB then we can use that as the domain and use nip.io for wildcard DNS on AWS/EKS

patrickleet commented 6 years ago

@jstrachan I saw this approach when I was trying to research yesterday as well. Saw this article but haven't gone through it yet, but maybe it's helpful: https://medium.com/ww-engineering/k8s-ws-aws-elb-nlb-5a23834d4f14

jstrachan commented 6 years ago

tried an NLB then used one of its IP addresses; seemed to work first time on kops but never worked on EKS. So its probably time to chuck in the towel and go back to using the ELB host name and relative paths on the Ingress instead

jstrachan commented 6 years ago

BTW the code should work right now for setting up the Route53 CNAME wildcard load balancer

thienlh commented 6 years ago

Do we have any update on this issue? Is there a way to config a DNS instead of using *.nip.io? This is a deal breaker for us really 😢

jstrachan commented 6 years ago

there's some docs here which we're hoping to publish on the AWS blog soon...

https://docs.google.com/document/d/1EgqGU3nrMTigPvfohNH7o0rHHZbGuoQbQ__KqhmQsUc/edit?usp=sharing

stormmore commented 5 years ago

I know I have mentioned this, I still believe that is wrong to have to give Jenkins X a full wildcard domain. Why isn't the choice to use something like external-dns to actually configure the required addresses inside the cloud provider's DNS (e.g. Route53, etc.). It is has a lot of the cloud provider dns options covered including Google, Azure and interestingly enough even CoreDNS.

jstrachan commented 5 years ago

external-dns sounds great to me - maybe we can create an addon for it?

stormmore commented 5 years ago

That would be awesome if it could do away with the need for providing a wildcard domain, it would be even better.

nia2k7 commented 5 years ago

Can we have multiple Jenkins X environments in the one hosted zone?

e.g. two CNAME record sets 1: *.jenkins1.jenkins.jx.konsek.cloud > route to ELB 1

  1. *.jenkins2.jenkins.jx.konsek.cloud > route to ELB 2
jenkins-x-bot commented 5 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with /close. Provide feedback via https://jenkins-x.io/community. /lifecycle stale

jenkins-x-bot commented 5 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with /close. Provide feedback via https://jenkins-x.io/community. /lifecycle rotten

jenkins-x-bot commented 5 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten. Provide feedback via https://jenkins-x.io/community. /close

jenkins-x-bot commented 5 years ago

@jenkins-x-bot: Closing this issue.

In response to [this](https://github.com/jenkins-x/jx/issues/1256#issuecomment-517964210): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. >Provide feedback via https://jenkins-x.io/community. >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.