[BUG] reddog.microsoft.com leads to unnecessary DNS resolution request to external DNS servers

[akorp]

Describe the bug All host in Azure clusters are getting search reddog.microsoft.com in their resolve.conf. Because of this all pods are also getting reddog.microsoft.com in their resolve.conf with a default coredns setup, for example:

## pod
>cat /etc/resolv.conf
search debug.svc.cluster.local svc.cluster.local cluster.local reddog.microsoft.com
nameserver 10.2.0.10
options ndots:5

also on the host with coredns we have search reddog.miscrosoft.com

## host
> cat /etc/resolv.conf 
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "systemd-resolve --status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0
search reddog.microsoft.com 

According to Microsoft documentation reddog.microsoft.com is a non-functional placeholder, which does not have any DNS records (https://learn.microsoft.com/en-us/azure/virtual-network/virtual-networks-name-resolution-for-vms-and-role-instances#vms-and-role-instances). However having this placeholder in pods' search force all non-cluster DNS requests with fewer than ndots:5 to be first looked with reddog.microsoft.com suffix (for example management.azure.com.reddog.microsoft.com., api.eu0.signalfx.com.reddog.microsoft.com.). This leads to a lot of necessary requests (non-cached) and traffic to our DNS servers. This also leads to longer DNS resolutions, since an extra unnecessary DNS resolution requests are made with suffix .reddog.microsoft.com. before proper DNS requests are made.

To Reproduce Intall AKS cluster with custom DNS servers on vnet with a default coredns.

Expected behavior No reddog.microsoft.com request should be made to external DNS servers.

Environment (please complete the following information):

• "azure-cli": "2.40.0"
• Kubernetes version v1.24.6

[ghost]

Action required from @Azure/aks-pm

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[matthiasritter]

Hi, we have the same problem. This has even caused our DNS server to stop responding, due to too many (unnecessary) DNS requests.

[srmars]

Having same issue

[ghost]

@aanandr, @phealy would you be able to assist?

Issue Details

---

**Describe the bug** All host in Azure clusters are getting `search reddog.microsoft.com` in their `resolve.conf`. Because of this all pods are also getting `reddog.microsoft.com` in their `resolve.conf` with a default `coredns` setup, for example: ``` ## pod >cat /etc/resolv.conf search debug.svc.cluster.local svc.cluster.local cluster.local reddog.microsoft.com nameserver 10.2.0.10 options ndots:5 ``` also on the host with coredns we have `search reddog.miscrosoft.com` ``` ## host > cat /etc/resolv.conf # This file is managed by man:systemd-resolved(8). Do not edit. # # This is a dynamic resolv.conf file for connecting local clients to the # internal DNS stub resolver of systemd-resolved. This file lists all # configured search domains. # # Run "systemd-resolve --status" to see details about the uplink DNS servers # currently in use. # # Third party programs must not access this file directly, but only through the # symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way, # replace this symlink by a static file or a different symlink. # # See man:systemd-resolved.service(8) for details about the supported modes of # operation for /etc/resolv.conf. nameserver 127.0.0.53 options edns0 search reddog.microsoft.com ``` According to Microsoft documentation `reddog.microsoft.com` is a non-functional placeholder, which does not have any DNS records (https://learn.microsoft.com/en-us/azure/virtual-network/virtual-networks-name-resolution-for-vms-and-role-instances#vms-and-role-instances). However having this placeholder in pods' `search` force all non-cluster DNS requests with fewer than `ndots:5` to be first looked with `reddog.microsoft.com` suffix (for example `management.azure.com.reddog.microsoft.com.`, `api.eu0.signalfx.com.reddog.microsoft.com.`). This leads to a lot of necessary requests (non-cached) and traffic to our DNS servers. This also leads to longer DNS resolutions, since an extra unnecessary DNS resolution requests are made with suffix `.reddog.microsoft.com.` before proper DNS requests are made. **To Reproduce** Intall AKS cluster with custom DNS servers on vnet with a default `coredns`. **Expected behavior** No `reddog.microsoft.com` request should be made to external DNS servers. **Environment (please complete the following information):** - "azure-cli": "2.40.0" - Kubernetes version v1.24.6

Author:	akorp
Assignees:	-
Labels:	`bug`, `networking/azcni`, `action-required`, `Needs Attention :wave:`
Milestone:	-

[ghost]

Action required from @Azure/aks-pm

[paulgmiller]

Initial take more to come.

So AKS is inheriting reddog.microsoft.com from the vm networking team. We're still trying to find why they don;'t use a blank serhc suffix. If there are reasons abut they don't matter for pods we might have coredns drop reddog.microsoft.com queries so they don't get forwarded to your upstram dns. Tryng to figure out if a fallthough or except config in coredns-custom would help people till we know more.

[aarongh2012]

We're seeing millions of failed DNS requests because of this issue. Is there any update?

[nnellanspdl]

According to this page we can customize the CoreDNS config used by AKS. We just have to create and apply a custom ConfigMap. And we have all of the built-in CoreDNS plugins at our disposal (which includes the ACL plugin).

So, would a ConfigMap like this be possible? (I haven't tested this yet)

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  reddog.server: |
    reddog.microsoft.com:53 {
      acl {
        drop
      }
    }

[smolit]

We have the same issue. @paulgmiller what about the vm networking team. Did they react to this issue? Is there a solution published in the meantime. I think a custom coredns configuration is a bit overengineered.

Thanks Sebastian

[msamad]

According to this page we can customize the CoreDNS config used by AKS. We just have to create and apply a custom ConfigMap. And we have all of the built-in CoreDNS plugins at our disposal (which includes the ACL plugin).

So, would a ConfigMap like this be possible? (I haven't tested this yet)

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  reddog.server: |
    reddog.microsoft.com:53 {
      acl {
        drop
      }
    }

Has anyone actually tried this in AKS? drop is introduced in coredns-1.10.1 https://coredns.io/2023/01/20/coredns-1.10.1-release/

As of today, the latest K8s version available for AKS GA is 1.27, which has Core DNS V1.9.4 https://learn.microsoft.com/en-us/azure/aks/supported-kubernetes-versions?tabs=azure-cli

So AKS doesn't have drop feature in core-dns at the moment.

Also, is drop a better option or block a better option? From the docs, it looks like block will return a REFUSE response code, which should allow DNS resolver to move onto the next search domain. What would drop return? and would the DNS resolver move on as usual?

[msamad]

Tested drop on a local k8s with coredns-1.10.1 and it results in timeout. The commit in https://github.com/coredns/coredns/pull/5722 is all about drop being added for DoS attacks. It doesn't write any response back causing the client to timeout.

block on the other end, returns a REFUSED error which ~seem the right one at this point~ does not let the dns resolver move onto the next in the search domain list or upstream. So in the end, host is not resolved.

Another option is to rewrite it to avoid any reddog.microsoft.com DNS queries from going upstream

rewrite stop {
  name regex (.*)\.reddog.microsoft.com\.$ {1}
  answer name (.*)\.$ {1}.reddog.microsoft.com

  .....
  forward . /etc/resolv.conf
  .....
  .....
}

or, use template to return NXDOMAIN for reddog.microsoft.com

reddog.microsoft.com:53 {
  ......
  ......
  template ANY ANY {
    rcode NXDOMAIN
  }
  ......
  ......
}

[Ehsaan-Azizi]

Just curious if there is any updates on this? Having the same issue and need some recommendations if there is a possibility creating a dynamic kubelet config for this to cascade clusterwise? has anyone tried it?

[asessomsblm]

Following

[msamad]

Just curious if there is any updates on this? Having the same issue and need some recommendations if there is a possibility creating a dynamic kubelet config for this to cascade clusterwise? has anyone tried it?

We added the following config in our clusters. Created a coredns-custom configmap and stopped it from going upstream for reddog.microsoft.com. It has been more than half a year now and things have been smooth since then, the load on our upstream DNS servers reduced too.

  reddog.server: |
    reddog.microsoft.com:53 {
      errors
      template ANY ANY {
        rcode NXDOMAIN
      }
      prometheus :9153
      cache 30
    }

[Ehsaan-Azizi]

Just curious if there is any updates on this? Having the same issue and need some recommendations if there is a possibility creating a dynamic kubelet config for this to cascade clusterwise? has anyone tried it?

We added the following config in our clusters. Created a coredns-custom configmap and stopped it from going upstream for reddog.microsoft.com. It has been more than half a year now and things have been smooth since then, the load on our upstream DNS servers reduced too.

  reddog.server: |

    reddog.microsoft.com:53 {

      errors

      template ANY ANY {

        rcode NXDOMAIN

      }

      prometheus :9153

      cache 30

    }

Can you please share how exactly you implemented this in coredns-custom and how did you tested and validated? I'm actively working on this and evaluating my options. I have implemented in ingress ngninx controller pod only in its dnsConfig settings. But will it cover the whole cluster workload if I do this in coredns-custom configmap? Appreciate the help!

[Ehsaan-Azizi]

Just curious if there is any updates on this? Having the same issue and need some recommendations if there is a possibility creating a dynamic kubelet config for this to cascade clusterwise? has anyone tried it?

We added the following config in our clusters. Created a coredns-custom configmap and stopped it from going upstream for reddog.microsoft.com. It has been more than half a year now and things have been smooth since then, the load on our upstream DNS servers reduced too.

  reddog.server: |
    reddog.microsoft.com:53 {
      errors
      template ANY ANY {
        rcode NXDOMAIN
      }
      prometheus :9153
      cache 30
    }

@msamad, the change in coredns-custom configmap crashed my ingress nginx along with kuma service mesh in lab cluster. I had to revert the change and everything started working.

[msamad]

Just curious if there is any updates on this? Having the same issue and need some recommendations if there is a possibility creating a dynamic kubelet config for this to cascade clusterwise? has anyone tried it?

We added the following config in our clusters. Created a coredns-custom configmap and stopped it from going upstream for reddog.microsoft.com. It has been more than half a year now and things have been smooth since then, the load on our upstream DNS servers reduced too.

  reddog.server: |
    reddog.microsoft.com:53 {
      errors
      template ANY ANY {
        rcode NXDOMAIN
      }
      prometheus :9153
      cache 30
    }

@msamad, the change in coredns-custom configmap crashed my ingress nginx along with kuma service mesh in lab cluster. I had to revert the change and everything started working.

Don't know how your whole cluster is set up so can't comment much. This is the docs page to follow on how to customize coredns https://learn.microsoft.com/en-us/azure/aks/coredns-custom#rewrite-dns