Actions are unable to reach namesever specified in invoker dns config

[jhawarchirag0]

I have a dns set up on 192.168.9.30. My actions contains urls which needs to be resolved on this server but the actions are unable to reach the server and still reach out to kube-dns. Can someone please guide me. My invoker configs are

invoker:
  imageName: "openwhisk/invoker"
  imageTag: "1.0.0"
  imagePullPolicy: "IfNotPresent"
  restartPolicy: "Always"
  port: 8080
  options: ""
  jvmHeapMB: "512"
  jvmOptions: ""
  loglevel: "INFO"
  containerFactory:
    useRunc: false
    impl: "kubernetes"
    enableConcurrency: false
    networkConfig:
      name: "bridge"
      dns:
        inheritInvokerConfig: false
        overrides:          # NOTE: if inheritInvokerConfig is true, all overrides are ignored
          # Nameservers, search, and options are space-separated lists
          # eg nameservers: "1.2.3.4 1.2.3.5 1.2.3.6" is a list of 3 nameservers
          nameservers: "192.168.9.30"
          search: "example.com"
          options: ""
    kubernetes:
      isolateUserActions: true
      replicaCount: 1

[style95]

Since a runtime pod runs as a separate pod, I think it's not related to the invoker pod. And this is rather related to Kubernetes DNS resolution.

Normally you can specify DNS configuration for a pod. https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-dns-config

But I doubt our container factory has such an option. Are you able to access your domain if you manually run a pod? If not, I think you need to ask the cluster admin.

[jhawarchirag0]

@style95 I thought the DNS section was for that purpose - https://github.com/apache/openwhisk-deploy-kube/pull/370

The answer to your question - yes I am if I change the dnsConfig and dnsPolicy of the deployment, but I am not sure if we can modify the same for wsk actions

[style95]

hm.. let me try that. It seems DockerContainer respects the option. But not sure it would apply to the Kubernetes pod too. Since a nameserver is specified this way, I think there should be some logic to translate the config into dnsConfig of a pod accordingly. And I couldn't find any.

apiVersion: v1
kind: Pod
metadata:
  namespace: default
  name: dns-example
spec:
  containers:
    - name: test
      image: nginx
  dnsPolicy: "None"
  dnsConfig:
    nameservers:
      - 1.2.3.4
    searches:
      - ns1.svc.cluster-domain.example
      - my.dns.search.suffix
    options:
      - name: ndots
        value: "2"
      - name: edns0

https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-dns-config

[style95]

@dgrove-oss Do you have any idea about this?

[dgrove-oss]

It looks like I added that logic in 2018. I think that was early enough that Kubernetes was still using Docker as the kubelet container engine (hadn't switched to containerd). We were only deploying the OpenWhisk control plane (invoker, controller, Kafka, etc) as proper Kubernetes pods. All of the user action containers were created outside of Kubernetes by having the invoker (which was deployed as a DaemonSet -- one invoker per node) going directly to Docker on its node and creating the user container. So I think all that this wiring did was to give us a way to pass the docker networking configuration into the invoker so it would then pass that information to Docker when it created the user containers.

But all of this is pretty hazy to be honest...2018 was a long time ago :(

[style95]

Thank you for the comment. It is helpful to know the history. I would try specifying the configuration.

[style95]

The dns config does not apply to the Kubernetes runtime pod.

[style95]

@jhawarchirag0 I think it would take some time to make the KubernetesContainerFactory respect the DNS config. In the meantime, I think you can try with a pod template if your DNS server IP does not dynamically change.

I think you can specify your DNS server in the pod template and use it as a base template to create a runtime pod.

[jhawarchirag0]

@style95 Thankyou for the help. But I am having difficulties in implementing this template. For my current setup I am using v1.0.0 of the helm chart.

openwhisk/whiskconfig.conf

include classpath("application.conf")

whisk {
  metrics {
    prometheus-enabled = false
  }
}

openwhisk/application.conf

whisk {
  kubernetes {
    pod-template = "file:/pod-template.yaml"
  }
}

The pod definition has

spec:
  dnsPolicy: ClusterFirst

Do I need to update my helm chart or is it something other I am doing wrong?

[jhawarchirag0]

I have created a file containing the pod-template.yaml

apiVersion: v1
kind: Pod
spec:
  dnsPolicy: None
  dnsConfig:
    nameservers:
      - 192.168.9.30
    searches:
      - example.com

I have added a cm

apiVersion: v1
kind: ConfigMap
data:
  pod-template.yaml: |
    ---
    apiVersion: v1
    kind: Pod
    spec:
      dnsPolicy: None
      dnsConfig:
        nameservers:
          - 192.168.9.30
        searches:
          - example.com
metadata:
  name: whisk-pod-template

I have added an env variable inside invoker-pod.yaml

      containers:
      - name: invoker
        env:
          - name: "WHISK_POD_TEMPLATE"
            valueFrom:
              configMapKeyRef:
                name: whisk-pod-template
                key: pod-template.yaml

I am getting this error in the logs for invoker

Defaulted container "invoker" out of: invoker, wait-for-controller (init)
Error: Could not find or load main class apiVersion:
Caused by: java.lang.ClassNotFoundException: apiVersion:

[style95]

@jhawarchirag0 As the PR described, you have 3 options.

1. File reference: You need to add the file in the container and specify the path of the pod template.
2. Env variable: Add an environment variable containing the pod template.
3. Literal inline yaml: Add the inline pod template in the configuration.

Did you configure the whisk config as well?

[jhawarchirag0]

@style95 Thanks for the help. I have resolved this issue

[style95]

@jhawarchirag0 Great. What was the problem?

[jhawarchirag0]

I was unable to pass it in the application.conf but used this in the owconfig.yaml

invoker:
  options: "-Dwhisk.kubernetes.user-pod-node-affinity.enabled=false  -Dwhisk.kubernetes.pod-template=file:/config/pod-template.yaml"

Used this configmap

apiVersion: v1
kind: ConfigMap
data:
  pod-template.yaml: |
    apiVersion: v1
    kind: Pod
    spec:
      dnsPolicy: None
      dnsConfig:
        nameservers:
          - 192.168.9.30
        searches:
          - example.com
metadata:
  name: whisk-pod-template

Modified invoker-pod.yaml with these changes

      volumes:
      - name: pod-template
        configMap:
          name: whisk-pod-template
          items:
          - key: pod-template.yaml
            path: pod-template.yaml
      containers:
      - name: invoker
        volumeMounts:
        - name: pod-template
          mountPath: /config/pod-template.yaml
          subPath: pod-template.yaml

For the env approach I don't quite understand the problem completely which is causing the issue

[jhawarchirag0]

I just discovered wsk -i api list does not work with this change.

wsk -i api list
error: Unable to obtain the API list: Unable to obtain API(s) from the API Gateway: "getaddrinfo EAI_AGAIN openwhisk-apigateway.openwhisk.svc.cluster.local openwhisk-apigateway.openwhisk.svc.cluster.local:9000"

[style95]

That's because it would not ask to the cluster DNS as you configured it as none.

[jhawarchirag0]

Fixed this by modifying the coredns and added a rule to forward all the requests associated to the domain to the dns ip