Catapult in not setting dns annotations in CAP pipelines

[prabalsharma]

from CI: post deployment of kubecf

Request error: Get https://api.prabal-gke.ci.kubecf.charmedquarks.me: dial tcp: lookup api.prabal-gke.ci.kubecf.charmedquarks.me on 172.31.2.5:53: no such host
TIP: If you are behind a firewall and require an HTTP proxy, verify the https_proxy environment variable is correctly set. Else, check your network connection.

286ca24c-1021-41ec-64c0-a3bddf9b24a5:/tmp/build/0a06c48c # cat catapult/buildprabal-gke/scf-config-values.yaml
---
system_domain: prabal-gke.ci.kubecf.charmedquarks.me
install_stacks:
- sle15
- cflinuxfs3
features:
  eirini:
    enabled: false
  autoscaler:
    enabled: false
high_availability: false
testing:
  brain_tests:
    enabled: true
  cf_acceptance_tests:
    enabled: true
  smoke_tests:
    enabled: true
  sync_integration_tests:
    enabled: true
properties:
  acceptance-tests:
    acceptance-tests:
      acceptance_tests:
        timeout_scale: 3
        ginkgo:
          slow_spec_threshold: 300
          extra_flags: ''
          nodes: 3
          flake_attempts: 5
  brain-tests:
    acceptance-tests-brain:
      acceptance_tests_brain:
        verbose: 'false'
        in_order: 'false'
        include: ''
        exclude: ''

No dns annotation https://github.com/SUSE/catapult/blob/master/modules/kubecf/gen_config.sh#L44-L67

286ca24c-1021-41ec-64c0-a3bddf9b24a5:/tmp/build/0a06c48c # source catapult/buildprabal-gke/.envrc 
286ca24c-1021-41ec-64c0-a3bddf9b24a5:/tmp/build/0a06c48c # kubectl get configmap -n kube-system cap-values -o json | jq -r '.data["services"]'
null
286ca24c-1021-41ec-64c0-a3bddf9b24a5:/tmp/build/0a06c48c # 

[prabalsharma]

proper service entry got created but then somehow don't exist after!

ℹ  🚀  gke ☸ prabal-gke  🎂  ./deploy.sh ➤  Configuring deployed GKE cluster…
configmap/cap-values created
{
    "apiVersion": "v1",
    "data": {
        "domain": "prabal-gke.ci.kubecf.charmedquarks.me",
        "garden-rootfs-driver": "overlay-xfs",
        "platform": "gke",
        "public-ip": "10.164.0.28",
        "services": "lb"
    },
    "kind": "ConfigMap",
    "metadata": {
        "creationTimestamp": "2020-10-05T21:25:21Z",
        "name": "cap-values",
        "namespace": "kube-system",
        "resourceVersion": "2406",
        "selfLink": "/api/v1/namespaces/kube-system/configmaps/cap-values",
        "uid": "3ea2c7fa-eb91-4f52-92e1-6509e044f30b"
    }
}
clusterrolebinding.rbac.authorization.k8s.io/admin created
clusterrolebinding.rbac.authorization.k8s.io/uaaadmin created
clusterrolebinding.rbac.authorization.k8s.io/scfadmin created
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
clusterrole.rbac.authorization.k8s.io/cluster-admin configured
clusterrolebinding.rbac.authorization.k8s.io/kube-system:default created
✅  🚀  gke ☸ prabal-gke 🎂  ./deploy.sh ➤  GKE cluster deployed

"services": "lb" can no longer be seen

[prabal:~/gop/src/github.com/SUSE/catapult] master(+1/-1)* 11m17s ± kubectl get configmap -n kube-system cap-values -o json
{
    "apiVersion": "v1",
    "data": {
        "chart": "/tmp/build/0a06c48c/helm-chart.kubecf-chart/kubecf-2.2.3.tgz",
        "domain": "prabal-gke.ci.kubecf.charmedquarks.me",
        "garden-rootfs-driver": "overlay-xfs",
        "platform": "gke",
        "public-ip": "10.164.0.28"
    },
    "kind": "ConfigMap",
    "metadata": {
        "creationTimestamp": "2020-10-05T21:25:21Z",
        "name": "cap-values",
        "namespace": "kube-system",
        "resourceVersion": "4736",
        "selfLink": "/api/v1/namespaces/kube-system/configmaps/cap-values",
        "uid": "3ea2c7fa-eb91-4f52-92e1-6509e044f30b"
    }
}

[prabalsharma]

gets over-written at this step for deploy job:

make -s -C modules/kubecf
/tmp/build/0a06c48c/catapult/buildprabal-gke /tmp/build/0a06c48c/catapult/modules/kubecf
[./clean.sh] [backend:gke] [cluster:prabal-gke] Loading
configmap/cap-values patched
[./clean.sh] [backend:gke] [cluster:prabal-gke] Cleaned up KubeCF from the k8s cluster
/tmp/build/0a06c48c/catapult/buildprabal-gke /tmp/build/0a06c48c/catapult/modules/kubecf
[./chart.sh] [backend:gke] [cluster:prabal-gke] Loading
[./chart.sh] [backend:gke] [cluster:prabal-gke] Grabbing chart from local file 
'/tmp/build/0a06c48c/helm-chart.kubecf-chart/kubecf-2.2.3.tgz' -> 'chart'

[prabalsharma]

what is going on here?

https://github.com/SUSE/catapult/blob/master/modules/kubecf/clean.sh#L59-L62

We are deleting services data entry from config-map.

@viccuad do you know?

I think we should not call clean.sh and leave it for explicit use by users.

[viccuad]

@prabalsharma I think you hit the nail with https://github.com/SUSE/catapult/blob/master/modules/kubecf/clean.sh#L59-L62.

The bug was introduced by https://github.com/SUSE/catapult/pull/294, when we started using services heavily, to accomodate for using the ingress deployments in cap.suse.de

services is cluster environment information, such as domain, and should not be deleted on kubecf-clean.