calico kubernetes cannot reach services (no route)

marvanto commented 5 years ago

Cannot reach service network I have HA multi-master cluster (3 masters on VM in same subnet). I configured cluster with --pod-network-cidr "192.168.0.0/16" --service-cidr "172.20.0.0/16", master1 - 10.0.3.2, master2 - 10.0.3.3, master3 - 10.0.3.4.

traceroute from "web" container

part of calico.yaml

kind: ConfigMap
apiVersion: v1
metadata:
  name: calico-config
  namespace: kube-system
data:
   Typha is disabled.
  typha_service_name: "none"
  ## Configure the backend to use.
  calico_backend: "bird"

  ## Configure the MTU to use
  veth_mtu: "1440"

  # The CNI network configuration to install on each node.  The special
  # values in this config will be automatically populated.

cni_network_config: |- { "name": "k8s-pod-network", "cniVersion": "0.3.1", "plugins": [ { "type": "calico", "log_level": "info", "datastore_type": "kubernetes", "nodename": "KUBERNETES_NODE_NAME", "mtu": __CNI_MTU__, "ipam": { "type": "calico-ipam" }, "policy": { "type": "k8s" }, "kubernetes": { "kubeconfig": "/etc/kubernetes/admin.conf" } }, { "type": "portmap", "snat": true, "capabilities": {"portMappings": true} } ] }

    - name: upgrade-ipam
      image: calico/cni:v3.9.1
      command: ["/opt/cni/bin/calico-ipam", "-upgrade"]
      env:
        - name: KUBERNETES_NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: CALICO_NETWORKING_BACKEND
          valueFrom:
            configMapKeyRef:
              name: calico-config
              key: calico_backend
      volumeMounts:
        - mountPath: /var/lib/cni/networks
          name: host-local-net-dir
        - mountPath: /host/opt/cni/bin
          name: cni-bin-dir
    ## This container installs the CNI binaries
    ## and CNI network config file on each node.
    - name: install-cni
      image: calico/cni:v3.9.1
      command: ["/install-cni.sh"]
      env:
        ##Name of the CNI config file to create.
        - name: CNI_CONF_NAME
          value: "10-calico.conflist"
        ## The CNI network config to install on each node.
        - name: CNI_NETWORK_CONFIG
          valueFrom:
            configMapKeyRef:
              name: calico-config
              key: cni_network_config
        ## Set the hostname based on the k8s node name.
        - name: KUBERNETES_NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        ## CNI MTU Config variable
        - name: CNI_MTU
          valueFrom:
            configMapKeyRef:
              name: calico-config
              key: veth_mtu
        ## Prevents the container from sleeping forever.
        - name: SLEEP
          value: "false"
      volumeMounts:
        - mountPath: /host/opt/cni/bin
          name: cni-bin-dir
        - mountPath: /host/etc/cni/net.d
          name: cni-net-dir
    ## Adds a Flex Volume Driver that creates a per-pod Unix Domain Socket to allow Dikastes
    ## to communicate with Felix over the Policy Sync API.
    - name: flexvol-driver
      image: calico/pod2daemon-flexvol:v3.9.1
      volumeMounts:
      - name: flexvol-driver-host
        mountPath: /host/driver
  containers:
    ## Runs calico-node container on each Kubernetes node.  This
    ## container programs network policy and routes on each
    ## host.
    - name: calico-node
      image: calico/node:v3.9.1
      env:
        ## Enabling service cluster IP advertisement
        - name: CALICO_ADVERTISE_CLUSTER_IPS
          value: "172.20.0.0/16"
        ## Use Kubernetes API as the backing datastore.
        - name: DATASTORE_TYPE
          value: "kubernetes"
          ## value: etcdv3
        ##Wait for the datastore.
        - name: WAIT_FOR_DATASTORE
          value: "true"
        ## Set based on the k8s node name.
        - name: ETCD_ENDPOINTS
          value: https://10.0.3.2:2379,https://10.0.3.3:2379,https://10.0.3.4:2379
        - name: KUBECONFIG
          value: "/etc/kubernetes/admin.conf"
        # - name: ETCD_KEY_FILE
        #   value: /etc/kubernetes/pki/etcd/server.key
        # - name: ETCD_CERT_FILE
        #   value: /etc/kubernetes/pki/etcd/server.crt
        # - name: ETCD_CA_CERT_FILE
        #   value: /etc/kubernetes/pki/etcd/ca.crt 
        # - name: K8S_API_ENDPOINT
        #   value: http://10.0.3.100:6443
        - name: NODENAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        # Choose the backend to use.
        - name: CALICO_NETWORKING_BACKEND
          valueFrom:
            configMapKeyRef:
              name: calico-config
              key: calico_backend
        # Cluster type to identify the deployment type
        - name: CLUSTER_TYPE
          value: "k8s,bgp"
        # Specify interface
        - name: IP_AUTODETECTION_METHOD
          value: "interface=eth.+"
        # Auto-detect the BGP IP address.
        - name: IP
          value: "autodetect"
        # Enable IPIP
        - name: CALICO_IPV4POOL_IPIP
          value: "Always"
        # Set MTU for tunnel device used if ipip is enabled
        - name: FELIX_IPINIPMTU
          valueFrom:
            configMapKeyRef:
              name: calico-config
              key: veth_mtu
        # The default IPv4 pool to create on startup if none exists. Pod IPs will be
        # chosen from this range. Changing this value after installation will have
        # no effect. This should fall within `--cluster-cidr`.
        - name: CALICO_IPV4POOL_CIDR
          value: "192.168.0.0/16"
        # Disable file logging so `kubectl logs` works.
        - name: CALICO_DISABLE_FILE_LOGGING
          value: "true"
        # Set Felix endpoint to host default action to ACCEPT.
        - name: FELIX_DEFAULTENDPOINTTOHOSTACTION
          value: "ACCEPT"
        # Disable IPv6 on Kubernetes.
        - name: FELIX_IPV6SUPPORT
          value: "false"
        # Set Felix logging to "info"
        - name: FELIX_LOGSEVERITYSCREEN
          value: "info"
        - name: FELIX_HEALTHENABLED
          value: "true"

Expected Behavior

Expected routes to services

Current Behavior

Possible Solution

I tried some options wich you can see in config, no effect, now i'm stuck

Steps to Reproduce (for bugs)

1. 2. 3. 4.

Context

Your Environment

Calico version
Orchestrator version (e.g. kubernetes, mesos, rkt):
Operating System and version:
Link to your project (optional):

caseydavenport commented 5 years ago

@marvanto Kubernetes services are TCP / UDP only. That means commands like ping which use ICMP won't work if you're targeting a Service IP address.

Try using a pod address to see if that works.

marvanto commented 5 years ago

@caseydavenport thank you, you have made my day. I've checked telnet to service and it works, thank you again.

tmjd commented 5 years ago

Sounds like things are resolved so I'm going to close this issue.

projectcalico / calico