Kubernetes gluster storage class not working with DNS name

[kevinpz]

Hi guys,

I have a little issue to use the GlusterFS storage class. I'm trying to add it with the DNS name inside the K8S cluster, but it seems that it doesn't work.

Here is my storage class creation file:

apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: gluster
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://heketi.default.svc.cluster.local:8080"
  restuser: "joe"
  restuserkey: "My Secret Life"

If I try to reach the domain name from a CentOS pod, it works well:

root@centos-68b945946c-dt87d /]# host heketi.storage.svc.cluster.local        
heketi.storage.svc.cluster.local has address 10.104.17.233

But, when I try to deploy a nginx pod, with a storage class:

kubectl create -f nginx.yaml

With this kind of volume reservation:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nginx-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: gluster
  resources:
    requests:
      storage: 1Gi

The deployment of the pod fails with the following error:

[centos@bastion-node ~]$ kubectl get pvc
NAME        STATUS    VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS   AGE
nginx-pvc   Pending                                       gluster        16s
[centos@bastion-node ~]$ kubectl describe pvc nginx-pvc
Name:          nginx-pvc
Namespace:     default
StorageClass:  gluster
Status:        Pending
Volume:
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/glusterfs
Capacity:
Access Modes:
Events:
  Type     Reason              Age               From                         Message
  ----     ------              ----              ----                         -------
  Warning  ProvisioningFailed  9s (x2 over 23s)  persistentvolume-controller  Failed to provision volume with StorageClass "gluster": create volume error: error creating volume Post http://heketi.default.svc.cluster.local:8080/volumes: dial tcp: lookup heketi.default.svc.cluster.local on 8.8.4.4:53: no such host

If in my storage class, I replace resturl: "http://heketi.default.svc.cluster.local:8080" with the corresponding IP address resturl: "http://10.104.17.233:8080" it works well.

Do you have any idea on why the PVC doesn't resolve the domain name inside the K8S cluster, but try to resolve it on internet?

Thanks!

[jarrpa]

I'm not sure what's going on there. Can you resolve cluster names from the kubernetes hosts themselves?

[jenciso]

@kevinpz I think that this problem has relation with this kubernetes issue https://github.com/kubernetes/kubernetes/issues/42306 Is for that reason that I'm using IP svc agains svc name

[kevinpz]

Hi @jenciso you're right. That's the same issue. So it's more a K8S issue than a gluster one?

[joejulian]

The problem seems to be that the heketi container's resolv.conf is searching 8.8.4.4 instead of your kubernetes dns.