Set up Heketi Rest URL in Storage Class

[BostjanBozic]

Hello,

Foremost I am sorry that I am setting this up as as issue, it is very likely that there I am doing something wrong on my part.

• Kubernetes version: v1.13.1
• Gluster version: 6.0

What I did was I deployed Heketi on Kubernetes and set up topology to use external servers for volume provisioning. From Heketi side, everything is working as intended - service is up and running, devices are created.

I have set up Heketi service, as I plan to use it for StorageClass:

apiVersion: v1
kind: Service
metadata:
  name: heketi
  namespace: kube-system
  labels:
    glusterfs: heketi-service
    heketi: service
  annotations:
    description: Exposes Heketi Service
spec:
  selector:
    glusterfs: heketi-pod
  ports:
  - name: heketi
    port: 8080
    targetPort: 8080

And I have created StorageClass with following configuration:

apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: heketi
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: http://10.109.154.186:8080
  restuser: "admin_from_secrets"
  restuserkey: "password_from_secrets"

Currently as you can see, resturl is hard coded IP. What I want to do is actually use service created above. I have tried multiple adresses, e.g. heketi.kube-system.svc.cluster.local, but I always receive following error when creating PVC:

Failed to provision volume with StorageClass "heketi": failed to create volume: failed to create volume: Post http://heketi.kube-system.svc.cluster.local:8080/volumes: dial tcp: lookup heketi.kube-system.svc.cluster.local on 10.1.50.10:53: no such host

Any kind of suggestion what I am doing wrong would be great.

Thank you and best regards, Bostjan

[SaravanaStorageNetwork]

resturl: http://10.109.154.186:8080

The rest url is coming from the heketi service.

You could check whether it is working as mentioned here: https://github.com/gluster/gluster-kubernetes/blob/master/README.md

$ curl $HEKETI_CLI_SERVER/hello

You are missing few parameters in the storageclass

Refer the sample storage class: https://github.com/gluster/gluster-kubernetes/blob/master/deploy/ocp-templates/gluster-s3-storageclass.yaml

[BostjanBozic]

Thank you for feedback.

Thing is if I am using service clusterIP (like I have used in example above), it is working as intended - PVC is created and PV assigned to it:

curl 10.109.154.186:8080/hello
Hello from Heketi

But I am worries that in case someone recreates service, IP would change. I could hard code clusterIP into service deployment, but I would prefer to use service name addressing it. I did check documentation on StorageClass, which mentiones:

resturl: Gluster REST service/Heketi service url which provision gluster volumes on demand. The general format should be IPaddress:Port and this is a mandatory parameter for GlusterFS dynamic provisioner. If Heketi service is exposed as a routable service in openshift/kubernetes setup, this can have a format similar to http://heketi-storage-project.cloudapps.mystorage.com where the fqdn is a resolvable Heketi service url

I thought I could use svc.cluster.local here, but this seems not to be the case.

Regarding missing parameters , for now I just want to see if everything is working and then add additional ones (e.g. replication factor).

[chenyg0911]

@BostjanBozic you should set kube-controller-manager dnsPolicy if it user hostnetwork.

  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet

[BostjanBozic]

Thank you very much for this, it indeed solved the issue.

I also found an issue at kubespray, where this was also discussed (in case someone needs it in future): https://github.com/kubernetes-sigs/kubespray/issues/3177

So basically I have updated /etc/kubernetes/manifests/kube-controller-manager.yaml by adding dnsPolicy: ClusterFirstWithHostNet and restarted kubelet.