This is not an issue really, it is more like a question and a feature idea. Please feel free to correct my understanding where it is wrong. There are three questions I am looking to please get some help with.
My goal is to be able to deploy Heketi, GlusterFS and the in tree Kubernetes Gluster Provisioner in such a way that the host network, and host IP addresses are unnecessary. In this new model, everything can be done with Pod IP addresses and the overlay network. I understand this is a currently unsupported feature (I am hoping to learn more about what is needed to change it).
Background
Currently the gk-deploy script creates a Daemonset with hostNetwork=true, so while Pods are created to execute the gluster daemon, the configuration expects to be serviced from the host network. This is related to the requirement to specify the host ip addresses in hostnames.storage in the topology file given to Heketi when initially creating the cluster.
gk-deploy then calls Heketi with the argument setup-openshift-heketi-storage to create heketi-storage.json which describes a Kubernetes Service and set of Endpoints that are used to provide Heketi with a volume in the new gluster cluster for it to store its data. An identical behavior occurs from the in tree Gluster Provisioner, where each PersistentVolumeClaim it serves, gets a Service and set of Endpoints.
In both cases the following is true of the endpoints:
a) They are explicitly created/specified, vs being created automatically by Kubernetes due to the associated Service
b) They are populated with the IP addresses of hosts given in the topology file.
c) They are associated with a Service that doesn't encode any useful TCP/Port information so clearly these Endpoints are not useful for the ports on the service.
Questions
Q1) Why is this the behavior with the created Services & EndPoints? Why explicitly specify the Endpoints and not just use a selector criteria on the Service to have Kubernetes automatically create the Endpoints?
Q2) Is there anything in Heketi and Gluster themselves (ignoring the Kubernetes Provisioner) that does not allow the use of FQDN specifiers in the Topology (my own research into this [see below] suggests it should work fine). It seems like provisioning a cluster via FQDN and then changing IP addresses doesn't cause glusters side of things to catch fire. Is this true?
Q3) Can we change the behavior of Heketi and the in tree Provisioner to create Endpoints by first creating a Service with a selector that matches the Pods we initially deployed in the Daemonset, I am specifically thinking we could deploy a Statefulset and get stable DNS names, which we use in the topology file, and then get the Endpoint Controller to create Endpoints based on a selector. When the IP addresses change the Endpoint Controller will update the Endpoints to reflect the new correct IP addresses.
Some Research Results
I have manually done the steps in gk-deploy but made the following changes.
1) Used a Statefulset instead of a Daemonset. This provides predictable/stable pod names.
2) Created a Headless service for the Statefulset which had a selector to the pods, such that Kubernetes created internal DNS entries to each Pod, and Endpoints to each pod.
3) Used a topology.json file that used the DNS names from (2) to use Heketi to create a cluster.
4) Edited the Heketi produced heketi-storage.json file such that the file no longer creates endpoints, and the Service has a selector to select for the same pods in the StatefulSet from (1)
5) Applied the edited heketi-storage.json file to get a working Heketi deployment. This results is a working service heketi-storage-endpoints to point to the Pods running gluster and heketi which is used as the REST url for heketi-cli and configuration in StorageClasses.
6) Used external-dns to ensure that the FQDN entries for my glusterd pods are visible to the hoste nodes via the entries internal cluster IP addresses.
At this point Heketi and gluster seem happy, I can manually delete gluster Pods (one at a time) and Kubernetes brings them back up at different IP addresses and once the DNS catches up the gluster cluster seems fine. Perhaps it is trying to fool me :-)
Where Everything Breaks
I currently have no control over the in tree Kubernetes Gluster Provisioner, Kubernetes 1.5+ doesn't allow specifying Endpoints, so I can't configure a StorageClass that tells the provisioner what EndPoints to use, it wants to generate them itself. Currently the Provisioner pulls what it expects to be IP addresses from Heketi, which in my case is providing DNS names, and thus provisioning fails because EndPoints can't have names in them only IP addresses.
failed to create endpoint/service default/glusterfs-dyanmic-XXXXX failed to create endpoint.
Endpoints "glusterfs-dynamic-XXXX" is invalid: [subsets[0].addresses[0].ip: Invalid value:
"glusterfs-7.gluster-storage-paas-gluster-small.XXXX": must be a valid IP address.
Bonus Round Question
If the in tree provisioner where changed to create Endpoints via a selector on a Service, can we eliminate the need for hostNetwork=true? The StorageClass that passes the Heketi REST url to the Provisioner would need to pass the selector criteria but that is not a difficult configuration to make. Security to a given heketi instance is maintained via the secret given to the StorageClass. This would allow any number independent deployments of gluster and heketi via containers on the same physical hosts (clearly we don't share block devices between clusters), which I don't believe is currently possible due to the dependency on the host network.
Thanks for reading this far and if there are no foreseeable technical hurdles in the approach I outlined above what is the path forward for trying to allow deployment in the way I described?
Thanks again!
This is not an issue really, it is more like a question and a feature idea. Please feel free to correct my understanding where it is wrong. There are three questions I am looking to please get some help with.
My goal is to be able to deploy Heketi, GlusterFS and the in tree Kubernetes Gluster Provisioner in such a way that the host network, and host IP addresses are unnecessary. In this new model, everything can be done with Pod IP addresses and the overlay network. I understand this is a currently unsupported feature (I am hoping to learn more about what is needed to change it).
Background
Currently the gk-deploy script creates a Daemonset with hostNetwork=true, so while Pods are created to execute the gluster daemon, the configuration expects to be serviced from the host network. This is related to the requirement to specify the host ip addresses in
hostnames.storagein the topology file given to Heketi when initially creating the cluster.gk-deploy then calls Heketi with the argument
setup-openshift-heketi-storageto createheketi-storage.jsonwhich describes a Kubernetes Service and set of Endpoints that are used to provide Heketi with a volume in the new gluster cluster for it to store its data. An identical behavior occurs from the in tree Gluster Provisioner, where each PersistentVolumeClaim it serves, gets a Service and set of Endpoints.In both cases the following is true of the endpoints: a) They are explicitly created/specified, vs being created automatically by Kubernetes due to the associated Service b) They are populated with the IP addresses of hosts given in the topology file. c) They are associated with a Service that doesn't encode any useful TCP/Port information so clearly these Endpoints are not useful for the ports on the service.
Questions
Q1) Why is this the behavior with the created Services & EndPoints? Why explicitly specify the Endpoints and not just use a selector criteria on the Service to have Kubernetes automatically create the Endpoints? Q2) Is there anything in Heketi and Gluster themselves (ignoring the Kubernetes Provisioner) that does not allow the use of FQDN specifiers in the Topology (my own research into this [see below] suggests it should work fine). It seems like provisioning a cluster via FQDN and then changing IP addresses doesn't cause glusters side of things to catch fire. Is this true? Q3) Can we change the behavior of Heketi and the in tree Provisioner to create Endpoints by first creating a Service with a selector that matches the Pods we initially deployed in the Daemonset, I am specifically thinking we could deploy a Statefulset and get stable DNS names, which we use in the topology file, and then get the Endpoint Controller to create Endpoints based on a selector. When the IP addresses change the Endpoint Controller will update the Endpoints to reflect the new correct IP addresses.
Some Research Results
I have manually done the steps in gk-deploy but made the following changes. 1) Used a Statefulset instead of a Daemonset. This provides predictable/stable pod names. 2) Created a Headless service for the Statefulset which had a selector to the pods, such that Kubernetes created internal DNS entries to each Pod, and Endpoints to each pod. 3) Used a topology.json file that used the DNS names from (2) to use Heketi to create a cluster. 4) Edited the Heketi produced
heketi-storage.jsonfile such that the file no longer creates endpoints, and the Service has a selector to select for the same pods in the StatefulSet from (1) 5) Applied the editedheketi-storage.jsonfile to get a working Heketi deployment. This results is a working serviceheketi-storage-endpointsto point to the Pods running gluster andheketiwhich is used as the REST url for heketi-cli and configuration in StorageClasses. 6) Used external-dns to ensure that the FQDN entries for my glusterd pods are visible to the hoste nodes via the entries internal cluster IP addresses.At this point Heketi and gluster seem happy, I can manually delete gluster Pods (one at a time) and Kubernetes brings them back up at different IP addresses and once the DNS catches up the gluster cluster seems fine. Perhaps it is trying to fool me :-)
Where Everything Breaks
I currently have no control over the in tree Kubernetes Gluster Provisioner, Kubernetes 1.5+ doesn't allow specifying Endpoints, so I can't configure a StorageClass that tells the provisioner what EndPoints to use, it wants to generate them itself. Currently the Provisioner pulls what it expects to be IP addresses from Heketi, which in my case is providing DNS names, and thus provisioning fails because EndPoints can't have names in them only IP addresses.
Bonus Round Question
If the in tree provisioner where changed to create Endpoints via a selector on a Service, can we eliminate the need for hostNetwork=true? The StorageClass that passes the Heketi REST url to the Provisioner would need to pass the selector criteria but that is not a difficult configuration to make. Security to a given heketi instance is maintained via the secret given to the StorageClass. This would allow any number independent deployments of gluster and heketi via containers on the same physical hosts (clearly we don't share block devices between clusters), which I don't believe is currently possible due to the dependency on the host network.
Thanks for reading this far and if there are no foreseeable technical hurdles in the approach I outlined above what is the path forward for trying to allow deployment in the way I described? Thanks again!