Infinispan pods are deployed using the Infinispan operator which utilises a default anti-affinity strategy that prefers, but does not require, pods to be deployed to different K8s nodes. This approach will ensure that we can tolerate a single node failure when caches are configured with num_owners=2 and the number of K8s nodes is greater than or equal to the number of Infinispan pods.
Limitation
If the number of K8s nodes is less than the number of Infinispan pods, then it's possible that a single K8s node will host multiple Infinispan pods. While not ideal, this scenario is acceptable as long as the Primary and Backup owner of data are not hosted on the same node. However, if both the Primary and Backup are on the same Node then a single K8s node failure will lead to data loss as both copes of our cache entity will be lost.
Proposal
We can leverage Infinispan Server Hinting to make our Infinispan deployments more resilient. Server Hinting ensures that the Infinispan hashing algorithm takes into account the host nodes when determining which Infinispan pods should host Primary/Backup data.
Current Behaviour
Infinispan pods are deployed using the Infinispan operator which utilises a default anti-affinity strategy that prefers, but does not require, pods to be deployed to different K8s nodes. This approach will ensure that we can tolerate a single node failure when caches are configured with
num_owners=2and the number of K8s nodes is greater than or equal to the number of Infinispan pods.Limitation
If the number of K8s nodes is less than the number of Infinispan pods, then it's possible that a single K8s node will host multiple Infinispan pods. While not ideal, this scenario is acceptable as long as the Primary and Backup owner of data are not hosted on the same node. However, if both the Primary and Backup are on the same Node then a single K8s node failure will lead to data loss as both copes of our cache entity will be lost.
Proposal
We can leverage Infinispan Server Hinting to make our Infinispan deployments more resilient. Server Hinting ensures that the Infinispan hashing algorithm takes into account the host nodes when determining which Infinispan pods should host Primary/Backup data.
Missing Pieces