Custom http selector overriding default selector

[lplazas]

Bug Report

What did you do?

I created two separate Elasticsearch resources, both with the exact same configuration except for the metadata.name field (elasticsearch-1 and elasticsearch-2). My use case is more complex but I managed to boil it down to this simple example, in which I create master and data nodes and I state I want to route the http traffic only to the data nodes (see resource definition)

What did you expect to see?

The http service selector should be pointing just to the data nodes of its cluster.

What did you see instead? Under which circumstances?

The default http selectors, which are:

• common.k8s.elastic.co/type: elasticsearch

• elasticsearch.k8s.elastic.co/cluster-name: $cluster_name

Get overridden by the custom selector, which in this case is the same for both clusters, causing the http service to target pods from different Elasticsearch clusters.

This is easily fixed from the user side by appending the cluster name (or any other cluster unique value) to the value of the node-type label (in my example).

The default selectors shouldn't be overridden if a custom selector is defined, as the service is anyways managed by the operator, and the targeted pods will always be of type elasticsearch and be part of the same elasticsearch cluster.

If considered as a good first-time-issue, I would be willing to start contributing with this.

Environment

• ECK version: 1.0

• Kubernetes information:

  • EKS
  • Kubernetes version: 1.14
  • kubectl version: 1.14

Resource definition

apiVersion: elasticsearch.k8s.elastic.co/v1beta1
kind: Elasticsearch
metadata:
  name: elasticsearch-1
spec:
  version: 7.4.0
  nodeSets:
    - name: master
      config:
        node.master: true
        node.data: false
        node.ingest: false
        node.ml: false
        node.store.allow_mmap: false
      podTemplate:
        metadata:
          labels:
            node-type: master
        spec:
          containers:
            - name: elasticsearch
              resources:
                limits:
                  memory: 2Gi
                  cpu: 1
              env:
                - name: ES_JAVA_OPTS
                  value: "-Xms1g -Xmx1g"
      count: 1
    - name: data
      config:
        node.master: false
        node.data: true
        node.ingest: false
        node.ml: false
        node.store.allow_mmap: false
      podTemplate:
        metadata:
          labels:
            node-type: data
        spec:
          containers:
            - name: elasticsearch
              resources:
                limits:
                  memory: 2Gi
                  cpu: 1
              env:
                - name: ES_JAVA_OPTS
                  value: "-Xms1g -Xmx1g"
      count: 3
  http:
    service:
      spec:
        type: LoadBalancer
        selector:
          node-type: data
        loadBalancerSourceRanges:
          - 64.69.69.69/32

[anyasabo]

We've tended towards not blacklisting certain settings in the past. For this specific use case I'm not sure though. I think we have a few options

• Leave it as is. people can override the service with selectors they'd like. This is a footgun, but is simple to understand the behavior

• Blacklist the selector as suggested in the initial issue. We could reject it at admission.

• Merge the selector with our default selectors. This is more complicated but might be useful. I'm also not sure of all the implications. I think it will cause some functionality like the prestop hook which expects all pods to be in the endpoints.

For what it's worth, for use cases like this in the past we have recommended creating a new service pointing to the data node labels rather than using the one the operator creates.

[Battleroid]

Merged with the defaults would be nice, as it's a bit odd that for a larger cluster the masters are tossed in the default http endpoint with the swath of data nodes.