GCP internal Ingress error with UNHEALTHY opensearch backends

[vmm-afonso]

I'm deploying an opensearch cluster on gke using the helm charts. I'm currently facing this problem where my Ingress is failing bcs the load balancer gke deploys requires a healthcheck but Opensearch won't return a status code 200 and instead is returning a 401.

This I believe might be because opensearch won't allow for unauthenticated health checks. If for example I open a shell session for an opensearch pod and run the following command: curl -vv localhost:9200/_cat/health -u 'user:pass' ; it returns the state 200 that I'm looking for, the problem is, the same command without passing a credential will return a 401.

At this point I've tried many things, I'm fairly certain that it's not a misconfiguration of my ingress or service or any other resource, but as I suggested above, the inability to query opensearch and get a state 200 response without authentication.

Is there any way to get around this?

I found that for opensearch-dashboards, it's possible to add the config "opensearch_security.auth.unauthenticated_routes: ['/api/stats']" to opensearch_dashboards.yaml. With this config I'm able to 'curl -vv localhost:5601/api/stats' from inside an opensearch-dashboards pod and It will return a response.

Is there any similar config for opensearch.yaml?

[vmm-afonso]

I found this blog where the author wrote the following:

"GKE deploys a Load Balancer with the ingress controller to route and manage the internal traffic. This Load Balancer requires a health check to function. One problem though, when auth is enabled, Elasticsearch returns a 401 instead of the required 200. They do not allow a TCP health check either."

I think the same applies here for opensearch, tho unfortunately after some testing I don't think the solution he provided works in this case.

Source: https://medium.com/@domainadmin/installing-elasticsearch-on-gke-with-the-bitnami-helm-chart-part-1-a8c24686b7e1

[prudhvigodithi]

[Untriage] Hey @vmm-afonso thanks, what do you think would be the proposed fix?

[GarryOne]

It didn't work with the GCP / GKE default ingress. But it did work instead with the nginx-ingress instead.

So, I installed Kibana using the helm chart:

helm install kibana elastic/kibana -n elastic-stack

Then, I've created kibana-ingress.yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kibana-ingress
  namespace: elastic-stack
spec:
  ingressClassName: "nginx"  # Specify NGINX Ingress Class
  tls:
  - hosts:
    - "kibana.test.com"
    secretName: cloudflare-origin-cert
  rules:
  - host: "kibana.test.com"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: kibana-kibana
            port:
              number: 5601

Then I run kubectl apply -f kibana-ingress.yaml

Then, you can run kubectl get svc -n ingress-nginx to get the External IP which you can use to point the DNS record to.