search

datastax / pulsar-helm-chart

Apache Pulsar Helm chart
Apache License 2.0
46 stars 38 forks source link

Istio service mesh compatibility #147

Open robparrott opened 2 years ago

robparrott commented 2 years ago

Currently the manner in which jobs start in this helm chart interferes with the use of an istio service mesh.

However, pulsar otherwise would be very compatible with a service mesh and this would reduce complexity for mTLS and ingress routing to a separate aspect of the environment, which is often desirable.

This issue can be reproduced as follows with a basic istio installation. From a new installation:

kubectl create namespace pulsar
kubectl label namespace pulsar istio-injection=enabled
helm upgrade --install  \
   pulsar  datastax-pulsar/pulsar \
  --namespace pulsar \
  --create-namespace

This will result in a set of initialization jobs that are hung up.

If instead we disable istio in the namespace kubectl label namespace pulsar istio-injection-, install datastax luna pulsar via the helm chart, then re-eanble istio, and cycle the various pods that take place in the data plane, the system works as expected.

This indicates that a set of tweaks to the jobs ( e.g. by setting pod labels sidecar.istio.io/inject=false ) may make this chart compatible with istio.

Jobs observed:

These jobs will not reach a completed state unless the associated istio-proxy container exits successfully (i.e. shell into the istio-proxy container and run kill -TERM 1). An alternative approach is to add a preStop condition to the main container to call curl -sf -XPOST http://127.0.0.1:15020/quitquitquit which will tell the istio-proxy to exit, or other ideas include a dedicated additional sidecar to manage this. See https://discuss.istio.io/t/best-practices-for-jobs/4968/2 for more.

robparrott commented 2 years ago

And as a quick additional note, it appears the incompatibility is primarily from the prometheus components, so setting kube-prometheus-stack.enabled=false addresses two of the issues.

The zookeeper metadata job is still outstanding.

robparrott commented 2 years ago

The pulsar helm chart from the upstream project introduced the ability to inject a final command to address this gap.

Here's an example of that in action:

The PR for this is here, and was merged last month: