search

kafkaesque-io / pulsar-helm-chart

Helm Chart for an Apache Pulsar Cluster
https://helm.kafkaesque.io
Apache License 2.0
31 stars 22 forks source link

Broker and Bookkeeper constantly crashing #63

Closed haris-zynka closed 4 years ago

haris-zynka commented 4 years ago

Hi, I'm using your helm chart for deploying test environment to Azure Kubernets, so I do not use persistence on Azure nor Anti Affinity. I'm unaware if those values case the problem but each time I deploy Pulsar, I use it a bit with my couple of services which use persistent topics and then it crashes. Currently I'm in a phase where I cannot even run the thing without it crashing instantly. I removed whole AKS so I can't get logs now. I'll try with AA and persistence to check if it will appear again but if anyone can advise me is it possible that my settings are causing it or maybe I need stronger VMs than Standard_B2s?

cdbartholomew commented 4 years ago

In AKS we run developer setups on Standard_DS2_v2, which has more memory than Standard_B2s (7 vs 4 GB) but the same number of CPUs (2).

haris-zynka commented 4 years ago

In AKS we run developer setups on Standard_DS2_v2, which has more memory than Standard_B2s (7 vs 4 GB) but the same number of CPUs (2).

Thanks, I did setup with DS2_v2 and I'll see how much it gets crashing or if it crashes at all. I did suspect that lack of RAM or poorer CPU might be effecting it and I think it will show that 4GB per machine is just not enough to run this.

cdbartholomew commented 4 years ago

Also, not using persistence will cause issues if BookKeeper pods crash. If they lose all their state, they will not recover gracefully. You will have to delete and recreate the Helm release (at least that's the easiest way to recover).

haris-zynka commented 4 years ago

Also, not using persistence will cause issues if BookKeeper pods crash. If they lose all their state, they will not recover gracefully. You will have to delete and recreate the Helm release (at least that's the easiest way to recover).

OK, thanks a lot. I will close the issue now as I believe that frist time it crashed it was out of memory and then it could not recover as it had no state to recover from while clients were trying to reconnect with backoff 100ms to 6s.