search

SigNoz / charts

Helm Charts for SigNoz
MIT License
77 stars 79 forks source link

how to configure clickhouse params #148

Open luizbafilho opened 1 year ago

luizbafilho commented 1 year ago

Hi folks, I'm having issues with clickhouse whenever there is an unexpected restart; it is exactly this issue: https://github.com/ClickHouse/ClickHouse/issues/41423, it seems that I need to set that param.

I tried doing:

clickhouse:
    settings:
         max_suspicious_broken_parts: 100

But that had no effect.

pranay01 commented 1 year ago

@luizbafilho Are you trying to deploy SigNoz in k8s via helm charts or in docker?

prashant-shahi commented 1 year ago 
clickhouse:
  settings:
    max_suspicious_broken_parts: 100

@luizbafilho I don't think above configuration is right.

  <merge_tree>
    <max_suspicious_broken_parts>5</max_suspicious_broken_parts>
  </merge_tree>

Above XML is right way. It should either be included as additional configuration under files section or manually create configmap and mount it under clickhouse configuration.

In both cases, chart changes would be required at the moment.

Either ways, I am not familiar with any unexpected restarts happening which could be resolved by this.

@srikanthccv @ankitnayan any idea about this particular scenario and the above solution?

luizbafilho commented 1 year ago

I'm using the chart

Just to be clear, that won't resolve the restart, it is the other way around, let's say I forced to restart the machine clickhouse goes into a crashloopback because it can't handle those broken parts, increasing that number allows clickhouse to gracefully handle those and move on.

A more in-depth explanation why that happens: https://kb.altinity.com/altinity-kb-setup-and-maintenance/suspiciously-many-broken-parts/

srikanthccv commented 1 year ago

@luizbafilho, you mentioned, you tried updating. Where did you do that?

srikanthccv commented 1 year ago

We need to add this to .spec.configuration.settings to take this effect.

luizbafilho commented 1 year ago

@luizbafilho, you mentioned, you tried updating. Where did you do that?

I'm not sure I follow the question, but, didn't do any updates, I just had to forcefully shutdown the node clickhouse was in.

srikanthccv commented 1 year ago

In the first comment description, you mentioned you tried doing ... I was asking how you updated that and where the spec config did you modify?

luizbafilho commented 1 year ago

I missed the last message; I did that in the helm chart values.yaml