SigNoz is an open-source observability platform native to OpenTelemetry with logs, traces and metrics in a single application. An open-source alternative to DataDog, NewRelic, etc. 🔥 🖥. 👉 Open source Application Performance Monitoring (APM) & Observability tool
The config value of MaxExecutionTime in query Service isn't being correctly set in the passed query to clickhouse database and it's still hardcoded to 64 seconds. This results in queries being aborted after 64 seconds. In both scenarios - clickhouse provided by SigNoz or external clickhouse setup on separate host.
The setting:
additionalEnvs: ClickHouseMaxExecutionTime: "3600"
Actual query information from clickhouse system.query_log shows for SELECT queries:
{'connect_timeout_with_failover_ms':'1000','distributed_aggregation_memory_efficient':'1','log_queries':'1','max_execution_time':'64','log_comment':'{"alertID":"","client":"api","dashboardID":"","path":"","servicesTab":"","source":"","viewName":""}','parallel_view_processing':'1','allow_nondeterministic_mutations':'1','allow_experimental_window_functions':'1','default_database_engine':'Ordinary'}
Setting ClickHouseMaxExecutionTimeLeaf variable doesn't help - the query is being aborted after 64 seconds as max_execution_time has higher priority then max_execution_time_leaf setting:
{'connect_timeout':'3600','receive_timeout':'3600','send_timeout':'3600','load_balancing':'random','max_execution_time':'64','max_execution_time_leaf':'3600','max_memory_usage':'10000000000','log_comment':'{"alertID":"","client":"browser","dashboardID":"ea6d4f10-9816-4812-9867-44752708e296","email":"","path":"/dashboard/ea6d4f10-9816-4812-9867-44752708e296","servicesTab":"","source":"dashboards","viewName":""}'}
Tested on K8S setup with clickhouse database on k8s pod (first example) and also with external clickhouse database set outside of K8S workers (second example).
Expected behavior
max_execution_time variable is being correctly passed to clickhouse database and the query doesn't get aborted after 64 seconds.
How to reproduce
Setup MaxExecutionTime to some value above 64
Run anything in GUI that produces a SELECT query (log viwer, dashboard)
Check clickhouse database system.query_log and filter for type = 'QueryFinish' and query_kind = 'Select'
Verify Settings: column and the value of max_execution_time variable
Version information
Signoz version: 0.53.0 (chart 0.51.1) and 0.58.2 (chart 0.56.2)
Bug description
The config value of MaxExecutionTime in query Service isn't being correctly set in the passed query to clickhouse database and it's still hardcoded to 64 seconds. This results in queries being aborted after 64 seconds. In both scenarios - clickhouse provided by SigNoz or external clickhouse setup on separate host.
The setting:
additionalEnvs: ClickHouseMaxExecutionTime: "3600"Actual query information from clickhouse system.query_log shows for SELECT queries:
{'connect_timeout_with_failover_ms':'1000','distributed_aggregation_memory_efficient':'1','log_queries':'1','max_execution_time':'64','log_comment':'{"alertID":"","client":"api","dashboardID":"","path":"","servicesTab":"","source":"","viewName":""}','parallel_view_processing':'1','allow_nondeterministic_mutations':'1','allow_experimental_window_functions':'1','default_database_engine':'Ordinary'}Setting
ClickHouseMaxExecutionTimeLeafvariable doesn't help - the query is being aborted after 64 seconds as max_execution_time has higher priority then max_execution_time_leaf setting:{'connect_timeout':'3600','receive_timeout':'3600','send_timeout':'3600','load_balancing':'random','max_execution_time':'64','max_execution_time_leaf':'3600','max_memory_usage':'10000000000','log_comment':'{"alertID":"","client":"browser","dashboardID":"ea6d4f10-9816-4812-9867-44752708e296","email":"","path":"/dashboard/ea6d4f10-9816-4812-9867-44752708e296","servicesTab":"","source":"dashboards","viewName":""}'}Tested on K8S setup with clickhouse database on k8s pod (first example) and also with external clickhouse database set outside of K8S workers (second example).
Expected behavior
max_execution_time variable is being correctly passed to clickhouse database and the query doesn't get aborted after 64 seconds.
How to reproduce
Version information
Additional context
The feature to add max_execution_time was open here: https://github.com/SigNoz/signoz/issues/5393 and merged to 0.53.0 release here: https://github.com/SigNoz/signoz/pull/5683
Thank you for your bug report – we love squashing them!