Currently logfmt is parsed inside the JS application. If the |logfmt pipe is encountered in the script, the raw information is dowmloaded from the clickhouse server and the further processing is done on the JS side.
It is incredibly slow and inefficient.
It limits the ability to support some complicated scripts like the one represented in https://github.com/metrico/qryn/issues/444 : topk(10, count(count_over_time({kind="exception", app="$app"} | logfmt [$__range])) by (value))
Current state
Currently logfmt is parsed inside the JS application. If the
|logfmtpipe is encountered in the script, the raw information is dowmloaded from the clickhouse server and the further processing is done on the JS side.It is incredibly slow and inefficient.
It limits the ability to support some complicated scripts like the one represented in https://github.com/metrico/qryn/issues/444 :
topk(10, count(count_over_time({kind="exception", app="$app"} | logfmt [$__range])) by (value))On the other hand the new clickhouse versions have a function providing the mechanism of "native" logfmt parsing: https://clickhouse.com/docs/en/sql-reference/functions/tuple-map-functions#extractkeyvaluepairs
Requirements
Useful Links
https://altinity.com/blog/boosting-performance-and-flexibility-of-clickhouse-key-value-pair-extraction