search

grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
22.76k stars 3.31k forks source link

Improve json parser performance #4328

Open cyriltovena opened 2 years ago

cyriltovena commented 2 years ago

Investigate if we can replace the current jsoniter parser with https://github.com/buger/jsonparser

I already used it for parsing json payload on the push side and it way better for avoiding alloc.

cyriltovena commented 2 years ago

When it's a metrics queries we could even use requiredLabels to only seek in json only key we need.

See https://github.com/grafana/loki/blob/main/pkg/logql/log/parser_hints.go#L17 The interface might need to expose the list though.

cyriltovena commented 2 years ago
stale[bot] commented 2 years ago

Hi! This issue has been automatically marked as stale because it has not had any activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project. A stalebot can be very useful in closing issues in a number of cases; the most common is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task, our sincere apologies if you find yourself at the mercy of the stalebot.

jeschkies commented 2 years ago

I was about to pitch simdjson. Do we have a dataset?

splitice commented 2 years ago

Simdjson-go looks as good as any and far better than the current?

stale[bot] commented 2 years ago

Hi! This issue has been automatically marked as stale because it has not had any activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project. A stalebot can be very useful in closing issues in a number of cases; the most common is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task, our sincere apologies if you find yourself at the mercy of the stalebot.

darxriggs commented 2 years ago

still relevant

shaunmansell commented 1 year ago

We've been trying Loki with services that log JSON and are seeing significant performance differences between queries with and without the JSON parser. We're seeing orders of difference in performance very similar to what's in https://github.com/grafana/loki/issues/5405.

Has there been any progress on testing the JSON parser switch in this issue?

wtchangdm commented 12 months ago

Hi, is there any update? This seems helpful in terms of performance