Support joins on LogQL

[vjsamuel]

What would you like to be added: Currently LogQL has robust features in being able to do counting and various aggregations on metric fields in the log lines. On PromQL, currently we have the ability to use state metrics to do joins so that metrics can be enriched at query time with additional labels that describe the source. It would be beneficial if we could do the same with LogQL.

Why is this needed: There can be external labels that define the log line more than just the labels that are coming with the log line. being able to joins allows a more robust querying experience.

https://www.section.io/blog/prometheus-querying/ provides a good read on how it is being used on PromQL today.

[vjsamuel]

Incorrectly raised in Grafana in the past: https://github.com/grafana/grafana/issues/29214

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[clouedoc]

I'm interested in this feature as well.

job runner

The job runner receives a job that contains a correlationId (string with random letters)

It publishes debug information to Loki.

Ex:

Jobmaster

The job master is the only one that knows if the job succeeded.

It publishes the job result (failure/success) along with the correlation Id:

What I want

I want to be able to know which IPs are failing the task

Workaround

The current workaround is to query the debug information inside the task master when the job finishes.

[jeschkies]

Joins definitely have value. However, we debated in the past if Loki is the tool to do these as we want to keep Loki scalable.

As I am keen to make it easier to export and import Logs. What would you think about joining or enriching logs externally through something like Spark, Kafka or server side expressions in Grafana?

[clouedoc]

Hi, I don't need joins anymore since I solved my issue using Postgres instead. That being said, I'll keep your answer in mind and try server-side expressions when needing a JOIN.

[jeschkies]

@clouedoc does it mean you've exported the logs to Postgres?

[clouedoc]

does it mean you've exported the logs to Postgres?

@jeschkies No, I used a Node PostgreSQL client to manually send the data to my PostgreSQL database.

That being said, it would've been helpful to export Loki logs to PostgreSQL. I don't know what the idiomatic way to do this is.

[jeschkies]

That being said, it would've been helpful to export Loki logs to PostgreSQL. I don't know what the idiomatic way to do this is.

There is none. However, I'm collecting use cases to build up a case.

[maxdialpad]

+1 on having the ability to do a join on a query. Here's an example of where I'd use it. We have log events that can contain two IDs, callId and traceId. I want to create a dashboard where the only input is a callId or even better, multiple call IDs, and have that return all logs that 1. contain that callId and 2. contain the traceId from all the logs in item 1.

Simple log example:

{
  environment: prod
  textPayload: callId = 36c550e2-31cb-43b6-98a7-accb72ac3105
  traceId: e94e6f6ab012cee08aca903b496d6355
}

{
  environment: prod
  textPayload:  Current user is abc123
  traceId: e94e6f6ab012cee08aca903b496d6355
}
{
  environment: prod
  textPayload:  Some other interesting message that does not contain a callId
  traceId: e94e6f6ab012cee08aca903b496d6355
}

Today, I would have to do this manually. I might first create a query like

{environment="prod"}
|= "36c550e2-31cb-43b6-98a7-accb72ac3105"
| json
| line_format "{{.textPayload}}"

Then gather all of the traceIds that show up, and then switch the query to search for those:

{environment="prod"}
|= "e94e6f6ab012cee08aca903b496d6355"
| json
| line_format "{{.textPayload}}"

If a join operator was available in LogQL, I think this could be done with a single query.

[soh-tetsu]

Vote for the request.

[Phara0h]

Looking to get all logs based on a search and then get all the logs from that results traceIds

[nadeda01]

Vote for Loki Joins..... without joins - what is the point of the data repo.

[nadeda01]

Still wishing for this - at this point I need to extract the logs and then submit another query based on output - I feel like I am back in the days of punch cards. I know there are better tools for this - but this tool is being pushed on us from the top.

[MartinEmrich]

+1, Again with the common use case of joining logs of different (micro-)services by a common "correlation id", "request id" or "trace id".

I guess usually either

• The time window where the same ID needs to be kept ist short (tracking synchronous API calls through multiple service levels, like a poor-mans tracing).
• Or the amount of events is rather small (like tracking aynchronous jobs from producer to consumer).

Would love to see such an option.

I could imagine forming such a query in the form of multiple subqueries, all ending up with a | joinlabel <label-with-my-id>, and then specifiying a reasonable working time window to keep and aggregate events with the same ID. it could then output single log events with all labels merged for each ID, timestamped with the first (or last) occurrence.