search

grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.95k stars 3.45k forks source link

LogCli: Ability to query logs without Loki server #3393

Open kavirajk opened 3 years ago

kavirajk commented 3 years ago

Is your feature request related to a problem? Please describe. Say I have "existing" file with several log lines, And I need to query the file via LogQL (I know we can use awk, sed, grep but I love LogQL more :) )

Describe the solution you'd like Support LogCli to read log lines from the stdin.

Describe alternatives you've considered Separate query tool to analyze existing log files without connecting to Loki server

Additional context LogQl is very powerful. Let's embrace it to use it on any log lines in the format labels + timestamp. Doesn't have to query only Loki server.

Some of the immediate use cases.

  1. Embrace logql as a text process tool for data in the keys=vals format

    cat mylogs | logcli query '{project="mypie", error=~"restart"}| json | reason_message ~= "censor-failed"'

  2. I can play with LogQL before actually using it in Grafana.

    echo "a,b,c,d" | logcli query "{} | regexp (?P<a>\w,) (?P<b\w,)"

  3. easy to {ask question about LogQL, experiment, show demo, etc.} without overhead of setting up Loki and promtail.

  4. I ❤️ LogQL

Possible issues:

  1. Currently LogQL query Loki, which treats data as labels + timestamp. But if we take logs from stdin, there are no timestamps. Potential Solution: We use timestamp for ordering. may be assume ordering as the order of log lines itself?? even we can treat timestamp from 0,1,2,3... to each log lines?
jeschkies commented 3 years ago

How about supporting --addr=file://my/file/location.log as well?

kavirajk commented 3 years ago

@jeschkies I like the idea. Default can be from stdout if --addr is not provided.

Also small additional notes

  1. Should be able to accept multiple --addr? for multiple files?
  2. Should be able to pass both directory and file to --addr.
jeschkies commented 3 years ago

I would use wildcards pattern know from gitignore and ls. So --addr=file:///var/logs/*.log or --addr=file:///var/logs/**.

vlad-diachenko commented 1 year ago

@kavirajk can we close the issue? it looks like the feature is already implemented

kavirajk commented 1 year ago

@vlad-diachenko It only implemented for log queries not for metric queries yet. I would recommend to keep it open for now.