search

fstab / grok_exporter

Export Prometheus metrics from arbitrary unstructured log data.
Apache License 2.0
898 stars 152 forks source link

Help with gork_exporter #70

Open sheevaeva opened 5 years ago

sheevaeva commented 5 years ago

Hello

I'm using grok exporter and here is what I want to achieve: I have a Java application whose log entry is in below format:

{"@version":1,"source_host":"fstest-stage-bm-62","message":"Known host file not configured, using user known host file: /home/.ssh/known_hosts","thread_name":"Camel (camel-1) thread #4 - aws-s3://fstest-stage-bm-62","@timestamp":"2019-08-28T07:52:12.526+00:00","level":"INFO","logger_name":"org.apache.cam.file.remote.oerations"}

I want to configure Prometheus alert for any 'ERROR' entry in the log level. Here is how the grok_exporter config.yml file look like:

global: config_version: 2 input: type: file path: ./example/test.log readall: true # Read from the beginning of the file? False means we start at the end of the file and read only new lines. grok: patterns_dir: ./patterns

metrics:

============================ The test log file has 4 log lines with one log line having log level as ERROR. I did try accessing http://IP:9144/metrics and I see the below but there is no metric created on Prometheus(grok_exporter is installed on Prometheus itself).

grok_exporter_line_processing_errors_total{metric="error_test"} 0

HELP grok_exporter_lines_matching_total Number of lines matched for each metric. Note that one line can be matched by multiple metrics.

TYPE grok_exporter_lines_matching_total counter

grok_exporter_lines_matching_total{metric="error_test"} 0

HELP grok_exporter_lines_processing_time_microseconds_total Processing time in microseconds for each metric. Divide by grok_exporter_lines_matching_total to get the average processing time for one log line.

TYPE grok_exporter_lines_processing_time_microseconds_total counter

grok_exporter_lines_processing_time_microseconds_total{metric="error_test"} 0

HELP grok_exporter_lines_total Total number of log lines processed by grok_exporter.

TYPE grok_exporter_lines_total counter

grok_exporter_lines_total{status="ignored"} 4 grok_exporter_lines_total{status="matched"} 0

I do see the metric on prometheus but ti doesn't yield any value. Can someone please help me with regex expression for my json log format as I couldn't get the correct matching format.

Thanks

fstab commented 5 years ago

There's something wrong with your match configuration, because it does not fit together with the format of your log line. If you just want to count the number of ERROR messages, the simplest thing you can do is as follows:

global:
  config_version: 2
input:
  type: file
  path: ./example/test.log
  readall: true # Read from the beginning of the file? False means we start at the end of the file and read only new lines.
grok:
  patterns_dir: ./patterns
metrics:
- type: counter
  name: error_test
  help: Counter metric example
  match: '"level":"ERROR"'
server:
  host: 0.0.0.0
  port: 9144
sheevaeva commented 5 years ago

Thank you Fabian for the response. The metric does match with the data I have added in the log file but i do not see the metric on Prometheus.

grok_exporter_lines_total{status="ignored"} 1 grok_exporter_lines_total{status="matched"} 1

I also tried assigning a label "severity" to the match string but that does not work either. Can you please take a look.

global: config_version: 2 input: type: file path: ./example/test.log readall: true # Read from the beginning of the file? False means we start at the end of the file and read only new lines. grok: patterns_dir: ./patterns metrics:

Thanks

On Fri, Sep 20, 2019 at 3:07 AM Fabian Stäber notifications@github.com wrote:

There's something wrong with your match configuration, because it does not fit together with the format of your log line. If you just want to count the number of ERROR messages, the simplest thing you can do is as follows:

global: config_version: 2input: type: file path: ./example/test.log readall: true # Read from the beginning of the file? False means we start at the end of the file and read only new lines.grok: patterns_dir: ./patternsmetrics:

  • type: counter name: error_test help: Counter metric example match: '"level":"ERROR"'server: host: 0.0.0.0 port: 9144

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/fstab/grok_exporter/issues/70?email_source=notifications&email_token=ANHIUPCXENY53CNWSDU2CPLQKPWINA5CNFSM4IX2CK3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7E4XTI#issuecomment-533318605, or mute the thread https://github.com/notifications/unsubscribe-auth/ANHIUPEYZMMNR2UR44NII73QKPWINANCNFSM4IX2CK3A .

fstab commented 5 years ago

i do not see the metric on Prometheus

If grok_exporterexposes the metric but you don't see it in Prometheus, then something with your Prometheus config is wrong. Could you post your Prometheus config?

And please, for better formatting in GitHub put a line with three ` characters before your yaml and a line with three ` characters after your yaml like this:

yaml here

sheevaeva commented 5 years ago

Here is how my preometheus config looks like :

 - job_name: 'grok'
    metrics_path: /grok
    static_configs:
      - targets:
          - IP:9144
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9144 # This exporter's real hostnane
fstab commented 5 years ago

Try removing metrics_path: /grok.

sheevaeva commented 5 years ago

Thanks Fabian, i will try this our shortly! I also need the "message" to be printed in case of "ERROR". How do i get this done in the grok config?

fstab commented 5 years ago

I think it's best to approach this step by step. Let's first try to see the metric in Prometheus without any labels, and then see where we can go from there. There are a few things to consider when using error messages as label values, such as cardinality explosion. You'll need to be careful with that.

sheevaeva commented 5 years ago

Thanks Fabian. I tried what you suggested and I do see the metric on Prometheus now. We now need to work on sending 'message' in the alert that is received. Appreciate your help.

Regards!

On Tue, Sep 24, 2019 at 2:43 AM Fabian Stäber notifications@github.com wrote:

I think it's best to approach this step by step. Let's first try to see the metric in Prometheus without any labels, and then see where we can go from there. There are a few things to consider when using error messages as label values, such as cardinality explosion. You'll need to be careful with that.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/fstab/grok_exporter/issues/70?email_source=notifications&email_token=ANHIUPH3KTSF5V25QDNIRMDQLEWOPA5CNFSM4IX2CK3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7MJGCQ#issuecomment-534287114, or mute the thread https://github.com/notifications/unsubscribe-auth/ANHIUPGBHPLRMBXKATW6UPDQLEWOPANCNFSM4IX2CK3A .

sheevaeva commented 5 years ago

Hi Fabian

Any update here please?

Thanks!

fstab commented 5 years ago

If you really want to use the error message as a label value you can do it like this:

global:
  config_version: 2
input:
  type: file
  path: ./test.log
  readall: true
grok:
  patterns_dir: ./patterns
  additional_patterns:
  - 'MESSAGE [^"]*'
metrics:
- type: counter
  name: error_test
  help: Counter metric example
  match: '(?=.*"level":"ERROR").*"message":"%{MESSAGE:message}"'
  labels:
    message: '{{.message}}'
server:
  host: 0.0.0.0
  port: 9144

However, you will get performance issues if you have many different log messages, because internally the Prometheus server will create a new time series for each message. Google "cardinality explosion" to learn more about this issue.

sheevaeva commented 5 years ago

Understood, Thanks Fabian.

sheevaeva commented 5 years ago

Hi Fabian, a quick question. Can you let me know if grok exporter is compatible with Prometheus version 2.3.2? It doesn't seem to working for me while it is fine with Prometheus version 2.9.1

sheevaeva commented 5 years ago

Hi Fabian Can you please let me knowif grok exporter is compatible with Prometheus version 2.3.2? It doesn't seem to be working fine for me , specially with alerting rules while it is fine with Prometheus version 2.9.1

fstab commented 5 years ago

Yes, it's compatible. There are no breaking changes between these Prometheus versions.

Alerting rules have nothing to do with grok_exporter anyway, grok_exporter is only focused on providing metrics, but not on alerting.

xuanyuanaosheng commented 4 years ago

@fstab Could you please add a grafana template? So we can use it, you can collect some basic log metrics like system log.