grok regex help - Githubissues

senattcsgit commented 3 years ago

Hello All- I have bunch of rest endpoint URIs with order numbers in it and they are quickly filling up my prometheus.

for e.g /order/status/101 , /order/status/102

and if I try to count the invocations on that URI- each order is a unique record for prometheus and filling it up. How do I just read until /order/status?

fstab commented 3 years ago

Please post your configuration and an example log line so I can have a look.

senattcsgit commented 3 years ago

here you go Log lines: 100.13.33.20 - - [26/Jan/2021:05:53:34 +0000] "GET /api/services/v1/accounts/8327522766/change HTTP/1.1" 200 975 "https://www.shop.dummy.com/shop/phones/details/Samsung/xxxxx" "Mozilla/5.0 (Linux; Android 10; LM-K500) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.4147.125 Mobile Safari/537.36" 0.125 "0.000" "0.122" "0.122" "200" "20503980" "3" "1.2.3.4:443" "www.shop.dummy.com"

100.13.33.20 - - [26/Jan/2021:05:53:35 +0000] "GET /api/services/v1/accounts/10075227667/change HTTP/1.1" 200 975 "https://www.shop.dummy.com/shop/phones/details/Samsung/xxxxx" "Mozilla/5.0 (Linux; Android 10; LM-K500) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Mobile Safari/537.36" 0.125 "0.000" "0.122" "0.122" "200" "20503980" "3" "1.2.3.4:443" "www.shop.dummy.com"

My goal is count no of invocations for /api/services/v1/accounts API.

pattern: %{IP:remote_addr} %{NOTSPACE:random1} %{NOTSPACE:remote_user} [%{HTTPDATE:time}] \"%{NOTSPACE:method} %{URIPATH:uri} %{NOTSPACE:protocol}\" %{INT:status} %{INT:body_bytes_sent} %{NOTSPACE:http_referer} \"%{DATA:http_user_agent}\" %{NOTSPACE:request_time} \"%{NOTSPACE:upstream_connect_time}\" \"%{NOTSPACE:upstream_header_time}\" \"%{NOTSPACE:upstream_response_time}\" \"%{NOTSPACE:upstream_status}\" \"%{NOTSPACE:connection}\" \"%{NOTSPACE:connection_requests}\" \"%{NOTSPACE:upstream_addr}\" \"%{NOTSPACE:http_host}\"

senattcsgit commented 3 years ago

global: config_version: 2 input: type: file path: /opt/app/logs/access.log readall: false # Read from the beginning of the file? False means we start at the end of the file and read only new lines. grok: patterns_dir: /opt/app/grok/patterns metrics:

type: counter name: upstream_invocations help: Upstream Invocations. match: '%{NGNIX}' labels: uri: '{{.uri}}' domain: '{{.http_host}}' job: NGNIX
type: counter name: upstream_status help: Upstream Response Status Code. match: '%{NGNIX}' labels: uri: '{{.uri}}' domain: '{{.http_host}}' job: NGNIX status: '{{.status}}'
type: gauge name: total_resp_time help: Ngnix Time spent waiting for a full HTTP request from the client. match: '%{NGNIX}' value: '{{.request_time}}' labels: uri: '{{.uri}}' domain: '{{.http_host}}' job: NGNIX
type: gauge name: upstream_resp_time help: Ngnix total time spent waiting for the upstream server to send a full HTTP response. match: '%{NGNIX}' value: '{{.upstream_response_time}}' labels: uri: '{{.uri}}' domain: '{{.http_host}}' job: NGNIX
type: summary name: total_latency help: Total Latency stats. match: '%{NGNIX}' value: '{{.request_time}}' labels: uri: '{{.uri}}' domain: '{{.http_host}}' job: NGNIX
type: summary name: upstream_latency help: Upstream Latency stats. match: '%{NGNIX}' value: '{{.upstream_response_time}}' labels: uri: '{{.uri}}' domain: '{{.http_host}}' job: NGNIX server: host: 0.0.0.0 port: 9142

senattcsgit commented 3 years ago

@fstab any help on this? Appreciate any help on this topic

senattcsgit commented 3 years ago

Any help on this topic? I tried several things , but its increasing my grok's CPU usage

senattcsgit commented 3 years ago

ANy help?

fstab / grok_exporter

grok regex help #147