search

GilbN / geoip2influx

A python script that will parse the nginx access.log and send geolocation metrics and log metrics to InfluxDB
MIT License
73 stars 15 forks source link

nginx_access_logs measurement not going to influx #2

Closed MervisDiamonds closed 3 years ago

MervisDiamonds commented 3 years ago

I'm having trouble shipping the nginx_access_logs measurement to influx. It is likely a configuration error but I'm not having any success at the moment. My current config is blow.

version: "2.1" services: geoip2influx: image: gilbn/geoip2influx container_name: geoip2influx environment:

  • PUID=1000
  • PGID=1000
  • TZ=America/New_York
  • INFLUX_HOST=xxx.xxx.xx.xx
  • INFLUX_HOST_PORT=xxx
  • MAXMINDDB_LICENSE_KEY=xxxxx
  • INFLUX_DATABASE=docker1
  • GEOIP2INFLUX_LOG_LEVEL=debug
  • SEND_NGINX_LOGS=true
  • LOG_MEASUREMENT=nginx_access_logs volumes:
  • /home/castanets/geo/:/config
  • /var/log/nginx/:/config/log/nginx/

Thank you for developing an excellent tool!

GilbN commented 3 years ago

What does the logs say?

MervisDiamonds commented 3 years ago

The log/nginx folder is empty. The geoip2influx.log is as follows:

19/Oct/2020 15:53:17 :: INFO :: Starting geoip2influx.. 19/Oct/2020 15:53:17 :: DEBUG :: Variables set: geoip_db_path :: /config/geoip2db/GeoLite2-City.mmdb -e LOG_PATH :: /config/log/nginx/access.log -e INFLUX_HOST :: xxx.xxx.xx.xx -e INFLUX_HOST_PORT :: xxxx -e INFLUX_DATABASE :: docker1 -e INFLUX_RETENTION :: 7d -e INFLUX_SHARD :: 1d -e INFLUX_USER :: xxxx -e INFLUX_PASS :: xxxx -e GEO_MEASUREMENT :: geoip2influx -e LOG_MEASUREMENT :: nginx_access_logs -e SEND_NGINX_LOGS :: true -e GEOIP2INFLUX_LOG_LEVEL :: DEBUG 19/Oct/2020 15:53:17 :: DEBUG :: Found: /config/log/nginx/access.log 19/Oct/2020 15:53:17 :: DEBUG :: Found: /config/geoip2db/GeoLite2-City.mmdb 19/Oct/2020 15:53:17 :: DEBUG :: Testing InfluxDB connection 19/Oct/2020 15:53:17 :: DEBUG :: Starting new HTTP connection (1): xxx.xxx.xxx.xx:xxxx 19/Oct/2020 15:53:17 :: DEBUG :: Starting new HTTP connection (1): xxx.xxx.xxx.xx:xxxx 19/Oct/2020 15:53:17 :: DEBUG :: http://xxx.xxx.xxx.xx:xxxx "GET /ping HTTP/1.1" 204 0 19/Oct/2020 15:53:17 :: DEBUG :: Influxdb version: 1.8.3 19/Oct/2020 15:53:17 :: DEBUG :: xxx.xxx.xxx.xx:xxxx "GET /query?q=SHOW+DATABASES&db=docker HTTP/1.1" 200 None 19/Oct/2020 15:53:17 :: DEBUG :: Found database: docker1 19/Oct/2020 16:41:50 :: DEBUG :: Testing regex on: /config/log/nginx/access.log 19/Oct/2020 16:41:52 :: DEBUG :: Testing regex on: /config/log/nginx/access.log 19/Oct/2020 16:41:54 :: DEBUG :: Testing regex on: /config/log/nginx/access.log 19/Oct/2020 16:41:56 :: DEBUG :: Testing regex on: /config/log/nginx/access.log 19/Oct/2020 16:41:58 :: DEBUG :: Testing regex on: /config/log/nginx/access.log 19/Oct/2020 16:42:00 :: DEBUG :: Testing regex on: /config/log/nginx/access.log 19/Oct/2020 16:42:02 :: DEBUG :: Testing regex on: /config/log/nginx/access.log 19/Oct/2020 16:42:04 :: DEBUG :: Testing regex on: /config/log/nginx/access.log 19/Oct/2020 16:42:06 :: DEBUG :: Testing regex on: /config/log/nginx/access.log 19/Oct/2020 16:42:08 :: DEBUG :: Testing regex on: /config/log/nginx/access.log 19/Oct/2020 16:42:10 :: DEBUG :: Testing regex on: /config/log/nginx/access.log 19/Oct/2020 16:42:12 :: DEBUG :: Testing regex on: /config/log/nginx/access.log 19/Oct/2020 16:42:14 :: WARNING :: Failed to match regex on: /config/log/nginx/access.log 19/Oct/2020 16:42:14 :: WARNING :: NGINX log metrics disabled! Double check your NGINX custom log format.. 19/Oct/2020 16:42:14 :: INFO :: Starting log parsing 19/Oct/2020 16:42:15 :: DEBUG :: Geo metrics: [{'tags': {'geohash': 'drt3n0792jyb', 'ip': '18.27.197.252', 'host': '97db8bc6e3ef', 'country_code': 'US', 'country_name': 'United States', 'state': 'Massachusetts', 'state_code': 'MA', 'city': 'Cambridge', 'postal_code': '02139', 'latitude': 42.3649, 'longitude': -71.0987}, 'fields': {'count': 1}, 'measurement': 'geoip2influx'}]

Below is the access.log line:

18.27.197.252 - user34 [19/Oct/2020:16:36:14 -0400]"GET /signalr/ping?apiKey=xxxxxx HTTP/1.1" 200 30"https://foobar.com" "Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0""0.011" "0.012""Cambridge" "US"

GilbN commented 3 years ago

The log/nginx folder is empty. The geoip2influx.log is as follows:

19/Oct/2020 16:42:12 :: DEBUG :: Testing regex on: /config/log/nginx/access.log 19/Oct/2020 16:42:14 :: WARNING :: Failed to match regex on: /config/log/nginx/access.log 19/Oct/2020 16:42:14 :: WARNING :: NGINX log metrics disabled! Double check your NGINX custom log format..

It says your custom nginx log format is wrong. https://github.com/gilbN/geoip2influx#sending-nginx-log-metrics

MervisDiamonds commented 3 years ago

Undone by a rogue space in my log format! I thought I copied directly...guess not. Thanks!