avito-tech / nginx-log-collector

nginx-log-collector
55 stars 11 forks source link

invalid json: cannot parse JSON #10

Closed Rodionc closed 4 years ago

Rodionc commented 4 years ago

Привет! Не знаю как тут лучше писАть, вроде все местные коллеги :) If needed, ready to provide English isuue report. Поставил\настроил\запустил - заработало, но в логах самого коллектора нижеприведённая ошибка... Подскажите, пожалуйста, где он пытается цифру найти, а видит букву и как с этим бороться?

Mar 4 00:27:27 savrus nginx-log-collector[11012]: {"level":"warn","component":"processor","host":"vm97373","error":"invalid json: cannot parse JSON: cannot parse number: expecting 0..9 digit, got e; unparsed tail: \"escape=json{\\"event_datetime\\": \\"2020-03-04T00:27:27+03:00\\", \\"server_name\\": \\"www.null.ru\\", \\"remote_addr\\": \\"144.76.68.17\\", \\"remote_user\\": \\"-\\", \\"http_x_real_ip\\": \\"-\\", \\"status\\": \\"200\\", \\"scheme\\": \\"http\\", \\"request_method\\": \\"GET\\", \\"request_uri\\": \\"/%D0%BC/%D1%83%D0%B0%D0%B7-patriot\\", \\"server_protocol\\": \\"HTTP/1.1\\", \\"body_bytes_sent\\": 6314, \\"http_referer\\": \\"-\\", \\"http_user_agent\\": \\"serpstatbot/1.0 (advanced backlink tracking bot; curl/7.58.0; http://serpstatbot.com/; abuse@serpstatbot.com)\\", \\"request_bytes\\": \\"458\\", \\"request_time\\": \\"0.457\\", \\"upstream_addr\\": \\"127.0.0.1:8080\\", \\"upstream_response_time\\": \\"0.457\\", \\"hostname\\": \\"vm97373.local\\", \\"host\\": \\"www.null.ru\\"}\"","time":"2020-03-04T00:27:27+03:00","message":"convert error"}

Настройки nginx.conf:

                         '{'
                         '"event_datetime": "$time_iso8601", '
                         '"server_name": "$server_name", '
                         '"remote_addr": "$remote_addr", '
                         '"remote_user": "$remote_user", '
                         '"http_x_real_ip": "$http_x_real_ip", '
                         '"status": "$status", '
                         '"scheme": "$scheme", '
                         '"request_method": "$request_method", '
                         '"request_uri": "$request_uri", '
                         '"server_protocol": "$server_protocol", '
                         '"body_bytes_sent": $body_bytes_sent, '
                         '"http_referer": "$http_referer", '
                         '"http_user_agent": "$http_user_agent", '
                         '"request_bytes": "$request_length", '
                         '"request_time": "$request_time", '
                         '"upstream_addr": "$upstream_addr", '
                         '"upstream_response_time": "$upstream_response_time", '
                         '"hostname": "$hostname", '
                         '"host": "$host"'
                         '}';```

Настройки nginx-log-collector:
```processor:
  workers: 8
receiver:
  addr: 0.0.0.0:4444
logging:
  level: debug
statsd:
  #  prefix: resources.monitoring.nginx_log_collector
  prefix: complex.delete_me.nginx_log_collector
  addr: localhost:2003
  enabled: false
pprof:
  enabled: true
  addr: 0.0.0.0:6060
backlog:
  dir: /tmp/backlog
collected_logs:
- tag: "nginx:"
  format: access  # access | error
  buffer_size: 104857600
  transformers:  # possible functions: ipToUint32 | limitMaxLength(int) | toArray | splitAndStore
    remote_addr:
      ipToUint32:
   # http_x_real_ip:
   #   ipToUint32:
    upstream_response_time:
      toArray:
    http_referer:
      limitMaxLength: 800
    request_uri:
      splitAndStore:
        delimiter: "?"
        store_to:
          request_uri: 0
          request_args: 1
  upload:
    table: nginx.access_log
    dsn: http://localhost:8123/
- tag: "nginx_error:"
  format: error  # access | error
  buffer_size: 1048576
  upload:
    table: nginx.error_log
    dsn: http://localhost:8123/
kolobaev commented 4 years ago

Первое что приходит в голову это что статус "200" должен быть числом.

Rodionc commented 4 years ago

Не помогло: убрал кавычки со всех числовых параметров, ничего не изменилось. Ошибка: Mar 4 09:36:01 savrus nginx-log-collector[13568]: {"level":"warn","component":"processor","host":"vm97373","error":"invalid json: cannot parse JSON: cannot parse number: expecting 0..9 digit, got e; unparsed tail: \"escape=json{\\"event_datetime\\": \\"2020-03-04T09:36:01+03:00\\", \\"server_name\\": \\"www.null.ru\\", \\"remote_addr\\": \\"54.36.150.83\\", \\"remote_user\\": \\"-\\", \\"http_x_real_ip\\": \\"-\\", \\"status\\": 200, \\"scheme\\": \\"http\\", \\"request_method\\": \\"GET\\", \\"request_uri\\": \\"/img/2014/140301_italy/pic_small/pic_038.jpg\\", \\"server_protocol\\": \\"HTTP/1.1\\", \\"body_bytes_sent\\": 86643, \\"http_referer\\": \\"-\\", \\"http_user_agent\\": \\"Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)\\", \\"request_bytes\\": 209, \\"request_time\\": 0.074, \\"upstream_addr\\": \\"-\\", \\"upstream_response_time\\": -, \\"hostname\\": \\"vm97373.local\\", \\"host\\": \\"null.ru\\"}\"","time":"2020-03-04T09:36:01+03:00","message":"convert error"}

Настройки nginx.conf:

                         '{'
                         '"event_datetime": "$time_iso8601", '
                         '"server_name": "$server_name", '
                         '"remote_addr": "$remote_addr", '
                         '"remote_user": "$remote_user", '
                         '"http_x_real_ip": "$http_x_real_ip", '
                         '"status": $status, '
                         '"scheme": "$scheme", '
                         '"request_method": "$request_method", '
                         '"request_uri": "$request_uri", '
                         '"server_protocol": "$server_protocol", '
                         '"body_bytes_sent": $body_bytes_sent, '
                         '"http_referer": "$http_referer", '
                         '"http_user_agent": "$http_user_agent", '
                         '"request_bytes": $request_length, '
                         '"request_time": $request_time, '
                         '"upstream_addr": "$upstream_addr", '
                         '"upstream_response_time": $upstream_response_time, '
                         '"hostname": "$hostname", '
                         '"host": "$host"'
                         '}';
Rodionc commented 4 years ago

Не сразу, но проблему удалось решить:

root@vm97373:~# cat /etc/issue
Ubuntu 16.04.6 LTS \n \l

root@vm97373:~# nginx -v
nginx version: nginx/1.10.3 (Ubuntu)

apt install software-properties-common
add-apt-repository ppa:nginx/stable
apt update
apt install nginx

nginx -v
nginx version: nginx/1.16.1

vi /etc/nginx/nginx.conf
 '"http_x_real_ip": "$http_x_real_ip", ' ->  '"http_x_real_ip": "remote_addr", '