search

mintance / nginx-clickhouse

Simple nginx logs parser & transporter to ClickHouse database.
Apache License 2.0
149 stars 30 forks source link

nginx-clickhouse Can`t save logs: rows and cols cannot be empty #12

Open yangdongxiang opened 4 years ago

yangdongxiang commented 4 years ago

After I successfully compile normally, I use a binary program to start, the steps are as follows: ./nginx-clickhouse -config_path config.yml Returned INFO[0000] Reading config file: config-sample.yml INFO[0000] Trying to open logfile: /home/service/nginx/logs/host.access.log INFO[0005] Preparing to save 99 new log entries. ERRO[0005] Can`t save logs: rows and cols cannot be empty

My nginx-clickhouse configuration file

Settings

settings: interval: 5 # in seconds log_path: /home/service/nginx/logs/host.access.log # path to logfile seek_from_end: false # start reading from the lasl line (to prevent duplicates after restart)

ClickHouse credentials

clickhouse: db: metrics table: nginx host: localhost port: 8123 credentials: user: default password: columns: RemoteAddr: remote_addr RemoteUser: remote_user TimeLocal: time_local Request: request Status: status BytesSent: bytes_sent HttpReferer: http_referer HttpUserAgent: http_user_agent

NGINX

nginx: log_type: main log_format: $remote_addr-$remote_user [$time_local] "$request" $status $bytes_sent "$http_referer" "$http_user_agent"

my clickhouse talbe schema ip-172-31-5-250.ap-northeast-1.compute.internal :) desc nginx

DESCRIBE TABLE nginx

┌─name─────────────────┬─type───────────┬─default_type─┬─default_expression─┬─comment─┬─codec_expression─ ┬─ttl_expression─┐ │ RemoteAddr │ String │ │ │ │ │ │ │ RemoteUser │ String │ │ │ │ │ │ │ TimeLocal │ DateTime │ │ │ │ │ │ │ Date │ Date │ DEFAULT │ toDate(TimeLocal) │ │ │ │ │ Request │ String │ │ │ │ │ │ │ RequestMethod │ String │ │ │ │ │ │ │ Status │ Int32 │ │ │ │ │ │ │ BytesSent │ Int64 │ │ │ │ │ │ │ HttpReferer │ String │ │ │ │ │ │ │ HttpUserAgent │ String │ │ │ │ │ │ │ RequestTime │ Float32 │ │ │ │ │ │ │ UpstreamConnectTime │ Float32 │ │ │ │ │ │ │ UpstreamHeaderTime │ Float32 │ │ │ │ │ │ │ UpstreamResponseTime │ Float32 │ │ │ │ │ │ │ Https │ FixedString(2) │ │ │ │ │ │ │ ConnectionsWaiting │ Int64 │ │ │ │ │ │ │ ConnectionsActive │ Int64 │ │ │ │ │ │ └──────────────────────┴──────────────────┴───────── ─────┴────────────────────┴─────────┴─────────────── ─────┴────────────────┘

17 rows in set. Elapsed: 0.002 sec.

Hope for your help~

peterborodatyy commented 3 years ago

Hi, can you send your logs file?