How does output to loki work?

[KVInventoR]

Hi all,

I need help with configuration for this wonderful project. I have simple config:

chsize: 1000

input:
  - type: file
    path: /logging/test_input.log
    start_position: end
    sincedb_write_interval: 5

filter:
  - type: grok
    match: ["%{IPTABLES_SRC}"]
    source: "message"
    patterns_path: "/iptables.grok"

  - type: geoip2
    db_path: "/GeoLite2-City.mmdb"
    ip_field: "src_ip"
    cache_size: 100000
    key: geoip
    flat_format: true

  - type: add_field
    key: 'geo2ip_city'
    value: '%{geoip.country_code}'

output:
  - type: stdout
    codec: json

  - type: loki
    urls:
      - "http://mloki.fr.com.ua:3100/loki/api/v1/push"

 echo 'May 11 10:02:05 zabbix kernel: [1609112.875635] FW_F_IN_DROP: IN=ens18 OUT= MAC=3a:e9:5f:c7:41:78:d0:07:ca:8c:10:01:08:00 SRC=104.156.15.12 DST=19.0.20.1 LEN=40 TOS=0x00 PREC=0x00 TTL=243 ID=8530 PROTO=TCP SPT=58399 DPT=3080 WINDOW=1024 RES=0x00 SYN URGP=0' >> /logging/test_input.log

Error:

2022/05/11 20:07:03 outputloki.go:89 [warning] key: geoip error:Unable to Cast to string
{"host":"gogstash","path":"/logging/test_input.log","@timestamp":"2022-05-11T20:07:03.018256841Z","message":"May 11 10:02:05 zabbix kernel: [1609112.875635] FW_F_IN_DROP: IN=ens18 OUT= MAC=3a:e9:5f:c7:41:78:d0:07:ca:8c:10:01:08:00 SRC=104.156.15.12 DST=19.0.20.1LEN=40 TOS=0x00 PREC=0x00 TTL=243 ID=8530 PROTO=TCP SPT=58399 DPT=3080 WINDOW=1024 RES=0x00 SYN URGP=0","offset":0,"src_ip":"104.156.15.12","geoip":{"longitude":-97.822,"timezone":"America/Chicago","continent_code":"NA","country_code":"US","country_name":"United States","ip":"104.156.15.12","latitude":37.751,"location":[-97.822,37.751]},"geo2ip_city":"US"}

also, I tried to use output stdout without codec json, and result was without geoip data, looks like geoip can be injected only in json mode.

Is there anything which can help me to resolve issue? I would like to send logs to loki with geoip data to build dashboard with worldmap and geoip data.

[helgeolav]

Hi

I had a quick look at this issue and from what I understand the loki output works and is passing data to the server? Everything except the geoenriched data.

The loki output does not support nested JSON structures, and "geoip" is not a simple type that can be processed. You have to copy all the fields like you do with geo2ip_city to get this right. And in the end you should delete the field "geoip" to get rid of the warning.

Disclaimer: I don't know anything about Loki but looked through the API documentation and I assume that for you to use this data it has to be passed in the stream attribute, which is done for all fields except the message field. Perhaps @yuangu can tell if loki supports nested inputs and if the module can be rewritten?

[KVInventoR]

hm, I also added:

  - remove_field:
    fields:
      - geoip

and in logs I got next:

2022/05/12 08:28:29 filtergeoip2.go:86 [info] geoip2 fsnotify initialized for /GeoLite2-City.mmdb
panic: interface conversion: interface {} is nil, not string

goroutine 1 [running]:
github.com/tsaikd/gogstash/config.GetFilters({0x13ca150, 0xc000380040}, {0xc0000a6180, 0x3, 0x13c6640?}, {0x13cb298, 0xc000312000})
        /build/config/filter.go:88 +0x42c
github.com/tsaikd/gogstash/config.(*Config).getFilters(...)
        /build/config/filter.go:104
github.com/tsaikd/gogstash/config.(*Config).startFilters(0xc000312000)
        /build/config/filter.go:108 +0x49
github.com/tsaikd/gogstash/config.(*Config).Start(0xc000312000, {0x13ca188?, 0xc000036040?})
        /build/config/config.go:138 +0x149
github.com/tsaikd/gogstash/cmd.gogstash({0x13ca188, 0xc000036040}, {0xc00003401f, 0xa}, 0xe0?, {0x0, 0x0}, 0x0)
        /build/cmd/gogstash.go:49 +0x285
github.com/tsaikd/gogstash/cmd.init.0.func2({0x13ca188, 0xc000036040}, 0x0?, {0x0?, 0x0?, 0xecb560?})
        /build/cmd/module.go:62 +0x85
github.com/tsaikd/KDGoLib/cliutil/cobrather.(*Module).MustNewCommand.func1(0xc00030c000?, {0x1c030e8, 0x0, 0x0})
        /go/pkg/mod/github.com/tsaikd/!k!d!go!lib@v0.0.0-20191001134900-7f3cf518e07d/cliutil/cobrather/module.go:58 +0xa5
github.com/spf13/cobra.(*Command).execute(0xc00030c000, {0xc000030290, 0x0, 0x0})
        /go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:842 +0x67c
github.com/spf13/cobra.(*Command).ExecuteC(0xc00030c000)
        /go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950 +0x39c
github.com/spf13/cobra.(*Command).Execute(...)
        /go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887
github.com/tsaikd/KDGoLib/cliutil/cobrather.(*Module).MustMainRun(0x0?, {0x0?, 0x1b56628?, 0xc0000021a0?})
        /go/pkg/mod/github.com/tsaikd/!k!d!go!lib@v0.0.0-20191001134900-7f3cf518e07d/cliutil/cobrather/module.go:130 +0x7e
main.main()
        /build/main.go:6 +0x27
2022/05/12 08:28:30 filtergeoip2.go:86 [info] geoip2 fsnotify initialized for /GeoLite2-City.mmdb
panic: interface conversion: interface {} is nil, not string

seems it's not possible to remove geoip map in simple way

[helgeolav]

This panic is part of initialization and should not occur. My first guess is a typo or indentation error in the config file. Could you provide the entire configuration file? Since geoip2 was initialized the error is below that section.

[yuangu]

I refer to this document that from https://grafana.com/docs/loki/latest/api/#post-lokiapiv1push 。In fact, it is found that the data received by Loki needs to be of type string. This is what ToStringE function does

[yuangu]

I think about how to solve this problem.

@helgeolav I looked at it because "geoip" is a nested type. However, Loki does not support nested types. Because it needs an accurate string value as the search key. I want to deal with it specially, but I'm afraid there will be some problems in the future.

How to solve it?

for key, value := range event.Extra {
        if key == "geoip" {
            for gp_key, gp_value := range value.(map[string]interface{}) {
                v, err := ToStringE(gp_value)
                if err != nil {
                    goglog.Logger.Warnf("geoip_key: %v error:%v", gp_key, err)
                }
                _stream[key+"_"+gp_key] = v
            }
            continue
        }

        v, err := ToStringE(value)
        if err != nil {
            goglog.Logger.Warnf("key: %v error:%v", key, err)
        }
        _stream[key] = v
    }

[KVInventoR]

@helgeolav this is my current config:

 cat config/config.yml
---
#

chsize: 1000

input:
  - type: file
    path: /logging/test_input.log
    start_position: end
    sincedb_write_interval: 5

filter:
  - type: grok
    match: ["%{IPTABLES_SRC}"]
    source: "message"
    patterns_path: "/iptables.grok"

  - type: geoip2
    db_path: "/GeoLite2-City.mmdb"
    ip_field: "src_ip"
    cache_size: 100000
    key: geoip
    flat_format: true

  - remove_field:
    fields:
      - geoip

output:
  - type: stdout
    codec: json

yamllint config/config.yml && echo $?
0

there are no yaml errors

[KVInventoR]

oh, sorry, I found an issue with remove_field

  - type: remove_field
    fields:
      - geoip

[helgeolav]

oh, sorry, I found an issue with remove_field

  - type: remove_field
    fields:
      - geoip

I see it now. That kind of configuration errors are easy to overlook. We should look into the code and rewrite it so the error is more clear than just a panic.

Does it work better for you now?

[KVInventoR]

Hi, please excuse the long-time delay, I switched to another project. Currently, going to restore work on this solution for monitoring system.