search

matusnovak / homelab

Homelab configuration via docker-compose
The Unlicense
27 stars 2 forks source link

Fluentbit is failing #3

Open m1k1o opened 4 years ago

m1k1o commented 4 years ago

fluent-bit-plugin-loki:latest is always restarting.

syslog:

Jul 12 07:22:07 icms kernel: [75859.791299] eth0: renamed from veth35032ea
Jul 12 07:22:07 icms systemd-networkd[9701]: veth8db42eb: Gained carrier
Jul 12 07:22:07 icms NetworkManager[9895]: <info>  [1594538527.5331] device (veth8db42eb): carrier: link connected
Jul 12 07:22:07 icms kernel: [75859.817752] IPv6: ADDRCONF(NETDEV_CHANGE): veth8db42eb: link becomes ready
Jul 12 07:22:07 icms kernel: [75859.817796] br-c96052879996: port 23(veth8db42eb) entered blocking state
Jul 12 07:22:07 icms kernel: [75859.817798] br-c96052879996: port 23(veth8db42eb) entered forwarding state
Jul 12 07:22:08 icms containerd[829]: time="2020-07-12T07:22:08.102424175Z" level=info msg="shim reaped" id=e24a51697164c271a44c63b9b7bfaa36aeb5027123a7aadf607c3579ad0a104e
Jul 12 07:22:08 icms dockerd[956]: time="2020-07-12T07:22:08.112412801Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jul 12 07:22:08 icms kernel: [75860.405463] br-c96052879996: port 23(veth8db42eb) entered disabled state
Jul 12 07:22:08 icms kernel: [75860.406570] veth35032ea: renamed from eth0
Jul 12 07:22:08 icms systemd-networkd[9701]: veth8db42eb: Lost carrier
Jul 12 07:22:08 icms systemd-udevd[193325]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jul 12 07:22:08 icms NetworkManager[9895]: <info>  [1594538528.1707] manager: (veth35032ea): new Veth device (/org/freedesktop/NetworkManager/Devices/3791)
Jul 12 07:22:08 icms networkd-dispatcher[782]: WARNING:Unknown index 2622 seen, reloading interface list
Jul 12 07:22:08 icms systemd-networkd[9701]: veth8db42eb: Link DOWN
Jul 12 07:22:08 icms kernel: [75860.463560] br-c96052879996: port 23(veth8db42eb) entered disabled state
Jul 12 07:22:08 icms kernel: [75860.468067] device veth8db42eb left promiscuous mode
Jul 12 07:22:08 icms kernel: [75860.468073] br-c96052879996: port 23(veth8db42eb) entered disabled state
Jul 12 07:22:08 icms systemd-networkd[9701]: rtnl: received neighbor for link '2623' we don't know about, ignoring.
Jul 12 07:22:08 icms systemd-networkd[9701]: rtnl: received neighbor for link '2623' we don't know about, ignoring.
Jul 12 07:22:08 icms NetworkManager[9895]: <info>  [1594538528.2027] device (veth8db42eb): released from master device br-c96052879996
Jul 12 07:22:08 icms networkd-dispatcher[782]: ERROR:Unknown interface index 2622 seen even after reload
Jul 12 07:22:08 icms networkd-dispatcher[782]: WARNING:Unknown index 2622 seen, reloading interface list
Jul 12 07:22:08 icms networkd-dispatcher[782]: ERROR:Unknown interface index 2622 seen even after reload
Jul 12 07:22:08 icms systemd[1]: run-docker-netns-2164d703997e.mount: Succeeded.
Jul 12 07:22:08 icms systemd[1]: var-lib-docker-containers-e24a51697164c271a44c63b9b7bfaa36aeb5027123a7aadf607c3579ad0a104e-mounts-shm.mount: Succeeded.
Jul 12 07:22:08 icms systemd[1]: var-lib-docker-overlay2-83a0a22709f48bbd27cd25b8d2284c891e34f18cee158a8b676e37cb2ad07ca0-merged.mount: Succeeded.
Jul 12 07:23:08 icms kernel: [75920.402438] br-c96052879996: port 23(vetha0a6db0) entered blocking state
Jul 12 07:23:08 icms kernel: [75920.402441] br-c96052879996: port 23(vetha0a6db0) entered disabled state
Jul 12 07:23:08 icms kernel: [75920.402579] device vetha0a6db0 entered promiscuous mode
Jul 12 07:23:08 icms kernel: [75920.402873] br-c96052879996: port 23(vetha0a6db0) entered blocking state
Jul 12 07:23:08 icms kernel: [75920.402875] br-c96052879996: port 23(vetha0a6db0) entered forwarding state
Jul 12 07:23:08 icms kernel: [75920.402984] br-c96052879996: port 23(vetha0a6db0) entered disabled state
Jul 12 07:23:08 icms networkd-dispatcher[782]: WARNING:Unknown index 2624 seen, reloading interface list
Jul 12 07:23:08 icms NetworkManager[9895]: <info>  [1594538588.1171] manager: (veth0e7c975): new Veth device (/org/freedesktop/NetworkManager/Devices/3792)
Jul 12 07:23:08 icms systemd-udevd[193457]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jul 12 07:23:08 icms systemd-udevd[193457]: Using default interface naming scheme 'v245'.
Jul 12 07:23:08 icms systemd-udevd[193457]: vetha0a6db0: Could not generate persistent MAC: No data available
Jul 12 07:23:08 icms systemd-udevd[193456]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jul 12 07:23:08 icms systemd-udevd[193456]: Using default interface naming scheme 'v245'.
Jul 12 07:23:08 icms systemd-udevd[193456]: veth0e7c975: Could not generate persistent MAC: No data available
Jul 12 07:23:08 icms systemd-networkd[9701]: vetha0a6db0: Link UP
Jul 12 07:23:08 icms NetworkManager[9895]: <info>  [1594538588.1197] manager: (vetha0a6db0): new Veth device (/org/freedesktop/NetworkManager/Devices/3793)
Jul 12 07:23:08 icms containerd[829]: time="2020-07-12T07:23:08.165338729Z" level=info msg="shim containerd-shim started" address=/containerd-shim/b599863014e3ca119e0579d40d5b0882de3e74ae76fe0121b93e46de4180006d.sock debug=false pid=193466
matusnovak commented 4 years ago

Can't reproduce. Maybe solved in newer version?

My fluentbit loki plugin logs:

Fluent Bit v1.4.6,
* Copyright (C) 2019-2020 The Fluent Bit Authors,
* Copyright (C) 2015-2018 Treasure Data,
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd,
* https://fluentbit.io,
,
[2020/09/09 20:11:57] [ info] [storage] version=1.0.3, initializing...,
[2020/09/09 20:11:57] [ info] [storage] in-memory,
[2020/09/09 20:11:57] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128,
[2020/09/09 20:11:57] [ info] [engine] started (pid=1),
level=info caller=out_loki.go:57 id=0 [flb-go]="Starting fluent-bit-go-loki" version="(version=master-750ab36, branch=master, revision=750ab36b)",
[2020/09/09 20:11:57] [ info] [input:forward:forward.0] listening on 0.0.0.0:24224,
level=info caller=out_loki.go:59 id=0 [flb-go]="provided parameter" URL=http://loki:3100/loki/api/v1/push,
level=info caller=out_loki.go:60 id=0 [flb-go]="provided parameter" TenantID=,
level=info caller=out_loki.go:61 id=0 [flb-go]="provided parameter" BatchWait=1.000s,
level=info caller=out_loki.go:62 id=0 [flb-go]="provided parameter" BatchSize=1001024,
level=info caller=out_loki.go:63 id=0 [flb-go]="provided parameter" Timeout=10.000s,
level=info caller=out_loki.go:64 id=0 [flb-go]="provided parameter" MinBackoff=0.500s,
level=info caller=out_loki.go:65 id=0 [flb-go]="provided parameter" MaxBackoff=300.000s,
level=info caller=out_loki.go:66 id=0 [flb-go]="provided parameter" MaxRetries=10,
level=info caller=out_loki.go:67 id=0 [flb-go]="provided parameter" Labels="[job=fluent-bit]",
level=info caller=out_loki.go:68 id=0 [flb-go]="provided parameter" LogLevel=info,
level=info caller=out_loki.go:69 id=0 [flb-go]="provided parameter" AutoKubernetesLabels=false,
level=info caller=out_loki.go:70 id=0 [flb-go]="provided parameter" RemoveKeys=[source],
level=info caller=out_loki.go:71 id=0 [flb-go]="provided parameter" LabelKeys=[container_name],
level=info caller=out_loki.go:72 id=0 [flb-go]="provided parameter" LineFormat=0,
level=info caller=out_loki.go:73 id=0 [flb-go]="provided parameter" DropSingleKey=true,
level=info caller=out_loki.go:74 id=0 [flb-go]="provided parameter" LabelMapPath=map[],
level=info caller=out_loki.go:75 id=0 [flb-go]="provided parameter" Buffer=false,
level=info caller=out_loki.go:76 id=0 [flb-go]="provided parameter" BufferType=dque,
level=info caller=out_loki.go:77 id=0 [flb-go]="provided parameter" DqueDir=/tmp/flb-storage/loki,
level=info caller=out_loki.go:78 id=0 [flb-go]="provided parameter" DqueSegmentSize=500,
level=info caller=out_loki.go:79 id=0 [flb-go]="provided parameter" DqueSync=false,
[2020/09/09 20:11:57] [ info] [sp] stream processor started,