concurrent map iteration and map write

[sequix]

Problem

Using the following conf to receive json log from logstash, then push it to loki. It quite a large amount of data, am I reach the limit of loki or this plugin?

...

Steps to replicate

[INPUT] Name tcp Listen 127.0.0.1 Port 8787 Chunk_Size 32 Buffer_Size 64 Tag loki-test

[OUTPUT] Name loki Match loki-test Url http://localhost:8100/api/prom/push BatchWait 1000 # (1sec) BatchSize 1024 # (1MiB) Labels {job="fluentbit-gz"} LabelKeys level,app,zone,type

Expected Behavior or What you need to ask

Fluent Bit v1.2.2 Copyright (C) Treasure Data

[2019/09/27 19:58:22] [ info] [storage] initializing... [2019/09/27 19:58:22] [ info] [storage] in-memory [2019/09/27 19:58:22] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128 [2019/09/27 19:58:22] [ info] [engine] started (pid=49268) [2019/09/27 19:58:22] [ info] [in_tcp] binding 127.0.0.1:8787 level=info caller=out_loki.go:89 [flb-go]="Starting fluent-bit-go-loki" version="(version=0.3.0, branch=, revision=)" level=info caller=out_loki.go:90 [flb-go]="provided parameter" URL=http://localhost:8100/api/prom/push level=info caller=out_loki.go:91 [flb-go]="provided parameter" BatchWait="1000 # (1sec)" level=info caller=out_loki.go:92 [flb-go]="provided parameter" BatchSize="1024 # (1MiB)" level=info caller=out_loki.go:93 [flb-go]="provided parameter" Labels="{job=\"fluentbit-gz\"}" level=info caller=out_loki.go:94 [flb-go]="provided parameter" LogLevel= level=info caller=out_loki.go:95 [flb-go]="provided parameter" RemoveKeys= level=info caller=out_loki.go:96 [flb-go]="provided parameter" LabelKeys=level,app,zone,type [2019/09/27 19:58:22] [ info] [sp] stream processor started time="2019-09-27T19:58:27+08:00" level=info component=client error="server returned HTTP status 400 Bad Request (400): entry out of order for stream: {app=\"invoker\", job=\"fluentbit-gz\", level=\"error\", type=\"zonea\", zone=\"gz\"}" fields.level=error fields.msg="final error sending batch" host="localhost:8100" status=400 fatal error: concurrent map iteration and map write

goroutine 151 [running]: runtime.throw(0x7f2b2adb9a31, 0x26) /home/squix/bin/go/src/runtime/panic.go:617 +0x74 fp=0xc00054dc28 sp=0xc00054dbf8 pc=0x7f2b2a281384 runtime.mapiternext(0xc00054dd10) /home/squix/bin/go/src/runtime/map.go:860 +0x599 fp=0xc00054dcb0 sp=0xc00054dc28 pc=0x7f2b2a262669 runtime.mapiterinit(0x7f2b2b5da500, 0xc000425350, 0xc00054dd10) /home/squix/bin/go/src/runtime/map.go:850 +0x1cf fp=0xc00054dcd0 sp=0xc00054dcb0 pc=0x7f2b2a261fdf github.com/cosmo0920/fluent-bit-go-loki/vendor/github.com/prometheus/common/model.labelSetToFastFingerprint(0xc000425350, 0xc00054dddc) /home/squix/code/go/gopath/src/github.com/cosmo0920/fluent-bit-go-loki/vendor/github.com/prometheus/common/model/signature.go:86 +0x8c fp=0xc00054dd80 sp=0xc00054dcd0 pc=0x7f2b2a3894ec github.com/cosmo0920/fluent-bit-go-loki/vendor/github.com/prometheus/common/model.LabelSet.FastFingerprint(...) /home/squix/code/go/gopath/src/github.com/cosmo0920/fluent-bit-go-loki/vendor/github.com/prometheus/common/model/labelset.go:150 github.com/cosmo0920/fluent-bit-go-loki/vendor/github.com/grafana/loki/pkg/promtail/client.(*client).run(0xc000164a20) /home/squix/code/go/gopath/src/github.com/cosmo0920/fluent-bit-go-loki/vendor/github.com/grafana/loki/pkg/promtail/client/client.go:133 +0x2d5 fp=0xc00054dfd8 sp=0xc00054dd80 pc=0x7f2b2ad8cf45 runtime.goexit() /home/squix/bin/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00054dfe0 sp=0xc00054dfd8 pc=0x7f2b2a2b1321 created by github.com/cosmo0920/fluent-bit-go-loki/vendor/github.com/grafana/loki/pkg/promtail/client.New /home/squix/code/go/gopath/src/github.com/cosmo0920/fluent-bit-go-loki/vendor/github.com/grafana/loki/pkg/promtail/client/client.go:105 +0x33a

goroutine 17 [chan send, locked to thread]: github.com/cosmo0920/fluent-bit-go-loki/vendor/github.com/grafana/loki/pkg/promtail/client.(client).Handle(0xc000164a20, 0xc000425350, 0x61e926d, 0xed51ff060, 0x7f2b2c1991c0, 0xc0001f9590, 0xe8, 0x1, 0xffffffffffffffff) /home/squix/code/go/gopath/src/github.com/cosmo0920/fluent-bit-go-loki/vendor/github.com/grafana/loki/pkg/promtail/client/client.go:243 +0xb2 main.(fluentPlugin).HandleLine(0x7f2b2c1b73e0, 0xc000425350, 0x61e926d, 0xed51ff060, 0x7f2b2c1991c0, 0xc0001f9590, 0xe8, 0x0, 0x0) /home/squix/code/go/gopath/src/github.com/cosmo0920/fluent-bit-go-loki/out_loki.go:60 +0x7d main.FLBPluginFlush(0x7f2b28085010, 0xc0001f4095, 0x98c860, 0x7f2b2b692e28) /home/squix/code/go/gopath/src/github.com/cosmo0920/fluent-bit-go-loki/out_loki.go:155 +0x39c main._cgoexpwrap_f8f5e2420747_FLBPluginFlush(0x7f2b28085010, 0x1f4095, 0x98c860, 0x4b2a92) _cgo_gotypes.go:75 +0x3f

goroutine 8 [syscall]: os/signal.signal_recv(0x0) /home/squix/bin/go/src/runtime/sigqueue.go:139 +0x9e os/signal.loop() /home/squix/bin/go/src/os/signal/signal_unix.go:23 +0x24 created by os/signal.init.0 /home/squix/bin/go/src/os/signal/signal_unix.go:29 +0x43

goroutine 36 [select]: github.com/cosmo0920/fluent-bit-go-loki/vendor/go.opencensus.io/stats/view.(*worker).start(0xc00012e240) /home/squix/code/go/gopath/src/github.com/cosmo0920/fluent-bit-go-loki/vendor/go.opencensus.io/stats/view/worker.go:152 +0xdf created by github.com/cosmo0920/fluent-bit-go-loki/vendor/go.opencensus.io/stats/view.init.0 /home/squix/code/go/gopath/src/github.com/cosmo0920/fluent-bit-go-loki/vendor/go.opencensus.io/stats/view/worker.go:29 +0x59

goroutine 42 [select]: github.com/cosmo0920/fluent-bit-go-loki/vendor/github.com/cortexproject/cortex/pkg/util.(*hashBucketHistogram).loop(0xc0003a4640) /home/squix/code/go/gopath/src/github.com/cosmo0920/fluent-bit-go-loki/vendor/github.com/cortexproject/cortex/pkg/util/hash_bucket_histogram.go:87 +0x15e created by github.com/cosmo0920/fluent-bit-go-loki/vendor/github.com/cortexproject/cortex/pkg/util.NewHashBucketHistogram /home/squix/code/go/gopath/src/github.com/cosmo0920/fluent-bit-go-loki/vendor/github.com/cortexproject/cortex/pkg/util/hash_bucket_histogram.go:64 +0x12e

goroutine 180 [IO wait]: internal/poll.runtime_pollWait(0x7f2b284a5f08, 0x72, 0xffffffffffffffff) /home/squix/bin/go/src/runtime/netpoll.go:182 +0x58 internal/poll.(pollDesc).wait(0xc00015c218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/squix/bin/go/src/internal/poll/fd_poll_runtime.go:87 +0x9d internal/poll.(pollDesc).waitRead(...) /home/squix/bin/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(FD).Read(0xc00015c200, 0xc000403000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/squix/bin/go/src/internal/poll/fd_unix.go:169 +0x19d net.(netFD).Read(0xc00015c200, 0xc000403000, 0x1000, 0x1000, 0xc0000e7c40, 0x7f2b2a5bc038, 0xc000165560) /home/squix/bin/go/src/net/fd_unix.go:202 +0x51 net.(conn).Read(0xc0000110a0, 0xc000403000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/squix/bin/go/src/net/net.go:177 +0x6b net/http.(persistConn).Read(0xc000293200, 0xc000403000, 0x1000, 0x1000, 0xc0000e7c88, 0x7f2b2a258485, 0xc00040cb40) /home/squix/bin/go/src/net/http/transport.go:1524 +0x7d bufio.(Reader).fill(0xc00044b260) /home/squix/bin/go/src/bufio/bufio.go:100 +0x111 bufio.(Reader).Peek(0xc00044b260, 0x1, 0x0, 0x0, 0x1, 0xc00040c400, 0x0) /home/squix/bin/go/src/bufio/bufio.go:138 +0x51 net/http.(persistConn).readLoop(0xc000293200) /home/squix/bin/go/src/net/http/transport.go:1677 +0x1a5 created by net/http.(Transport).dialConn /home/squix/bin/go/src/net/http/transport.go:1357 +0xaea

goroutine 181 [select]: net/http.(persistConn).writeLoop(0xc000293200) /home/squix/bin/go/src/net/http/transport.go:1958 +0x115 created by net/http.(Transport).dialConn /home/squix/bin/go/src/net/http/transport.go:1358 +0xb0f

...

Using Fluentd and loki plugin versions

• OS version CentOS release 6.3 (Final)

• Bare Metal or within Docker or Kubernetes or others? Bare Metal, 14 cores, 378GiB Mem

• fluent-bit version Fluent Bit v1.2.2

• fluent-bit-go-loki version 0.3.0

• Grafana and Loki version Grafana v6.2.5 Loki v0.3.0

[cosmo0920]

This repository is deprecated. Could you try loki repository's fluent-bit plugin?

[sequix]

ok, let me give it a try.

[sequix]

It turns out that the fluentd plugin from loki's is work. and sorry for this late feedback.

[cosmo0920]

No problem. Thanks for the confirmation.