search

Graylog2 / collector-sidecar

Manage log collectors through Graylog
https://www.graylog.org/
Other
268 stars 56 forks source link

Ubuntu 22 - Filebeat issue. #461

Closed Pavlo9999 closed 1 year ago

Pavlo9999 commented 1 year ago

Problem description

I am using Graylog sidecar (filebeat) for Centos 7 systems and Ubuntu 20 and Ubuntu 22. I have an issue with the Ubuntu 2 system only. On Centos systems, I am using the 1.1.0 version of the agent. After I apply config to the system pass some time or it could be immediately and I get the error:

runtime/cgo: pthread_create failed: Operation not permitted
SIGABRT: abort
PC=0x7f11f8378a7c m=8 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: unknown pc 0x7f11f8378a7c
stack: frame={sp:0x7f11e6fbc820, fp:0x0} stack=[0x7f11e67bd1e8,0x7f11e6fbcde8)
00007f11e6fbc720:  0000000000000080  00007f1100000000 
00007f11e6fbc730:  00000000015bfafd <runtime.scanstack.func1+61>  00007f1100000000 
00007f11e6fbc740:  0000000000000000  0000000000000000 
00007f11e6fbc750:  0000000000000000  0000000000000000 
00007f11e6fbc760:  0000000000000000  0000000000000000 
00007f11e6fbc770:  0000000000000000  0000000000000000 
00007f11e6fbc780:  0000000000000000  0000000000000000 
00007f11e6fbc790:  0000000000000000  0000000000000000 
00007f11e6fbc7a0:  0000000000000000  0000000000000000 
00007f11e6fbc7b0:  0000000000000000  0000000000000130 
00007f11e6fbc7c0:  0000000000000000  0000000000000000 
00007f11e6fbc7d0:  0000001300000000  0000000000000120 
00007f11e6fbc7e0:  0000000000000000  0000000000000000 
00007f11e6fbc7f0:  0000000000000000  0000000000000000 
00007f11e6fbc800:  0000000000000004  0000003400000013 
00007f11e6fbc810:  0000000000000000  00007f11f8378a6e 
00007f11e6fbc820: <0000000000000000  000000770000007c 
00007f11e6fbc830:  0000005b0000006e  0000000000000000 
00007f11e6fbc840:  0000000000000000  00007f11f8408a51 
00007f11e6fbc850:  00007f11e665c640  00007f11e6fbcb30 
00007f11e6fbc860:  00007f11e6fbc9ae  00007f11e6fbc9af 
00007f11e6fbc870:  0000000000000000  00007f11f8376759 
00007f11e6fbc880:  00000000007fff00  0000000000000000 
00007f11e6fbc890:  00000000003d0f00  00007f11e665c910 
00007f11e6fbc8a0:  00007f11e665c910  cc0ab6632dec7000 
00007f11e6fbc8b0:  00007f11e6fbd640  0000000000000006 
00007f11e6fbc8c0:  0000000006278970  0000000000000000 
00007f11e6fbc8d0:  0000000003e98b64  00007f11f8324476 
00007f11e6fbc8e0:  00007f11f84fce90  00007f11f830a7f3 
00007f11e6fbc8f0:  0000000000000020  00007f11e665c640 
00007f11e6fbc900:  0000000000000000  0000000000000001 
00007f11e6fbc910:  00007f11e665c640  00007f11f83775c4 
runtime: unknown pc 0x7f11f8378a7c
stack: frame={sp:0x7f11e6fbc820, fp:0x0} stack=[0x7f11e67bd1e8,0x7f11e6fbcde8)
00007f11e6fbc720:  0000000000000080  00007f1100000000 
00007f11e6fbc730:  00000000015bfafd <runtime.scanstack.func1+61>  00007f1100000000 
00007f11e6fbc740:  0000000000000000  0000000000000000 
00007f11e6fbc750:  0000000000000000  0000000000000000 
00007f11e6fbc760:  0000000000000000  0000000000000000 
00007f11e6fbc770:  0000000000000000  0000000000000000 
00007f11e6fbc780:  0000000000000000  0000000000000000 
00007f11e6fbc790:  0000000000000000  0000000000000000 
00007f11e6fbc7a0:  0000000000000000  0000000000000000 
00007f11e6fbc7b0:  0000000000000000  0000000000000130 
00007f11e6fbc7c0:  0000000000000000  0000000000000000 
00007f11e6fbc7d0:  0000001300000000  0000000000000120 
00007f11e6fbc7e0:  0000000000000000  0000000000000000 
00007f11e6fbc7f0:  0000000000000000  0000000000000000 
00007f11e6fbc800:  0000000000000004  0000003400000013 
00007f11e6fbc810:  0000000000000000  00007f11f8378a6e 
00007f11e6fbc820: <0000000000000000  000000770000007c 
00007f11e6fbc830:  0000005b0000006e  0000000000000000 
00007f11e6fbc840:  0000000000000000  00007f11f8408a51 
00007f11e6fbc850:  00007f11e665c640  00007f11e6fbcb30 
00007f11e6fbc860:  00007f11e6fbc9ae  00007f11e6fbc9af 
00007f11e6fbc870:  0000000000000000  00007f11f8376759 
00007f11e6fbc880:  00000000007fff00  0000000000000000 
00007f11e6fbc890:  00000000003d0f00  00007f11e665c910 
00007f11e6fbc8a0:  00007f11e665c910  cc0ab6632dec7000 
00007f11e6fbc8b0:  00007f11e6fbd640  0000000000000006 
00007f11e6fbc8c0:  0000000006278970  0000000000000000 
00007f11e6fbc8d0:  0000000003e98b64  00007f11f8324476 
00007f11e6fbc8e0:  00007f11f84fce90  00007f11f830a7f3 
00007f11e6fbc8f0:  0000000000000020  00007f11e665c640 
00007f11e6fbc900:  0000000000000000  0000000000000001 
00007f11e6fbc910:  00007f11e665c640  00007f11f83775c4 

goroutine 21 [syscall]:
runtime.notetsleepg(0x5b2aa00, 0x38e962b2, 0x0)
    /usr/local/go/src/runtime/lock_futex.go:227 +0x34 fp=0xc00006af60 sp=0xc00006af30 pc=0x156f864
runtime.timerproc(0x5b2a9e0)
    /usr/local/go/src/runtime/time.go:311 +0x2f1 fp=0xc00006afd8 sp=0xc00006af60 pc=0x15b3861
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00006afe0 sp=0xc00006afd8 pc=0x15c4a71
created by runtime.(*timersBucket).addtimerLocked
    /usr/local/go/src/runtime/time.go:169 +0x10e

goroutine 1 [chan receive]:
github.com/elastic/beats/v7/filebeat/beater.(*signalWait).Wait(...)
    /go/src/github.com/elastic/beats/filebeat/beater/signalwait.go:45
github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Run(0xc0001844c0, 0xc0005ba500, 0x0, 0x0)
    /go/src/github.com/elastic/beats/filebeat/beater/filebeat.go:423 +0x10a1
github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch(0xc0005ba500, 0x3a742bb, 0x8, 0x3a742bb, 0x8, 0x0, 0x0, 0x1, 0x0, 0x0, ...)
    /go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:469 +0x733
github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1(0x3a742bb, 0x8, 0x3a742bb, 0x8, 0x0, 0x0, 0xc00083dc10, 0x3b98168, 0x0, 0x0)
    /go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:200 +0x56b
github.com/elastic/beats/v7/libbeat/cmd/instance.Run(0x3a742bb, 0x8, 0x3a742bb, 0x8, 0x0, 0x0, 0x1, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:201 +0x148
github.com/elastic/beats/v7/libbeat/cmd.genRunCmd.func1(0xc0002a2580, 0xc0009182c0, 0x0, 0x2)
    /go/src/github.com/elastic/beats/libbeat/cmd/run.go:36 +0x9d
github.com/spf13/cobra.(*Command).execute(0xc0002a2580, 0xc0000b8070, 0x2, 0x2, 0xc0002a2580, 0xc0000b8070)
    /go/src/github.com/elastic/beats/vendor/github.com/spf13/cobra/command.go:766 +0x2aa
github.com/spf13/cobra.(*Command).ExecuteC(0xc0002a2580, 0x0, 0xa6000, 0xc0000a4058)
    /go/src/github.com/elastic/beats/vendor/github.com/spf13/cobra/command.go:852 +0x2ea
github.com/spf13/cobra.(*Command).Execute(...)
    /go/src/github.com/elastic/beats/vendor/github.com/spf13/cobra/command.go:800
main.main()
    /go/src/github.com/elastic/beats/x-pack/filebeat/main.go:22 +0x2f

goroutine 13 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc00029e870)
    /go/src/github.com/elastic/beats/vendor/go.opencensus.io/stats/view/worker.go:154 +0x100
created by go.opencensus.io/stats/view.init.0
    /go/src/github.com/elastic/beats/vendor/go.opencensus.io/stats/view/worker.go:32 +0x57

goroutine 26 [select]:
github.com/elastic/beats/v7/filebeat/input/log.(*Input).stopWhenDone(0xc000538000)
    /go/src/github.com/elastic/beats/filebeat/input/log/input.go:753 +0x11c
created by github.com/elastic/beats/v7/filebeat/input/log.NewInput
    /go/src/github.com/elastic/beats/filebeat/input/log/input.go:155 +0x7e2

goroutine 27 [select]:
github.com/elastic/beats/v7/filebeat/channel.(*subOutlet).OnEvent(0xc000e35c80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x390e220, 0xc0008e06c0, 0x0, 0x3a9e600)
    /go/src/github.com/elastic/beats/filebeat/channel/util.go:84 +0x231
github.com/elastic/beats/v7/filebeat/input/log.(*Input).createHarvester.func1(0x0, 0x0, 0x0, 0x3f86060, 0xc00014be10, 0xc000197220, 0xf, 0x5b988, 0xc0e4f017925f65aa, 0x2e30acf, ...)
    /go/src/github.com/elastic/beats/filebeat/input/log/input.go:658 +0x107
github.com/elastic/beats/v7/filebeat/input/log.(*Harvester).SendStateUpdate(0xc00055f180)
    /go/src/github.com/elastic/beats/filebeat/input/log/harvester.go:476 +0xd6
github.com/elastic/beats/v7/filebeat/input/log.(*Input).startHarvester(0xc000538000, 0x0, 0x0, 0x0, 0x3f86060, 0xc00014be10, 0xc000197220, 0xf, 0x5b988, 0xc0e4f017925f65aa, ...)
    /go/src/github.com/elastic/beats/filebeat/input/log/input.go:696 +0x1da
github.com/elastic/beats/v7/filebeat/input/log.(*Input).harvestExistingFile(0xc000538000, 0x0, 0x0, 0x0, 0x3f86060, 0xc00014be10, 0xc000197220, 0xf, 0x0, 0xc0e4f017925f65aa, ...)
    /go/src/github.com/elastic/beats/filebeat/input/log/input.go:521 +0xa95
github.com/elastic/beats/v7/filebeat/input/log.(*Input).scan(0xc000538000)
    /go/src/github.com/elastic/beats/filebeat/input/log/input.go:504 +0x68f
github.com/elastic/beats/v7/filebeat/input/log.(*Input).Run(0xc000538000)
    /go/src/github.com/elastic/beats/filebeat/input/log/input.go:206 +0x8f
github.com/elastic/beats/v7/filebeat/input.(*Runner).Run(0xc0000c7ce0)
    /go/src/github.com/elastic/beats/filebeat/input/input.go:128 +0x49
github.com/elastic/beats/v7/filebeat/input.(*Runner).Start.func1(0xc0001967f0, 0xc0000c7ce0)
    /go/src/github.com/elastic/beats/filebeat/input/input.go:121 +0x65
created by github.com/elastic/beats/v7/filebeat/input.(*Runner).Start
    /go/src/github.com/elastic/beats/filebeat/input/input.go:114 +0xc6

goroutine 24 [syscall]:
os/signal.signal_recv(0x0)
    /usr/local/go/src/runtime/sigqueue.go:147 +0x9c
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
    /usr/local/go/src/os/signal/signal_unix.go:29 +0x41

goroutine 25 [chan receive]:
k8s.io/klog.(*loggingT).flushDaemon(0x5b23760)
    /go/src/github.com/elastic/beats/vendor/k8s.io/klog/klog.go:1000 +0x8b
created by k8s.io/klog.init.0
    /go/src/github.com/elastic/beats/vendor/k8s.io/klog/klog.go:404 +0x6c

goroutine 57 [chan receive]:
github.com/elastic/go-lumber/client/v2.(*AsyncClient).ackLoop(0xc000e340c0)
    /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-lumber/client/v2/async.go:149 +0x117
created by github.com/elastic/go-lumber/client/v2.(*AsyncClient).startACK
    /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-lumber/client/v2/async.go:123 +0x8e

goroutine 58 [select]:
github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*bufferingEventLoop).run(0xc0002388c0)
    /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/eventloop.go:316 +0x1cf
github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue.func1(0xc0001aabd0, 0x3f1d960, 0xc0002388c0)
    /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:171 +0x67
created by github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue
    /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:169 +0x373

goroutine 59 [select]:
github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*ackLoop).run(0xc0000b7cc0)
    /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/ackloop.go:60 +0x107
github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue.func2(0xc0001aabd0, 0xc0000b7cc0)
    /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:175 +0x61
created by github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue
    /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:173 +0x3a2

goroutine 60 [select]:
github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*consumer).Get(0xc0001842e0, 0x800, 0x0, 0x0, 0x1, 0x1)
    /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/consume.go:65 +0xe9
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*eventConsumer).loop(0xc0000b6000, 0x3f1d9a0, 0xc0001842e0)
    /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/consumer.go:167 +0x497
created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer
    /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/consumer.go:77 +0x154

goroutine 61 [select]:
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*retryer).loop(0xc0000c6ba0)
    /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/retry.go:137 +0x25a
created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.newRetryer
    /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/retry.go:96 +0x14f

goroutine 62 [select]:
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run(0xc0000c7440)
    /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:128 +0xda
created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.makeClientWorker
    /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:80 +0x1c8

goroutine 63 [select]:
github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).snapshotLoop(0xc000e34c00)
    /go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:129 +0x410
github.com/elastic/beats/v7/libbeat/monitoring/report/log.MakeReporter.func1(0xc000e34c00)
    /go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:107 +0x5d
created by github.com/elastic/beats/v7/libbeat/monitoring/report/log.MakeReporter
    /go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:105 +0x157

goroutine 65 [chan receive]:
github.com/elastic/beats/v7/libbeat/service.HandleSignals.func1(0xc0000c77a0, 0xc0005d87d0, 0xc0005d87c0, 0xc0001e24d0, 0xc0001844e0)
    /go/src/github.com/elastic/beats/libbeat/service/service.go:50 +0x52
created by github.com/elastic/beats/v7/libbeat/service.HandleSignals
    /go/src/github.com/elastic/beats/libbeat/service/service.go:49 +0x172

goroutine 67 [select]:
github.com/elastic/beats/v7/filebeat/beater.(*registrarLogger).Published(0xc0005d8900, 0xc0008e05a0, 0x1, 0x1)
    /go/src/github.com/elastic/beats/filebeat/beater/channels.go:53 +0xd3
github.com/elastic/beats/v7/filebeat/beater.(*eventACKer).ackEvents(0xc000e34e40, 0xc0005840c0, 0x1, 0x1)
    /go/src/github.com/elastic/beats/filebeat/beater/acker.go:65 +0x3b7
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*pipelineEventCB).reportEventsData(0xc000912700, 0xc0005840c0, 0x1, 0x1, 0x1)
    /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/pipeline_ack.go:319 +0x91
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*pipelineEventCB).worker(0xc000912700)
    /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/pipeline_ack.go:255 +0x264
created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.newPipelineEventCB
    /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/pipeline_ack.go:196 +0x15b

goroutine 68 [runnable]:
syscall.Syscall(0x4a, 0x7, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/syscall/asm_linux_amd64.s:18 +0x5
syscall.Fsync(0x7, 0x5b21f01, 0xc0000de680)
    /usr/local/go/src/syscall/zsyscall_linux_amd64.go:449 +0x40
internal/poll.(*FD).Fsync(0xc000098c60, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_fsync_posix.go:17 +0x8f
os.(*File).Sync(0xc000010750, 0x3464600, 0xc000185720)
    /usr/local/go/src/os/file_posix.go:113 +0x4c
github.com/elastic/beats/v7/filebeat/registrar.writeTmpFile(0xc000054780, 0x4d, 0x180, 0xc00011cc00, 0x7, 0x7, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/elastic/beats/filebeat/registrar/registrar.go:430 +0x471
github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).writeRegistry(0xc0000c78c0, 0x2c, 0xc000081e88)
    /go/src/github.com/elastic/beats/filebeat/registrar/registrar.go:392 +0xa9
github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).flushRegistry(0xc0000c78c0)
    /go/src/github.com/elastic/beats/filebeat/registrar/registrar.go:373 +0x2f
github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Run(0xc0000c78c0)
    /go/src/github.com/elastic/beats/filebeat/registrar/registrar.go:302 +0x271
created by github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Start
    /go/src/github.com/elastic/beats/filebeat/registrar/registrar.go:272 +0xe8

goroutine 69 [chan send]:
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*pipelineEventCB).pushMsg(...)
    /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/pipeline_ack.go:226
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*pipelineEventCB).onEvents(...)
    /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/pipeline_ack.go:217
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*eventDataACK).onACK(0xc000913580, 0x1, 0x0)
    /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:396 +0xeb
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*boundGapCountACK).onACK(0xc000098660, 0x1, 0x0)
    /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:332 +0x5a
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*gapCountACK).ackLoop(0xc000098670)
    /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:165 +0x1a2
created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*gapCountACK).init
    /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:129 +0x182

goroutine 70 [select]:
github.com/elastic/beats/v7/filebeat/channel.CloseOnSignal.func1(0x3f50d60, 0xc000e35c50, 0xc0001832c0)
    /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7
created by github.com/elastic/beats/v7/filebeat/channel.CloseOnSignal
    /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76

goroutine 71 [chan send]:
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*gapCountACK).addDropEvent(0xc000098670)
    /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:257 +0x106
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*gapCountACK).addEvent(0xc000098670, 0x0, 0x0, 0x0, 0x0, 0x0, 0x390e220, 0xc0008e06c0, 0x0, 0xc000199500, ...)
    /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:234 +0x57
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*boundGapCountACK).addEvent(0xc000098660, 0x0, 0x0, 0x0, 0x0, 0x0, 0x390e220, 0xc0008e06c0, 0x0, 0x0, ...)
    /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:326 +0x82
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*eventDataACK).addEvent(0xc000913580, 0x0, 0x0, 0x0, 0x0, 0x0, 0x390e220, 0xc0008e06c0, 0x0, 0x0, ...)
    /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:381 +0x10c
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*client).publish(0xc000156100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x390e220, 0xc0008e06c0, 0x0)
    /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/client.go:102 +0x1a2
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*client).Publish(0xc000156100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x390e220, 0xc0008e06c0, 0x0)
    /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/client.go:68 +0xc2
github.com/elastic/beats/v7/filebeat/channel.(*outlet).OnEvent(0xc000e35c50, 0x0, 0x0, 0x0, 0x0, 0x0, 0x390e220, 0xc0008e06c0, 0x0, 0x0)
    /go/src/github.com/elastic/beats/filebeat/channel/outlet.go:64 +0x7e
github.com/elastic/beats/v7/filebeat/channel.SubOutlet.func1(0xc000e35c80, 0x3f50d60, 0xc000e35c50)
    /go/src/github.com/elastic/beats/filebeat/channel/util.go:45 +0x10d
created by github.com/elastic/beats/v7/filebeat/channel.SubOutlet
    /go/src/github.com/elastic/beats/filebeat/channel/util.go:43 +0xf8

goroutine 72 [select]:
github.com/elastic/beats/v7/filebeat/channel.CloseOnSignal.func1(0x3f50da0, 0xc000e35c80, 0xc000182e40)
    /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7
created by github.com/elastic/beats/v7/filebeat/channel.CloseOnSignal
    /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76

goroutine 28 [chan receive]:
github.com/elastic/beats/v7/filebeat/beater.waitChannel.func1()
    /go/src/github.com/elastic/beats/filebeat/beater/signalwait.go:79 +0x33
github.com/elastic/beats/v7/filebeat/beater.(*signalWait).Add.func1(0xc0001982a0, 0xc0005d8830)
    /go/src/github.com/elastic/beats/filebeat/beater/signalwait.go:52 +0x27
created by github.com/elastic/beats/v7/filebeat/beater.(*signalWait).Add
    /go/src/github.com/elastic/beats/filebeat/beater/signalwait.go:51 +0x4c

rax    0x0
rbx    0x7f11e6fbd640
rcx    0x7f11f8378a7c
rdx    0x6
rdi    0x19b764
rsi    0x19b76b
rbp    0x19b76b
rsp    0x7f11e6fbc820
r8     0x7f11e6fbc8f0
r9     0x7fffffff
r10    0x8
r11    0x246
r12    0x6
r13    0x16
r14    0x3e98b64
r15    0x0
rip    0x7f11f8378a7c
rflags 0x246
cs     0x33
fs     0x0
gs     0x0

Environment

Filebeat Config file: 

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

filebeat.inputs:
- input_type: log
  paths:
    - /var/log/syslog
    - /var/log/zabbix-agent/zabbix_agentd.log
    - /var/log/amazon/ssm/amazon-ssm-agent.log
    - /var/log/auth.log
    - /var/log/dmesg
    - /var/log/dpkg.log
    - /var/log/php8.0-fpm.log
    - /var/log/rabbitmq/rabbit\@*.log
    - /var/log/mongo/mongod.log
    - /var/log/helpAround/lynis.log
    - /var/log/clickhouse-server/*.log
    - /var/log/vector/*.log
    - /usr/local/helpAround/stuff/mattermost/logs/mattermost.log
    - /usr/local/helpAround/mattermost/logs/mattermost.log
    - /var/lib/docker/containers/*/*-json.log
  type: log
output.logstash:
   hosts: ["graylog_ip:port"]
path:
  data: /var/lib/graylog-sidecar/collectors/filebeat/data
  logs: /var/lib/graylog-sidecar/collectors/filebeat/log

Also I have tried default config, and during some times I get the same issue(

bernd commented 1 year ago

@Pavlo9999 Can you please add the filebeat version you are using to the "Environment" section? Thanks!

Pavlo9999 commented 1 year ago

@bernd sure Sidecar Version: 1.2.0 Graylog Version: 4.3.10+6cb2860 Operating System: Centos 7 Elasticsearch Version: 7.10.2 MongoDB Version: 4.2.23 Filebeat Version: 7.8.0

mpfz0r commented 1 year ago

I think that's https://github.com/elastic/beats/pull/28117

@Pavlo9999 can you try and add seccomp.enabed: false to your config as a workaround?

Pavlo9999 commented 1 year ago

image I have added to Filebeat config, and restarted service and Graylog sidecar, unfortunately, it's doesn't help and I get the same error

drewmiranda-gl commented 1 year ago

This appears to have been addressed in filebeat 7.16. Can you retry using the latest 7.x filebeat (7.17.8)?

Pavlo9999 commented 1 year ago

@drewmiranda-gl yesterday I upgraded Filebeat to the latest version 7.17.8 and the issue was resolved. I have checked logs during the last 12 hours, and everything works and Graylog-sidecar didn't fail. Issue fixed, thanks for help)