search

loggie-io / loggie

A lightweight, cloud-native data transfer agent and aggregator
https://loggie-io.github.io/docs-en/
Apache License 2.0
1.26k stars 167 forks source link

通过CRD配置采集Pod容器日志报错 #595

Closed ixiaoyi93 closed 1 year ago

ixiaoyi93 commented 1 year ago

Ask your question here:

初次体验，通过 helm 部署的 v1.4.1 版本。在添加完 logConfig 之后，loggie 就打印报错日志，如下：

{"level":"debug","time":"2023-07-19 09:43:42","caller":"/pkg/discovery/kubernetes/controller/controller.go:491","message":"syncHandler start process: {Type:pod Key:kube-system/ingress-nginx-public-test-outer-cqxqh SelectorType:pod}"}
Wed, Jul 19 2023 09:43:42 | {"level":"debug","time":"2023-07-19 09:43:42","caller":"/pkg/discovery/kubernetes/controller/reconcile.go:316","message":"pod: kube-system/ingress-nginx-public-test-outer-cqxqh add or update event received"}
Wed, Jul 19 2023 09:43:42 | {"level":"debug","time":"2023-07-19 09:43:42","caller":"/pkg/discovery/kubernetes/controller/controller.go:491","message":"syncHandler start process: {Type:pod Key:monitoring/loggie-s96ls SelectorType:pod}"}
Wed, Jul 19 2023 09:43:42 | {"level":"debug","time":"2023-07-19 09:43:42","caller":"/pkg/discovery/kubernetes/controller/reconcile.go:316","message":"pod: monitoring/loggie-s96ls add or update event received"}
Wed, Jul 19 2023 09:43:51 | {"level":"debug","time":"2023-07-19 09:43:51","caller":"/pkg/discovery/kubernetes/controller/controller.go:491","message":"syncHandler start process: {Type:pod Key:vinci-dev/vinci-wechat-5f98d48ccf-vfbwl SelectorType:pod}"}
Wed, Jul 19 2023 09:43:51 | {"level":"debug","time":"2023-07-19 09:43:51","caller":"/pkg/discovery/kubernetes/controller/reconcile.go:316","message":"pod: vinci-dev/vinci-wechat-5f98d48ccf-vfbwl add or update event received"}
Wed, Jul 19 2023 09:43:52 | {"level":"debug","time":"2023-07-19 09:43:52","caller":"/pkg/source/file/watch.go:415","message":"scan paths /var/log/pods/675af80f-9d02-4a35-817a-1befcc492bf4/matter/*.log , matches: []"}
Wed, Jul 19 2023 09:43:52 | {"level":"debug","time":"2023-07-19 09:43:52","caller":"/pkg/source/file/watch.go:415","message":"scan paths /var/log/pods/alpha-dev_matter-69fc75944b-4j5n5_675af80f-9d02-4a35-817a-1befcc492bf4/matter/*.log , matches: [/var/log/pods/alpha-dev_matter-69fc75944b-4j5n5_675af80f-9d02-4a35-817a-1befcc492bf4/matter/0.log]"}
Wed, Jul 19 2023 09:43:52 | {"level":"info","time":"2023-07-19 09:43:52","caller":"/pkg/source/file/watch.go:322","message":"[alpha-dev/matter-matter-69fc75944b-4j5n5/matter/common] start collect file: /var/log/pods/alpha-dev_matter-69fc75944b-4j5n5_675af80f-9d02-4a35-817a-1befcc492bf4/matter/0.log"}
Wed, Jul 19 2023 09:43:52 | fatal error: unexpected signal during runtime execution
Wed, Jul 19 2023 09:43:52 | [signal SIGSEGV: segmentation violation code=0x1 addr=0x47 pc=0x7fd210376360]
Wed, Jul 19 2023 09:43:52 |  
Wed, Jul 19 2023 09:43:52 | runtime stack:
Wed, Jul 19 2023 09:43:52 | runtime.throw({0x283d688?, 0x7fd2100886e0?})
Wed, Jul 19 2023 09:43:52 | /usr/local/go/src/runtime/panic.go:992 +0x71
Wed, Jul 19 2023 09:43:52 | runtime.sigpanic()
Wed, Jul 19 2023 09:43:52 | /usr/local/go/src/runtime/signal_unix.go:802 +0x389
Wed, Jul 19 2023 09:43:52 |  
Wed, Jul 19 2023 09:43:52 | goroutine 199 [syscall]:
Wed, Jul 19 2023 09:43:52 | runtime.cgocall(0x1f1c370, 0xc00082dd98)
Wed, Jul 19 2023 09:43:52 | /usr/local/go/src/runtime/cgocall.go:157 +0x5c fp=0xc00082dd70 sp=0xc00082dd38 pc=0x405a3c
Wed, Jul 19 2023 09:43:52 | net._C2func_getaddrinfo(0xc000da2210, 0x0, 0xc000a75ec0, 0xc000b231a8)
Wed, Jul 19 2023 09:43:52 | _cgo_gotypes.go:94 +0x56 fp=0xc00082dd98 sp=0xc00082dd70 pc=0x5fecd6
Wed, Jul 19 2023 09:43:52 | net.cgoLookupIPCNAME.func1({0xc000da2210, 0x7fd23855c5b8?, 0x280?}, 0xc000da2180?, 0x60549b?)
Wed, Jul 19 2023 09:43:52 | /usr/local/go/src/net/cgo_unix.go:160 +0x9f fp=0xc00082ddf0 sp=0xc00082dd98 pc=0x6009ff
Wed, Jul 19 2023 09:43:52 | net.cgoLookupIPCNAME({0x27e8a5f, 0x3}, {0xc000da2180, 0x29})
Wed, Jul 19 2023 09:43:52 | /usr/local/go/src/net/cgo_unix.go:160 +0x16d fp=0xc00082df38 sp=0xc00082ddf0 pc=0x60026d
Wed, Jul 19 2023 09:43:52 | net.cgoIPLookup(0x4066770?, {0x27e8a5f?, 0xc000da21e0?}, {0xc000da2180?, 0xc00016e000?})
Wed, Jul 19 2023 09:43:52 | /usr/local/go/src/net/cgo_unix.go:217 +0x3b fp=0xc00082dfa8 sp=0xc00082df38 pc=0x600abb
Wed, Jul 19 2023 09:43:52 | net.cgoLookupIP.func1()
Wed, Jul 19 2023 09:43:52 | /usr/local/go/src/net/cgo_unix.go:227 +0x36 fp=0xc00082dfe0 sp=0xc00082dfa8 pc=0x600ef6
Wed, Jul 19 2023 09:43:52 | runtime.goexit()
Wed, Jul 19 2023 09:43:52 | /usr/local/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00082dfe8 sp=0xc00082dfe0 pc=0x46d981
Wed, Jul 19 2023 09:43:52 | created by net.cgoLookupIP
Wed, Jul 19 2023 09:43:52 | /usr/local/go/src/net/cgo_unix.go:227 +0x12a
Wed, Jul 19 2023 09:43:52 |  
Wed, Jul 19 2023 09:43:52 | goroutine 1 [chan receive]:
Wed, Jul 19 2023 09:43:52 | main.main()
Wed, Jul 19 2023 09:43:52 | /cmd/loggie/main.go:139 +0x92a
Wed, Jul 19 2023 09:43:52 |  
Wed, Jul 19 2023 09:43:52 | goroutine 6 [sleep]:
Wed, Jul 19 2023 09:43:52 | time.Sleep(0x6fc23ac00)
Wed, Jul 19 2023 09:43:52 | /usr/local/go/src/runtime/time.go:194 +0x12e
Wed, Jul 19 2023 09:43:52 | sigs.k8s.io/controller-runtime/pkg/log.init.0.func1()
Wed, Jul 19 2023 09:43:52 | /vendor/sigs.k8s.io/controller-runtime/pkg/log/log.go:63 +0x38
Wed, Jul 19 2023 09:43:52 | created by sigs.k8s.io/controller-runtime/pkg/log.init.0
Wed, Jul 19 2023 09:43:52 | /vendor/sigs.k8s.io/controller-runtime/pkg/log/log.go:62 +0x25
Wed, Jul 19 2023 09:43:52 |  
Wed, Jul 19 2023 09:43:52 | goroutine 39 [chan receive]:
Wed, Jul 19 2023 09:43:52 | github.com/panjf2000/ants/v2.(*Pool).purgePeriodically(0xc000299810)
Wed, Jul 19 2023 09:43:52 | /vendor/github.com/panjf2000/ants/v2/pool.go:69 +0x8b
Wed, Jul 19 2023 09:43:52 | created by github.com/panjf2000/ants/v2.NewPool
Wed, Jul 19 2023 09:43:52 | /vendor/github.com/panjf2000/ants/v2/pool.go:137 +0x34a
Wed, Jul 19 2023 09:43:52 |  
Wed, Jul 19 2023 09:43:52 | goroutine 15 [select]:
Wed, Jul 19 2023 09:43:52 | github.com/loggie-io/loggie/pkg/eventbus/export/logger.(*logger).run(0xc00018a2c0, {0x1, 0x6fc23ac00, 0x0, 0x0, {{0x0, 0x0}, 0x0, 0x0, 0x0, ...}})
Wed, Jul 19 2023 09:43:52 | /pkg/eventbus/export/logger/logger.go:107 +0x19c
Wed, Jul 19 2023 09:43:52 | created by github.com/loggie-io/loggie/pkg/eventbus/export/logger.Run
Wed, Jul 19 2023 09:43:52 | /pkg/eventbus/export/logger/logger.go:94 +0xdd
Wed, Jul 19 2023 09:43:52 |  
Wed, Jul 19 2023 09:43:52 | goroutine 66 [syscall]:
Wed, Jul 19 2023 09:43:52 | os/signal.signal_recv()
Wed, Jul 19 2023 09:43:52 | /usr/local/go/src/runtime/sigqueue.go:151 +0x2f
Wed, Jul 19 2023 09:43:52 | os/signal.loop()
Wed, Jul 19 2023 09:43:52 | /usr/local/go/src/os/signal/signal_unix.go:23 +0x19
Wed, Jul 19 2023 09:43:52 | created by os/signal.Notify.func1.1
Wed, Jul 19 2023 09:43:52 | /usr/local/go/src/os/signal/signal.go:151 +0x2a
Wed, Jul 19 2023 09:43:52 |  
Wed, Jul 19 2023 09:43:52 | goroutine 67 [chan receive]:
Wed, Jul 19 2023 09:43:52 | github.com/loggie-io/loggie/pkg/core/signals.SetupSignalHandler.func1()
Wed, Jul 19 2023 09:43:52 | /pkg/core/signals/signal.go:39 +0x2d
Wed, Jul 19 2023 09:43:52 | created by github.com/loggie-io/loggie/pkg/core/signals.SetupSignalHandler
Wed, Jul 19 2023 09:43:52 | /pkg/core/signals/signal.go:38 +0xbc
Wed, Jul 19 2023 09:43:52 |  
Wed, Jul 19 2023 09:43:52 | goroutine 16 [select]:
Wed, Jul 19 2023 09:43:52 | github.com/loggie-io/loggie/pkg/eventbus/listener/sink.(*Listener).run(0xc00026e760)
Wed, Jul 19 2023 09:43:52 | /pkg/eventbus/listener/sink/listener.go:103 +0x12f
Wed, Jul 19 2023 09:43:52 | created by github.com/loggie-io/loggie/pkg/eventbus/listener/sink.(*Listener).Start
Wed, Jul 19 2023 09:43:52 | /pkg/eventbus/listener/sink/listener.go:79 +0x56
Wed, Jul 19 2023 09:43:52 |  
Wed, Jul 19 2023 09:43:52 | goroutine 82 [select]:
Wed, Jul 19 2023 09:43:52 | github.com/loggie-io/loggie/pkg/eventbus/listener/filesource.(*Listener).run(0xc00026e780)
Wed, Jul 19 2023 09:43:52 | /pkg/eventbus/listener/filesource/listener.go:192 +0x13d
Wed, Jul 19 2023 09:43:52 | created by github.com/loggie-io/loggie/pkg/eventbus/listener/filesource.(*Listener).Start
Wed, Jul 19 2023 09:43:52 | /pkg/eventbus/listener/filesource/listener.go:87 +0x56
Wed, Jul 19 2023 09:43:52 |  
Wed, Jul 19 2023 09:43:52 | goroutine 83 [select]:
Wed, Jul 19 2023 09:43:52 | github.com/loggie-io/loggie/pkg/eventbus/listener/filewatcher.(*Listener).run(0xc00026e7c0)
Wed, Jul 19 2023 09:43:52 | /pkg/eventbus/listener/filewatcher/listener.go:201 +0x13d
Wed, Jul 19 2023 09:43:52 | created by github.com/loggie-io/loggie/pkg/eventbus/listener/filewatcher.(*Listener).Start
Wed, Jul 19 2023 09:43:52 | /pkg/eventbus/listener/filewatcher/listener.go:90 +0x56
Wed, Jul 19 2023 09:43:52 |  
Wed, Jul 19 2023 09:43:52 | goroutine 84 [select]:
Wed, Jul 19 2023 09:43:52 | github.com/loggie-io/loggie/pkg/eventbus/listener/pipeline.(*Listener).run(0xc0009b4690)
Wed, Jul 19 2023 09:43:52 | /pkg/eventbus/listener/pipeline/listener.go:87 +0x114
Wed, Jul 19 2023 09:43:52 | created by github.com/loggie-io/loggie/pkg/eventbus/listener/pipeline.(*Listener).Start
Wed, Jul 19 2023 09:43:52 | /pkg/eventbus/listener/pipeline/listener.go:67 +0x56
Wed, Jul 19 2023 09:43:52 |  
Wed, Jul 19 2023 09:43:52 | goroutine 85 [select]:
Wed, Jul 19 2023 09:43:52 | github.com/loggie-io/loggie/pkg/eventbus/listener/queue.(*Listener).run(0xc00026e800)
Wed, Jul 19 2023 09:43:52 | /pkg/eventbus/listener/queue/listener.go:102 +0x126
Wed, Jul 19 2023 09:43:52 | created by github.com/loggie-io/loggie/pkg/eventbus/listener/queue.(*Listener).Start
Wed, Jul 19 2023 09:43:52 | /pkg/eventbus/listener/queue/listener.go:78 +0x56
Wed, Jul 19 2023 09:43:52 |  
Wed, Jul 19 2023 09:43:52 | goroutine 86 [select]:
Wed, Jul 19 2023 09:43:52 | github.com/loggie-io/loggie/pkg/eventbus/listener/reload.(*Listener).export(0xc00037c918)
Wed, Jul 19 2023 09:43:52 | /pkg/eventbus/listener/reload/listener.go:96 +0xf4
Wed, Jul 19 2023 09:43:52 | created by github.com/loggie-io/loggie/pkg/eventbus/listener/reload.(*Listener).Start
Wed, Jul 19 2023 09:43:52 | /pkg/eventbus/listener/reload/listener.go:71 +0x56
Wed, Jul 19 2023 09:43:52 |  
Wed, Jul 19 2023 09:43:52 | goroutine 87 [select]:
Wed, Jul 19 2023 09:43:52 | github.com/loggie-io/loggie/pkg/eventbus.(*EventCenter).run(0xc000497590)
Wed, Jul 19 2023 09:43:52 | /pkg/eventbus/center.go:191 +0xb9
Wed, Jul 19 2023 09:43:52 | created by github.com/loggie-io/loggie/pkg/eventbus.StartAndRun
Wed, Jul 19 2023 09:43:52 | /pkg/eventbus/center.go:88 +0x65
Wed, Jul 19 2023 09:43:52 |  
Wed, Jul 19 2023 09:43:52 | goroutine 88 [select]:
Wed, Jul 19 2023 09:43:52 | github.com/loggie-io/loggie/pkg/interceptor/retry.(*Interceptor).run(0xc000415500)
Wed, Jul 19 2023 09:43:52 | /pkg/interceptor/retry/interceptor.go:177 +0x1ec
Wed, Jul 19 2023 09:43:52 | created by github.com/loggie-io/loggie/pkg/interceptor/retry.(*Interceptor).Start
Wed, Jul 19 2023 09:43:52 | /pkg/interceptor/retry/interceptor.go:118 +0x5a
Wed, Jul 19 2023 09:43:52 |  
Wed, Jul 19 2023 09:43:52 | goroutine 89 [select]:
Wed, Jul 19 2023 09:43:52 | database/sql.(*DB).connectionOpener(0xc0009b76c0, {0x2c1c9e8, 0xc0004531c0})
Wed, Jul 19 2023 09:43:52 | /usr/local/go/src/database/sql/sql.go:1226 +0x8d
Wed, Jul 19 2023 09:43:52 | created by database/sql.OpenDB
Wed, Jul 19 2023 09:43:52 | /usr/local/go/src/database/sql/sql.go:794 +0x18d
Wed, Jul 19 2023 09:43:52 |  
Wed, Jul 19 2023 09:43:52 | goroutine 90 [select]:
Wed, Jul 19 2023 09:43:52 | github.com/loggie-io/loggie/pkg/source/file.(*dbHandler).run(0xc000415800)
Wed, Jul 19 2023 09:43:52 | /pkg/source/file/persistence.go:201 +0x265
Wed, Jul 19 2023 09:43:52 | created by github.com/loggie-io/loggie/pkg/source/file.newDbHandler
Wed, Jul 19 2023 09:43:52 | /pkg/source/file/persistence.go:115 +0x20c
Wed, Jul 19 2023 09:43:52 |  
Wed, Jul 19 2023 09:43:52 | goroutine 91 [select]:
Wed, Jul 19 2023 09:43:52 | github.com/loggie-io/loggie/pkg/source/file.(*AckChainHandler).run(0xc000378550)
Wed, Jul 19 2023 09:43:52 | /pkg/source/file/ack.go:304 +0x1ec
Wed, Jul 19 2023 09:43:52 | created by github.com/loggie-io/loggie/pkg/source/file.NewAckChainHandler
Wed, Jul 19 2023 09:43:52 | /pkg/source/file/ack.go:269 +0x1c5
Wed, Jul 19 2023 09:43:52 |  
Wed, Jul 19 2023 09:43:52 | goroutine 92 [syscall]:
Wed, Jul 19 2023 09:43:52 | syscall.Syscall6(0xe8, 0x8, 0xc00069dc14, 0x7, 0xffffffffffffffff, 0x0, 0x0)
Wed, Jul 19 2023 09:43:52 | /usr/local/go/src/syscall/asm_linux_amd64.s:43 +0x5
Wed, Jul 19 2023 09:43:52 | golang.org/x/sys/unix.EpollWait(0x0?, {0xc00069dc14?, 0x0?, 0x0?}, 0x0?)
Wed, Jul 19 2023 09:43:52 | /vendor/golang.org/x/sys/unix/zsyscall_linux_amd64.go:56 +0x58
Wed, Jul 19 2023 09:43:52 | github.com/fsnotify/fsnotify.(*fdPoller).wait(0xc000580080)
Wed, Jul 19 2023 09:43:52 | /vendor/github.com/fsnotify/fsnotify/inotify_poller.go:86 +0x7d
Wed, Jul 19 2023 09:43:52 | github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc0003785a0)
Wed, Jul 19 2023 09:43:52 | /vendor/github.com/fsnotify/fsnotify/inotify.go:206 +0x26e
Wed, Jul 19 2023 09:43:52 | created by github.com/fsnotify/fsnotify.NewWatcher
Wed, Jul 19 2023 09:43:52 | /vendor/github.com/fsnotify/fsnotify/inotify.go:60 +0x1c5
Wed, Jul 19 2023 09:43:52 |  
Wed, Jul 19 2023 09:43:52 | goroutine 93 [select]:
Wed, Jul 19 2023 09:43:52 | github.com/loggie-io/loggie/pkg/source/file.(*Watcher).run(0xc00027efc0)
Wed, Jul 19 2023 09:43:52 | /pkg/source/file/watch.go:682 +0x217
Wed, Jul 19 2023 09:43:52 | created by github.com/loggie-io/loggie/pkg/source/file.newWatcher
Wed, Jul 19 2023 09:43:52 | /pkg/source/file/watch.go:81 +0x2ac

其中 Interceptor 配置的 yaml 如下：

apiVersion: loggie.io/v1beta1
kind: Interceptor
metadata:
  name: default
spec:
  interceptors: |
    - type: rateLimit
      qps: 90000

Sink 的配置的 yaml 如下：

apiVersion: loggie.io/v1beta1
kind: Sink
metadata:
  name: default
spec:
  sink: |
    type: loki
    url: "http://loki-gateway.monitoring.svc.cluster.local/loki/api/v1/push"

LogConf 的配置 yaml 如下：

apiVersion: loggie.io/v1beta1
kind: LogConfig
metadata:
  name: matter
  namespace: alpha-dev
spec:
  selector:
    type: pod
    labelSelector:
      app: "matter"
  pipeline:
    sources: |
      - type: file
        name: common
        paths:
          - stdout
    sinkRef: default
    interceptorRef: default

根据官方文档案例进行配置的，还请解惑。

ethfoo commented 1 year ago

cgo导致的问题，请参考：https://loggie-io.github.io/docs/main/user-guide/troubleshot/problems/#netcgolookupip-segmentation-violation

可以考虑试用非CGO版本，参考：https://loggie-io.github.io/docs/main/developer-guide/build/#_2