KindlingProject / kindling

eBPF-based Cloud Native Monitoring Tool
http://kindling.harmonycloud.cn
Apache License 2.0
1.11k stars 182 forks source link

目标容器和目标ip不一致 #483

Closed xiaodaiit closed 1 year ago

xiaodaiit commented 1 year ago

{"comm":"postgres","container_id":"cf6c2ebecb20","content_key":"","dnat_ip":"","dnat_port":"-1","dns_domain":"","dns_id":"0","dns_ip":"","dns_rcode":"0","dst_container_id":"cf6c2ebecb20","dst_ip":"172.21.0.7","dst_node_ip":"unknown","dst_port":"6379","error_type":"2","http_method":"","http_status_code":"0","http_url":"","is_error":"true","is_server":"true","is_slow":"false","pid":"16373","protocol":"redis","request_payload":"","response_payload":"","sql":"","src_container_id":"","src_ip":"172.21.0.10","src_node":"","src_node_ip":"","src_port":"37052"}

这里面dst_container_id 是另外一个服务器,不是dst_ip这个ip对应的容器,是为啥

dxsup commented 1 year ago

这条数据是通过networkanalyzer的debug日志看到的吗?看起来数据来自服务端,所以dst_container_id和dst_ip都是从本地获取到的,一般不会出现错误。能描述一下环境信息吗,哪个信息是对的?错误的信息是属于哪个pod的?

xiaodaiit commented 1 year ago

这条数据是通过networkanalyzer的debug日志看到的吗?看起来数据来自服务端,所以dst_container_id和dst_ip都是从本地获取到的,一般不会出现错误。能描述一下环境信息吗,哪个信息是对的?错误的信息是属于哪个pod的?

对是通过debug看到的源数据, 这里面redis信息是对的,但是对应的容器id是不对的

dxsup commented 1 year ago

这个containerid是postgres的吗?是否真的有postgres的请求存在?如果是的话,看起来像是DataGroup复用导致的之前的信息被复用了,新的redis信息还没有覆盖旧信息。我建议查看cgoreceiver的debug事件日志,看是否存在这种错误的事件,如果有说明是底层问题,否则要继续排查networkanalyzer的问题。

xiaodaiit commented 1 year ago

这个containerid是postgres的吗?是否真的有postgres的请求存在?如果是的话,看起来像是DataGroup复用导致的之前的信息被复用了,新的redis信息还没有覆盖旧信息。我建议查看cgoreceiver的debug事件日志,看是否存在这种错误的事件,如果有说明是底层问题,否则要继续排查networkanalyzer的问题。

计划用这个新版本做测试,有没有开发者文档,中文版的,发来研究一下

dxsup commented 1 year ago

目前只有官网有的那些: http://kindling.harmonycloud.cn/blogs/development/collector-pipeline/ http://kindling.harmonycloud.cn/docs/developer-guide/build-kindling-container-image/

需要哪些方面的开发者文档?后面补充起来。

xiaodaiit commented 1 year ago

开源项目我觉得最重要的第一个是让别人能本地构建出来,能debug, git clone https://github.com/kindlingproject/kindling.git

Run the compiling container to build the binaries

./run_docker.sh cd deploy/scripts sh build.sh 这个操作都走不下去。。。

dxsup commented 1 year ago

main分支的代码在构建上发生了变化,文档是0.7.1版本的,还没有及时同步,建议先用release-v0.7.1分支的代码

xiaodaiit commented 1 year ago

[root@centos7-4 scripts]# pwd /home/test/kindling/deploy/scripts [root@centos7-4 scripts]# ls build.sh run_docker.sh [root@centos7-4 scripts]# ./run_docker.sh latest: Pulling from kindlingproject/agent-builder Digest: sha256:96798e58ee4653633c04d0b9636a57fe5b826b8d7582cf5fd74cf8507d18dc33 Status: Image is up to date for kindlingproject/agent-builder:latest [root@34b031e0709d kindling]# cd deploy/scripts/ [root@34b031e0709d scripts]# sh build.sh -- Using local falcosecurity/libs in '/kindling/probe/libs/agent-libs/' -- No strlcpy found, will use local definition CMake Error at cmake/modules/agent-libs.cmake:74 (include): include could not find load file:

libscap

Call Stack (most recent call first): CMakeLists.txt:75 (include)

CMake Error at cmake/modules/agent-libs.cmake:75 (include): include could not find load file:

libsinsp

Call Stack (most recent call first): CMakeLists.txt:75 (include)

CMake Error at src/CMakeLists.txt:2 (include): include could not find load file:

libscap

CMake Error at src/CMakeLists.txt:3 (include): include could not find load file:

libsinsp

CMake Error at src/CMakeLists.txt:6 (include_directories): include_directories given empty-string as include directory.

-- Configuring incomplete, errors occurred! See also "/kindling/probe/build/CMakeFiles/CMakeOutput.log". See also "/kindling/probe/build/CMakeFiles/CMakeError.log". make: *** No targets specified and no makefile found. Stop. compiler libkindling failed! exit!

基于0.7.1分支构建的结果。。。

xiaodaiit commented 1 year ago

你们有没有自己开发环境的流程部署,和上线服务的部署流程,包括宿主机和docker,k8s环境的

dxsup commented 1 year ago
[root@34b031e0709d scripts]# sh build.sh
-- Using local falcosecurity/libs in '/kindling/probe/libs/agent-libs/'

这一行的输出结果表明使用的agent-libs路径是'/kindling/probe/libs/agent-libs/',这个路径是最新代码才改的,看起来并不是0.7.1分支的代码。

xiaodaiit commented 1 year ago
image
xiaodaiit commented 1 year ago

那个联系加群的加不进去啊,有没有交流群给一个

dxsup commented 1 year ago

有,已经申请加微信好友了吧,稍等我确认一下

xiaodaiit commented 1 year ago

嗯 扫的是git页面给的那个微信,通过一下谢谢

dxsup commented 1 year ago

你们有没有自己开发环境的流程部署,和上线服务的部署流程,包括宿主机和docker,k8s环境的

这些后面可以更新到文档里。

现在有在k8s环境的启动的yaml,在deploy/agent路径下,可以执行install.sh。 除非对依赖库和编译流程很了解,否则不推荐在宿主机上直接启动,因为不同环境下兼容性可能有问题。推荐用docker启动测试,命令是:

docker run -d --rm --privileged --pid=host --net=host -v /dev:/host/dev -v /proc:/host/proc -v /root/.kube:/root/.kube kindling-agent:latest

如果内核版本没有适配过,需要提前针对内核版本编译eBPF,然后把产物.o文件挂载到/opt/.kindling/$(uname -r).o

dxsup commented 1 year ago

嗯 扫的是git页面给的那个微信,通过一下谢谢

已经通过了。

image

git statusgit diff看一下

dxsup commented 1 year ago

这个问题有解决吗?