容器集群-安全组件安装问题

[AceCoronnet9034]

问题描述： 我在Elkeid平台系统管理->容器集群->添加集群页面添加了一个k8s集群。添加完安装安全组件安装指引中提供的步骤(三个master节点均有操作)，完成了所有步骤。更改完kube-apiserver.yaml后，集群状态正常，无报错日志。但elkeid平台依然显示入侵&威胁检测状态：未安装。且确实未获取数据。

针对这个现象，我做了哪些操作： 1，删除集群重新在平台添加后再次尝试那些步骤。 2，检查核对了audit-policy.yaml和audit.kubeconfig文件内容，确定与平台生成的内容一致。 3，依次重启了k8s集群的三个master节点。 4，查看apiserver日志(无异常)。

环境信息： OS：Ubuntu 20.04.6 LTS K8S：v1.22.10 内核版本：1 5.4.0-189-generic

文件路径： /etc/kubernetes/elkeid-audit/audit-policy.yaml /etc/kubernetes/elkeid-audit/audit.kubeconfig

kube-apiserver.yaml内容： apiVersion: v1 kind: Pod metadata: annotations: kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 172.20.1.1:6443 creationTimestamp: null labels: component: kube-apiserver tier: control-plane name: kube-apiserver namespace: kube-system spec: containers:

• command:
  • kube-apiserver
  • --advertise-address=172.20.1.1
  • --allow-privileged=true
  • --authorization-mode=Node,RBAC
  • --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
  • --audit-webhook-config-file=/etc/kubernetes/audit/audit.kubeconfig
  • --client-ca-file=/etc/kubernetes/pki/ca.crt
  • --enable-admission-plugins=NodeRestriction
  • --enable-bootstrap-token-auth=true
  • --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
  • --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
  • --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
  • --etcd-servers=https://127.0.0.1:2379
  • --feature-gates=TTLAfterFinished=true
  • --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
  • --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
  • --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
  • --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
  • --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
  • --requestheader-allowed-names=front-proxy-client
  • --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
  • --requestheader-extra-headers-prefix=X-Remote-Extra-
  • --requestheader-group-headers=X-Remote-Group
  • --requestheader-username-headers=X-Remote-User
  • --secure-port=6443
  • --service-account-issuer=https://kubernetes.default.svc.cluster.local
  • --service-account-key-file=/etc/kubernetes/pki/sa.pub
  • --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
  • --service-cluster-ip-range=10.96.0.0/12
  • --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
  • --tls-private-key-file=/etc/kubernetes/pki/apiserver.key image: k8s.gcr.io/kube-apiserver:v1.22.10 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 8 httpGet: host: 172.20.1.1 path: /livez port: 6443 scheme: HTTPS initialDelaySeconds: 10 periodSeconds: 10 timeoutSeconds: 15 name: kube-apiserver readinessProbe: failureThreshold: 3 httpGet: host: 172.20.1.1 path: /readyz port: 6443 scheme: HTTPS periodSeconds: 1 timeoutSeconds: 15 resources: requests: cpu: 250m startupProbe: failureThreshold: 24 httpGet: host: 172.20.1.1 path: /livez port: 6443 scheme: HTTPS initialDelaySeconds: 10 periodSeconds: 10 timeoutSeconds: 15 volumeMounts:
  • mountPath: /etc/ssl/certs name: ca-certs readOnly: true
  • mountPath: /etc/kubernetes/audit/ name: elkeid-audit readOnly: true
  • mountPath: /etc/ca-certificates name: etc-ca-certificates readOnly: true
  • mountPath: /etc/pki name: etc-pki readOnly: true
  • mountPath: /etc/kubernetes/pki name: k8s-certs readOnly: true
  • mountPath: /etc/localtime name: localtime readOnly: true
  • mountPath: /usr/local/share/ca-certificates name: usr-local-share-ca-certificates readOnly: true
  • mountPath: /usr/share/ca-certificates name: usr-share-ca-certificates readOnly: true hostNetwork: true priorityClassName: system-node-critical securityContext: seccompProfile: type: RuntimeDefault volumes:
• hostPath: path: /etc/ssl/certs type: DirectoryOrCreate name: ca-certs
• hostPath: path: /etc/kubernetes/elkeid-audit type: Directory name: elkeid-audit
• hostPath: path: /etc/ca-certificates type: DirectoryOrCreate name: etc-ca-certificates
• hostPath: path: /etc/pki type: DirectoryOrCreate name: etc-pki
• hostPath: path: /etc/kubernetes/pki type: DirectoryOrCreate name: k8s-certs
• hostPath: path: /etc/localtime type: File name: localtime
• hostPath: path: /usr/local/share/ca-certificates type: DirectoryOrCreate name: usr-local-share-ca-certificates
• hostPath: path: /usr/share/ca-certificates type: DirectoryOrCreate name: usr-share-ca-certificates status: {}

[UgOrange]

可以使用 curl -k 测试下 agent_center 审计日志是否正常监听，地址在下载到的 audit webhook config file 中 server 字段中（端口为 6754）

[UgOrange]

比如curl -k -X POST -H "Content-Type: application/json" --data '{"kind":"Event"}' https://{URL}

[AceCoronnet9034]

默认生成的文件中，server字段为server: https://127.0.0.1:6754/rawdata/audit。这里127.0.0.1:6754是表示我需要在k8s节点主机上安装agent_center服务吗？我elkeid部署是按照"单机docker快速部署 (单机测试环境推荐)"文档使用docker部署的，elkeid本身并没有监听6754端口。

[UgOrange]

需要确保 k8s 能够访问通 agent_center 所在的 6754 端口，该端口用于接收审计日志。docker 部署的 ac 应该也会监听对应端口，可以在docker run的时候指定端口映射

[AceCoronnet9034]

我给elkeid_community容器添加了6754端口映射(allinone部署方式只有这个容器)，然后使用命令curl -k -X POST -H "Content-Type: application/json" --data '{"kind":"Event"}' https://172.20.1.10:6754显示结果curl: (56) OpenSSL SSL_read: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate, errno 0。在容器内执行curl得到的结果为curl: (58) NSS: client certificate not found (nickname not specified)。正常的返回结果应该是什么？

[UgOrange]

可以重新配置一下 k8s 看看能不能正常接收到审计日志

[AceCoronnet9034]

重新配置了k8s依然是未安装，我查看master1节点的kube-apiserver日志如下：AUDIT: id="a5b2abcc-17b3-4d0e-967c-e8d8e0e23ffb" stage="ResponseComplete" ip="172.20.1.1" method="patch" user="system:node:master-1" groups="\"system:nodes\",\"system:authenticated\"" as="" asgroups="" user-agent="kubelet/v1.22.10 (linux/amd64) kubernetes/eae22ba" namespace="" uri="/api/v1/nodes/master-1/status?timeout=10s" response="200" 看起来审计似乎配置是生效了的，k8s ping elkeid的6754端口也是通的。安全组件状态始终无法工作。

[UgOrange]

需要确保证书正确，重启 kube-apiserver（可以通过移动并恢复apiserver YAML文件的方式触发重启），然后建议通过 netstat 等工具查看是否有请求 6754 的网络连接

[DirtyPipe]

hello，这个问题有结果吗？我也遇到了。我使用的docker部署，6754端口我是通过iptables转发进去的，在容器里面看有收到容器集群发给6754端口的网络连接。但是找不到为啥安全组件一直不上线

[DirtyPipe]

已经解决，k8s会验证服务端证书的IP范围和证书是否一直，docker生成的证书为127.0.0.1 用脚本/elkeid/agent_center/k8s_cert_gen.sh重新生成证书覆盖原证书并重启服务elkeid_ac.service 然后再去界面重新生成audit.kubeconfig配置文件