root-q commented 1 year ago

yucq01@risleysrv:/usr/local/bin$ kubectl create -f /yucq/test/1/policy-init.yaml varmorpolicy.crd.varmor.org/demo-1 created yucq01@risleysrv:/usr/local/bin$ kubectl get VarmorPolicy -n demo NAME ENFORCER MODE TARGET-KIND TARGET-NAME TARGET-SELECTOR PROFILE-NAME READY STATUS AGE

demo-1 AppArmor AlwaysAllow Deployment {"matchLabels":{"app":"demo-1"}} varmor-demo-demo-1 false Error 14s

root-q commented 1 year ago

apiVersion: crd.varmor.org/v1beta1 kind: VarmorPolicy metadata: name: demo-1 namespace: demo spec: target: kind: Deployment selector: matchLabels: app: demo-1 policy: enforcer: AppArmor mode: AlwaysAllow

Danny-Wei commented 1 year ago

请提供更多信息，方便定位原因：

CR 对象的状态

kubectl get VarmorPolicy -n demo demo-1 -o jsonpath='{.status}'
kubectl get ArmorProfile -n demo varmor-demo-demo-1 -o jsonpath='{.status}'

Manager & Agent 的日志（如有 Error 则提供）
系统内核版本和 LSM 启用状态
```
uname -a
cat /sys/kernel/security/lsm
```

root-q commented 1 year ago

yucq01@risleysrv:/usr/local/bin$ kubectl get VarmorPolicy -n demo demo-1 -o jsonpath='{.status}' {"conditions":[{"lastTransitionTime":"2023-11-30T05:12:48Z","status":"True","type":"Created"}],"phase":"Error","profileName":"varmor-demo-demo-1","ready":false} yucq01@risleysrv:/usr/local/bin$ kubectl get ArmorProfile -n demo varmor-demo-demo-1 -o jsonpath='{.status}' {"conditions":[{"lastTransitionTime":"2023-11-30T05:12:48Z","message":"loadAppArmorProfile(): exit status 1 Cache read/write disabled: interface file missing. (Kernel needs AppArmor 2.4 compatibility patch.)\nWarning: unable to find a suitable fs in /proc/mounts, is it mounted?\nUse --subdomainfs to override.\n","nodeName":"minikube","status":"False","type":"Ready"}],"currentNumberLoaded":0,"desiredNumberLoaded":1} yucq01@risleysrv:/usr/local/bin$ uname -a Linux risleysrv 5.4.0-21-generic #25-Ubuntu SMP Sat Mar 28 13:10:28 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux yucq01@risleysrv:/usr/local/bin$ cat /sys/kernel/security/lsm lockdown,capability,yama,apparmor

Danny-Wei commented 1 year ago

你使用的是 Ubuntu 发行版的哪个版本？猜测你使用的内核裁剪并关闭了 AppArmor LSM 功能。请通过 cat /sys/module/apparmor/parameters/enabled 确认 AppArmor LSM 是否开启。

root-q commented 1 year ago

yucq01@risleysrv:/usr/local/bin$ cat /etc/os-release NAME="Ubuntu" VERSION="20.04.6 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.6 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal yucq01@risleysrv:/usr/local/bin$ cat /sys/module/apparmor/parameters/enabled Y

root-q commented 1 year ago

系统是Ubantu20.04

感谢大佬的反馈，我无法解决问题

yucq01@risleysrv:/root$ sudo aa-status [sudo] yucq01 的密码： apparmor module is loaded. 41 profiles are loaded. 39 profiles are in enforce mode. /snap/core/16202/usr/lib/snapd/snap-confine /snap/core/16202/usr/lib/snapd/snap-confine//mount-namespace-capture-helper /usr/bin/evince /usr/bin/evince-previewer /usr/bin/evince-previewer//sanitized_helper /usr/bin/evince-thumbnailer /usr/bin/evince//sanitized_helper /usr/bin/man /usr/lib/NetworkManager/nm-dhcp-client.action /usr/lib/NetworkManager/nm-dhcp-helper /usr/lib/connman/scripts/dhclient-script /usr/lib/cups/backend/cups-pdf /usr/lib/snapd/snap-confine /usr/lib/snapd/snap-confine//mount-namespace-capture-helper /usr/sbin/cups-browsed /usr/sbin/cupsd /usr/sbin/cupsd//third_party /usr/sbin/tcpdump /{,usr/}sbin/dhclient docker-default ippusbxd libreoffice-senddoc libreoffice-soffice//gpg libreoffice-xpdfimport libvirtd libvirtd//qemu_bridge_helper lsb_release man_filter man_groff nvidia_modprobe nvidia_modprobe//kmod snap-update-ns.core snap-update-ns.snap-store snap.core.hook.configure snap.snap-store.hook.configure snap.snap-store.snap-store snap.snap-store.ubuntu-software snap.snap-store.ubuntu-software-local-file virt-aa-helper 2 profiles are in complain mode. libreoffice-oopslash libreoffice-soffice 2 processes have profiles defined. 2 processes are in enforce mode. /usr/sbin/cups-browsed (910270) /usr/sbin/cupsd (910268) 0 processes are in complain mode. 0 processes are unconfined but have a profile defined.

Danny-Wei commented 1 year ago

感谢反馈，bug 已经定位了，是因为加载 AppArmor Profile 时无法 feature abi 不匹配导致的。近期我们会修复这个问题并发布新版本。

Danny-Wei commented 1 year ago

抱歉，之前的判断可能有误。我在本地 Ubuntu 20.04.5 测试环境中没有复现这个问题。

请你提供以下信息：

Agent 的日志
Agent DaemonSet 的 YAML 文件 cat kubectl get ds -n varmor varmor-agent -o yaml
进入 Agent 容器后，执行 aa-features-abi -x 的输出结果

root-q commented 1 year ago

1.Agent 的日志

yucq01@risleysrv:/usr/local/bin$ I1130 07:30:31.875811 I1130 07:30:31.876745 I1130 07:30:31.878650 I1130 07:30:31.895575 I1130 07:30:31.895627 I1130 07:30:31.895647 I1130 07:30:31.984607 I1130 07:30:31.987252 I1130 07:30:31.987658 I1130 07:31:02.499413 I1130 07:31:02.499647 I1130 07:31:02.500594 E1130 07:31:02.503667 I1201 02:26:55.001355 I1201 02:26:55.001454 E1201 02:26:55.001685 E1201 02:26:55.001732 I1201 02:26:55.011140 I1201 02:26:55.011188 E1201 02:26:55.011259 E1201 02:26:55.011309 I1201 02:26:55.026560 I1201 02:26:55.026633 E1201 02:26:55.026693 E1201 02:26:55.026746 I1201 02:26:55.050119 I1201 02:26:55.050149 E1201 02:26:55.050196 E1201 02:26:55.050214 I1201 02:26:55.093687 I1201 02:26:55.093728 E1201 02:26:55.093782 E1201 02:26:55.093813 I1201 02:26:55.177255 I1201 02:26:55.177290 E1201 02:26:55.177331 E1201 02:26:55.177360 I1201 02:26:55.340934 I1201 02:26:55.341005 E1201 02:26:55.341063 E1201 02:26:55.341085 I1201 02:26:55.665222 I1201 02:26:55.665270 E1201 02:26:55.665322 E1201 02:26:55.665344 I1201 02:26:56.310153 I1201 02:26:56.310195 E1201 02:26:56.310250 E1201 02:26:56.310280 I1201 02:26:57.593996 I1201 02:26:57.594039 E1201 02:26:57.594090 E1201 02:26:57.594115 I1201 02:27:00.158956 I1201 02:27:00.159002 E1201 02:27:00.159054 E1201 02:27:00.159090 E1201 02:27:00.159161 I1201 02:27:12.454995 I1201 02:27:12.455036 I1201 02:27:12.455691 E1201 02:27:12.459676 kubectl logs varmor-agent-4frnk -n varmor 1 config.go:136] CreateClientConfig "msg"="Using in-cluster configuration" 1 main.go:129] SETUP "msg"="vArmor agent startup" 1 agent.go:132] AGENT "msg"="the BPF enforcer is not enabled (use --enableBpfEnforcer to enable it)" 1 agent.go:145] AGENT "msg"="NewAgent" "nodeName"="minikube" 1 agent.go:149] AGENT "msg"="initialize the AppArmor LSM" 1 agent.go:152] AGENT "msg"="setup the AppArmor feature ABI, abstractions, tunables and default profiles to /etc/apparmor.d" 1 agent.go:160] AGENT "msg"="setup the mock cri-containerd.apparmor.d profile for containerd" 1 main.go:153] SETUP "msg"="vArmor agent is online" 1 agent.go:543] AGENT "msg"="starting" 1 agent.go:311] AGENT/handleCreateOrUpdateArmorProfile() "msg"="ArmorProfile created or updated" "labels"=null "name"="varmor-demo-demo-1" "namespace"="demo" "profile mode"="enforce" "profile name"="varmor-demo-demo-1" 1 agent.go:383] AGENT/handleCreateOrUpdateArmorProfile() "msg"="saving the AppArmor profile ('varmor-demo-demo-1') to Node/minikube" 1 agent.go:393] AGENT/handleCreateOrUpdateArmorProfile() "msg"="loading 'varmor-demo-demo-1 (enforce)' to Node/minikube's kernel" 1 agent.go:396] AGENT/handleCreateOrUpdateArmorProfile() "msg"="loadAppArmorProfile()" "error"="exit status 1" "output"="Cache read/write disabled: interface file missing. (Kernel needs AppArmor 2.4 compatibility patch.)\nWarning: unable to find a suitable fs in /proc/mounts, is it mounted?\nUse --subdomainfs to override.\n" 1 agent.go:427] AGENT/handleDeleteArmorProfile() "msg"="ArmorProfile deleted" "name"="varmor-demo-demo-1" "namespace"="demo" 1 agent.go:446] AGENT/handleDeleteArmorProfile() "msg"="unloading the AppArmor profile ('varmor-demo-demo-1') from Node/minikube's kernel" 1 agent.go:449] AGENT/handleDeleteArmorProfile() "msg"="isAppArmorProfileLoaded()" "error"="open /sys/kernel/security/apparmor/profiles: no such file or directory" 1 agent.go:514] AGENT "msg"="failed to sync profile" "error"="open /sys/kernel/security/apparmor/profiles: no such file or directory" "key"="demo/varmor-demo-demo-1" 1 agent.go:427] AGENT/handleDeleteArmorProfile() "msg"="ArmorProfile deleted" "name"="varmor-demo-demo-1" "namespace"="demo" 1 agent.go:446] AGENT/handleDeleteArmorProfile() "msg"="unloading the AppArmor profile ('varmor-demo-demo-1') from Node/minikube's kernel" 1 agent.go:449] AGENT/handleDeleteArmorProfile() "msg"="isAppArmorProfileLoaded()" "error"="open /sys/kernel/security/apparmor/profiles: no such file or directory" 1 agent.go:514] AGENT "msg"="failed to sync profile" "error"="open /sys/kernel/security/apparmor/profiles: no such file or directory" "key"="demo/varmor-demo-demo-1" 1 agent.go:427] AGENT/handleDeleteArmorProfile() "msg"="ArmorProfile deleted" "name"="varmor-demo-demo-1" "namespace"="demo" 1 agent.go:446] AGENT/handleDeleteArmorProfile() "msg"="unloading the AppArmor profile ('varmor-demo-demo-1') from Node/minikube's kernel" 1 agent.go:449] AGENT/handleDeleteArmorProfile() "msg"="isAppArmorProfileLoaded()" "error"="open /sys/kernel/security/apparmor/profiles: no such file or directory" 1 agent.go:514] AGENT "msg"="failed to sync profile" "error"="open /sys/kernel/security/apparmor/profiles: no such file or directory" "key"="demo/varmor-demo-demo-1" 1 agent.go:427] AGENT/handleDeleteArmorProfile() "msg"="ArmorProfile deleted" "name"="varmor-demo-demo-1" "namespace"="demo" 1 agent.go:446] AGENT/handleDeleteArmorProfile() "msg"="unloading the AppArmor profile ('varmor-demo-demo-1') from Node/minikube's kernel" 1 agent.go:449] AGENT/handleDeleteArmorProfile() "msg"="isAppArmorProfileLoaded()" "error"="open /sys/kernel/security/apparmor/profiles: no such file or directory" 1 agent.go:514] AGENT "msg"="failed to sync profile" "error"="open /sys/kernel/security/apparmor/profiles: no such file or directory" "key"="demo/varmor-demo-demo-1" 1 agent.go:427] AGENT/handleDeleteArmorProfile() "msg"="ArmorProfile deleted" "name"="varmor-demo-demo-1" "namespace"="demo" 1 agent.go:446] AGENT/handleDeleteArmorProfile() "msg"="unloading the AppArmor profile ('varmor-demo-demo-1') from Node/minikube's kernel" 1 agent.go:449] AGENT/handleDeleteArmorProfile() "msg"="isAppArmorProfileLoaded()" "error"="open /sys/kernel/security/apparmor/profiles: no such file or directory" 1 agent.go:514] AGENT "msg"="failed to sync profile" "error"="open /sys/kernel/security/apparmor/profiles: no such file or directory" "key"="demo/varmor-demo-demo-1" 1 agent.go:427] AGENT/handleDeleteArmorProfile() "msg"="ArmorProfile deleted" "name"="varmor-demo-demo-1" "namespace"="demo" 1 agent.go:446] AGENT/handleDeleteArmorProfile() "msg"="unloading the AppArmor profile ('varmor-demo-demo-1') from Node/minikube's kernel" 1 agent.go:449] AGENT/handleDeleteArmorProfile() "msg"="isAppArmorProfileLoaded()" "error"="open /sys/kernel/security/apparmor/profiles: no such file or directory" 1 agent.go:514] AGENT "msg"="failed to sync profile" "error"="open /sys/kernel/security/apparmor/profiles: no such file or directory" "key"="demo/varmor-demo-demo-1" 1 agent.go:427] AGENT/handleDeleteArmorProfile() "msg"="ArmorProfile deleted" "name"="varmor-demo-demo-1" "namespace"="demo" 1 agent.go:446] AGENT/handleDeleteArmorProfile() "msg"="unloading the AppArmor profile ('varmor-demo-demo-1') from Node/minikube's kernel" 1 agent.go:449] AGENT/handleDeleteArmorProfile() "msg"="isAppArmorProfileLoaded()" "error"="open /sys/kernel/security/apparmor/profiles: no such file or directory" 1 agent.go:514] AGENT "msg"="failed to sync profile" "error"="open /sys/kernel/security/apparmor/profiles: no such file or directory" "key"="demo/varmor-demo-demo-1" 1 agent.go:427] AGENT/handleDeleteArmorProfile() "msg"="ArmorProfile deleted" "name"="varmor-demo-demo-1" "namespace"="demo" 1 agent.go:446] AGENT/handleDeleteArmorProfile() "msg"="unloading the AppArmor profile ('varmor-demo-demo-1') from Node/minikube's kernel" 1 agent.go:449] AGENT/handleDeleteArmorProfile() "msg"="isAppArmorProfileLoaded()" "error"="open /sys/kernel/security/apparmor/profiles: no such file or directory" 1 agent.go:514] AGENT "msg"="failed to sync profile" "error"="open /sys/kernel/security/apparmor/profiles: no such file or directory" "key"="demo/varmor-demo-demo-1" 1 agent.go:427] AGENT/handleDeleteArmorProfile() "msg"="ArmorProfile deleted" "name"="varmor-demo-demo-1" "namespace"="demo" 1 agent.go:446] AGENT/handleDeleteArmorProfile() "msg"="unloading the AppArmor profile ('varmor-demo-demo-1') from Node/minikube's kernel" 1 agent.go:449] AGENT/handleDeleteArmorProfile() "msg"="isAppArmorProfileLoaded()" "error"="open /sys/kernel/security/apparmor/profiles: no such file or directory" 1 agent.go:514] AGENT "msg"="failed to sync profile" "error"="open /sys/kernel/security/apparmor/profiles: no such file or directory" "key"="demo/varmor-demo-demo-1" 1 agent.go:427] AGENT/handleDeleteArmorProfile() "msg"="ArmorProfile deleted" "name"="varmor-demo-demo-1" "namespace"="demo" 1 agent.go:446] AGENT/handleDeleteArmorProfile() "msg"="unloading the AppArmor profile ('varmor-demo-demo-1') from Node/minikube's kernel" 1 agent.go:449] AGENT/handleDeleteArmorProfile() "msg"="isAppArmorProfileLoaded()" "error"="open /sys/kernel/security/apparmor/profiles: no such file or directory" 1 agent.go:514] AGENT "msg"="failed to sync profile" "error"="open /sys/kernel/security/apparmor/profiles: no such file or directory" "key"="demo/varmor-demo-demo-1" 1 agent.go:427] AGENT/handleDeleteArmorProfile() "msg"="ArmorProfile deleted" "name"="varmor-demo-demo-1" "namespace"="demo" 1 agent.go:446] AGENT/handleDeleteArmorProfile() "msg"="unloading the AppArmor profile ('varmor-demo-demo-1') from Node/minikube's kernel" 1 agent.go:449] AGENT/handleDeleteArmorProfile() "msg"="isAppArmorProfileLoaded()" "error"="open /sys/kernel/security/apparmor/profiles: no such file or directory" 1 agent.go:519] open /sys/kernel/security/apparmor/profiles: no such file or directory 1 agent.go:520] AGENT "msg"="max retries exceeded, dropping profile out of queue" "error"="open /sys/kernel/security/apparmor/profiles: no such file or directory" "key"="demo/varmor-demo-demo-1" 1 agent.go:311] AGENT/handleCreateOrUpdateArmorProfile() "msg"="ArmorProfile created or updated" "labels"=null "name"="varmor-demo-demo-1" "namespace"="demo" "profile mode"="enforce" "profile name"="varmor-demo-demo-1" 1 agent.go:383] AGENT/handleCreateOrUpdateArmorProfile() "msg"="saving the AppArmor profile ('varmor-demo-demo-1') to Node/minikube" 1 agent.go:393] AGENT/handleCreateOrUpdateArmorProfile() "msg"="loading 'varmor-demo-demo-1 (enforce)' to Node/minikube's kernel" 1 agent.go:396] AGENT/handleCreateOrUpdateArmorProfile() "msg"="loadAppArmorProfile()" "error"="exit status 1" "output"="Cache read/write disabled: interface file missing. (Kernel needs AppArmor 2.4 compatibility patch.)\nWarning: unable to find a suitable fs in /proc/mounts, is it mounted?\nUse --subdomainfs to override.\n"

2.Agent DaemonSet 的 YAML 文件

yucq01@risleysrv:/usr/local/bin$ kubectl get ds -n varmor varmor-agent -o yaml apiVersion: apps/v1 kind: DaemonSet metadata: annotations: deprecated.daemonset.template.generation: "1" meta.helm.sh/release-name: varmor meta.helm.sh/release-namespace: varmor creationTimestamp: "2023-11-30T07:30:16Z" generation: 1 labels: app.kubernetes.io/component: varmor-agent app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: varmor app.kubernetes.io/version: v0.5.4 helm.sh/chart: varmor-0.5.4 name: varmor-agent namespace: varmor resourceVersion: "573" uid: 12d9361c-13af-4d52-b28b-896829d87833 spec: revisionHistoryLimit: 10 selector: matchLabels: app.kubernetes.io/component: varmor-agent app.kubernetes.io/name: varmor template: metadata: creationTimestamp: null labels: app.kubernetes.io/component: varmor-agent app.kubernetes.io/name: varmor spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms:

matchExpressions:
- key: kubernetes.io/os operator: In values:
  - linux
- key: node.kubernetes.io/instance-type operator: NotIn values:
  - virtual-node
matchExpressions:
- key: beta.kubernetes.io/os operator: In values:
  - linux
- key: beta.kubernetes.io/instance-type operator: NotIn values:
  - virtual-node containers:
    - command:
    - /varmor/vArmor
    - --agent image: elkeid-cn-beijing.cr.volces.com/varmor/varmor:v0.5.4 imagePullPolicy: Always name: agent resources: limits: cpu: 200m memory: 100Mi requests: cpu: 100m memory: 40Mi securityContext: privileged: true runAsUser: 0 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts:
    - mountPath: /sys/kernel/security name: securityfs
    - mountPath: /sys/module/apparmor name: apparmor
    - mountPath: /etc/apparmor.d name: apparmor-dir dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: {} serviceAccount: varmor-agent serviceAccountName: varmor-agent terminationGracePeriodSeconds: 30 tolerations:
    - effect: NoSchedule operator: Exists volumes:
    - hostPath: path: /sys/kernel/security type: Directory name: securityfs
    - hostPath: path: /sys/module/apparmor type: Directory name: apparmor
    - hostPath: path: /var/run/varmor/apparmor.d type: DirectoryOrCreate name: apparmor-dir updateStrategy: rollingUpdate: maxSurge: 0 maxUnavailable: 1 type: RollingUpdate status: currentNumberScheduled: 1 desiredNumberScheduled: 1 numberAvailable: 1 numberMisscheduled: 0 numberReady: 1 observedGeneration: 1 updatedNumberScheduled: 1

3.进入 Agent 容器后，执行 aa-features-abi -x 的输出结果

yucq01@risleysrv:/usr/local/bin$ kubectl exec -n varmor varmor-agent-4frnk -it -- /bin/sh

aa-features-abi -x

aa-features-abi: ERROR: failed to extract features abi from the kernel - No such file or directory

Danny-Wei commented 1 year ago

你能在宿主机访问 /sys/kernel/security/apparmor/features 吗？看起来是因为 /sys/kernel/security/apparmor/features 不可用，导致无法获取当前内核的 feature abi。

我将系统升级至 Ubuntu 20.04.6 LTS （5.4.0-169-generic）后仍然没有复现此问题。你可以尝试升级内核版本后再次测试。

root-q commented 1 year ago

感谢大佬反馈，放弃使用minikube后，问题得到解决。

Danny-Wei commented 12 months ago

好的，谢谢反馈。我们后面会在 minikube 环境下尝试复现此问题。

bytedance / vArmor

Step 3. 示例创建 VarmorPolicy后，VarmorPolicy状态为Error #10

aa-features-abi -x