Closed andresmareca-ibm closed 2 years ago
@jotak can you take a look ??? is this connected to the latest PR that removed the decode stage ???
@andresmareca-ibm thanks for looking into this .... I think that this might be more connected to FLP and NOO repo's but having the issue here is also ok.
Hi @andresmareca-ibm ,
What is the image versions of the operator, and of flowlogs-pipeline ? Are they both main
?
This is likely due to a version mismatch, as @eranra mentioned there was a recent breaking change that has a corresponding update on the operator side. I guess you don't have the operator patch? If you built and deployed the operator from source, maybe you wasn't up to date? Or maybe you still have an old image in your cluster, in which case you may double-check if the operator's image pull policy is set to Always
.
Another option is to prefer using released versions rather than main
(but of course you won't get the very last updates)
By the way,
After that I try to deploy what is on netobserv-ebpf-agent. And here it fails for the first time due to lack of serviceAccount permissions. After adding on the serviceAccount I was able to start the actual pods
@mariomac , any idea about that?
@andresmareca-ibm could I see your FlowCollector
deployment file?
Also, if possible, can I see the netobserv-controller-manager pod logs?
I just deployed the main
version and it worked. Are you using any other version?
I've been digging into the default permissions that we grant to the eBPF agent. The netobserv-manager-role
grants security context constraints for the host network but there isn't any rolebinding that assigns the permissions for the netobserv agent to use security context constraints. By any reason, in our installations it's granted by default but not in yours.
In order to try to reproduce the issue, and verify that we provide a patch that will actually work, what version of OpenShift are you using?
I just deployed the
main
version and it worked. Are you using any other version?
I just pull both repos yesterday at 7:30 UTC and run the make commands on the main branch
I've been digging into the default permissions that we grant to the eBPF agent. The
netobserv-manager-role
grants security context constraints for the host network but there isn't any rolebinding that assigns the permissions for the netobserv agent to use security context constraints. By any reason, in our installations it's granted by default but not in yours.In order to try to reproduce the issue, and verify that we provide a patch that will actually work, what version of OpenShift are you using?
I deploy the cluster using the IBM Cloud procedure. The version is: 4.10.16_1521
I'm going to create a new cluster and reapply the scripts in the following order:
make ocp-deploy
make ocp-deploy
At the end I should have the same pods as you have on the pictures from above, right??
@andresmareca-ibm you don't need to clone the netobserv-ebpf-agent
repo, as the newtork-observability-operator will directly refer to the latest deployed image in quay.
In the NOO repo, you should do:
make deploy ocp-deploy
Then you can deploy the example flowcollector:
oc apply -f config/samples/flows_v1alpha1_flowcollector.yaml
If you want the eBPF agent to be deployed, you should set the agent: ebpf property in the descriptor
@mariomac BTW: I use make ocp-run
which does all of those including deployment of a sample workload etc ... then I just change the CR to use eBPF ./... I this ends up with the same thing.
FYI:: @ctrath ^^^
I'm getting this error during the ebpf container creation.
Error: unknown capability "CAP_BPF" to add
@andresmareca-ibm this means that the kernel does not support this capability. By curiosity, which Linux distribution and Kernel version are you using?
Anyway, you can workaround this issue by adding privileged: true
property to the ebpf:
section of the flowCollector: https://github.com/netobserv/network-observability-operator/blob/main/config/samples/flows_v1alpha1_flowcollector.yaml#L21
@andresmareca-ibm this means that the kernel does not support this capability. By curiosity, which Linux distribution and Kernel version are you using?
Anyway, you can workaround this issue by adding
privileged: true
property to theebpf:
section of the flowCollector: https://github.com/netobserv/network-observability-operator/blob/main/config/samples/flows_v1alpha1_flowcollector.yaml#L21
The error that the pods do not start we have already solved it. We have had to create a service account with the necessary permissions as shown below.
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: netobserv-ebpf-agent-test
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: netobserv-clusterrole
rules:
- apiGroups: [""] # "" indicates the core API group
resources:
- nodes
- nodes/proxy
- services
- endpoints
- pods
verbs: ["get", "watch", "list"]
- apiGroups:
- security.openshift.io
resourceNames:
- hostmount-anyuid
- privileged
- cgroup
resources:
- securitycontextconstraints
verbs:
- use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: netobserv-rolebinding
subjects:
- kind: ServiceAccount
name: netobserv-ebpf-agent-test
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: netobserv-clusterrole
I close this issue but there will be another one because another error has come out. Thank you very much!!
I'm deploying this agent in an openshift managed cluster by IBM Cloud. During the deploy face I have an issue with some of the pods.
First I have deployed the operator network-observability-operator following the instructions in the README file.
After that I try to deploy what is on netobserv-ebpf-agent. And here it fails for the first time due to lack of serviceAccount permissions. After adding on the serviceAccount I was able to start the actual pods:
This is where it fails a second time. The output of the pods are:
Any ideas on how to fix it?