search

falcosecurity / falco

Cloud Native Runtime Security
https://falco.org
Apache License 2.0
7.37k stars 901 forks source link

Integration between Falco and FalcoSidekick #2565

Closed chaitu577 closed 5 months ago

chaitu577 commented 1 year ago

Hi Team,

Can anyone point me to the documentation or guide me on how to integrate Falco (setup using helm on Kubernetes - https://falco.org/docs/getting-started/try-falco/try-falco-on-kubernetes/) and Falco Sidekick / UI(setup reference: https://github.com/falcosecurity/falcosidekick, https://github.com/falcosecurity/falcosidekick-ui) coz I don't see any events on the Web UI, thanks.

Regards, Che

Issif commented 1 year ago

Hi, Do you use helm for your deployment? If you do, by just setting --set falcosidekick.enabled=true --set falcosidekick.webui.enabled=true with the Falco chart, everything will be configured for you

chaitu577 commented 1 year ago

Hi Issif,

I did the deployment using Helm, but when I tried setting the same, the WebUI is not accessible due to storage issues (I mean we don't have any persistent storage in k8's env. which Redis is expecting)

Issif commented 1 year ago

I see, you can disable the PVC but all events will be kept in memory: --set falcosidekick.webui.redis.storageEnabled=false

chaitu577 commented 1 year ago

Thanks Issif, deployed the same by setting the above flag, but i cant access the webUI via 2802 port.

kubectl port-forward svc/falco-1684835778-falcosidekick-ui 2802:2802 Forwarding from 127.0.0.1:2802 -> 2802

-----------------------------------kubectl get pods----------------------------------------------------------------------- NAME READY STATUS RESTARTS AGE alpine-deployment-65c65b9476-b5b2g 1/1 Running 0 3d21h falco-1684835778-9pmjd 0/2 Init:CrashLoopBackOff 10 (2m2s ago) 41m falco-1684835778-falcosidekick-5fbdc8b559-nzbwv 1/1 Running 0 41m falco-1684835778-falcosidekick-5fbdc8b559-tq2zc 1/1 Running 0 41m falco-1684835778-falcosidekick-ui-799b77d96-282rv 0/1 CrashLoopBackOff 16 (4m54s ago) 41m falco-1684835778-falcosidekick-ui-799b77d96-qnkhr 0/1 Running 17 (5m14s ago) 41m falco-1684835778-falcosidekick-ui-redis-0 1/1 Running 0 41m falco-1684835778-j4xnm 0/2 Init:CrashLoopBackOff 10 (92s ago) 41m falco-1684835778-n2pqd 0/2 Init:CrashLoopBackOff 10 (116s ago) 41m falco-1684835778-sd2qj 0/2 Init:CrashLoopBackOff 10 (2m12s ago) 41m falco-1684835778-wpdkj 0/2 Init:CrashLoopBackOff 10 (2m18s ago) 41m falco-1684835778-wxbxr 0/2 Init:CrashLoopBackOff 10 (118s ago) 41m falco-56fmc 2/2 Running 0 3d22h falco-6xhjv 2/2 Running 0 3d22h falco-cg9dg 2/2 Running 0 3d22h falco-klpjd 2/2 Running 0 3d22h falco-q4cr7 2/2 Running 0 3d22h falco-xd497 2/2 Running 0 3d22h

Issif commented 1 year ago

what errors and logs do you have?

chaitu577 commented 1 year ago

image

this is wat I see

chaitu577 commented 1 year ago

kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE falco-1684835778-falcosidekick ClusterIP 10.1.1.6 2801/TCP 93m falco-1684835778-falcosidekick-ui ClusterIP 10.1.1.1 2802/TCP 93m falco-1684835778-falcosidekick-ui-redis ClusterIP 10.1.9.2 6379/TCP 93m

Issif commented 1 year ago

At falcosidekick level it's ok, and for falcosidekick-ui? Is the redis correctly running? When you try to access through the port-forward, what http error do you get?

chaitu577 commented 1 year ago

Yep redis is up & running, PFB snip.

image

docker exec -it 7d705638e61b redis-cli

127.0.0.1:6379> ping PONG 127.0.0.1:6379>

this is the error I see now, thanks.

image

Issif commented 1 year ago

Your service name is strange, on my side, I start the port-forward with just:

kubectl port-forward svc/falco-falcosidekick-ui 2802:2802 -n falco

Can you list your services to be sure please?

chaitu577 commented 1 year ago

Here we go:

image

Issif commented 1 year ago

Everything is ok at this level, I don't understand why the port-forward fails. It doesn't seem related to falcosidekick-ui but more on your K8s config. Any CNI that could create an issue? Are you able to create a port-forward to another svc? Can you try directly to the pod?

chaitu577 commented 1 year ago

I've deployed a new Redis server on a different machine, can you please check the below & correct it, as executing the below command pick the IP of Redis Cluster IP(from kubectl get svc) not the one passed below, thanks.

helm install falco -n falco --set tty=true falcosecurity/falco \   --set falcosidekick.enabled=true \   --set redis.address=rndcrs101 \   --set FALCOSIDEKICK_UI_REDIS_URL=10.X.X.X:6379 \   --set falcosidekick.webui.enabled=true

Issif commented 1 year ago

Here's the correct syntax:

helm install falco -n falco --set tty=true falcosecurity/falco
  --set falcosidekick.enabled=true
  --set falcosidekick.webui.enabled=true
  --set falcosidekick.webui.externalRedis.enabled=true
  --set falcosidekick.webui.externalRedis.url=10.X.X.X
  --set falcosidekick.webui.externalRedis.port=6379

I agree falcosidekick.webui.externalRedis.url is not well named.

chaitu577 commented 1 year ago

Thanks, Issif, I see the below error where I can't find the deployment-ui in the falcosidekick GitHub repo, but it exists in the charts repo here(https://github.com/falcosecurity/charts/tree/master/falcosidekick/templates). do I need to clone it and do something?

Error: INSTALLATION FAILED: execution error at (falco/charts/falcosidekick/templates/deployment-ui.yaml:3:5): Both webui.redis and webui.externalRedis modules are enabled. Please disable one of them.

Issif commented 1 year ago

Add --set falcosidekick.webui.redis.enabled=false.

All values are there: https://github.com/falcosecurity/charts/blob/master/falcosidekick/values.yaml

chaitu577 commented 1 year ago

it worked but the UI pod's arent coming up..!

kubectl get pods NAME READY STATUS RESTARTS AGE alpine-deployment-65c65b9476-g6vft 1/1 Running 0 3d21h falco-b5sqt 2/2 Running 0 114s falco-falcosidekick-5dd98bcdc4-4vw24 1/1 Running 0 114s falco-falcosidekick-5dd98bcdc4-84qtg 1/1 Running 0 114s falco-falcosidekick-ui-595b78c7c7-4q6f5 0/1 CrashLoopBackOff 4 (7s ago) 114s falco-falcosidekick-ui-595b78c7c7-w7t7f 0/1 CrashLoopBackOff 4 (16s ago) 114s falco-fcmgm 2/2 Running 0 114s falco-h5fb5 2/2 Running 0 114s falco-m27jk 2/2 Running 0 114s falco-pbf54 2/2 Running 0 114s falco-s2dt2 2/2 Running 0 114s

kubectl logs falco-falcosidekick-ui-595b78c7c7-w7t7f 2023/05/29 15:32:23 [WARN] : Index does not exist 2023/05/29 15:32:23 [WARN] : Create Index 2023/05/29 15:32:23 ERR unknown command FT.CREATE, with args beginning with: eventIndex, SCHEMA, output, TEXT, rule, TEXT, priority, TEXT, hostname, TEXT, source, TEXT, tags, TEXT,

Issif commented 1 year ago

Are you sure to run a redis instance with redisearch module? It's a requirement.

chaitu577 commented 1 year ago

Yes, installed redisearch and all pods are up and running, also i did port-forward the UI pod to 2802 and i see the UI console as well, very much thanks Issif.

now the issue is I'm unable to see the events triggered on Web UI console. Any ideas? image

I see the config is pointing to the newly installed Redis instance as well. image

UI image

Issif commented 1 year ago

Can you check in logs if :

chaitu577 commented 1 year ago

Sure, PFB logs.

Triggered Alert kubectl exec -it alpine-deployment-65c65b9476-g6vft -- sh -c "cat /etc/passwd" root:x:0:0:root:/root:/bin/ash

Falco Logs docky]# for pod in $(kubectl get pods -o=name); do kubectl logs "$pod" --all-containers | grep "passwd"; done {"hostname":"falco-pbf54","output":"07:32:08.067095951: Notice A shell was spawned in a container with an attached terminal (user=root user_loginuid=-1 k8s.ns=falco k8s.pod=alpine-deployment-65c65b9476-g6vft container=4dca6289b5c9 shell=sh parent=runc cmdline=sh -c cat /etc/passwd pid=1876022 terminal=34816 container_id=4dca6289b5c9 image=docker.io/library/alpine)","priority":"Notice","rule":"Terminal shell in container","source":"syscall","tags":["T1059","container","mitre_execution","shell"],"time":"2023-06-02T07:32:08.067095951Z", "output_fields": {"container.id":"4dca6289b5c9","container.image.repository":"docker.io/library/alpine","evt.time":1685691128067095951,"k8s.ns.name":"falco","k8s.pod.name":"alpine-deployment-65c65b9476-g6vft","proc.cmdline":"sh -c cat /etc/passwd","proc.name":"sh","proc.pid":1876022,"proc.pname":"runc","proc.tty":34816,"user.loginuid":-1,"user.name":"root"}}

Falco Sidekick Logs

kubectl logs falco-falcosidekick-5dd98bcdc4-84qtg -c falcosidekick 2023/05/29 15:29:25 [INFO] : Falco Sidekick version: 2.27.0 2023/05/29 15:29:25 [INFO] : Enabled Outputs : [WebUI] 2023/05/29 15:29:25 [INFO] : Falco Sidekick is up and listening on :2801

Falco Sidekick UI Logs kubectl logs falco-falcosidekick-ui-595b78c7c7-kbbc5 -c falcosidekick-ui 2023/06/01 20:22:17 [INFO] : Falcosidekick UI is listening on 0.0.0.0:2802 2023/06/01 20:22:17 [INFO] : log level is info 2023/06/02 07:31:40 [INFO] : user 'admin' authenticated 2023/06/05 08:26:04 [INFO] : user 'admin' authenticated 2023/06/07 07:22:05 [INFO] : user 'admin' authenticated [root@ docky]# kubectl logs falco-falcosidekick-ui-595b78c7c7-rl9s8 2023/06/01 20:23:40 [INFO] : Falcosidekick UI is listening on 0.0.0.0:2802 2023/06/01 20:23:40 [INFO] : log level is info

Issif commented 1 year ago

Are you running falcosidekick-ui on arm64? We discovered a strange bug https://github.com/falcosecurity/falcosidekick-ui/issues/95

For now, I don't have access to an arm64 machine to reproduce.

chaitu577 commented 1 year ago

oops, its x86 64-bit architecture ("AMD64" or "Intel 64") but not arm64.

uname -mvnrs Linux abc.xx.com 3.10.0-1160.88.1.el7.x86_64 #1 SMP Tue Mar 7 15:41:52 UTC 2023 x86_64

Issif commented 1 year ago

Can you paste me your whole values.yaml please? Just edit the sensitive data. Right now, I don't see any issue.

chaitu577 commented 1 year ago

Here's the correct syntax:

helm install falco -n falco --set tty=true falcosecurity/falco
  --set falcosidekick.enabled=true
  --set falcosidekick.webui.enabled=true
  --set falcosidekick.webui.externalRedis.enabled=true
  --set falcosidekick.webui.externalRedis.url=10.X.X.X
  --set falcosidekick.webui.externalRedis.port=6379

I agree falcosidekick.webui.externalRedis.url is not well named.

I've used helm to install, its fetching directly from falcosecurity/falco.

Issif commented 1 year ago

I don't see any issue with this config. Can you try to create a port-forward to a falcosidekick pod and trigger a test event:

kubectl port-forward svc/falco-falcosidekick 2801:2801 -n falco
curl -XPOST "http://localhost:2801/test"

If you see something in the falcosidekick logs and in the webui, it means the issue is between falco and falcosidekick.

Do you have any specific networkpolicies? or cilium? or calico? that could interfere?

chaitu577 commented 1 year ago

I see no errors/handling connection when using 127.0.0.1 and it throws an connection refused error while using localhost/hostname:2801, I'm using calico but i dont see anything that's blocking, any specific checks/info that u need? please let me know, thanks.

curl -XPOST "http://XXXX:2801/test" curl: (7) Failed connect to XXXX:2801; Connection refused

kubectl port-forward svc/falco-falcosidekick 2801:2801 -n falco Forwarding from 127.0.0.1:2801 -> 2801 Handling connection for 2801

kubectl logs falco-falcosidekick-5dd98bcdc4-4vw24 2023/05/29 15:29:24 [INFO] : Falco Sidekick version: 2.27.0 2023/05/29 15:29:24 [INFO] : Enabled Outputs : [WebUI] 2023/05/29 15:29:24 [INFO] : Falco Sidekick is up and listening on :2801 2023/06/08 11:47:38 [ERROR] : WebUI - Post "http://falco-falcosidekick-ui:2802": dial tcp 10.X:2802: i/o timeout 2023/06/08 11:47:38 [ERROR] : WebUI - Post "http://falco-falcosidekick-ui:2802": dial tcp 10.X:2802: i/o timeout 2023/06/08 11:51:21 [ERROR] : WebUI - Post "http://falco-falcosidekick-ui:2802": dial tcp 10.X:2802: i/o timeout 2023/06/08 11:51:21 [ERROR] : WebUI - Post "http://falco-falcosidekick-ui:2802": dial tcp 10.X:2802: i/o timeout 2023/06/08 11:53:33 [ERROR] : WebUI - Post "http://falco-falcosidekick-ui:2802": dial tcp 10.X:2802: i/o timeout 2023/06/08 11:53:33 [ERROR] : WebUI - Post "http://falco-falcosidekick-ui:2802": dial tcp 10.X:2802: i/o timeout

Issif commented 1 year ago

I see the issue now. For a reason I don't understand, your svc for falcosidekick-ui is falco-1684835778-falcosidekick-ui but the helm chart has used falco-falcosidekick-ui.

The template is this one:

  {{- if .Values.webui.enabled -}}
  {{ $weburl := printf "http://%s-ui:2802" (include "falcosidekick.fullname" .) }}
  WEBUI_URL: "{{ $weburl | b64enc }}"
  {{- end }}

With falcosidekick.fullname defines as:

{{- define "falcosidekick.fullname" -}}
{{- if .Values.fullnameOverride -}}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- if contains $name .Release.Name -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}
{{- end -}}

It should work, but it doesn't, I need to search where the bug is.

chaitu577 commented 1 year ago

Yes, previously it was like that due to prior installations may be, but currently my SVC are below, please check & assist.

kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE falco-falcosidekick ClusterIP 10.XXX 2801/TCP 9d falco-falcosidekick-ui ClusterIP 10.1XXXX 2802/TCP 9d

Issif commented 1 year ago

As everything is stateless, can you delete your deployments and all resources and redeploy everything, by checking the names?

dneto82 commented 10 months ago

I see, you can disable the PVC but all events will be kept in memory: --set falcosidekick.webui.redis.storageEnabled=false

This save me when was evaluating at stateless scenario such minikube / eks. I hope we should quote this in doc as note in "evaluation / trying mode".

Issif commented 10 months ago

I see, you can disable the PVC but all events will be kept in memory: --set falcosidekick.webui.redis.storageEnabled=false

This save me when was evaluating at stateless scenario such minikube / eks. I hope we should quote this in doc as note in "evaluation / trying mode".

Feel free to submit a PR to add it :wink:

poiana commented 7 months ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

poiana commented 6 months ago

Stale issues rot after 30d of inactivity.

Mark the issue as fresh with /remove-lifecycle rotten.

Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle rotten

poiana commented 5 months ago

Rotten issues close after 30d of inactivity.

Reopen the issue with /reopen.

Mark the issue as fresh with /remove-lifecycle rotten.

Provide feedback via https://github.com/falcosecurity/community. /close

poiana commented 5 months ago

@poiana: Closing this issue.

In response to [this](https://github.com/falcosecurity/falco/issues/2565#issuecomment-2119283271): >Rotten issues close after 30d of inactivity. > >Reopen the issue with `/reopen`. > >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Provide feedback via https://github.com/falcosecurity/community. >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
Github-user191 commented 2 months ago

I see, you can disable the PVC but all events will be kept in memory: --set falcosidekick.webui.redis.storageEnabled=false

Thank you! Was stuck for an hour and setting this flag in my helm install worked and my redis could start