Falco starts but nothing happens

Diliz commented 2 years ago

Describe the bug

I managed to run falco on openshift (not so easy if you try to push it with a least privileged argo instance xD), the problem is it seems stuck at launch, maybe I failed something somewhere, but I got no real state of what's going wrong there :/

How to reproduce it

Installed latest falco on openshift with the latest falco helm chart, here the falco values:


falco:

  image:
    pullPolicy: *pullPolicy
    pullSecrets: []
    registry: *registry
    repository: falcosecurity/falco
    tag: 0.32.2-slim # tried non slim version as well

  driver:
    enabled: true
    kind: "module"
    loader:
      enabled: true
      initContainer:
        enabled: true
        image:
          pullPolicy: *pullPolicy
          registry: *registry
          repository: falcosecurity/falco-driver-loader
          tag: 0.32.2
        args:
          - "--compile"

  auditLog:
    enabled: true

  falco:
    load_plugins: [json, k8saudit]
    watch_config_files: true
    rules_file:
      - /etc/falco/falco_rules.yaml
      - /etc/falco/falco_rules.local.yaml
      - /etc/falco/k8s_audit_rules.yaml
    json_output: true
    log_level: debug
    priority: debug

    stdout_output:
      enabled: true

  http_output: # enable for sidekick
    enabled: true

  scc: # true here enabled creation of Security Context Constraints in Openshift
    create: false #manually created for testing purpose (can't give the rights to argo for this)

  serviceAccount:
    create: true

And here the falco output:

Fri Aug 19 14:48:01 2022: Falco version 0.32.2
Fri Aug 19 14:48:01 2022: Falco initialized with configuration file /etc/falco/falco.yaml
Fri Aug 19 14:48:01 2022: Loading plugin (k8saudit) from file /usr/share/falco/plugins/libk8saudit.so
Fri Aug 19 14:48:01 2022: Loading plugin (json) from file /usr/share/falco/plugins/libjson.so
Fri Aug 19 14:48:01 2022: Configured rules filenames:
Fri Aug 19 14:48:01 2022: /etc/falco/falco_rules.yaml
Fri Aug 19 14:48:01 2022: /etc/falco/falco_rules.local.yaml
Fri Aug 19 14:48:01 2022: /etc/falco/k8s_audit_rules.yaml
Fri Aug 19 14:48:01 2022: Loading rules from file /etc/falco/falco_rules.yaml:
Fri Aug 19 14:48:02 2022: Loading rules from file /etc/falco/falco_rules.local.yaml:
Fri Aug 19 14:48:02 2022: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
Fri Aug 19 14:48:02 2022: Watching /etc/falco/falco.yaml
Fri Aug 19 14:48:02 2022: Watching /etc/falco/falco_rules.yaml.
Fri Aug 19 14:48:02 2022: Watching /etc/falco/falco_rules.local.yaml.
Fri Aug 19 14:48:02 2022: Watching /etc/falco/k8s_audit_rules.yaml.
Fri Aug 19 14:48:02 2022: Starting internal webserver, listening on port 8765

Then nothing more :/

Expected behaviour

Events logged by falco

Environment

Falco version: 0.32.2
System info: rhcos 4.18.x
Cloud provider or hardware configuration:
OS: rhcos
Kernel: 4.18.x
Installation method: helm chart + argo

Additional context

Openshift cluster:

The security context constraints was manually added for testing purpose
Machine config with kernel-devel added for kernel module building purpose

alacuku commented 2 years ago

Hi @Diliz, since falco 0.32.0 the kubernetes audit events support has been refactored to become a plugin. One thing that changed is that syscalls and k8s audit events sources cannot be active at the same time anymore. I think this is the issue why you are not seeing the syscalls events in the falco output since falco loads the k8s audit plugin. At the time being you need to run two different falco instances: one for the syscalls and the other one for the k8s audit events.

jasondellaluce commented 2 years ago

Agree with @alacuku. We expect to restore running both in the same Falco instance in the near future.

Diliz commented 2 years ago

Hello!

Thanks for the replies!

I actually tried to stop split my falco deployment in 2, one for syscalls and one for k8s audit events, but still no events detected, nothing happens, I dunno what's wrong here, got no errors nor clues about what's going on there

jasondellaluce commented 2 years ago

Can you share the logs of both deployments? You can making logs more verbose by adding -o libs_logger.enabled=true in the Falco CLI command.

Diliz commented 2 years ago

Thanks for the replies! Here an update!

syscall only falco instances

For the syscalls only falco, it is logging now, I had the following issues with it:

The behaviour without any custom rules other than the falco_rules.yaml, without the json plugin loaded: Error: Plugin 'json' is required but not loaded
The behaviour without any custom rules other than the falco_rules.yaml, with the json plugin loaded: Error: Plugin 'json' has field extraction capability but is not compatible with any enabled event source

To fix it I honestly don't know what I changed that triggered the service to run correctly, I tried to fallback to falco 0.32.1, 0.31.x, tried too many things to remember correctly what changed that made it work as expected, but here my falco.yml file:

  buffered_outputs: false
  file_output:
    enabled: false
    filename: ./events.txt
    keep_alive: false
  grpc:
    bind_address: unix:///var/run/falco/falco.sock
    enabled: false
    threadiness: 0
  grpc_output:
    enabled: false
  http_output:
    enabled: false
    url: ""
    user_agent: falcosecurity/falco
  json_include_output_property: true
  json_include_tags_property: true
  json_output: true
  libs_logger:
    enabled: false
    severity: debug
  load_plugins: []
  log_level: debug
  log_stderr: true
  log_syslog: true
  metadata_download:
    chunk_wait_us: 1000
    max_mb: 100
    watch_freq_sec: 1
  output_timeout: 2000
  outputs:
    max_burst: 1000
    rate: 1
  plugins:
  - init_config: null
    library_path: libk8saudit.so
    name: k8saudit
    open_params: http://:9765/k8s-audit
  - library_path: libcloudtrail.so
    name: cloudtrail
  - init_config: ""
    library_path: libjson.so
    name: json
  priority: debug
  program_output:
    enabled: false
    keep_alive: false
    program: 'jq ''{text: .output}'' | curl -d @- -X POST https://hooks.slack.com/services/XXX'
  rules_file:
  - /etc/falco/falco_rules.yaml
  stdout_output:
    enabled: true
  syscall_event_drops:
    actions:
    - log
    - alert
    max_burst: 1
    rate: 0.03333
    threshold: 0.1
  syscall_event_timeouts:
    max_consecutives: 1000
  syslog_output:
    enabled: true
  time_format_iso_8601: false
  watch_config_files: true
  webserver:
    enabled: true
    k8s_healthz_endpoint: /healthz
    listen_port: 8765
    ssl_certificate: /etc/falco/falco.pem
    ssl_enabled: false

k8s audit only falco instances

Here goes the logs for the k8s only falco 0.32.2:

Tue Aug 23 15:03:10 2022: Falco version 0.32.1
Tue Aug 23 15:03:10 2022: Falco initialized with configuration file /etc/falco/falco.yaml
Tue Aug 23 15:03:10 2022: Loading plugin (k8saudit) from file /usr/share/falco/plugins/libk8saudit.so
Tue Aug 23 15:03:10 2022: [libs]: Parsing Fields JSON=[{"name":"ka.auditid","type":"string","isList":false,"arg":{"isRequired":false,"isIndex":false,"isKey":false},"display":"","desc":"The unique id of the audit event","properties":null},{"name":"ka.stage","type":"string","isList":false,"arg":{"isRequired":false,"isIndex":false,"isKey":false},"display":"","desc":"Stage of the request (e.g. RequestReceived, ResponseComplete, etc.)","properties":null},{"name":"ka.auth.decision","type":"string","isList":false,"arg":{"isRequired":false,"isIndex":false,"isKey":false},"display":"","desc":"The authorization decision","properties":null},{"name":"ka.auth.reason","type":"string","isList":false,"arg":{"isRequired":false,"isIndex":false,"isKey":false},"display":"","desc":"The authorization reason","properties":null},{"name":"ka.user.name","type":"string","isList":false,"arg":{"isRequired":false,"isIndex":false,"isKey":false},"display":"","desc":"The user name performing the request","properties":null},{"name":"ka.user...
Tue Aug 23 15:03:10 2022: Loading plugin (json) from file /usr/share/falco/plugins/libjson.so
Tue Aug 23 15:03:10 2022: [libs]: Parsing Fields JSON=[{"name":"json.value","type":"string","isList":false,"arg":{"isRequired":true,"isIndex":false,"isKey":true},"display":"","desc":"Extracts a value from a JSON-encoded input. Syntax is json.value[\u003cjson pointer\u003e], where \u003cjson pointer\u003e is a json pointer (see https://datatracker.ietf.org/doc/html/rfc6901)","properties":null},{"name":"json.obj","type":"string","isList":false,"arg":{"isRequired":false,"isIndex":false,"isKey":false},"display":"","desc":"The full json message as a text string.","properties":null},{"name":"json.rawtime","type":"string","isList":false,"arg":{"isRequired":false,"isIndex":false,"isKey":false},"display":"","desc":"The time of the event, identical to evt.rawtime.","properties":null},{"name":"jevt.value","type":"string","isList":false,"arg":{"isRequired":true,"isIndex":false,"isKey":true},"display":"","desc":"Alias for json.value, provided for backwards compatibility.","properties":null},{"name":"jevt.obj","type":"s...
Tue Aug 23 15:03:10 2022: Configured rules filenames:
Tue Aug 23 15:03:10 2022: /etc/falco/k8s_audit_rules.yaml
Tue Aug 23 15:03:10 2022: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
Tue Aug 23 15:03:10 2022: Watching /etc/falco/falco.yaml
Tue Aug 23 15:03:10 2022: Watching /etc/falco/k8s_audit_rules.yaml.
Tue Aug 23 15:03:10 2022: Starting internal webserver, listening on port 8765
Tue Aug 23 15:03:10 2022: [libs]: starting live capture

And here the config:

buffered_outputs: false
file_output:
  enabled: false
  filename: ./events.txt
  keep_alive: false
grpc:
  bind_address: unix:///var/run/falco/falco.sock
  enabled: false
  threadiness: 0
grpc_output:
  enabled: false
http_output:
  enabled: false
  url: ""
  user_agent: falcosecurity/falco
json_include_output_property: true
json_include_tags_property: true
json_output: true
libs_logger:
  enabled: false
  severity: debug
load_plugins:
- json
- k8saudit
log_level: debug
log_stderr: true
log_syslog: true
metadata_download:
  chunk_wait_us: 1000
  max_mb: 100
  watch_freq_sec: 1
output_timeout: 2000
outputs:
  max_burst: 1000
  rate: 1
plugins:
- init_config: null
  library_path: libk8saudit.so
  name: k8saudit
  open_params: http://:9765/k8s-audit
- library_path: libcloudtrail.so
  name: cloudtrail
- init_config: ""
  library_path: libjson.so
  name: json
priority: debug
program_output:
  enabled: false
  keep_alive: false
  program: 'jq ''{text: .output}'' | curl -d @- -X POST https://hooks.slack.com/services/XXX'
rules_file:
- /etc/falco/k8s_audit_rules.yaml
stdout_output:
  enabled: true
syscall_event_drops:
  actions:
  - log
  - alert
  max_burst: 1
  rate: 0.03333
  threshold: 0.1
syscall_event_timeouts:
  max_consecutives: 1000
syslog_output:
  enabled: true
time_format_iso_8601: false
watch_config_files: true
webserver:
  enabled: true
  k8s_healthz_endpoint: /healthz
  listen_port: 8765
  ssl_certificate: /etc/falco/falco.pem
  ssl_enabled: false

jasondellaluce commented 2 years ago

Your k8saudit-only setup seems to be working fine, you've only issue with the syscall-only one. That seems fine to me as well, but you need to prevent it from loading the /etc/falco/k8s_audit_rules.yaml ruleset. The error codes are indicating that the /etc/falco/k8s_audit_rules.yaml ruleset requires the json plugin and it is not loaded (because you don't need it). This should solve your issue.

Diliz commented 2 years ago

Hello,

It's stuck there for the k8s only one, even with the falco event generator (which is in loop mode) + manually triggerred events that should be monitored by falco :/

Weird for the json plugin, I'll check it tomorrow, but I'm pretty sure I added it before retrying it.

Thanks for the support 👍

EDIT:

The json plugin is loaded in the k8s only instances, but still no events, even when I manually trigger them :/

alacuku commented 2 years ago

You need to configure the k8s api-server to forward the audit log events to the falco instance. We do not have a step-by-step guide on how to do that but here you can find an example: https://github.com/falcosecurity/charts/tree/master/falco#kubernetes-audit-log

BicycleWalrus commented 2 years ago

Wanted to bump this issue. I also am having issues with getting the k8s-audit plugin up and working. The examples shown in the post from @alacuku has some good documentation for kOps, but not much else.

I had Falco running for syscalls just fine. Beautifully even. Then I converted it to k8s-audit. When I attempt restarting my api server with the appropriate webhook configured, the API Server never comes up. I'm assuming this may be an authentication issue, but when attempting to pass falco and approved key and cert I generated just for it, it still doesn't work.

It would seem to me that a change of this magnitude some come with some documentation along with it. I've met quite a few organizations which have also struggled with this change since v0.32.0.

Some updates:

This is my webhook. According to a sysdig article, this is the appropriate method to set it up. We're using http so the certificate authority shouldn't be required, as per the guidance @alacuku posted.

apiVersion: v1
kind: Config
clusters:
- cluster:
    server: http://172.16.3.229:8765/k8s-audit # IP Address is the Service ClusterIP
  name: falco
contexts:
- context:
    cluster: falco
    user: ""
  name: default-context
current-context: default-context
preferences: {}
users: []

After using this webhook, the API Server does return! Everything seems to be appropriately configured, and even when manually trigger events which should violate the built-in falco rules (a ConfigMap which passes plain text credentials, for example), Falco neither alerts nor captures. Instead, it does absolutely nothing at all.

Thu Sep 15 01:16:46 2022: Falco version 0.32.2
Thu Sep 15 01:16:46 2022: Falco initialized with configuration file /etc/falco/falco.yaml
Thu Sep 15 01:16:46 2022: Loading plugin (k8saudit) from file /usr/share/falco/plugins/libk8saudit.so
Thu Sep 15 01:16:46 2022: Loading plugin (json) from file /usr/share/falco/plugins/libjson.so
Thu Sep 15 01:16:46 2022: Loading rules from file /etc/falco/falco_rules.yaml:
Thu Sep 15 01:16:46 2022: Loading rules from file /etc/falco/falco_rules.local.yaml:
Thu Sep 15 01:16:46 2022: Starting internal webserver, listening on port 8765

I installed Falco with helm, following the procedure outlined at the link @alacuku gave.

I do have an active audit policy, and logs are populating. I've tested with both a simple auditpolicy which captures all metadata, and a more specified audit policy (the one from kubernetes.io, which the sysdig article linked earlier suggested we use for this test).

The big concern here, is that Falco is tested on the CKS. This change was quite significant, and if we can't get it working while following all of the documentation on the subject, this leaves me feeling quite concerned. After going through this experience, troubleshooting for now over 18 hours, conferring with other professionals in the space who also cannot get it working; I'm beginning to suspect one of two conclusions:

The k8s-audit plugin does not work.
The documentation with appropriate guidance to get it working is not available, thus rendering the product 'unavailable'.

I'm honestly mystified about this change. Falco was a very stable product, but ... it seems v0.32.0 broke it.

alacuku commented 2 years ago

Hi @BicycleWalrus, sorry to hear you are struggling to set up Flaco with k8s audit enabled.

It seems that the api-server is not able to reach the Falco instance. In the webhook configuration:

apiVersion: v1
kind: Config
clusters:
- cluster:
    server: http://172.16.3.229:8765/k8s-audit # IP Address is the Service ClusterIP
  name: falco
contexts:
- context:
    cluster: falco
    user: ""
  name: default-context
current-context: default-context
preferences: {}
users: []

the ClusterIP is used to contact the Falco instance. Have you checked that the api-server is able to reach the falco instance using that address? Furthermore, it seems that the port 8765 configured in the webhook is different from the default one of the falco configured by the chart 9765. Please be sure that the port and cluster ip address configured in the webhook match the ones configured for falco.

If for some reason the api-server is not able to reach falco over the ClusterIP you can use the following server address: http://localhost:30007/k8s-audit. If you are asking why that address the answer is clear if you check how we configured the values.yaml file of the helm chart:

driver:
  enabled: false

collectors:
  enabled: false

controller:
  kind: deployment

services:
  - name: k8saudit-webhook
    type: NodePort
    ports:
      - port: 9765 # See plugin open_params
        nodePort: 30007
        protocol: TCP

falco:
  rulesFile:
    - /etc/falco/k8s_audit_rules.yaml
    - /etc/falco/rules.d
  plugins:
    - name: k8saudit
      library_path: libk8saudit.so
      init_config:
        ""
        # maxEventBytes: 1048576
        # sslCertificate: /etc/falco/falco.pem
      open_params: "http://:9765/k8s-audit"
    - name: json
      library_path: libjson.so
      init_config: ""
  load_plugins: [k8saudit, json]

It configures a service of type NodePort for falco reachable at the nodePort 30007. If the nodePort 30007 is not available in your cluster then change it and remember to set the new values in the server address in the webhook configuration.

leogr commented 2 years ago

The documentation with appropriate guidance to get it working is not available, thus rendering the product 'unavailable'.

This is likely the real issue and I apologize for that.

I'm honestly mystified about this change. Falco was a very stable product, but ... it seems v0.32.0 broke it.

Plugins introduced new challenges and we had to change some assumptions in the Falco architecture (also the AuditSink was removed meanwhile). Anyway, I'm pretty sure the real issue is that we lack proper documentation.

@BicycleWalrus if you have any feedback on how we can improve that please share. For example, would a blog post have helped in such a case?

cc @falcosecurity/falco-maintainers

BicycleWalrus commented 2 years ago

@alacuku -- Thanks for the tips. I've been testing with both port 8765 and 9765 -- Figuring 9765 was more appropriate considering the service. I've run both of the configurations you recommended just now, and neither returned successful results. Falco is still not able to monitor the Audit Log being produced by the Api Server.

My next step was to attempt playing around with the service configuration and making other manifest adjustments. Sadly, my attempts were to no avail, so I've re-installed the non-edited helm chart (with the provided values file you cited above).

Unfortunately, no progress was made with any of these attempts.

@leogr -- Biggest things I could recommend is to not worry about what documentation is in, or out of scope. I noticed on several pages, both on the falco.org site, and in the github that detailing connecting the actual webhook was outside of the scope of the documentation. Setting up webhooks is certainly something I've done before, and not really the hurdle here. It seems falco needs some fine-tuning in terms of getting this work.

I've personally scoured the net looking for a successful run of the k8s-audit plugin since the v0.32.0 update. I haven't found one (not saying it doesn't exist, I just haven't personally found one).

Normally, I'd just resort to installing v0.31.* -- but my organization makes it a point to stay up-to-date, so I really need to figure out how to get Falco going in v0.32.2.

leogr commented 2 years ago

@BicycleWalrus thanks for your feedback.

I think the problem is in the documentation because I'm confident the k8saudit plugin works. The helm chart setup should work too, but perhaps some issue is hidden in the configuration, and we are not seeing it. AFAIK, also CKS is not up-to-date with the latest Falco version.

Anyhow, let's try to solve this issue. I want to help. Before debugging it, I want to add a bit of context and what changed with 0.32. Hoping that can clarify and also help to find a solution.

Before and after Falco 0.32

	Falco 0.31.1	Falco 0.32.2
Required chart ver	v1.18.6	v2.0.17
Container image	`falcosecurity/falco`	`falcosecurity/falco-no-driver`
Chart AuditSink support	yes, used by defult	no
Multi-source (syscalls + k8saudit)	yes	no
Deploy kind	DaemonSet	Deployment
K8s audit support	built-in	plugin
K8s audit enabled by default	yes	no (`k8saudit` plugin must be configured)
Listening ports (default)	8765 (for both k8saudit and healthz endpoint)	9765 (for the k8saudit) / 8765 (only for healthz endpoint)
Service kind (default)	ClusterIP	NodePort

Important notes:

Configuration settings to enable k8s audit are completely different between 0.31 and 0.32
Old blog posts and previous documentation won't work with Falco 0.32

Webhook configuration caveats

kube-apiserver `--audit-policy-file`

If the flag is omitted, no events are logged. (see https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/#audit-policy). This is a common mistake since this configuration was not required for the dynamic webhook (AuditSink), IIRC. It's mandatory now. It's particularly important because, depending on the policy file content, you may or not receive some kind of audit event.

kube-apiserver `--audit-webhook-config-file`

The webhook config file should be:

apiVersion: v1
kind: Config
clusters:
  - name: # name of your cluster here
    cluster:
      server: # webhook URL, see below
contexts:
  - context:
      cluster: # name of your cluster here
      user: ""
    name: default-context
current-context: default-context
preferences: {}
users: []

Now, assuming no SSL (so http://) and a NodePort configured in this way:

    type: NodePort
    ports:
      - port: 9765 # This must match the plugin open_params value
        nodePort: 30007
        protocol: TCP

There're two alternatives for server::

Using the ClusterIP: http://<ClusterIP>:9765/k8s-audit This is the common configuration. One caveat is that the ClusterIP is not known before starting the kube-apiserver. So one has to start the kube-apiserver, get the ClusterIP, then modify the audit webhook confing file and restart the kube-apiserver. Also, DNS can't be used.
Using the local port: http://localhost:30007/k8s-audit This has the advantage that the URL is known before starting the kube-apiserver. Since the nodePort will be available on all nodes, it will be available also in the kube-apiserver's node. So one can use localhost or the node IP (using the node IP may be mandatory depending on the K8s configuration).

Using NodePort is preferable for the simplicity of the configuration.

Troubleshooting

Is the Falco webhook listening?

Yes, if the Falco start-up log include: Loading plugin (k8saudit) from file /usr/share/falco/plugins/libk8saudit.so

Is the Falco webhook reachable?

Try curl http://<ClusterIP>:9765/k8s-audit.

If you make a GET request, it will reply with GET method not allowed.

Is the API server sending events?

Check the kube-apiserver logs, for example:

k --namespace kube-system logs -f kube-apiserver-kind-control-plane

Audit Log messages and errors should be reported.

Falco output may be buffered

In such case, events may appear only after a while.

Try tty: true.

Repro

If all tips above have not worked, we need a repro. In particular, we need:

Exact version of the Falco Helm chart
Helm values used (please use helm get values falco)
Contents of the policy file and the webhook config file
Kubernetes version

And any other details (i.e., any customization) you believe can be relevant.

If you can't publicly share some info, you can contact me via DM on Kubernetes Slack (if that works for you).

BicycleWalrus commented 2 years ago

Excellent! Will work this today, and should have at least the results of the attempt with this information, if nots steps to reproduce.

Thanks!

ashuec90 commented 1 year ago

Hello,

The above configuration is for cluster setup by using kubeadm. What about when using EKS? How to make sure api server is sending the event to falco webhook? Followed the configuration given in https://github.com/falcosecurity/charts/blob/master/falco/values-k8saudit.yaml but it is not working and i dont see any logs coming in falco pods. I am able to send the syscall events to falco but k8s-event is not coming.

Kindly guide me if i need to do the extra configurations or point me to the docs using which i can make it work. Environment EKS 1.21 falco version 0.32.2

Falco logs gets stuck here.

cc @leogr @alacuku

leogr commented 1 year ago

Hello,

The above configuration is for cluster setup by using kubeadm. What about when using EKS? How to make sure api server is sending the event to falco webhook? Followed the configuration given in https://github.com/falcosecurity/charts/blob/master/falco/values-k8saudit.yaml but it is not working and i dont see any logs coming in falco pods. I am able to send the syscall events to falco but k8s-event is not coming.

Kindly guide me if i need to do the extra configurations or point me to the docs using which i can make it work. Environment EKS 1.21 falco version 0.32.2

Falco logs gets stuck here.

cc @leogr @alacuku

That configuration does not work with managed k8s, like eks or gke. We need to implement dedicated plugins to interface with their proprietary APIs. For EKS there's already a plugin in the making :point_right: https://github.com/falcosecurity/plugins/pull/134

Likely, it will be released together with Falco 0.33 (very soon).

leogr commented 1 year ago

cc @jasondellaluce

ashuec90 commented 1 year ago

Great!! We will wait for it to be present in the helm charts.

Thanks a lot!!

CindyLizi commented 1 year ago

not sure if anyone managed to get the k8saudit plugin work, I stuck since day1 with the deployment.

I also posted the question on stackoverflow to look for help. https://stackoverflow.com/questions/73301538/falco-couldnt-get-in-datasource-k8s-audit-in-falcosidekick-ui-after-deploy-fal/74003029#74003029

Yesterday i found this issue, it's exactly the same with my problem, it's great that @leogr provided the detail information, it will be better if such information could be found from https://github.com/falcosecurity/charts/tree/master/falco#kubernetes-audit-log

I went through the comments from @leogr https://github.com/falcosecurity/falco/issues/2172#issuecomment-1249101560 tired both options of the server for webhook, neither works for me, i also followed the troubleshooting suggestion to check my deployment, all looks good.

and as I mentioned in my question, there is no data dump from port 9765, only the health check from port 8765, is it correct?

$tcpdump -i cali2b1a048d315
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on cali2b1a048d315, link-type EN10MB (Ethernet), capture size 262144 bytes
09:46:47.206455 IP node1.52642 > 192.168.5.136.8765: Flags [S], seq 3186172356, win 62370, options [mss 8910,sackOK,TS val 2279860630 ecr 0,nop,wscale 10], length 0
09:46:47.206483 IP 192.168.5.136.8765 > node1.52642: Flags [S.], seq 3676579593, ack 3186172357, win 62286, options [mss 8910,sackOK,TS val 2809267901 ecr 2279860630,nop,wscale 10], length 0
09:46:47.206501 IP node1.52642 > 192.168.5.136.8765: Flags [.], ack 1, win 61, options [nop,nop,TS val 2279860630 ecr 2809267901], length 0
09:46:47.206528 IP node1.52644 > 192.168.5.136.8765: Flags [S], seq 2907831321, win 62370, options [mss 8910,sackOK,TS val 2279860630 ecr 0,nop,wscale 10], length 0

and does falcosidekick UI support the k8saudit plugin? need any update to adopt it?

BicycleWalrus commented 1 year ago

Just to make clear, I still haven't gotten the K8s Audit plugin to work.

On Mon, Oct 10, 2022, 2:12 AM CindyLizi @.***> wrote:

not sure if anyone managed to get the k8saudit plugin work, I've struggled since day1 with the deployment.

I also posted the question on stackoverflow to look for help.

https://stackoverflow.com/questions/73301538/falco-couldnt-get-in-datasource-k8s-audit-in-falcosidekick-ui-after-deploy-fal/74003029#74003029

Yesterday i found this issue, it's exactly the same with my problem. I went through the comments from @leogr https://github.com/leogr #2172 (comment) https://github.com/falcosecurity/falco/issues/2172#issuecomment-1249101560 tired both options of the server for webhook, neither works for me, i also followed the troubleshooting suggestion to check my deployment, all looks good.

and as I mentioned in my question, there is no data dump from port 9765, only the health check from port 8765, is it correct?

$tcpdump -i cali2b1a048d315 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on cali2b1a048d315, link-type EN10MB (Ethernet), capture size 262144 bytes 09:46:47.206455 IP node1.52642 > 192.168.5.136.8765: Flags [S], seq 3186172356, win 62370, options [mss 8910,sackOK,TS val 2279860630 ecr 0,nop,wscale 10], length 0 09:46:47.206483 IP 192.168.5.136.8765 > node1.52642: Flags [S.], seq 3676579593, ack 3186172357, win 62286, options [mss 8910,sackOK,TS val 2809267901 ecr 2279860630,nop,wscale 10], length 0 09:46:47.206501 IP node1.52642 > 192.168.5.136.8765: Flags [.], ack 1, win 61, options [nop,nop,TS val 2279860630 ecr 2809267901], length 0 09:46:47.206528 IP node1.52644 > 192.168.5.136.8765: Flags [S], seq 2907831321, win 62370, options [mss 8910,sackOK,TS val 2279860630 ecr 0,nop,wscale 10], length 0

— Reply to this email directly, view it on GitHub https://github.com/falcosecurity/falco/issues/2172#issuecomment-1272826859, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHVVAC6IE64AFMEOPI7WRGLWCOXWVANCNFSM57GPMN5Q . You are receiving this because you were mentioned.Message ID: @.***>

leogr commented 1 year ago

Hey @CindyLizi

I also posted the question on stackoverflow to look for help. https://stackoverflow.com/questions/73301538/falco-couldnt-get-in-datasource-k8s-audit-in-falcosidekick-ui-after-deploy-fal/74003029#74003029

The answer in StackOverflow seems correct to me. You should use port 9765.

Yesterday i found this issue, it's exactly the same with my problem, it's great that @leogr provided the detail information, it will be better if such information could be found from https://github.com/falcosecurity/charts/tree/master/falco#kubernetes-audit-log

Totally agree. I've opened an issue to track this :point_right: https://github.com/falcosecurity/charts/issues/410

I went through the comments from @leogr #2172 (comment) tired both options of the server for webhook, neither works for me, i also followed the troubleshooting suggestion to check my deployment, all looks good.

and as I mentioned in my question, there is no data dump from port 9765, only the health check from port 8765, is it correct?

Note that if you don't receive any data on port 9765, the issue is not in the Falco configuration but in how the Kubernetes Audig Log webhook backend is configured.

and does falcosidekick UI support the k8saudit plugin? need any update to adopt it?

Falcosedikick just consumes Falco's output. If you get k8sudit alerts work in Falco, you should see them in falcosidekick UI too.

leogr commented 1 year ago

Hey @BicycleWalrus

Could you provide more details? (even privately) I'm willing to help, but I can't debug without all information.

Let me know! And feel free to contact me via Slack

PS Maybe my reply will be delayed next week because of KubeCon :smiley_cat:

Just to make clear, I still haven't gotten the K8s Audit plugin to work. … On Mon, Oct 10, 2022, 2:12 AM CindyLizi @.> wrote: not sure if anyone managed to get the k8saudit plugin work, I've struggled since day1 with the deployment. I also posted the question on stackoverflow to look for help. https://stackoverflow.com/questions/73301538/falco-couldnt-get-in-datasource-k8s-audit-in-falcosidekick-ui-after-deploy-fal/74003029#74003029 Yesterday i found this issue, it's exactly the same with my problem. I went through the comments from @leogr https://github.com/leogr #2172 (comment) <#2172 (comment)> tired both options of the server for webhook, neither works for me, i also followed the troubleshooting suggestion to check my deployment, all looks good. and as I mentioned in my question, there is no data dump from port 9765, only the health check from port 8765, is it correct? $tcpdump -i cali2b1a048d315 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on cali2b1a048d315, link-type EN10MB (Ethernet), capture size 262144 bytes 09:46:47.206455 IP node1.52642 > 192.168.5.136.8765: Flags [S], seq 3186172356, win 62370, options [mss 8910,sackOK,TS val 2279860630 ecr 0,nop,wscale 10], length 0 09:46:47.206483 IP 192.168.5.136.8765 > node1.52642: Flags [S.], seq 3676579593, ack 3186172357, win 62286, options [mss 8910,sackOK,TS val 2809267901 ecr 2279860630,nop,wscale 10], length 0 09:46:47.206501 IP node1.52642 > 192.168.5.136.8765: Flags [.], ack 1, win 61, options [nop,nop,TS val 2279860630 ecr 2809267901], length 0 09:46:47.206528 IP node1.52644 > 192.168.5.136.8765: Flags [S], seq 2907831321, win 62370, options [mss 8910,sackOK,TS val 2279860630 ecr 0,nop,wscale 10], length 0 — Reply to this email directly, view it on GitHub <#2172 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHVVAC6IE64AFMEOPI7WRGLWCOXWVANCNFSM57GPMN5Q . You are receiving this because you were mentioned.Message ID: @.>

poiana commented 1 year ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

leogr commented 1 year ago

Hey folks,

Is this still an issue?

Reme240400 commented 1 year ago

Yes, I'm still having issues, can I contact you by slack?

leogr commented 1 year ago

Yes, I'm still having issues, can I contact you by slack?

Sure. I'll be happy to help. Just consider that we are currently very busy with the upcoming release.

Reme240400 commented 1 year ago

Sure. I'll be happy to help. Just consider that we are currently very busy with the upcoming release. I have a question, i nthe deamonset I have to leave the driver's flag set to false if I use a plugin(k8saudit). But if is set to false my 2 container don't have logs, if I set the flag to true the container generate the logs but I'm not sure that are ok k8s. Seems that the 2 container watch the same things, when they should control or syscall or k8s audit

poiana commented 1 year ago

Stale issues rot after 30d of inactivity.

Mark the issue as fresh with /remove-lifecycle rotten.

Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle rotten

poiana commented 1 year ago

Rotten issues close after 30d of inactivity.

Reopen the issue with /reopen.

Mark the issue as fresh with /remove-lifecycle rotten.

Provide feedback via https://github.com/falcosecurity/community. /close

poiana commented 1 year ago

@poiana: Closing this issue.

In response to [this](https://github.com/falcosecurity/falco/issues/2172#issuecomment-1483987611): >Rotten issues close after 30d of inactivity. > >Reopen the issue with `/reopen`. > >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Provide feedback via https://github.com/falcosecurity/community. >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

falcosecurity / falco