search

falcosecurity / falco

Cloud Native Runtime Security
https://falco.org
Apache License 2.0
7.25k stars 893 forks source link

http_output not working, libcurl error: Problem with the SSL CA cert (path? access rights?) #2274

Closed exiett closed 1 year ago

exiett commented 1 year ago

Describe the bug After installing Falco in a Kubernetes cluster using the official Helm chart, whenever I try to output the results in a JSON format to an HTTP endpoint, Falco gives me a SSL error. Isn't this ca-certificate package supposed to come with the default installation?

How to reproduce it

  1. helm repo add falcosecurity https://falcosecurity.github.io/charts
  2. helm repo update
  3. helm install falco falcosecurity/falco
  4. exec into one of the pods running Falco.
  5. falco -o json_ouput=true http_output.enabled=true http_output.url=<REQUESTBIN_ADDRESS>
  6. exec into another pod (or even the same pod) to generate a Falco event.

Expected behaviour Falco sends an HTTP call to the endpoint containing the JSON of the syscall it just registered.

Screenshots

  1. Output when running curl <MY_REQUEST_BIN> from inside a pod running Falco: 
    {"about":"Pipedream is the fastest way to connects APIs. Build and run workflows with code-level control when you need it — and no code when you don't.","event_id":"2GfreizFr8I0hJjji77gNQEoP8Y","workflow_id":"p_LQCDnkR","owner_id":"u_KDh7vnG","deployment_id":"d_zqs7eNxx","timestamp":"2022-10-26T13:22:56.700Z","inspect":"https://pipedream.com/@/p_LQCDnkR","quickstart":"https://pipedream.com/quickstart/"}
  2. Output when Falco receives an event with json_output=true and http_output=true enabled to the same requestbin address: 
    Fri Oct 21 18:02:03 2022: libcurl error: Problem with the SSL CA cert (path? access rights?)
  3. Command we are running from within the pod: falco -o json_ouput=true -o http_output.enabled=true -o http_output.url=<MY_REQUEST_BIN_ADDRESS>

Environment

Additional context TL;DR: We're trying to use the built-in functionality of HTTP Output that comes with Falco to send JSON output about syscalls that Falco catches inside our containers. The main idea is to understand if this works or Sidekick is the only alternative.

alacuku commented 1 year ago

Hi @exiett, thanks for reporting the issue. The following PR was made to fix the bug: https://github.com/falcosecurity/libs/pull/442 but it seems that the issue is still present. We are going to investigate further and fix the bug.

trallnag commented 1 year ago

The issue is still there btw.

Wed Jan 18 20:47:02 2023: Falco version: 0.33.1-105+c1985a7 (x86_64)
...
Wed Jan 18 20:47:58 2023: libcurl error: Problem with the SSL CA cert (path? access rights?)
Wed Jan 18 20:48:18 2023: libcurl error: Problem with the SSL CA cert (path? access rights?)

I am testing it by pointing Falco to a pipedream request bin like https://xxx.m.pipedream.net.

Issif commented 1 year ago

The certificates for CA were missing in the docker image, the PR #2355 fixes that for the upcoming release

jasondellaluce commented 1 year ago

/milestone 0.34.0

leogr commented 1 year ago

I am assuming https://github.com/falcosecurity/falco/pull/2355 has fixed this PR, so /close

poiana commented 1 year ago

@leogr: Closing this issue.

In response to [this](https://github.com/falcosecurity/falco/issues/2274#issuecomment-1400154872): >I am assuming https://github.com/falcosecurity/falco/pull/2355 has fixed this PR, so >/close > > > Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
movd commented 1 year ago

Reopening this I'm experiencing the same problem but in normal package install on a Debian Linux. I want to talk to Falcosidekick via HTTPS.

$ falco --version
Falco version: 0.33.1
Libs version:  0.9.2
Plugin API:    2.0.0
Driver:
  API version:    2.0.0
  Schema version: 2.0.0
  Default driver: 3.0.1+driver
$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 11 (bullseye)
Release:    11
Codename:   bullseye
ca-certificates/stable,now 20210119 all [installed,automatic]
$ curl -vL 'https://falcosidekick.apps.example.com' 2>&1 | grep -A6 'Server certificate:'
* Server certificate:
*  subject: CN=*.apps.example.com
*  start date: May  3 00:00:00 2022 GMT
*  expire date: May  3 23:59:59 2023 GMT
*  subjectAltName: host "falcosidekick.apps.example.com" matched cert's "*.apps.example.com"
*  issuer: C=US; O=DigiCert Inc; CN=RapidSSL TLS DV RSA Mixed SHA256 2020 CA-1
*  SSL certificate verify ok.
$ sudo journalctl -fu falco
...
Jan 24 09:10:35 node01.example.com falco[890543]: libcurl error: Problem with the SSL CA cert (path? access rights?)
...
movd commented 1 year ago

/reopen

poiana commented 1 year ago

@movd: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to [this](https://github.com/falcosecurity/falco/issues/2274#issuecomment-1401537068): >/reopen Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
leogr commented 1 year ago

Quoting @movd :

Reopening this I'm experiencing the same problem but in normal package install on a Debian Linux. I want to talk to Falcosidekick via HTTPS.

:+1:

/reopen

Note the issue may be different. Could you provide more detail about your OS/distro/environment? Or a detailed repro. Thanks in advance

cc @Issif @jasondellaluce @LucaGuerra

poiana commented 1 year ago

@leogr: Reopened this issue.

In response to [this](https://github.com/falcosecurity/falco/issues/2274#issuecomment-1402256737): >Quoting @movd : >> Reopening this I'm experiencing the same problem but in normal package install on a Debian Linux. I want to talk to Falcosidekick via HTTPS. > >:+1: > >/reopen > >Note the issue may be different. Could you provide more detail about your OS/distro/environment? Or a detailed repro. Thanks in advance > >cc @Issif @jasondellaluce @LucaGuerra Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
Issif commented 1 year ago

I'm pretty sure it's same issue. If it's ok in the next release 0.34 we'll be sure.

movd commented 1 year ago

Yes sure. I can update the issue once 0.34 is out. Our setup will automatically install it.

Describe the bug

When I set up http_output with an https URL like so:

http_output:
  enabled: true
  url: https://falcosidekick.apps.example.com

I'll receive the following errors: 

Jan 24 09:10:35 node01.example.com falco[890543]: libcurl error: Problem with the SSL CA cert (path? access rights?)

As the URL states, I deploy a Falcosidekick inside a Kubernetes Cluster, but our Falco processes lives outside of the cluster and are installed via apt. When using unencrypted HTTP, it works just fine.

How to reproduce it

Enable http_output with a valid https endpoint (see above). Do something to trigger an Falco alert that should get forwarded to your https destination. E. g.:

Trigger Error File below /etc opened for writing rule:

root@node01:~# echo "bad file" > /etc/falco_test.sh

Falco can't forward the alert:

$ sudo journalctl -fu falco.service
...
Jan 25 09:44:05 node01.example.com falco[939076]: libcurl error: Problem with the SSL CA cert (path? access rights?)
Jan 25 09:44:05 node01.example.com falco[939076]: {"hostname":"node01.example.com","output":"09:44:05.330918780: Error File below /etc opened for writing (user=<NA> user_loginuid=-1 command=prometheus-conf --listen-address=:8080 --reload-url=http://127.0.0.1:9093/-/reload --config-file=/etc/alertmanager/config/alertmanager.yaml.gz --config-envsubst-file=/etc/alertmanager/config_out/alertmanager.env.yaml pid=66005 parent=containerd-shim pcmdline=containerd-shim -namespace k8s.io -id 5c3c2ab4942c4dec45c48a32a06d56843706ba94748a4ed33d2e8cad9d6dae71 -address /run/containerd/containerd.sock file=/etc/alertmanager/config_out/alertmanager.env.yaml.tmp program=prometheus-conf gparent=systemd ggparent=<NA> gggparent=<NA> container_id=033b1aeea6a0 image=quay.io/prometheus-operator/prometheus-config-reloader)","priority":"Error","rule":"Write below etc","source":"syscall","tags":["filesystem","mitre_persistence"],"time":"2023-01-25T08:44:05.330918780Z", "output_fields": {"container.id":"033b1aeea6a0","container.image.repository":"quay.io/prometheus-operator/prometheus-config-reloader","evt.time":1674636245330918780,"fd.name":"/etc/alertmanager/config_out/alertmanager.env.yaml.tmp","proc.aname[2]":"systemd","proc.aname[3]":null,"proc.aname[4]":null,"proc.cmdline":"prometheus-conf --listen-address=:8080 --reload-url=http://127.0.0.1:9093/-/reload --config-file=/etc/alertmanager/config/alertmanager.yaml.gz --config-envsubst-file=/etc/alertmanager/config_out/alertmanager.env.yaml","proc.name":"prometheus-conf","proc.pcmdline":"containerd-shim -namespace k8s.io -id 5c3c2ab4942c4dec45c48a32a06d56843706ba94748a4ed33d2e8cad9d6dae71 -address /run/containerd/containerd.sock","proc.pid":66005,"proc.pname":Wed Jan 25 09:44:05 2023: libcurl error: Problem with the SSL CA cert (path? access rights?)

Expected behaviour

Event gets forwarded to the https destination

Environment

root@node01:~# falco --version
Falco version: 0.33.1
Libs version:  0.9.2
Plugin API:    2.0.0
Driver:
  API version:    2.0.0
  Schema version: 2.0.0
  Default driver: 3.0.1+driver
root@node01:~# falco --support | jq .system_info
Wed Jan 25 09:46:01 2023: Falco version: 0.33.1 (x86_64)
Wed Jan 25 09:46:01 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Wed Jan 25 09:46:01 2023: Loading rules from file /etc/falco/falco_rules.yaml
Wed Jan 25 09:46:01 2023: Loading rules from file /etc/falco/falco_rules.local.yaml
{
  "machine": "x86_64",
  "nodename": "node01.example.com",
  "release": "5.10.0-21-amd64",
  "sysname": "Linux",
  "version": "#1 SMP Debian 5.10.162-1 (2023-01-21)"
}
# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@node01:~# uname -a
Linux node01.example.com 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux
# systemctl status falco.service
● falco.service - Falco: Container Native Runtime Security
     Loaded: loaded (/lib/systemd/system/falco.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/falco.service.d
             └─ebpf.conf
     Active: active (running) since Wed 2023-01-25 09:43:16 CET; 3min 36s ago
       Docs: https://falco.org/docs/
   Main PID: 939076 (falco)
      Tasks: 18 (limit: 19165)
     Memory: 62.7M
        CPU: 7.204s
     CGroup: /system.slice/falco.service
             └─939076 /usr/bin/falco --pidfile=/var/run/falco.pid
...
root@node01:~# cat /etc/systemd/system/falco.service.d/ebpf.conf
# Ansible managed
# Ref: https://falco.org/docs/event-sources/drivers/#ebpf-probe
# Empty FALCO_BPF_PROBE environment variable enables eBPF support
[Service]
Environment=FALCO_BPF_PROBE=""
ExecStartPre=
ExecStopPost=

Additional context

The certificate is valid and works fine with plain curl (from same host to the same destination).

$ curl -vL 'https://falcosidekick.apps.example.com' 2>&1 | grep -A6 'Server certificate:'
* Server certificate:
*  subject: CN=*.apps.example.com
*  start date: May  3 00:00:00 2022 GMT
*  expire date: May  3 23:59:59 2023 GMT
*  subjectAltName: host "falcosidekick.apps.example.com" matched cert's "*.apps.example.com"
*  issuer: C=US; O=DigiCert Inc; CN=RapidSSL TLS DV RSA Mixed SHA256 2020 CA-1
*  SSL certificate verify ok.
Issif commented 1 year ago

Sadly, even with 0.34 we face SSL issues https://github.com/falcosecurity/falco/issues/2395#issuecomment-1422635105

movd commented 1 year ago

Can confirm I'm still sadly facing the same issue.

falco.yaml

http_output:
  enabled: true
  url: https://falcosidekick.apps.example.com
  user_agent: "falcosecurity/falco"

unit journal:

root@node04:~# journalctl -fu falco-bpf.service
-- Journal begins at Wed 2023-02-08 06:28:02 CET. --
Feb 09 08:51:02 node04.k8s-dgi-dev.dg-i.net falco[850382]: libcurl error: Problem with the SSL CA cert (path? access rights?)
Feb 09 08:51:07 node04.k8s-dgi-dev.dg-i.net falco[850382]: libcurl error: Problem with the SSL CA cert (path? access rights?)
Feb 09 08:51:12 node04.k8s-dgi-dev.dg-i.net falco[850382]: libcurl error: Problem with the SSL CA cert (path? access rights?)
Feb 09 08:51:17 node04.k8s-dgi-dev.dg-i.net falco[850382]: libcurl error: Problem with the SSL CA cert (path? access rights?)

I'm using a valid certificate:

root@node04:~# curl -vL 'https://falcosidekick.apps.example.com' 2>&1 | grep -A6 'Server certificate:'
* Server certificate:
*  subject: CN=*.apps.example.com
*  start date: May  3 00:00:00 2022 GMT
*  expire date: May  3 23:59:59 2023 GMT
*  subjectAltName: host "falcosidekick.apps.example.com" matched cert's "*.apps.example.com"
*  issuer: C=US; O=DigiCert Inc; CN=RapidSSL TLS DV RSA Mixed SHA256 2020 CA-1
*  SSL certificate verify ok.

Environment:

root@node04:~# falco --support | jq .system_info
Thu Feb  9 08:54:52 2023: Falco version: 0.34.0 (x86_64)
Thu Feb  9 08:54:52 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Thu Feb  9 08:54:52 2023: Loading rules from file /etc/falco/falco_rules.yaml
Thu Feb  9 08:54:52 2023: Loading rules from file /etc/falco/falco_rules.local.yaml
{
  "machine": "x86_64",
  "nodename": "node04.example.com",
  "release": "5.10.0-21-amd64",
  "sysname": "Linux",
  "version": "#1 SMP Debian 5.10.162-1 (2023-01-21)"
}

Let me know If I can be of any help :)

alacuku commented 1 year ago

I'm working on that: #2412 . I'm gonna run some tests to make sure that solves the problem!

FedeDP commented 1 year ago

Moving to next milestone for now; possibly a 0.34.1 patch-release will be released though! /milestone 0.35.0

alacuku commented 1 year ago

Falco 0.34.1 is out. The patch release should solve this issue. Could you please check it out? cc @movd @exiett

PaulFurtado commented 1 year ago

Even after those fixes, this is still not fixed for RHEL-based distros, I left more details in a comment here: https://github.com/falcosecurity/libs/pull/871/files#r1142781849

It would also be great if the HTTP feature allowed users to specify a custom path to a CA bundle (ex: for using internal purpose-specific CAs). If you added an option ca_file to the yaml confi, you could pass it to curl with CURLOPT_CAINFO. Having this option would also allow users to workaround this bug without recompiling.

leogr commented 1 year ago

cc @alacuku

alacuku commented 1 year ago

Hi @PaulFurtado, thanks for testing and reporting this. We definitely need to make the CA/certs path configurable.