sboschman commented 5 months ago

What type of PR is this?

/kind feature

Any specific area of the project related to this PR?

/area plugins

What this PR does / why we need it: This k8saudit-gke plugin tries to bridge te gap between Google Cloud audit logs for GKE and the Falco ruleset for k8s audit events. This way the existing Falco ruleset for k8s audit events can be used to monitor GKE clusters as well.

Which issue(s) this PR fixes: Add a k8saudit-gke plugin to the plugins suite (requested plugins are tracked in #228 )

Special notes for your reviewer:

is it acceptable to bump golang to 1.21 (from 1.18) for plugins?
default k8s rules have to be fine-tuned a bit for gke (e.g. add the allowed gke images). Currently the rule overrides are in the dir rule_extensions as the ci workflow validator fails for override rules. Obviously it would be nice to release/ship this rule file as part of the plugin, so it can be installed with falcoctl. Is this fixable in the ci pipeline?
the plugin id is currently 666, if we can register a "real" plugin id that would be great

sboschman commented 5 months ago

@Issif as I mentioned earlier in #228 here is the pr for a k8saudit-gke plugin. I did put down some issues/questions in the reviewer notes, some guidance with these would be appreciated.

github-actions[bot] commented 5 months ago

Rules files suggestions

Issif commented 5 months ago

Can you sign off all the commits please.

For the version of Go, @jasondellaluce could help us, last time I tried I got issues, don't remember which ones.

I guess we need to add your plugin here to allow it to use the file:

https://github.com/falcosecurity/plugins/blob/master/plugins/k8saudit/rules/k8s_audit_rules.yaml#L21,L26

github-actions[bot] commented 5 months ago

Rules files suggestions

sboschman commented 5 months ago

I guess we need to add your plugin here to allow it to use the file:

https://github.com/falcosecurity/plugins/blob/master/plugins/k8saudit/rules/k8s_audit_rules.yaml#L21,L26

yes indeed, modded my own rules files with:

- required_plugin_versions:
  - name: k8saudit
    version: 0.7.0
    alternatives:
      - name: k8saudit-gke
        version: 0.1.0

Don't know how the release versioning in the ci workflows does it thing, but I can just add it with version 0.1.0 ?

github-actions[bot] commented 5 months ago

Rules files suggestions

github-actions[bot] commented 5 months ago

Rules files suggestions

sboschman commented 5 months ago

This is what happens when I place the gke rule overrides into the rules dir (job):

Error: /__w/plugins/plugins/plugins/k8saudit-gke/rules/k8s_audit_rules_gke.yaml: Invalid
1 Errors:
In rules content: (/__w/plugins/plugins/plugins/k8saudit-gke/rules/k8s_audit_rules_gke.yaml:0:0)
    rule 'Disallowed K8s User': (/__w/plugins/plugins/plugins/k8saudit-gke/rules/k8s_audit_rules_gke.yaml:10:2)
------
- rule: Disallowed K8s User
  ^
------
LOAD_ERR_YAML_VALIDATE (Error validating internal structure of YAML file): Item has no mapping for key 'output'

The rule override:

- macro: gke_service_agent
  condition: (ka.user.name endswith "@container-engine-robot.iam.gserviceaccount.com")

- rule: Disallowed K8s User
  condition: and not gke_service_agent
  override: 
    condition: append

any idea if this is fixable @Issif (or perhaps @jasondellaluce ) ? Or is this kind of plugin rule overrides not allowed by design?

Issif commented 5 months ago

cc @alacuku @LucaGuerra can you help us with the CI for the rules?

leogr commented 5 months ago

This is what happens when I place the gke rule overrides into the rules dir (job):

Error: /__w/plugins/plugins/plugins/k8saudit-gke/rules/k8s_audit_rules_gke.yaml: Invalid
1 Errors:
In rules content: (/__w/plugins/plugins/plugins/k8saudit-gke/rules/k8s_audit_rules_gke.yaml:0:0)
    rule 'Disallowed K8s User': (/__w/plugins/plugins/plugins/k8saudit-gke/rules/k8s_audit_rules_gke.yaml:10:2)
------
- rule: Disallowed K8s User
  ^
------
LOAD_ERR_YAML_VALIDATE (Error validating internal structure of YAML file): Item has no mapping for key 'output'

The rule override:

- macro: gke_service_agent
  condition: (ka.user.name endswith "@container-engine-robot.iam.gserviceaccount.com")

- rule: Disallowed K8s User
  condition: and not gke_service_agent
  override: 
    condition: append

any idea if this is fixable @Issif (or perhaps @jasondellaluce ) ? Or is this kind of plugin rule overrides not allowed by design?

The issue is because override was introduced in Falco 0.37, but the CI still uses Falco 0.36.2. @alacuku @LucaGuerra I guess we can upgrade the CI to use the latest Falco version.

Issif commented 5 months ago

This is what happens when I place the gke rule overrides into the rules dir (job):
Error: /__w/plugins/plugins/plugins/k8saudit-gke/rules/k8s_audit_rules_gke.yaml: Invalid
1 Errors:
In rules content: (/__w/plugins/plugins/plugins/k8saudit-gke/rules/k8s_audit_rules_gke.yaml:0:0)
    rule 'Disallowed K8s User': (/__w/plugins/plugins/plugins/k8saudit-gke/rules/k8s_audit_rules_gke.yaml:10:2)
------
- rule: Disallowed K8s User
  ^
------
LOAD_ERR_YAML_VALIDATE (Error validating internal structure of YAML file): Item has no mapping for key 'output'
The rule override:
- macro: gke_service_agent
  condition: (ka.user.name endswith "@container-engine-robot.iam.gserviceaccount.com")

- rule: Disallowed K8s User
  condition: and not gke_service_agent
  override: 
    condition: append
any idea if this is fixable @Issif (or perhaps @jasondellaluce ) ? Or is this kind of plugin rule overrides not allowed by design?
The issue is because override was introduced in Falco 0.37, but the CI still uses Falco 0.36.2. @alacuku @LucaGuerra I guess we can upgrade the CI to use the latest Falco version.

I prepared a PR for https://github.com/falcosecurity/plugins/pull/426

leogr commented 5 months ago

I prepared a PR for #426

426 is merged now.

@sboschman could you rebase this, pls? The CI issue should be addressed then. Thanks!

github-actions[bot] commented 5 months ago

Rules files suggestions

sboschman commented 5 months ago

@leogr check now runs with 0.37.1, but we somehow have to tell te checker it has to load the default/base k8saudit rules as well, any ideas on this?

Issif commented 5 months ago

@leogr check now runs with 0.37.1, but we somehow have to tell te checker it has to load the default/base k8saudit rules as well, any ideas on this?

Can you try to add a symlink to the k8s_audit_rules.yaml, I'm not sure but it could work, because of this https://github.com/falcosecurity/plugins/blob/ff78f4218758e1dd0d0e4e0a06b7b5d8198a3458/.github/setup-plugin-config-rules.sh#L22

github-actions[bot] commented 5 months ago

Rules files suggestions

sboschman commented 5 months ago

@Issif that seems to have solved the validate-local job, but the validate-falcoctl job still gives an error:

Error: rules validation command failed, open /usr/share/falco/plugins/libk8saudit-gke.so: no such file or directory

I am not sure what this Validate plugin and rules step is doing, is it trying to d/l this new unreleased plugin with falcoctl from the oci registry? In the step script it says something 'echo Attempt installing locally-built plugin "${plugin_name}"...' so should it be able to grab the plugin .so file from the build?

The plugins-x86_64-424.tar.gz build artifiact does contain a k8saudit-gke-0.1.0-0.0.0-0+4cb2951-linux-x86_64 file

leogr commented 5 months ago

@Issif that seems to have solved the validate-local job, but the validate-falcoctl job still gives an error:
Error: rules validation command failed, open /usr/share/falco/plugins/libk8saudit-gke.so: no such file or directory
I am not sure what this Validate plugin and rules step is doing, is it trying to d/l this new unreleased plugin with falcoctl from the oci registry? In the step script it says something 'echo Attempt installing locally-built plugin "${plugin_name}"...' so should it be able to grab the plugin .so file from the build?

The plugins-x86_64-424.tar.gz build artifiact does contain a k8saudit-gke-0.1.0-0.0.0-0+4cb2951-linux-x86_64 file

The error simply means that the plugin is not present in the container image. I can't recall the full workflow right now. @LucaGuerra @alacuku, any clue? :thinking:

leogr commented 5 months ago

@sboschman also note that the DCO is missing in this commit :point_down: Could you sign it off pls? :pray:

The list of commits missing DCO signoff:

df0b409 ci(k8saudit-gke): try symlink to base k8saudit rules

sboschman commented 5 months ago

    deps=$(cat $rules_file | yq -r '.[].required_plugin_versions | select(. != null and . != "")[] | [.name + ":" + .version] | @csv')

I have an idea.. I have no

- required_engine_version: 15
- required_plugin_versions:

fields in my rules file

alacuku commented 5 months ago

@Issif that seems to have solved the validate-local job, but the validate-falcoctl job still gives an error:
Error: rules validation command failed, open /usr/share/falco/plugins/libk8saudit-gke.so: no such file or directory
I am not sure what this Validate plugin and rules step is doing, is it trying to d/l this new unreleased plugin with falcoctl from the oci registry? In the step script it says something 'echo Attempt installing locally-built plugin "${plugin_name}"...' so should it be able to grab the plugin .so file from the build? The plugins-x86_64-424.tar.gz build artifiact does contain a k8saudit-gke-0.1.0-0.0.0-0+4cb2951-linux-x86_64 file
The error simply means that the plugin is not present in the container image. I can't recall the full workflow right now. @LucaGuerra @alacuku, any clue? 🤔

It seems that the tool is installing the wrong plugin: k8saudit plugin instead of the k8saudit-gke..

github-actions[bot] commented 5 months ago

Rules files suggestions

alacuku commented 5 months ago

@sboschman, could you try to remove the rulesfile from the k8saudit-gke plugin? From my understanding, the CI should behave the same way as with k8saudit-eks plugin which shares the same rules file as the k8saudit plugin. The CI creates an empty rulesfile and validates the plugin.

sboschman commented 5 months ago

@sboschman, could you try to remove the rulesfile from the k8saudit-gke plugin? From my understanding, the CI should behave the same way as with k8saudit-eks plugin which shares the same rules file as the k8saudit plugin. The CI creates an empty rulesfile and validates the plugin.

yes that works to pass the ci, but how do we publish the gke specific rule overrides?

for example:

# Allow the Google Container Engine robot serviceaccount
- rule: Disallowed K8s User
  exceptions:
    - name: google-container-engine-robot
      fields: [ka.user.name]
      comps: [endswith]
      values:
        - ["@container-engine-robot.iam.gserviceaccount.com"]
  override: 
    exceptions: append

github-actions[bot] commented 5 months ago

Rules files suggestions

github-actions[bot] commented 5 months ago

Rules files suggestions

sboschman commented 5 months ago

The error simply means that the plugin is not present in the container image. I can't recall the full workflow right now. @LucaGuerra @alacuku, any clue? 🤔

@leogr , indeed, not sure what the expected behaviour is here.

1) Without the k8saudit rules, validation fails; sort of expected as the rules append/override the k8saudit rules and are not standalone rules by themselves 2) As @Issif suggested, we can add a symlink to include the k8saudit rules, now we can pass the validate-local job, but validate-falcoctl job will fail as it now messes up rules and deps; it loads the deps from the k8saudit rules into the container, and tries to validate the k8saudit rules with the k8saudit-gke.so plugin, which it did not load into the container... and it never gets to the point to validate the k8saudit-gke rules.

Furthermore:

what do we have to bundle in the k8saudit-gke rules image for falcoctl to install? Only our gke overrides, or both the base k8saudit rules and our gke extensions? I suppose, symlinking the base k8saudit rules, also means they get included into the oci image for falcoctl... Issueing two falcoctl artifact install command for both the base rules and gke rules does not sound like a problem to me? Compared to only have to install the gke rules artifact. If you install the eks plugin, you also have to falcoctl install the base rules.
if we revert the symlink idea, how do we load additional rule files for validation, as we have to load both the falco-stable and base k8saudit rules in the ci container

leogr commented 5 months ago

/assign

leogr commented 5 months ago

Hey @sboschman

I just wanted to let you know that @LucaGuerra, @alacuku and I are working to find a solution to the two distinct problems affecting this PR:

There's (at least) one missing list definition in k8s_audit_rules.yaml (the list is falco_privileged_images, the one you tried to extend. @LucaGuerra will open a separate PR to address this. Once this is fixed, we will ask you to rebase this PR (and the symlink idea will likely be okay).
The missing plugin artifact (as you reported). We are currently evaluating different options. We also were in the process of reviving our CI pipeline (which accumulated some technical debt).

We need a bit of time to address these, then we will get back to you :)

Btw, thanks again for this significant contribution. I look forward to seeing this merged :clap:

:pray:

sboschman commented 5 months ago

Thanks for the heads up, @leogr . All the help is much appreciated 🙏

alacuku commented 5 months ago

Hi @sboschman, could you please rebase your PR on top of #432? I think you should drop all the changes you have made to the CI for debugging purposes.

The #432 unifies how the deps are extracted from the rulesfiles and handles the case when the tested plugin is set as an alternative. Furthermore, the validate-falcoctl job loads all the rules at once, which is needed for the k8saudit-gke plugin.

github-actions[bot] commented 5 months ago

Rules files suggestions

alacuku commented 5 months ago

Hi @sboschman, could you please rebase your PR on top of #432? I think you should drop all the changes you have made to the CI for debugging purposes.

The #432 unifies how the deps are extracted from the rulesfiles and handles the case when the tested plugin is set as an alternative. Furthermore, the validate-falcoctl job loads all the rules at once, which is needed for the k8saudit-gke plugin.

The validation jobs are now working as intended. Failing because the rules seem not to be valid.

sboschman commented 5 months ago

Hi @sboschman, could you please rebase your PR on top of #432? I think you should drop all the changes you have made to the CI for debugging purposes. The #432 unifies how the deps are extracted from the rulesfiles and handles the case when the tested plugin is set as an alternative. Furthermore, the validate-falcoctl job loads all the rules at once, which is needed for the k8saudit-gke plugin.

The validation jobs are now working as intended. Failing because the rules seem not to be valid.

yeah wrong engine version, but we haven't really validated any rules yet, so I am still expecting it to fail b/c the deps on stable and/or sandbox rules.

alacuku commented 5 months ago

yeah wrong engine version, but we haven't really validated any rules yet, so I am still expecting it to fail b/c the deps on stable and/or sandbox rules.

This #431 PR should solve the deps issue with stable/sandbox rules. @LucaGuerra

github-actions[bot] commented 5 months ago

Rules files suggestions

sboschman commented 5 months ago

yeah wrong engine version, but we haven't really validated any rules yet, so I am still expecting it to fail b/c the deps on stable and/or sandbox rules.

This #431 PR should solve the deps issue with stable/sandbox rules. @LucaGuerra

I see... but I rebased on #432, so that change isn't used yet... should I cherrypick the change from #431 and update the gke rules to use this new list?

github-actions[bot] commented 5 months ago

Rules files suggestions

gcp_auditlog_rules.yaml

Comparing ecaf1b1e98080e69abeb67ecbd5a2a9facccaa4c with latest tag gcpaudit-0.3.0

No changes detected

github-actions[bot] commented 5 months ago

Rules files suggestions

gcp_auditlog_rules.yaml

Comparing 49d4d974e7d02e73881ff45249fe950e7525bd2d with latest tag gcpaudit-0.3.0

No changes detected

Rules files suggestions

sboschman commented 5 months ago

we do have a green build, @leogr @alacuku @Issif 🥳

github-actions[bot] commented 5 months ago

Rules files suggestions

gcp_auditlog_rules.yaml

Comparing 3c7c4fdec4a43908bc142d408da8654c273b266b with latest tag gcpaudit-0.3.0

No changes detected

alacuku commented 5 months ago

432 merged.

poiana commented 5 months ago

LGTM label has been added.

Git tree hash: aebd4d4508476c84f4a265c399ec2b127fc82ee9

poiana commented 5 months ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: leogr, sboschman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/falcosecurity/plugins/blob/master/OWNERS)~~ [leogr] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

falcosecurity / plugins

A plugin to collect K8S Audit Logs from GKE (k8saudit-gke plugin) #424

Rules files suggestions

Rules files suggestions

Rules files suggestions

Rules files suggestions

426 is merged now.

Rules files suggestions

Rules files suggestions

Rules files suggestions

Rules files suggestions

Rules files suggestions

Rules files suggestions

Rules files suggestions

Rules files suggestions

gcp_auditlog_rules.yaml

Rules files suggestions

gcp_auditlog_rules.yaml

Rules files suggestions

Rules files suggestions

gcp_auditlog_rules.yaml

432 merged.