[tracking] missing Falco regression tests

jasondellaluce commented 1 year ago

This is a tracker for all the tests related to the Falco CLI executable that are still missing.

Each of the following bullet point is a wonderful opportunity for easy first contributions. This tracker will be kept updated whenever PRs will get opened for any of the bullet points.

Falco command line

Test the behavior and expected output of the Falco CLI depending on its options, considered both individually and in combinations:

[ ] Commands printing information:
- [ ] -h
- [ ] --help
- [ ] --support
- [x] -i
- [x] -L
- [ ] -l
- [ ] --list
- [ ] --list-syscall-events
- [ ] --markdown
- [ ] -N
- [ ] --gvisor-generate-config
- [ ] --page-size
[ ] Metadata collection and container runtime:
- [ ] --cri
- [ ] --disable-cri-async
- [x] -k [DEPRECATED]
- [x] --k8s-api [DEPRECATED]
- [x] -K [DEPRECATED]
- [x] --k8s-api-cert [DEPRECATED]
- [x] --k8s-node [DEPRECATED]
- [x] -m [DEPRECATED]
- [x] --mesos-api [DEPRECATED]
[ ] Falco event collection modes:
- [x] -g [DEPRECATED]
- [x] --gvisor-config [DEPRECATED]
- [x] --gvisor-root [DEPRECATED]
- [x] --modern-bpf [DEPRECATED]
[ ] Changers of Falco's behavior:
- [ ] --disable-source
- [ ] --enable-source
- [ ] -A
- [ ] -P
- [ ] --pidfile
- [ ] -p
- [ ] --print
- [ ] -b
- [ ] --print-base64
- [ ] -S
- [ ] --snaplen
[ ] Misc Falco features:
- [x] -s [DEPRECATED]
- [x] --stats-interval [DEPRECATED]
- [ ] -U
- [ ] --unbuffered

Falco configuration fields

Test the behavior and expected output of Falco depending on the falco.yaml fields:

[x] watch_config_files
[ ] libs_logger
[ ] buffered_outputs
[ ] syscall_event_timeouts
[x] syscall_buf_size_preset [DEPRECATED]
[x] modern_bpf [DEPRECATED]
[ ] output_timeout
[ ] outputs
[ ] syslog_output
[ ] file_output
[ ] stdout_output
[ ] webserver
[ ] program_output
[ ] http_output
[x] metadata_download

Falco environment variables

Test the Falco behavior on the supported environment variables and their priorities, also in combination with their args/configs/cmds counterparts:

[x] FALCO_K8S_API [DEPRECATED]
[x] FALCO_K8S_API_CERT [DEPRECATED]
[x] FALCO_MESOS_API [DEPRECATED]
[ ] FALCO_HOSTNAME
[ ] FALCO_GRPC_HOSTNAME
[x] FALCO_BPF_PROBE [DEPRECATED]
[ ] HOME (used for bpf probe)

Other Falco things

Umbrella category for all the rest of things done by Falco not fitting in the above lists. This will likely be moved to another issue because they are more oriented to the integration with other tools (such as the driver loader), but I'm gonna list them here for now:

Falco reaction to signals SIGINT, SIGUSR1, SIGHUP
Collection of live events with kmod, bpf, modern-bpf, gvisor, userspace
Collection of live events with multiple event sources active at the same
Stress test with event generator, checking memory usage and event drops

poiana commented 1 year ago

There is not a label identifying the kind of this issue. Please specify it either using /kind <group> or manually from the side menu.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

poiana commented 1 year ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

poiana commented 1 year ago

Stale issues rot after 30d of inactivity.

Mark the issue as fresh with /remove-lifecycle rotten.

Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle rotten

leogr commented 1 year ago

/kind feature /remove-lifecycle rotten.

poiana commented 1 year ago

@leogr: The label(s) kind/feature cannot be applied, because the repository doesn't have them.

In response to [this](https://github.com/falcosecurity/testing/issues/7#issuecomment-1708033871): >/kind feature >/remove-lifecycle rotten. Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

leogr commented 1 year ago

/kind feature

incertum commented 11 months ago

@jasondellaluce we can remove the deprecated config or cmd args options.

poiana commented 10 months ago

Rotten issues close after 30d of inactivity.

Reopen the issue with /reopen.

Mark the issue as fresh with /remove-lifecycle rotten.

Provide feedback via https://github.com/falcosecurity/community. /close

poiana commented 10 months ago

@poiana: Closing this issue.

In response to [this](https://github.com/falcosecurity/testing/issues/7#issuecomment-1792136271): >Rotten issues close after 30d of inactivity. > >Reopen the issue with `/reopen`. > >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Provide feedback via https://github.com/falcosecurity/community. >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

leogr commented 10 months ago

/reopen

poiana commented 10 months ago

@leogr: Reopened this issue.

In response to [this](https://github.com/falcosecurity/testing/issues/7#issuecomment-1802206953): >/reopen Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

leogr commented 10 months ago

/remove-lifecycle rotten

leogr commented 10 months ago

/help

poiana commented 10 months ago

@leogr: This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed by commenting with the /remove-help command.

In response to [this](https://github.com/falcosecurity/testing/issues/7): >/help Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

leogr commented 9 months ago

/assign

leogr commented 9 months ago

This list must be reviewed considering the new deprecations:

@Andreagit97 @FedeDP could you help me with that please?

Andreagit97 commented 9 months ago

I 've already done it putting [DEPRECATED] near the items, but please double check

incertum commented 9 months ago

Check on some Legacy tests around enabling rules with or without tags and in combination with using the enabled key or not. Some tests seem to assert final rules that are enabled (e.g. TestFalco_Legacy_DisabledTagsB), other test could be missing it? e.g. TestFalco_Legacy_DisabledRulesUsingEnabledFlag or TestFalco_Legacy_DisabledRuleUsingFalseEnabledFlagOnly etc.

poiana commented 6 months ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

Andreagit97 commented 6 months ago

/remove-lifecycle stale

poiana commented 3 months ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

FedeDP commented 3 months ago

/remove-lifecycle stale

Marked watch_config_files as done since testing framework now tests it: https://github.com/falcosecurity/testing/pull/54

poiana commented 2 weeks ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

Andreagit97 commented 2 weeks ago

/remove-lifecycle stale

falcosecurity / testing