yonahd / kor

A Golang Tool to discover unused Kubernetes Resources
MIT License
1.04k stars 96 forks source link

feat: add regex in resource exceptions #277

Closed Itaykal closed 5 months ago

Itaykal commented 6 months ago

What this PR does / why we need it

Adds regular expression support to exceptions/*/*.json files.

PR Checklist

Github Issue

Fixes #276

Notes for your reviewers

Not sure this is how this impacts performance. The behavior of the exception filtration in configmaps and serviceaccounts is now changed to be aligned with all other objects, this as well has some impact on performance.

codecov-commenter commented 6 months ago

Codecov Report

Attention: Patch coverage is 37.68116% with 43 lines in your changes missing coverage. Please review.

Project coverage is 40.87%. Comparing base (5b9c270) to head (60fb008). Report is 7 commits behind head on main.

Files Patch % Lines
pkg/kor/configmaps.go 50.00% 3 Missing and 3 partials :warning:
pkg/kor/serviceaccounts.go 50.00% 3 Missing and 3 partials :warning:
pkg/kor/crds.go 0.00% 4 Missing :warning:
pkg/kor/kor.go 60.00% 2 Missing and 2 partials :warning:
pkg/kor/roles.go 20.00% 2 Missing and 2 partials :warning:
pkg/kor/secrets.go 33.33% 2 Missing and 2 partials :warning:
pkg/kor/clusterroles.go 25.00% 1 Missing and 2 partials :warning:
pkg/kor/daemonsets.go 25.00% 1 Missing and 2 partials :warning:
pkg/kor/jobs.go 25.00% 1 Missing and 2 partials :warning:
pkg/kor/services.go 25.00% 1 Missing and 2 partials :warning:
... and 1 more

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #277 +/- ## ========================================== - Coverage 40.96% 40.87% -0.09% ========================================== Files 58 58 Lines 2910 2953 +43 ========================================== + Hits 1192 1207 +15 - Misses 1530 1545 +15 - Partials 188 201 +13 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

yonahd commented 6 months ago

The PR looks like it would work well. The issue I have with it is the readability, searchability and the extra work to add new resources here. The question is if supporting prefixes and suffixes would be better codewise as that would cover the functionality we are currently lacking. WDYT?

Itaykal commented 5 months ago

It's quite frustrating to work with... I thought about adding a flag to enable regex searching on specific exceptions that require them and thus simplifying all "non-regex" searches, something along the lines of this:

{
    "Namespace": ".*-example",
    "ResourceName": "example",
    "Regex": true
}

This is also quite unreadable but it grants the efficiency and ease of use of regex while still allowing more readable exceptions when not needed.

Another topic that scares me a bit more then readability is that with some minor human error this could cause false positives. I think this should be discussed before moving on to readability (and ease of use) as I don't see any way to avoid this risk.

yonahd commented 5 months ago

It's quite frustrating to work with... I thought about adding a flag to enable regex searching on specific exceptions that require them and thus simplifying all "non-regex" searches, something along the lines of this:

{
    "Namespace": ".*-example",
    "ResourceName": "example",
    "Regex": true
}

This is also quite unreadable but it grants the efficiency and ease of use of regex while still allowing more readable exceptions when not needed.

Another topic that scares me a bit more then readability is that with some minor human error this could cause false positives. I think this should be discussed before moving on to readability (and ease of use) as I don't see any way to avoid this risk.

The damage of a regex too broad would marking unused resources as used which is not great but the damage is less than the reverse.

We obviously want kor to find as many unused resources as possible so we would need to be cautious in this area.

The solution might be to create a separate json for regexes or prefixes and handle it separately. This or the solution you provided with the additional field are likely cleaner and safer

Itaykal commented 5 months ago

I've implemented the flag solution as it's easier to handle later on, wdyt?

yonahd commented 5 months ago

Looks much better. I'd like to have tests in place for this. Also (not necessarily belonging to this PR but we need to have validators for the exception jsons)

Itaykal commented 5 months ago

What should be tested here? the regex exception's usability or the validity of the expressions themselves?

doronkg commented 5 months ago

@Itaykal can you rebase the branch so the exceptions folder will go under validation of the new GitHub action added in #295 and for the kor binary to be ignored as applied in #290?

yonahd commented 5 months ago

What should be tested here? the regex exception's usability or the validity of the expressions themselves?

I think we need to test the that it excludes

Itaykal commented 5 months ago

Rebased and added testing.

doronkg commented 5 months ago

@Itaykal many exception duplications were added in this PR (probably following the rebasing), can you create a new PR that removes them?

Bonus - enhance the validation workflow (#298) to remove duplications.