search

argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes
https://argo-cd.readthedocs.io
Apache License 2.0
17.91k stars 5.46k forks source link

Using multiple scmProvider filters doesn't work properly and causes unexpected behaviour #14420

Open jjsiv opened 1 year ago

jjsiv commented 1 year ago

Checklist:

Describe the bug In essence, using multiple filters (which should be OR'ed, according to the docs) doesn't really work. Also using "multitype" (internally repositoryMatch is FilterTypeRepo and pathsExist is FilterTypeBranch) match within a single filter, e.g.

- repositoryMatch: ^myapp pathsExist: [kubernetes/kustomization.yaml]

Will cause the generator to needlessly query all repos in the org/project for branches, causing for example issues with hitting the API request rate limit as described in issues #10788 or #9413. The reason this happens is because a single Filter can only be of one type (FilterTypeRepo or FilterTypeBranch): https://github.com/argoproj/argo-cd/blob/master/applicationset/services/scm_provider/types.go#L32 And if you define a repositoryMatch + branchMatch/pathMatch, it will be of FilterTypeBranch due to the order of checking in this function: https://github.com/argoproj/argo-cd/blob/master/applicationset/services/scm_provider/utils.go#L12 Since the count of repo type filters is subsequently used for filtering repos by name/label: https://github.com/argoproj/argo-cd/blob/master/applicationset/services/scm_provider/utils.go#L111 setting any of the pathsExist, branchMatch, pathsDoNotExist alongside repositoryMatch or labelMatch will pretty much invalidate them.

The other issue is with using multiple filters, e.g.:

- repositoryMatch: ^myapp pathsExist: [kubernetes/kustomization.yaml]

  • repositoryMatch: ^otherapp

This is supposed to generate apps from (repo names starting with myapp AND having paths kubernetes/kustomization.yaml) OR (repo names starting with otherapp) but in reality in most cases it will not generate any apps at all. This is due to again having a "pathsExist" alongside "repositoryMatch" in Filter1 invalidating repositoryMatch in that filter. Only repositoryMatch from Filter2 will be applied, and then, pathsExist from Filter1. In this case, the previously defined filters effectively become a single filter:

- repositoryMatch: ^otherapp pathsExist: [kubernetes/kustomization.yaml]

This happens due to the repositoryMatch from Filter1 being invalidated due to presence of pathsExist in Filter1 + how branch filters will always apply to all filters (e.g. pathsExist from Filter1 will apply to Filter2). This is caused by this function call always returning branch filters not from just one filter, but from all filters, and then being looped over: https://github.com/argoproj/argo-cd/blob/master/applicationset/services/scm_provider/utils.go#L148

I tried to make this as clear as possible but I imagine it might still be a little confusing :smile:

To Reproduce There isn't enough debug logs in scmProvider modules to see what is actually going/what requests are being made, so it's needed to add some additional logs in various places. Or just follow the code flow. Examples from docs can be used: https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators-SCM-Provider/#filters

Expected behavior

Screenshots

Version All

Paste the output from `argocd version` here.

Logs

Paste any relevant application logs here.
jjsiv commented 1 year ago

I've already got a fix for this on my fork so I will try to raise a PR in the upcoming days.