I am new to Kyverno (not new to the Kubernetes ecosystem, though, if that is relevant) and am trying to set up some policies and test cases so that I can validate Kubernetes manifests outside of a cluster (the goal is to implement GitHub Actions pull request checks in a repository managed by ArgoCD).
However, when I run the same kyverno test . command, I get this failure response:
$ kyverno test .
WARNING: test file (kyverno-test.yaml) uses a deprecated schema that will be removed in 1.13
Loading test ( kyverno-test.yaml ) ...
Loading values/variables ...
Loading policies ...
Loading resources ...
Loading exceptions ...
Applying 1 policy to 1 resource ...
Checking results ...
│────│─────────────────────│────────────────────│───────────────│────────│───────────│
│ ID │ POLICY │ RULE │ RESOURCE │ RESULT │ REASON │
│────│─────────────────────│────────────────────│───────────────│────────│───────────│
│ 1 │ disallow-latest-tag │ require-image-tag │ Pod/myapp-pod │ Fail │ Not found │
│ 2 │ disallow-latest-tag │ validate-image-tag │ Pod/myapp-pod │ Fail │ Not found │
│────│─────────────────────│────────────────────│───────────────│────────│───────────│
Test Summary: 0 tests passed and 2 tests failed
Aggregated Failed Test Cases :
│────│─────────────────────│────────────────────│───────────────│────────│───────────│
│ ID │ POLICY │ RULE │ RESOURCE │ RESULT │ REASON │
│────│─────────────────────│────────────────────│───────────────│────────│───────────│
│ 1 │ disallow-latest-tag │ require-image-tag │ Pod/myapp-pod │ Fail │ Not found │
│ 2 │ disallow-latest-tag │ validate-image-tag │ Pod/myapp-pod │ Fail │ Not found │
│────│─────────────────────│────────────────────│───────────────│────────│───────────│
Error: 2 tests failed
I searched the Kyverno discussions (link to search) and came across this comment, which suggested adding the -v=8 flag for more detailed output.
When I pass -v=8, I get an additional ERROR log message in the output:
2024-05-10T08:46:27-05:00 ERROR kubectl-kyverno test/test.go:150 skipping invalid policy {"name": "disallow-latest-tag", "error": "only select variables are allowed in background mode. Set spec.background=false to disable background mode for this policy rule: invalid variable used at path: spec/rules[0]/match/any[0]/clusterRoles "}
Full output:
$ kyverno test . -v=8
WARNING: test file (kyverno-test.yaml) uses a deprecated schema that will be removed in 1.13
Loading test ( kyverno-test.yaml ) ...
Loading values/variables ...
Loading policies ...
Loading resources ...
Loading exceptions ...
Applying 1 policy to 1 resource ...
2024-05-10T08:46:27-05:00 ERROR kubectl-kyverno test/test.go:150 skipping invalid policy {"name": "disallow-latest-tag", "error": "only select variables are allowed in background mode. Set spec.background=false to disable background mode for this policy rule: invalid variable used at path: spec/rules[0]/match/any[0]/clusterRoles "}
Checking results ...
│────│─────────────────────│────────────────────│───────────────│────────│───────────│
│ ID │ POLICY │ RULE │ RESOURCE │ RESULT │ REASON │
│────│─────────────────────│────────────────────│───────────────│────────│───────────│
│ 1 │ disallow-latest-tag │ require-image-tag │ Pod/myapp-pod │ Fail │ Not found │
│ 2 │ disallow-latest-tag │ validate-image-tag │ Pod/myapp-pod │ Fail │ Not found │
│────│─────────────────────│────────────────────│───────────────│────────│───────────│
Test Summary: 0 tests passed and 2 tests failed
Aggregated Failed Test Cases :
│────│─────────────────────│────────────────────│───────────────│────────│───────────│
│ ID │ POLICY │ RULE │ RESOURCE │ RESULT │ REASON │
│────│─────────────────────│────────────────────│───────────────│────────│───────────│
│ 1 │ disallow-latest-tag │ require-image-tag │ Pod/myapp-pod │ Fail │ Not found │
│ 2 │ disallow-latest-tag │ validate-image-tag │ Pod/myapp-pod │ Fail │ Not found │
│────│─────────────────────│────────────────────│───────────────│────────│───────────│
Error: 2 tests failed
If I remove the clusterRoles property from the require-image-tag rule:
- resources:
kinds:
- Pod
- clusterRoles:
- - cluster-admin
validate:
message: "An image tag is required."
pattern:
Then I get the same behavior (success) as the documentation example:
$ kyverno test . -v=8
WARNING: test file (kyverno-test.yaml) uses a deprecated schema that will be removed in 1.13
Loading test ( kyverno-test.yaml ) ...
Loading values/variables ...
Loading policies ...
Loading resources ...
Loading exceptions ...
Applying 1 policy to 1 resource ...
Checking results ...
│────│─────────────────────│────────────────────│───────────────│────────│────────│
│ ID │ POLICY │ RULE │ RESOURCE │ RESULT │ REASON │
│────│─────────────────────│────────────────────│───────────────│────────│────────│
│ 1 │ disallow-latest-tag │ require-image-tag │ Pod/myapp-pod │ Pass │ Ok │
│ 2 │ disallow-latest-tag │ validate-image-tag │ Pod/myapp-pod │ Pass │ Ok │
│────│─────────────────────│────────────────────│───────────────│────────│────────│
Test Summary: 2 tests passed and 0 tests failed
There is still a difference between the output in the documentation and my local output, in that the documentation shows the default namespace in output (in the resource column), while my local output does not - that doesn't feel significant to me but wanted to mention it here regardless.
I also was curious about the spec.background=false bit in the error message, so searched for that and found the Background Scans documentation. Following that, I set background: false in the policy:
Page link
https://kyverno.io/docs/kyverno-cli/usage/test/#examples
Description
I am new to Kyverno (not new to the Kubernetes ecosystem, though, if that is relevant) and am trying to set up some policies and test cases so that I can validate Kubernetes manifests outside of a cluster (the goal is to implement GitHub Actions pull request checks in a repository managed by ArgoCD).
I am working through the Examples section of the documentation on the
kyverno test
CLI command (https://kyverno.io/docs/kyverno-cli/usage/test/#examples) and I wasn't seeing the same output that the documentation showed.I copied the YAML files provided by that documentation page (have them in a MWE git repo here too):
Contents of
disallow_latest_tag.yaml
:Contents of
resource.yaml
:Contents of
kyverno-test.yaml
:With these three files, the documentation shows
kyverno test .
returning this success response:However, when I run the same
kyverno test .
command, I get this failure response:I searched the Kyverno discussions (link to search) and came across this comment, which suggested adding the
-v=8
flag for more detailed output.When I pass
-v=8
, I get an additionalERROR
log message in the output:Full output:
If I remove the clusterRoles property from the
require-image-tag
rule:Then I get the same behavior (success) as the documentation example:
There is still a difference between the output in the documentation and my local output, in that the documentation shows the
default
namespace in output (in the resource column), while my local output does not - that doesn't feel significant to me but wanted to mention it here regardless.I also was curious about the
spec.background=false
bit in the error message, so searched for that and found the Background Scans documentation. Following that, I setbackground: false
in the policy:And this also allows the
kyverno test .
command to pass (even if theclusterRoles
property is present).Expected behavior
I expect that when running the same commands on the same files provided in the documentation (using the same CLI version), that I get the same output.
Slack discussion
n/a