[Bug] [CLI] Pass result in test when it should be skip

[anushkamittal2001]

Kyverno CLI Version

1.12.0

Description

Recently we added this PR https://github.com/kyverno/kyverno/pull/9274/files, to deal with

When resources are excluded the EngineResponse doesn't contain any RuleResponses. This wasn't handled in the CLI and the resources were reported as Not Found.

Now for excluded resources we get send pass result, it should actually be skip as stated here: https://kyverno.io/docs/kyverno-cli/usage/test/#:~:text=The%20resource%20does%20not%20meet%20either%20the%20match%20or%20exclude%20block%2C%20or%20does%20not%20pass%20the%20preconditions%20statements.

Steps to reproduce

1. policy

   ## Test
   apiVersion: kyverno.io/v1
   kind: ClusterPolicy
   metadata:
   name: deny-zero-grace-delete
   spec:
   validationFailureAction: Enforce
   background: false
   rules:
   - name: deny-zero-grace-delete
   match:
     any:
     - resources:
         kinds:
         - Pod
         - Deployment
         - Service
         - PersistentVolumeClaim
         operations:
         - DELETE
   exclude:
     any:
     - clusterRoles:
       - cluster-admin
   validate:
     message: "Force deletion of resources with '--grace-period=0 --force' is not allowed for non-admin users."
     deny:
       conditions:
         all:
         - key: "{{ request.options.gracePeriodSeconds }}"
           operator: LessThan
           value: 1

2. resource

   apiVersion: v1
   kind: Pod
   metadata:
   name: badpod01
   spec:
   initContainers:
   - name: initcontainer01
   image: dummyimagename
   - name: initcontainer02
   image: dummyimagename
   securityContext:
     allowPrivilegeEscalation: true
   containers:
   - name: container01
   image: dummyimagename
   securityContext:
     allowPrivilegeEscalation: false
   options:
   gracePeriodSeconds: 0

3. user_info

   apiVersion: cli.kyverno.io/v1alpha1
   kind: UserInfo
   metadata:
   name: user-info
   clusterRoles:
   - cluster-admin
   userInfo:
   username: someone@somecorp.com

4. values

   apiVersion: cli.kyverno.io/v1alpha1
   kind: Values
   globalValues:
   request.operation: DELETE
   policies:
   - name: deny-zero-grace-delete
   resources:
   - name: badpod01
   values:
     request.options.gracePeriodSeconds: 0

5. test

   apiVersion: cli.kyverno.io/v1alpha1
   kind: Test
   metadata:
   name: deny-zero-grace-delete
   policies:
   - ./deny-force-delete.yaml
   resources:
   - resource.yaml
   results:
   - kind: Deployment
   policy: deny-zero-grace-delete
   resources:
   - baddeployment01
   result: pass
   rule: deny-zero-grace-delete
   variables: values1.yaml
   userinfo: user_info.yaml

on running test command:

kyverno test .          
Loading test  ( kyverno-test.yaml ) ...
  Loading values/variables ...
  Loading user infos ...
  Loading policies ...
  Loading resources ...
  Loading exceptions ...
  Applying 1 policy to 1 resource ...
  Checking results ...

│────│────────────────────────│────────────────────────│──────────────│────────│──────────│
│ ID │ POLICY                 │ RULE                   │ RESOURCE     │ RESULT │ REASON   │
│────│────────────────────────│────────────────────────│──────────────│────────│──────────│
│ 1  │ deny-zero-grace-delete │ deny-zero-grace-delete │ Pod/badpod01 │ Pass   │ Excluded │
│────│────────────────────────│────────────────────────│──────────────│────────│──────────│

Expected behavior

The result should show Skip because resource excluded

Screenshots

No response

Kyverno logs

No response

Slack discussion

No response

Troubleshooting

• [X] I have read and followed the troubleshooting guide.
• [X] I have searched other issues in this repository and mine is not recorded.

[chipzoller]

The value of the RESULT column in the test output is always Pass when the tested outcome matches the outcome stated in the test manifest's results[].result field. If this field is populated with skip and the actual outcome of the test is a skip then the RESULT column is still expected to say Pass. I'm in favor of introducing a new results[].result value called excluded so that these test results align with Policy Report results. In a Policy Report, a resource which is not matched as a result of the combined match and/or exclude blocks does not receive a result called skip; there is no Policy Report written at all. The CLI should align with this so as not to create confusion for users.

~And not that it's probably relevant, but I've no idea what this options.gracePeriodSeconds field is you're coming up with in your Pod definition.~ This is actually a valid field populated in the AdmissionReview when a DELETE request is sent with flags.