Add Framework for Conformance Tests

tssurya commented 1 year ago

NOTE: Most of the framework is blatantly adopted from https://github.com/kubernetes-sigs/gateway-api/tree/main/conformance gateway-api code without reinventing the wheel so that we can also adopt the new conformance profile GEP. Of course I have modified some of this to suite the needs of netpol conformance. Kudos/Credit to the gateway-api team.

NOTE2: This PR only adds the test framework. The actual tests will be added in a second PR.

NOTE3: I contemplated using gateway-api imports into this repo to refer to their conformance test utilities but that didn't seem like the right thing to do because moving forward we would still need to customize some of these utils according to how we modify our tests and so on. @shaneutt has brought up some great points around consolidating some of these pieces into a separate repo that both parties and more parties in future can use extensibly but that is something to ponder in the future.

Sample Output:

[surya@hidden-temple network-policy-api]$ make conformance                                                                                                                   
go test -v ./conformance/... -args                                                                                                                                           
=== RUN   TestConformance                                                                                                                                                    
    conformance_test.go:56: Running conformance tests with cleanup: true                                                                                                     
         debug: false                                                                                                                                                        
         enable all features: false                                                                                                                                          
         supported features: []                                                                                                                                              
         exempt features: []                                                                                                                                                 
    suite.go:116: Test Setup: Applying base manifests                                                                                                                        
    apply.go:124: Creating network-policy-conformance-gryffindor Namespace                                                                                                   
    apply.go:124: Creating network-policy-conformance-slytherin Namespace                                                                                                    
    apply.go:124: Creating network-policy-conformance-hufflepuff Namespace                                                                                                   
    apply.go:124: Creating network-policy-conformance-ravenclaw Namespace                                                                                                    
    apply.go:124: Creating harry-potter StatefulSet                                                                                                                          
    apply.go:124: Creating draco-malfoy StatefulSet                                                                                                                          
    apply.go:124: Creating cedric-diggory StatefulSet                                                                                                                        
    apply.go:124: Creating luna-lovegood StatefulSet                                                                                                                         
    suite.go:119: Test Setup: Ensuring Namespaces and Pods from base manifests are ready                                                                                     
    helper.go:32: Ready was not in conditions list                                                                                                                           
    helper.go:34: network-policy-conformance-gryffindor/harry-potter-0 Pod not ready yet                                                                                     
    helper.go:32: Ready condition set to False, expected True                                                                                                                
    helper.go:32: Ready was not in conditions list                                                                                                                           
    helper.go:34: network-policy-conformance-gryffindor/harry-potter-0 Pod not ready yet                                                                                     
    helper.go:32: Ready condition set to False, expected True                                                                                                                
    helper.go:32: Ready was not in conditions list                                                                                                                           
    helper.go:34: network-policy-conformance-gryffindor/harry-potter-1 Pod not ready yet                                                                                     
    helper.go:32: Ready condition set to False, expected True                                                                                                                
    helper.go:32: Ready was not in conditions list                                                                                                                           
    helper.go:34: network-policy-conformance-gryffindor/harry-potter-1 Pod not ready yet                                                                                     
    helper.go:32: Ready condition set to False, expected True                                                                                                                
    helper.go:32: Ready was not in conditions list                                                                                                                           
    helper.go:34: network-policy-conformance-hufflepuff/cedric-diggory-1 Pod not ready yet                                                                                   
    helper.go:39: Namespaces and Pods in network-policy-conformance-gryffindor, network-policy-conformance-slytherin, network-policy-conformance-hufflepuff, network-policy-conformance-ravenclaw namespaces ready
   apply.go:132: Deleting luna-lovegood StatefulSet
    apply.go:132: Deleting cedric-diggory StatefulSet
    apply.go:132: Deleting draco-malfoy StatefulSet
    apply.go:132: Deleting harry-potter StatefulSet
    apply.go:132: Deleting network-policy-conformance-ravenclaw Namespace
    apply.go:132: Deleting network-policy-conformance-hufflepuff Namespace
    apply.go:132: Deleting network-policy-conformance-slytherin Namespace
    apply.go:132: Deleting network-policy-conformance-gryffindor Namespace
--- PASS: TestConformance (5.19s)
PASS
ok      sigs.k8s.io/network-policy-api/conformance      5.217s
?       sigs.k8s.io/network-policy-api/conformance/tests        [no test files]
?       sigs.k8s.io/network-policy-api/conformance/utils/config [no test files]
?       sigs.k8s.io/network-policy-api/conformance/utils/flags  [no test files]
=== RUN   TestPrepareResources
=== RUN   TestPrepareResources/empty_namespace_labels
=== RUN   TestPrepareResources/simple_namespace_labels
=== RUN   TestPrepareResources/overwrite_namespace_labels
--- PASS: TestPrepareResources (0.00s)
    --- PASS: TestPrepareResources/empty_namespace_labels (0.00s)
    --- PASS: TestPrepareResources/simple_namespace_labels (0.00s)
    --- PASS: TestPrepareResources/overwrite_namespace_labels (0.00s)
PASS
ok      sigs.k8s.io/network-policy-api/conformance/utils/kubernetes     0.014s
?       sigs.k8s.io/network-policy-api/conformance/utils/suite  [no test files]

Closes #35

netlify[bot] commented 1 year ago

Deploy Preview for kubernetes-sigs-network-policy-api ready!

Name	Link
Latest commit	00bf1ffcd7b8bf9725bd05c99ddbd65321dbd963
Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-network-policy-api/deploys/646baea4e462070009f08de2
Deploy Preview	https://deploy-preview-98--kubernetes-sigs-network-policy-api.netlify.app
Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

astoycos commented 1 year ago

Tested locally things look good! Just some small NITs

astoycos commented 1 year ago

/lgtm /approve

k8s-ci-robot commented 1 year ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: astoycos, tssurya

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/kubernetes-sigs/network-policy-api/blob/master/OWNERS)~~ [astoycos] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

kubernetes-sigs / network-policy-api