vmware-tanzu / secrets-manager

VMware Secrets Manager is a lightweight secrets manager to protect your sensitive data. It’s perfect for edge deployments where energy and footprint requirements are strict—See more: https://vsecm.com/
https://vsecm.com/
BSD 2-Clause "Simplified" License
153 stars 25 forks source link

`make test` shall print a warning if the project does not meet the coverage target #398

Open v0lkan opened 8 months ago

v0lkan commented 8 months ago

When running make test-local if we don’t reach a coverage target, a warning shall be displayed.

muratmirgun commented 7 months ago

@v0lkan maybe we can close this issue 💯

v0lkan commented 7 months ago

Closed :)

v0lkan commented 7 months ago

Actually I’m taking a note to verify it on the build server; if it fails, I’ll repoen this one.

v0lkan commented 7 months ago

Re-opening the issue. There are two things that I’ve seen:

  1. The report output is confusing
➜  secrets-manager git:(ovolkan/fix) make cover
Running tests with coverage...
Checking test coverage...
Test coverage: 24.9
Test Threshold: 70
Test coverage is less than 70
Test coverage is greater than 70

one of these should not have been here:

Test coverage is less than 70
Test coverage is greater than 70
  1. Although the coverage is below target, I didn’t see a warning when executing make test-local.
muratmirgun commented 7 months ago

ow I can check this again. Thanks for return

v0lkan commented 7 months ago

Sure thing. I’m sure the fix is a few liners.

v0lkan commented 7 months ago

test output for reference:

I think if we add the cover script immediately after the part that runs unit tests, we’ll get the warning we want there too.

This script assumes that you have a local minikube cluster running,
and you have already installed SPIRE and VMware Secrets Manager.
Also, make sure you have executed 'eval $(minikube docker-env)'
before running this script.

Running Go unit tests...
?     github.com/vmware-tanzu/secrets-manager/app/init-container/cmd  [no test files]
ok    github.com/vmware-tanzu/secrets-manager/app/keygen/cmd  (cached)  coverage: 0.0% of statements [no tests to run]
?     github.com/vmware-tanzu/secrets-manager/app/safe/cmd  [no test files]
ok    github.com/vmware-tanzu/secrets-manager/app/safe/internal/bootstrap (cached)  coverage: 0.0% of statements [no tests to run]
ok    github.com/vmware-tanzu/secrets-manager/app/safe/internal/server  (cached)  coverage: 0.0% of statements [no tests to run]
ok    github.com/vmware-tanzu/secrets-manager/app/safe/internal/server/handle (cached)  coverage: 0.0% of statements [no tests to run]
ok    github.com/vmware-tanzu/secrets-manager/app/safe/internal/server/route  (cached)  coverage: 0.0% of statements [no tests to run]
?     github.com/vmware-tanzu/secrets-manager/app/safe/internal/state [no test files]
?     github.com/vmware-tanzu/secrets-manager/app/sentinel/busywait [no test files]
ok    github.com/vmware-tanzu/secrets-manager/app/sentinel/cmd  (cached)  coverage: 0.0% of statements [no tests to run]
?     github.com/vmware-tanzu/secrets-manager/app/sidecar/cmd [no test files]
ok    github.com/vmware-tanzu/secrets-manager/app/sentinel/internal/safe  (cached)  coverage: 0.0% of statements [no tests to run]
?     github.com/vmware-tanzu/secrets-manager/ci/poll [no test files]
ok    github.com/vmware-tanzu/secrets-manager/core/audit  (cached)  coverage: 100.0% of statements
ok    github.com/vmware-tanzu/secrets-manager/core/crypto (cached)  coverage: 64.1% of statements
ok    github.com/vmware-tanzu/secrets-manager/core/entity/data/v1 (cached)  coverage: 87.8% of statements
ok    github.com/vmware-tanzu/secrets-manager/core/entity/reqres/safe/v1  (cached)  coverage: [no statements] [no tests to run]
ok    github.com/vmware-tanzu/secrets-manager/core/env  (cached)  coverage: 87.5% of statements
ok    github.com/vmware-tanzu/secrets-manager/core/log  (cached)  coverage: 13.8% of statements
ok    github.com/vmware-tanzu/secrets-manager/core/probe  (cached)  coverage: 12.5% of statements
ok    github.com/vmware-tanzu/secrets-manager/core/system (cached)  coverage: 0.0% of statements [no tests to run]
ok    github.com/vmware-tanzu/secrets-manager/core/template (cached)  coverage: 90.0% of statements
?     github.com/vmware-tanzu/secrets-manager/examples/multiple-secrets [no test files]
?     github.com/vmware-tanzu/secrets-manager/examples/multiple-secrets/busywait  [no test files]
?     github.com/vmware-tanzu/secrets-manager/examples/using-init-container [no test files]
ok    github.com/vmware-tanzu/secrets-manager/core/validation (cached)  coverage: 100.0% of statements
?     github.com/vmware-tanzu/secrets-manager/examples/using-sdk  [no test files]
?     github.com/vmware-tanzu/secrets-manager/examples/using-sdk/helper/env [no test files]
?     github.com/vmware-tanzu/secrets-manager/examples/using-sidecar  [no test files]
?     github.com/vmware-tanzu/secrets-manager/examples/using-sidecar/helper/env [no test files]
ok    github.com/vmware-tanzu/secrets-manager/sdk/internal/timer  (cached)  coverage: 0.0% of statements [no tests to run]
ok    github.com/vmware-tanzu/secrets-manager/sdk/sentry  (cached)  coverage: 0.0% of statements [no tests to run]
ok    github.com/vmware-tanzu/secrets-manager/sdk/startup (cached)  coverage: 0.0% of statements [no tests to run]
Testing: Encrypting secrets…

PASS \o/

Deploying workload that uses the SDK…
make[1]: Entering directory '/home/aegis/WORKSPACE/VSecM'
serviceaccount/example unchanged
deployment.apps/example created
clusterspiffeid.spire.spiffe.io/example unchanged
secret/vsecm-secret-example unchanged
make[1]: Entering directory '/home/aegis/WORKSPACE/VSecM'
serviceaccount/example unchanged
deployment.apps/example created
clusterspiffeid.spire.spiffe.io/example unchanged
secret/vsecm-secret-example unchanged
make[1]: Leaving directory '/home/aegis/WORKSPACE/VSecM'
Waiting for example workload…
Deployed workload that uses the SDK.
set_encrypted_secret()
value: '!VSecMRocks!'
sentinel: 'vsecm-sentinel-d6f45b5f8-8qtjx'
res: '
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA5NGNvV2g5bTlNQXp4OFdsbjFhNTNuRDhXUldJVXlnUlJTOXJiUUdRMW00CnVEMXJhT3F6RXFvSzU1eWhlM2RlSkQ4aGF0UVMyRzFOVmpoL2RNQWxmemcKLS0tIDF6cVcyTENndVorQkQrUEdqMWRkamFyRVhPZVR3ai9oQ1RCN2c4V3JldGcKwz1y0dm0ZfA8T1UMao2mczGISyu9Jh2VHw5DGSP2zAhOd/JwcLHTDgwG6Zk='                                                                                                                                                                                                                                                                                                                                                            OK                                                                                                                                                                                                                                                                                                                                                                                                                                    done: set_encrypted_secret()                                                                                                                                                                                       assert_workload_secret_value()                                                                                                                                                                                     workload: 'example-65b4cfd5c5-2t4ql'                                                                                                                                                                               value: '!VSecMRocks!'                                                                                                                                                                                                                                                                                                                                                                                                                 PASS \o/                                                                                                                                                                                                                                                                                                                                                                                                                              Cleanup…                                                                                                                                                                                                                                                                                                                                                                                                                              OK                                                                                                                                                                                                                                                                                                                                                                                                                                    deployment.apps "example" deleted                                                                                                                                                                                  Waiting for example workload deletion…                                                                                                                                                                             Tested: Encrypting secrets.                                                                                                                                                                                        Cleanup…                                                                                                                                                                                                                                                                                                                                                                                                                              OK                                                                                                                                                                                                                                                                                                                                                                                                                                    Deployment does not exist, skipping delete step.                                                                                                                                                                   Waiting for example workload deletion…                                                                                                                                                                                                                                                                                                                                                                                                ________________________________________                                                                                                                                                                           Case: Workload using VSecM SDK…                                                                                                                                                                                                                                                                                                                                                                                                       Deploying workload that uses the SDK…                                                                                                                                                                              make[1]: Entering directory '/home/aegis/WORKSPACE/VSecM'                                                                                                                                                          serviceaccount/example unchanged                                                                                                                                                                                   deployment.apps/example created                                                                                                                                                                                    clusterspiffeid.spire.spiffe.io/example unchanged                                                                                                                                                                  secret/vsecm-secret-example unchanged                                                                                                                                                                              make[1]: Leaving directory '/home/aegis/WORKSPACE/VSecM'                                                                                                                                                           Waiting for example workload…                                                                                                                                                                                      Deployed workload that uses the SDK.                                                                                                                                                                               Testing: Secret registration…                                                                                                                                                                                      set_secret()
                                                                                                                                                                                                                   OK                                                                                                                                                                                                                                                                                                                                                                                                                                    done: set_secret()                                                                                                                                                                                                 assert_workload_secret_value()                                                                                                                                                                                     workload: 'example-65b4cfd5c5-9tflj'                                                                                                                                                                               value: '!VSecMRocks!'                                                                                                                                                                                                                                                                                                                                                                                                                 PASS \o/                                                                                                                                                                                                                                                                                                                                                                                                                              Tested: Secret registration.                                                                                                                                                                                       Testing: Secret deletion…                                                                                                                                                                                          Deleting secret…                                                                                                                                                                                                                                                                                                                                                                                                                      OK                                                                                                                                                                                                                                                                                                                                                                                                                                    Deleted secret.                                                                                                                                                                                                    assert_workload_secret_no_value()                                                                                                                                                                                  workload: 'example-65b4cfd5c5-9tflj'                                                                                                                                                                               res: 'NO_SECRET'                                                                                                                                                                                                                                                                                                                                                                                                                      PASS \o/                                                                                                                                                                                                                                                                                                                                                                                                                              Tested: Secret deletion.                                                                                                                                                                                           Testing: Secret registration (append mode)…                                                                                                                                                                        append_secret()                                                                                                                                                                                                                                                                                                                                                                                                                       OK                                                                                                                                                                                                                                                                                                                                                                                                                                    done: append_secret()                                                                                                                                                                                              append_secret()                                                                                                                                                                                                                                                                                                                                                                                                                       OK                                                                                                                                                                                                                                                                                                                                                                                                                                    done: append_secret()                                                                                                                                                                                              assert_workload_secret_value()                                                                                                                                                                                     workload: 'example-65b4cfd5c5-9tflj'                                                                                                                                                                               value: '["Rocks!","!VSecM"]'                                                                                                                                                                                                                                                                                                                                                                                                          PASS \o/                                                                                                                                                                                                                                                                                                                                                                                                                              Deleting secret…                                                                                                                                                                                                                                                                                                                                                                                                                      OK                                                                                                                                                                                                                                                                                                                                                                                                                                    Deleted secret.                                                                                                                                                                                                    Tested: Secret registration (append mode).                                                                                                                                                                         Testing: Secret registration (JSON transformation)…                                                                                                                                                                set_json_secret()
OK                                                                                                                                                                                                                                                                                                                                                                                                                                    done: set_json_secret()                                                                                                                                                                                            assert_workload_secret_value()                                                                                                                                                                                     workload: 'example-65b4cfd5c5-9tflj'                                                                                                                                                                               value: '{"USERNAME":"*root*", "PASSWORD":"*CasHC0w*"}'                                                                                                                                                                                                                                                                                                                                                                                PASS \o/                                                                                                                                                                                                                                                                                                                                                                                                                              Deleting secret…                                                                                                                                                                                                                                                                                                                                                                                                                      OK                                                                                                                                                                                                                                                                                                                                                                                                                                    Deleted secret.                                                                                                                                                                                                    Tested: Secret registration (JSON transformation).                                                                                                                                                                 Testing: Secret registration (YAML transformation)…                                                                                                                                                                set_yaml_secret()                                                                                                                                                                                                                                                                                                                                                                                                                     OK                                                                                                                                                                                                                                                                                                                                                                                                                                    done: set_yaml_secret()                                                                                                                                                                                            assert_workload_secret_value()                                                                                                                                                                                     workload: 'example-65b4cfd5c5-9tflj'                                                                                                                                                                               value: 'PASSWORD: '*CasHC0w*'                                                                                                                                                                                      USERNAME: '*root*''                                                                                                                                                                                                                                                                                                                                                                                                                   PASS \o/                                                                                                                                                                                                                                                                                                                                                                                                                              Deleting secret…                                                                                                                                                                                                                                                                                                                                                                                                                      OK                                                                                                                                                                                                                                                                                                                                                                                                                                    Deleted secret.                                                                                                                                                                                                    Tested: Secret registration (YAML transformation).                                                                                                                                                                 Cleanup…                                                                                                                                                                                                                                                                                                                                                                                                                              OK                                                                                                                                                                                                                                                                                                                                                                                                                                    deployment.apps "example" deleted                                                                                                                                                                                  Waiting for example workload deletion…                                                                                                                                                                                                                                                                                                                                                                                                ________________________________________                                                                                                                                                                           Case: Workload using VSecM Sidecar…                                                                                                                                                                                                                                                                                                                                                                                                   Deploying workload that uses the sidecar…                                                                                                                                                                          make[1]: Entering directory '/home/aegis/WORKSPACE/VSecM'                                                                                                                                                          serviceaccount/example unchanged                                                                                                                                                                                   deployment.apps/example created                                                                                                                                                                                    clusterspiffeid.spire.spiffe.io/example unchanged                                                                                                                                                                  secret/vsecm-secret-example unchanged                                                                                                                                                                              make[1]: Leaving directory '/home/aegis/WORKSPACE/VSecM'
Waiting for example workload…                                                                                                                                                                                      Deployed workload that uses the sidecar.                                                                                                                                                                           Testing: Secret registration…                                                                                                                                                                                      set_secret()                                                                                                                                                                                                                                                                                                                                                                                                                          OK                                                                                                                                                                                                                                                                                                                                                                                                                                    done: set_secret()                                                                                                                                                                                                 Waiting for 15 seconds to let the sidecar poll the secret…                                                                                                                                                         assert_workload_secret_value()                                                                                                                                                                                     workload: 'example-6598d68d8f-g975k'                                                                                                                                                                               value: '!VSecMRocks!'                                                                                                                                                                                                                                                                                                                                                                                                                 PASS \o/                                                                                                                                                                                                                                                                                                                                                                                                                              Tested: Secret registration.                                                                                                                                                                                       Testing: Secret deletion (sidecar)…                                                                                                                                                                                Deleting secret…                                                                                                                                                                                                                                                                                                                                                                                                                      OK                                                                                                                                                                                                                                                                                                                                                                                                                                    Deleted secret.                                                                                                                                                                                                    Waiting for 15 seconds to let the sidecar poll the secret…                                                                                                                                                         assert_workload_secret_no_value()                                                                                                                                                                                  workload: 'example-6598d68d8f-g975k'                                                                                                                                                                               res: ''                                                                                                                                                                                                                                                                                                                                                                                                                               PASS \o/                                                                                                                                                                                                                                                                                                                                                                                                                              Tested: Secret deletion (sidecar).                                                                                                                                                                                 Testing Secret registration (append mode)…                                                                                                                                                                         append_secret()                                                                                                                                                                                                                                                                                                                                                                                                                       OK                                                                                                                                                                                                                                                                                                                                                                                                                                    done: append_secret()                                                                                                                                                                                              append_secret()                                                                                                                                                                                                                                                                                                                                                                                                                       OK                                                                                                                                                                                                                                                                                                                                                                                                                                    done: append_secret()                                                                                                                                                                                              Waiting for 15 seconds to let the sidecar poll the secret…                                                                                                                                                         assert_workload_secret_value()                                                                                                                                                                                     workload: 'example-6598d68d8f-g975k'                                                                                                                                                                               value: '["Rocks!","!VSecM"]'                                                                                                                                                                                                                                                                                                                                                                                                          PASS \o/                                                                                                                                                                                                                                                                                                                                                                                                                              Deleting secret…                                                                                                                                                                                                                                                                                                                                                                                                                      OK                                                                                                                                                                                                                                                                                                                                                                                                                                    Deleted secret.
Tested: Secret registration (append mode).                                                                                                                                                                         Testing Secret registration (JSON transformation)…                                                                                                                                                                 set_json_secret()                                                                                                                                                                                                                                                                                                                                                                                                                     OK                                                                                                                                                                                                                                                                                                                                                                                                                                    done: set_json_secret()                                                                                                                                                                                            Waiting for 15 seconds to let the sidecar poll the secret…                                                                                                                                                         assert_workload_secret_value()                                                                                                                                                                                     workload: 'example-6598d68d8f-g975k'                                                                                                                                                                               value: '{"USERNAME":"*root*", "PASSWORD":"*CasHC0w*"}'                                                                                                                                                                                                                                                                                                                                                                                PASS \o/                                                                                                                                                                                                                                                                                                                                                                                                                              Deleting secret…                                                                                                                                                                                                                                                                                                                                                                                                                      OK                                                                                                                                                                                                                                                                                                                                                                                                                                    Deleted secret.                                                                                                                                                                                                    Tested: Secret registration (JSON transformation).                                                                                                                                                                 Testing Secret registration (YAML transformation)…                                                                                                                                                                 set_yaml_secret()                                                                                                                                                                                                                                                                                                                                                                                                                     OK                                                                                                                                                                                                                                                                                                                                                                                                                                    done: set_yaml_secret()                                                                                                                                                                                            Waiting for 15 seconds to let the sidecar poll the secret…                                                                                                                                                         assert_workload_secret_value()                                                                                                                                                                                     workload: 'example-6598d68d8f-g975k'                                                                                                                                                                               value: 'PASSWORD: '*CasHC0w*'                                                                                                                                                                                      USERNAME: '*root*''                                                                                                                                                                                                                                                                                                                                                                                                                   PASS \o/                                                                                                                                                                                                                                                                                                                                                                                                                              Deleting secret…                                                                                                                                                                                                                                                                                                                                                                                                                      OK                                                                                                                                                                                                                                                                                                                                                                                                                                    Deleted secret.                                                                                                                                                                                                    Tested: Secret registration (YAML transformation).                                                                                                                                                                 Cleanup…                                                                                                                                                                                                                                                                                                                                                                                                                              OK                                                                                                                                                                                                                                                                                                                                                                                                                                    deployment.apps "example" deleted                                                                                                                                                                                  Waiting for example workload deletion…                                                                                                                                                                                                                                                                                                                                                                                                ________________________________________                                                                                                                                                                           Case: Workload using VSecM Init Container…                                                                                                                                                                                                                                                                                                                                                                                            All done. Cleaning up…                                                                                                                                                                                             Cleanup…