Venafi Issuer tests are consistently failing with "common name test-common-name-bvxqgardto is not allowed in this policy"

[maelvls]

TL;DR: On Friday 15 Sept 2022, the policy folder \VED\Policy was changed, which affected the policy folder used during the cert-manager e2e tests (\VED\Policy\Jetstack). Before the change, there was no restriction on the common name or DNS names. After the change, the common name and DNS names became restricted to example.com or *.example.com. On 18 Sept 2022, I removed the example.com domain restriction in \VED\Policy which fixed the build failures.

Testgrid has been notifying us of the Venafi tests failing over and over:

The tests seem to be failing with this error:

common name test-common-name-bvxqgardto is not allowed in this policy: ^([\p{L}\p{N}-*]+\.)*example\.com$

This error comes from the vcert library, in the SimpleValidateCertificateRequest) function. Vcert fetches the policies attached to the policy folder \VED\Policy\Jetstack.

It might be due to a change in the policies of the folder \VED\Policy\Jetstack in TPP. I looked at the JSON file produced by vcert getpolicy to see:

TOKEN=$(vcert getcred -u $VENAFI_TPP_URL --username $VENAFI_TPP_USERNAME --password $VENAFI_TPP_PASSWORD --client-id=$VENAFI_TPP_CLIENT_ID --scope='certificate:manage;configuration:manage' --format json | jq -r .access_token)
vcert getpolicy -u $VENAFI_TPP_URL -t $TOKEN -z 'Jetstack'

which returns:

{
  "users": [
    "jetstack-platform",
    "jetstack_user"
  ],
  "policy": {
    "domains": [
      "example.com"
    ],
    "wildcardAllowed": true,
    "certificateAuthority": "\\VED\\Policy\\Administration\\CAs\\Microsoft CA Web Server 1 Year",
    "keyPair": {
      "reuseAllowed": true
    },
    "subjectAltNames": {
      "dnsAllowed": true,
      "ipAllowed": false,
      "emailAllowed": false,
      "uriAllowed": false,
      "upnAllowed": false
    }
  },
  "defaults": {
    "keyPair": {
      "keyType": "RSA",
      "rsaKeySize": 2048,
      "serviceGenerated": false
    },
    "autoInstalled": true
  }
}

The enforces common names and DNS names to end with example.com. But our tests submit CSRs with a common name that doesn't end with example.com; I looked at old tests, and the common names we have been using never ends with example.com. It always look something like this:

test-common-name-tkewxmuslq
test-common-name-ucgwdafybr

I also looked in the UI to see whether this policy is inherited from the root folder \VED\Policy or not. The answer is yes: the domain example.com is enforced from the root folder:

This (seemingly new) example.com policy seems to have been added to \VED\Policy on Friday 15 Sept 2022.

I learned that TPP relies on a "Log Server", and that log server (which is a feature of the SQL database as I understand it) allows us to audit everything that happens in TPP. So I RDP'd into the VM, and opened the application "Venafi Configuration Admin" as suggested here. I had to "enable" something first:

I think the "events" I am looking for are "updates to the Admin UI":

Unfortunately, the tab "All events" shows an error. Creating custom filters (e.g., filtering on "updates to Admin UI") shows the same error:

[maelvls]

On 18 Sept 2022, I removed the example.com domain restriction in \VED\Policy which fixed the build failures. For context, the UI looked like this:

[maelvls]

No more failures due to the Venafi issuer in Prow! I will close this issue now.