TLS CA Signed certs for API and Router - Azure Domains and Vanity Domains

[julienstroheker]

In order to be able to enable TLS CA Signed certificates either using Azure or Vanity domains we'll need some changes into the API Data Model.

In this design, the OSA RP will deploy a Key Vault into the cluster RG named Cluster KV

On the plugin side, the master nodes should be able to pull the generated certificates from the cluster KV using the cloud provider. The cloud provider will have restraint access to this KV Get access policy.

In order to a "bring your own certificates" scenario some changes needs to be apply.

Here is a proposal open for discussion.

A new stuct named CertProfile will be used allowing the customer to refer certificate information for the API (into the Properties structure) and for the router (into the RouterProfile)

// Properties represents the cluster definition.
type Properties struct {
    // ProvisioningState (out): current state of the OSA resource.
    ProvisioningState ProvisioningState `json:"provisioningState,omitempty"`

    // OpenShiftVersion (in): OpenShift version to be created/updated, e.g.
    // `v3.11`.
    OpenShiftVersion string `json:"openShiftVersion,omitempty"`

    // PublicHostname (in,optional): Optional user-specified FQDN for OpenShift
    // API server.  If specified, after OSA cluster creation, user must create a
    // PublicHostname CNAME record forwarding to the returned FQDN value.
    PublicHostname string `json:"publicHostname,omitempty"`

    // FQDN (in): FQDN for OpenShift API server.  User-specified FQDN for
    // OpenShift API server loadbalancer internal hostname.
    FQDN string `json:"fqdn,omitempty"`

    // NetworkProfile (in): Configuration for OpenShift networking.
    NetworkProfile NetworkProfile `json:"networkProfile,omitempty"`

    // RouterProfiles (in,optional/out): Configuration for OpenShift router(s).
    RouterProfiles []RouterProfile `json:"routerProfiles,omitempty"`

    // AgentPoolProfiles (in): configuration of OpenShift cluster VMs.
    AgentPoolProfiles []AgentPoolProfile `json:"agentPoolProfiles,omitempty"`

    // AuthProfile (in): configures OpenShift authentication
    AuthProfile AuthProfile `json:"authProfile,omitempty"`

    ServicePrincipalProfile []ServicePrincipalProfile `json:"servicePrincipalProfile,omitempty"`

    AzProfile AzProfile `json:"azProfile,omitempty"`

    // APICertProfile (in, optional): configures OpenShift API certificate
    APICertProfile CertProfile `json:"apiCertProfile,omitempty"`
}

// RouterProfile represents an OpenShift router.
type RouterProfile struct {
    Name string `json:"name,omitempty"`

    // PublicSubdomain (in,optional/out): DNS subdomain for OpenShift router. If
    // specified, after OSA cluster creation, user must create a (wildcard)
    // *.PublicSubdomain CNAME record forwarding to the returned FQDN value.  If
    // not specified, OSA will auto-allocate and setup a PublicSubdomain and
    // return it.  The OpenShift master is configured with the PublicSubdomain
    // of the "default" RouterProfile.
    PublicSubdomain string `json:"publicSubdomain,omitempty"`

    // FQDN (out): Auto-allocated FQDN for the OpenShift router.
    FQDN string `json:"fqdn,omitempty"`

    // CertsProfile (in, optional): configures OpenShift Router certificate
    RouterCertProfile CertProfile `json:"routerCertProfile,omitempty"`
}

// CertsProfile contains configuration for OpenShift certificates.
type CertProfile struct {
    // KeyVaultCertificateID (in, optional): the customer certificate ID
    KeyVaultCertificateID string `json:"keyVaultCertificateID,omitempty"`
    // KeyVaultSecretID (in, optional): the customer secret ID
    KeyVaultSecretID string `json:"keyVaultSecretID,omitempty"`
    // KeyVaultKeyID (in, optional): the customer key ID
    KeyVaultKeyID string `json:"keyVaultKeyID,omitempty"`
}

SPN

2 SPN will be passed as array to ServicePrincipalProfile : https://github.com/openshift/openshift-azure/blob/0fe6e2859fcc56e27860e5dff58b0281a4530956/pkg/api/types.go#L61

A new field has to be added into the ServicePrincipalProfile structure to identify the SPN type (Master, Worker) : https://github.com/openshift/openshift-azure/blob/0fe6e2859fcc56e27860e5dff58b0281a4530956/pkg/api/types.go#L234

[julienstroheker]

cc @jim-minter @amanohar

[jim-minter]

Some quick comments:

• I guess this is internal data model API only, so most (all?) of the pointers should go away
• Do we need/want CertificateExpirationDate in this part of the model? Just looking for confirmation.
• Need to cover the fact that we will need 2 service principals now - one for the masters, one for the workers
• Please can you describe how KeyVaultID / KeyVaultSecretID map to the arguments to the API functions we'd end up calling (e.g. Get Key, Get Certificate). Can they map more closely to those arguments? Will you pass in a key name and certificate name? Will you be passing in versions for each of those as well? e.g. from https://docs.microsoft.com/en-us/rest/api/keyvault/getkey/getkey:

vaultBaseUrl | path | True | string | The vault name, for example https://myvault.vault.azure.net. key-name | path | True | string | The name of the key to get. key-version | path | True | string | Adding the version parameter retrieves a specific version of a key.

and from https://docs.microsoft.com/en-us/rest/api/keyvault/getcertificate/getcertificate:

vaultBaseUrl | path | True | string | The vault name, for example https://myvault.vault.azure.net. certificate-name | path | True | string | The name of the certificate in the given vault. certificate-version | path | True | string | The version of the certificate.

[julienstroheker]

Thanks @jim-minter

I updated the description, still missing the last point, let me double check on my side first by running the whole scenario manually using the API.

Thanks

[jim-minter]

OK, so I think:

• it'd be useful if you could additionally to pass the keyvault ID in Properties. That way we have it available if/when we create more keys and certs.
• fwiw I prefer MasterServicePrincipalProfile ServicePrincipalProfile; WorkerServicePrincipalProfile ServicePrincipalProfile to ServicePrincipalProfiles []ServicePrincipalProfile
• we can reuse CertProfile in the config struct when the time comes.

[jim-minter]

@julienstroheker what's the next step here - are you planning to draw up a final proposal bearing in mind the feedback?

[jim-minter]

@julienstroheker please can you give us an example KeyVaultCertificateID / KeyID / SecretID so we know what they look like?

[jim-minter]

@julienstroheker @amanohar please can you confirm: when you place the certificate into the key vault, will you be placing both the certificate and the key in the certificate object?

@asalkeld fyi

[asalkeld]

also is it going to be a secret, key or certificate ... i.e. pick just one please.

[julienstroheker]

@jim-minter @asalkeld sorry for the delay.

Here a last version about the DM based on the comments :

// Properties represents the cluster definition.
type Properties struct {
    // ProvisioningState (out): current state of the OSA resource.
    ProvisioningState ProvisioningState `json:"provisioningState,omitempty"`

    // OpenShiftVersion (in): OpenShift version to be created/updated, e.g.
    // `v3.11`.
    OpenShiftVersion string `json:"openShiftVersion,omitempty"`

    // PublicHostname (in,optional): Optional user-specified FQDN for OpenShift
    // API server.  If specified, after OSA cluster creation, user must create a
    // PublicHostname CNAME record forwarding to the returned FQDN value.
    PublicHostname string `json:"publicHostname,omitempty"`

    // FQDN (in): FQDN for OpenShift API server.  User-specified FQDN for
    // OpenShift API server loadbalancer internal hostname.
    FQDN string `json:"fqdn,omitempty"`

    // NetworkProfile (in): Configuration for OpenShift networking.
    NetworkProfile NetworkProfile `json:"networkProfile,omitempty"`

    // RouterProfiles (in,optional/out): Configuration for OpenShift router(s).
    RouterProfiles []RouterProfile `json:"routerProfiles,omitempty"`

    // AgentPoolProfiles (in): configuration of OpenShift cluster VMs.
    AgentPoolProfiles []AgentPoolProfile `json:"agentPoolProfiles,omitempty"`

    // AuthProfile (in): configures OpenShift authentication
    AuthProfile AuthProfile `json:"authProfile,omitempty"`

    MasterServicePrincipalProfile ServicePrincipalProfile `json:"masterServicePrincipalProfile,omitempty"`
    WorkerServicePrincipalProfile ServicePrincipalProfile `json:"workerServicePrincipalProfile,omitempty"`

    AzProfile AzProfile `json:"azProfile,omitempty"`

    // APICertProfile (in, optional): configures OpenShift API certificate
    APICertProfile CertProfile `json:"apiCertProfile,omitempty"`
}

// RouterProfile represents an OpenShift router.
type RouterProfile struct {
    Name string `json:"name,omitempty"`

    // PublicSubdomain (in,optional/out): DNS subdomain for OpenShift router. If
    // specified, after OSA cluster creation, user must create a (wildcard)
    // *.PublicSubdomain CNAME record forwarding to the returned FQDN value.  If
    // not specified, OSA will auto-allocate and setup a PublicSubdomain and
    // return it.  The OpenShift master is configured with the PublicSubdomain
    // of the "default" RouterProfile.
    PublicSubdomain string `json:"publicSubdomain,omitempty"`

    // FQDN (out): Auto-allocated FQDN for the OpenShift router.
    FQDN string `json:"fqdn,omitempty"`

    // CertsProfile (in, optional): configures OpenShift Router certificate
    RouterCertProfile CertProfile `json:"routerCertProfile,omitempty"`
}

// CertsProfile contains configuration for OpenShift certificates.
type CertProfile struct {
    // KeyVaultSecretURL (in, optional): the customer secret ID
    KeyVaultSecretURL string `json:"keyVaultSecretURL,omitempty"`
}

KeyVaultSecretID will be the URI from where you'll be able to pull a secretBundle from the KV : Ex : https://osa-julien-certs.vault.azure.net/secrets/testjulienazmosaio

The secretBundle : https://docs.microsoft.com/en-ca/rest/api/keyvault/getsecret/getsecret#secretbundle will allows you to parse the Cert and the Private Key.

Since the KeyVaultSecretID will contain the KV name into the URI, do you still wants a seperate value with it ? Eg . https://<KvName>.vault.azure.net/secrets/<CertName>

Secret version will not be stored. Plugin will pull the last version per default.

Let me know if i am missing something.

Thanks

[asalkeld]

So shouldn't this be under Config.Certificates instead of being moved to [API/Router]Certprofile each in different places? Seems a bit weird to me... While I am at it, you should change KeyVaultSecretID to KeyVaultSecretURL

[amanohar]

@asalkeld the API proposal is anticipating future feature of bring your own certs and ensuring that this proposal works with both current design (service generated certs and BYOC).

The post above is proposing that internal DM mirrors the public DM in terms of cert organization. Otherwise in Config.Certificates we create some sort of array to map each router's cert with its name and then validate that everything is setup correctly.

However, I am open to putting everything under Config.Certificates if that works better. Do you have a structure in mind?

cc: @jim-minter @julienstroheker

[julienstroheker]

@asalkeld Agreed KeyVaultSecretURL makes more sense, I put KeyVaultSecretID because this is the Azure terminology for KV for a secret.

I changed it for KeyVaultSecretURL

[asalkeld]

thanks @julienstroheker

@amanohar @jim-minter reminded me that Config is for items that we pass in, and properties is for external items. So this seems totally reasonable, sorry for the confusion. I'll get cracking on updating my PR with what we have here.

[jim-minter]

Changes merged at https://github.com/openshift/openshift-azure/pull/1192/files#diff-9ff06ad723720f9428a65da3710cf436