Update Stack Architecture documentation (not diagram)

[nimojg-microsoft]

It's been a while since we've updated this. Right now the written description of the stack architecture does not include the work around Application Gateway, Key Vault, Tika VMs, DDOS Protection, etc. Can we update it?

[SorraTheOrc]

All these features are documented in the parameters file (this is autogenerated so should always represent the complete feature set).

It would be good to document properly, I agree.

[nimojg-microsoft]

Would be good to have someone review what I added. I'm a bit unclear on how many Tika VMs we provide for example, and all the ways in which we use App Gateway, Key Vault, etc. today in the template. Also, it'd be good to have some more notes in the section where we talk about "Observations about the current template" - unless we plan to have more documentation on these features elsewhere.

[taylora3]

Is there any further documentation around the use of App Gateway and Key Vault etc.? I am trying to do a deployment using App gateway and I'm getting a few error

1. When using the key vault and VMSS https termination I get the following..

is not a valid versioned Key Vault Secret URL. It should be in the format https://<vaultEndpoint>/secrets/<secretName>/<secretVersion>

.. I think that this is a case of the parameter for SSL Cert KeyVault URL needing to be the Secret URL rather than the certificate one.

2. When using the app gateway or VMSS https termination I get the "VM has reported a failure when processing extension 'install_moodle'" on the controller vm.

Can you tell me the format of the parameters required to get the App gateway working?

Is this an issue with the setup of the cert in the keyvault?

Thanks :)

[hosungsmsft]

Hello @taylora3,

1. The KeyVault-related documentation for our templates is available at docs/SslCert.md. You should follow the instructions there precisely to obtain the 5 required template parameters. Yes, unfortunately, ARM requires a KeyVault "secret", not a "certificate" in order for the cert to be deployed on VMs at the ARM deployment time. That's why the scrtipt etc/keyvault.sh has to convert the PEM files into PFX files and upload them to the KeyVault as "secrets", not as "certificates." This is ARM's restriction that we have to follow. From reading your problem description, I suppose you uploaded your SSL cert to your Azure KeyVault as a "certificate", and unfortunately, that won't work with our templates. If you want us to take more look at this, please provide the 5 template parameter values (keyVaultResourceId, sslCertKeyVaultURL, sslCertThumbprint, caCertKeyVaultURL, caCertThumbprint) that you entered when you deploy the templates with your own SSL cert stored on your KeyVault.

2. That generic failure message alone is not enough for us to help on this failure. Could you please click the "Error details" link and provide the failure messages? Also is this failure new? There was some recent code change on the install_moodle.sh script that's tested only on the generic CI scenario, not tested on other cases, so it might be affecting your deployment. Apologies if that's the case, and please help us fix the issue. With the usual VMSS https termination option, it shouldn't fail, but you said it fails on either app gateway or VMSS https termination, so I think we need to have failure messages for both failures.

Apologies for the lack of the AppGw documentation in our templates. We'll add more documentation on this soon, but currently the only available one is at the corresponding template source code. Look at the "description" values in here. The same convention (your KeyVault's Azure resource ID and the KeyVault URL for your cert uploaded as a "secret"--in the pfx format--onto your KeyVault) applies here. So if you have a PEM-format cert, you'll need to convert it to a PFX cert (with no password) and upload it to your KeyVault as a "secret", NOT as a "certificate". This is quite inconvenient, I think, for most people from non-Windows world, but that's how the current Azure KeyVault and Resource Manager secret handling is done.

For more information on how App Gateway can accept an SSL cert, please refer to this. As you can see, App Gateway API accepts only a base64-encoded cert content as its parameter. Since we don't want customers to pass their SSL certs this way, we had to rely on the general ARM secret handling as described here, and this is where the PFX format and "secret"-not-"certificate" quirkiness get in.

Hope this helps and feel free to reach out to me by email if you have some things that you don't feel comfortable sharing over this public forum. Thanks.

[taylora3]

@hosungsmsft thanks for the reply. I was given a .crt .cer .pfx .key files by my ssl certificate provider.

Do I still need to regenerate the .pfx file from the .crt file?

Is the below a legitimate version of running the script: keyvault.sh 'moodlekeyvault RGMoodle NorthEurope certsecretname certpem.cer nginx.key nginx.crt'

I think the "secret", NOT as a "certificate" part of this is tripping me up. Is there a way of adding the pfx file as a secret to Azure without using the CLI or the script. I was uploading the PFX as a certificate and getting all the values from the UI.

[hosungsmsft]

@taylora3 -- Just wanted to make sure that you are trying this for https termination at your VMSS resource, NOT for an App Gateway https termination (because for App Gateway, the keyvault.sh can't be used unfortunately---the format's a little different. The keyvault.sh script is only for the VMSS-based https termination. If you are trying to use an App Gateway for your https termination, please let us know again).

Assuming that you are trying to use your valid SSL cert (that you purchased from your SSL cert provider), hopefully the keyvault.sh is a good enough tool, but that script has some specific assumptions/requirements. I assume your certpem.cer is the certificate of your CA, but the keyvault.sh requires it to be in PEM format. I believe *.cer files are usually in the binary DER format, so that needs to be converted. This can be done by following the "Convert a DER file (.cer .der) to PEM" command in this link. Another important requirement is that your CA cert must be signed by a trusted root CA, not by another CA. If your CA cert is signed by a non-root CA, then unfortunately our current templates can't import that cert chain into the deployed VMs, so it won't work in that case.

Please note that a *.cer file might not be in the binary DER format, but in the PEM (base64-encoded) format. It feels like that might be the case here, because the file name (certpem.cer) seems to suggest that. In that case, the above conversion is not necessary. Various providers/vendors mix up the file extensions and formats, so the only way to tell is to actually inspect the file content. You might want to read carefully some related articles like this, to see how messy the SSL cert standards/conventions are.

Assuming your nginx.crt and nginx.key are in PEM format, signed by the CA whose cert is certpem.cer, yes, the command you entered is correct, except you shouldn't put the quotations. Just run the following command on a Linux shell (or git-bash should work on Windows):

$ keyvault.sh moodlekeyvault RGMoodle NorthEurope certsecretname certpem.cer nginx.key nginx.crt

Of course you should run az login first and set the subscription accordingly.

Yes I find it confusing to have to upload a cert as a secret after encoding the .pfx file into base64 format, and compose a specific JSON-formatted content as in the keyvault.sh (here), but it's just the way Azure's VM REST API is defined (here). I suppose this is a legacy from the early KeyVault (when they didn't have a separate "Certificate" type), but I can't be sure. I'm afraid this is unnecessarily inconvenient and complex than necessary for most customers, and hopefully we can make it less so with tools and support.

Please do let us know again if you have further questions. Best luck on your SSL setup. Thanks.

[taylora3]

@hosungsmsft thankyou for the very clear response. I am specifically trying to get it to work for an App Gateway. Could you point me in the direction of what I would need to do to get the App gateway working?

Thanks again :)

[hosungsmsft]

@taylora3 -- For AppGw, please note that I haven't tested the AppGw option that much. The only test case I could conduct was to create, using opensll, a self-signed SSL cert and private key (not purchased, thus not signed by a valid CA), which are in PEM format, convert it to PFX (Azure requirements, done using openssl, and this PFX should NOT be password-protected currently by our templates), upload it to my Azure KeyVault and use it with our templates for the AppGw test. So what I'm going to tell you may not work exactly as hoped, and in that case, please feel free to reach out to me by email and we may try some remote screen-shared online meeting session to resolve this.

Not sure if your cert provider gave you a PFX file which is password protected, but if it is, we need to first remove the password (because our appgw.json template currently doesn't accept a PFX password---see this). You can use openssl to conduct this step using examples like this. Note the file name samples used in the example (e.g., protected.p12.orig and unprotected.p12) might be different from yours (which might be something like pkcs12.pfx as you entered above).

Once you have a PFX file for your cert without a password, you need to upload the PFX file's content to your Azure KeyVault after encoding the binary file content in base64 format. This can be achieved using a command like the following (on a Linux shell with Azure CLI. Let me know if your environment is different):

$ PFX_CONTENT_ENCODED=$(cat <your_pfx_filename> | base64)
$ az keyvault secret set --vault-name <your_azure_keyvault_name> --name <your_secret_name_for_your_uploaded_pfx> --value $PFX_CONTENT_ENCODED

Then finally you can pass <your_secrete_name_for_your_uploaded_pfx to our template parameter appGwSslCertKeyVaultSecretName (together with appGwSslCertKeyVaultResourceId, which can be obtained by the command az keyvault show --name <your_azure_keyvatul_name> --query id -o tsv.

Hope this helps and feel free to reach out to me by email if you'd like to go over this together on a screen-shared online meeting. Of course feel free to leave more questions here as well.

Thanks, Hosung

[hosungsmsft]

Sorry, the command above has a bug. The base64 conversion is missing an option not to break into multiple lines. The correct command should be as follows:

$ PFX_CONTENT_ENCODED=$(cat <your_pfx_filename> | base64 -w 0)
$ az keyvault secret set --vault-name <your_azure_keyvault_name> --name <your_secret_name_for_your_uploaded_pfx> --value $PFX_CONTENT_ENCODED