`az vm format-secret` fails with error `Parameter ‘resource_group_name’ can not be None.`

[iainfoulds]

Environment summary

Install Method: How did you install the CLI? (e.g. pip, interactive script, apt-get, Docker, MSI, nightly)

Docker: docker run -v ${HOME}:/root -it azuresdk/azure-cli-python:latest

CLI Version: What version of the CLI and modules are installed? (Use az --version)

azure-cli (2.0.2+dev)

acr (2.0.0+dev) acs (2.0.2+dev) appservice (0.1.2+dev) batch (2.0.0+dev) cloud (2.0.0+dev) component (2.0.0+dev) configure (2.0.2+dev) container (0.1.2+dev) core (2.0.2+dev) dla (0.0.1+dev) dls (0.0.1+dev) documentdb (0.1.2+dev) feedback (2.0.0+dev) find (0.0.1b1+dev) iot (0.1.2+dev) keyvault (2.0.0+dev) lab (0.0.1+dev) monitor (0.0.1+dev) network (2.0.2+dev) nspkg (2.0.0+dev) profile (2.0.2+dev) redis (0.1.1b3+dev) resource (2.0.2+dev) role (2.0.1+dev) sql (2.0.0+dev) storage (2.0.2+dev) taskhelp (0.1.1b4+dev) vm (2.0.2+dev)

Python (Linux) 3.5.2 (default, Dec 27 2016, 21:33:11) [GCC 5.3.0]

OS Version: What OS and version are you using?
Alpine 3.4.6

Shell Type: What shell are you using? (e.g. bash, cmd.exe, Bash on Windows)

GNU bash, version 4.3.42(1)-release (x86_64-alpine-linux-musl)

---

Description

I'm following through the steps add a secret from Key Vault when creating a VM. The steps as indicated in various parts of the CLI help and ref content should be:

az keyvault create --resource-group resourcegroupname --name vaultname --enabled-for-deployment

az keyvault certificate create --vault-name vaultname -n cert1 \
    -p "$(az keyvault certificate get-default-policy)"

secrets=$(az keyvault secret list-versions --vault-name vaultname \
    -n cert1 --query "[?attributes.enabled].id" -o tsv)

vm_secrets=$(az vm format-secret -s "$secrets")

az vm create -g group-name -n vm-name --admin-username deploy  \
    --image debian --secrets "$vm_secrets"

When it comes to az vm format-secret, I receive the error: Parameter ‘resource_group_name’ can not be None.. I can't see anything in the commands that have me define a resource group, other than when you first create the Key Vault.

$secrets is defined:

echo $secrets                                                                                                                                                                                                                
https://mykeyvault9780.vault.azure.net/secrets/myCert/5e7e56fa361a45b1bdb7bfb61c8b2d0b

Full debug output:

bash-4.3# az vm format-secret --secrets $secrets --debug
Command arguments [‘vm’, ‘format-secret’, ‘--secrets’, ’https://mykeyvault9780.vault.azure.net/secrets/myCert/5e7e56fa361a45b1bdb7bfb61c8b2d0b']
Current active cloud ‘AzureCloud’
{‘active_directory’: ‘https://login.microsoftonline.com’,
 ‘active_directory_graph_resource_id’: ‘https://graph.windows.net/’,
 ‘active_directory_resource_id’: ‘https://management.core.windows.net/’,
 ‘batch_resource_id’: None,
 ‘gallery’: ‘https://gallery.azure.com/’,
 ‘management’: ‘https://management.core.windows.net/’,
 ‘resource_manager’: ‘https://management.azure.com/’,
 ‘sql_management’: ‘https://management.core.windows.net:8443/’}
{‘azure_datalake_analytics_catalog_and_job_endpoint’: ‘azuredatalakeanalytics.net’,
 ‘azure_datalake_store_file_system_endpoint’: ‘azuredatalakestore.net’,
 ‘keyvault_dns’: ‘.vault.azure.net’,
 ‘sql_server_hostname’: ‘.database.windows.net’,
 ‘storage_endpoint’: ‘core.windows.net’}
Registered application event handler ‘CommandTableParams.Loaded’ at <function add_id_parameters at 0x7f5b4c539f28>
Registered application event handler ‘CommandTable.Loaded’ at <function add_id_parameters at 0x7f5b4c539f28>
Successfully loaded command table from module ‘vm’.
Application event ‘CommandTable.Loaded’ with event data {‘command_table’: {‘vm format-secret’: <azure.cli.core.commands.CliCommand object at 0x7f5b4c52ba20>}}
Application event ‘CommandParser.Loaded’ with event data {‘parser’: AzCliCommandParser(prog=‘az’, usage=None, description=None, formatter_class=<class ‘argparse.HelpFormatter’>, conflict_handler=‘error’, add_help=True)}
Application event ‘CommandTableParams.Loaded’ with event data {‘command_table’: {‘vm format-secret’: <azure.cli.core.commands.CliCommand object at 0x7f5b4c52ba20>}}
Application event ‘CommandParser.Parsed’ with event data {‘command’: ‘vm format-secret’, ‘args’: Namespace(_command_package=‘vm’, _jmespath_query=None, _log_verbosity_debug=False, _log_verbosity_verbose=False, _output_format=‘json’, _parser=AzCliCommandParser(prog=‘az vm format-secret’, usage=None, description=‘Format secrets to be used in `az vm create --secrets`’, formatter_class=<class ‘argparse.HelpFormatter’>, conflict_handler=‘error’, add_help=True), _validators=[], certificate_store=None, command=‘vm format-secret’, func=<function create_ [...]
Getting management service client client_type=KeyVaultManagementClient
msrest.pipeline : Adding ‘log_request’ callback before event: ‘request’
msrest.pipeline : Callback to overwrite original call: False
msrest.pipeline : Adding ‘log_response’ callback after event: ‘response’
msrest.pipeline : Callback to overwrite original call: False
Getting management service client client_type=KeyVaultManagementClient
msrest.pipeline : Adding ‘log_request’ callback before event: ‘request’
msrest.pipeline : Callback to overwrite original call: False
msrest.pipeline : Adding ‘log_response’ callback after event: ‘response’
msrest.pipeline : Callback to overwrite original call: False
adal-python : 41f7dd6a-7e13-458a-9461-32437bfbd54f - Authority:Performing instance discovery: https://login.microsoftonline.com/a4442dcc-ce0e-4940-a367-3029b469f0e5
adal-python : 41f7dd6a-7e13-458a-9461-32437bfbd54f - Authority:Performing static instance discovery
adal-python : 41f7dd6a-7e13-458a-9461-32437bfbd54f - Authority:Authority validated via static instance discovery
adal-python : 41f7dd6a-7e13-458a-9461-32437bfbd54f - TokenRequest:Getting token from cache with refresh if necessary.
adal-python : 41f7dd6a-7e13-458a-9461-32437bfbd54f - OAuth2Client:finding with query: {“userId”: “i_foulds@live.com”, “_clientId”: “04b07795-8ddb-461a-bbee-02f9e1bf7b46"}
adal-python : 41f7dd6a-7e13-458a-9461-32437bfbd54f - OAuth2Client:Looking for potential cache entries:
adal-python : 41f7dd6a-7e13-458a-9461-32437bfbd54f - OAuth2Client:{“userId”: “i_foulds@live.com”, “_clientId”: “04b07795-8ddb-461a-bbee-02f9e1bf7b46"}
adal-python : 41f7dd6a-7e13-458a-9461-32437bfbd54f - OAuth2Client:Found 4 potential entries.
adal-python : 41f7dd6a-7e13-458a-9461-32437bfbd54f - OAuth2Client:Resource specific token found.
adal-python : 41f7dd6a-7e13-458a-9461-32437bfbd54f - OAuth2Client:Returning token from cache lookup, AccessTokenId: b’MqjdOlWQCE29wpKQ75otCiN7o7SUEQhrU7T08BSJa8M=‘, RefreshTokenId: b’vTxz8td408Z0YP/0aj1wL3IR3kOXlwZW0DxqD5KnrUs=’
msrest.pipeline : Configuring request: timeout=100, verify=True, cert=None
msrest.pipeline : Configuring redirects: allow=True, max=30
msrest.pipeline : Configuring proxies: ‘’
msrest.pipeline : Evaluate proxies against ENV settings: True
msrest.pipeline : Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90
msrest.http_logger : Request URL: ‘https://management.azure.com/subscriptions/<snip>/resources?$filter=resourceType%20eq%20%27Microsoft.KeyVault%2Fvaults%27&api-version=2015-11-01’
msrest.http_logger : Request method: ‘GET’
msrest.http_logger : Request headers:
msrest.http_logger :     ‘User-Agent’: ‘python/3.5.2 (Linux-4.9.12-moby-x86_64-with) requests/2.13.0 msrest/0.4.7 msrest_azure/0.4.7 keyvaultmanagementclient/0.30.0 Azure-SDK-For-Python AZURECLI/2.0.2+dev’
msrest.http_logger :     ‘Connection’: ‘keep-alive’
msrest.http_logger :     ‘Accept-Encoding’: ‘gzip, deflate’
msrest.http_logger :     ‘Accept’: ‘application/json’
msrest.http_logger :     ‘Authorization’: ‘*****’
msrest.http_logger :     ‘x-ms-client-request-id’: ‘d8127b9e-1ef2-11e7-bbd5-0242ac110002’
msrest.http_logger :     ‘CommandName’: ‘vm format-secret’
msrest.http_logger :     ‘Content-Type’: ‘application/json; charset=utf-8’
msrest.http_logger :     ‘accept-language’: ‘en-US’
msrest.http_logger : Request body:
msrest.http_logger : None
requests.packages.urllib3.connectionpool : Starting new HTTPS connection (1): management.azure.com
requests.packages.urllib3.connectionpool : https://management.azure.com:443 “GET /subscriptions/<snip>/resources?$filter=resourceType%20eq%20%27Microsoft.KeyVault%2Fvaults%27&api-version=2015-11-01 HTTP/1.1” 200 305
msrest.http_logger : Response status: 200
msrest.http_logger : Response headers:
msrest.http_logger :     ‘Cache-Control’: ‘no-cache’
msrest.http_logger :     ‘Pragma’: ‘no-cache’
msrest.http_logger :     ‘Content-Type’: ‘application/json; charset=utf-8’
msrest.http_logger :     ‘Content-Encoding’: ‘gzip’
msrest.http_logger :     ‘Expires’: ‘-1’
msrest.http_logger :     ‘Vary’: ‘Accept-Encoding’
msrest.http_logger :     ‘x-ms-ratelimit-remaining-subscription-reads’: ‘14999’
msrest.http_logger :     ‘x-ms-request-id’: ‘fd5a10bc-04a6-4185-a915-8bc6a89d8221’
msrest.http_logger :     ‘x-ms-correlation-request-id’: ‘fd5a10bc-04a6-4185-a915-8bc6a89d8221’
msrest.http_logger :     ‘x-ms-routing-request-id’: ‘WESTUS:20170411T201002Z:fd5a10bc-04a6-4185-a915-8bc6a89d8221’
msrest.http_logger :     ‘Strict-Transport-Security’: ‘max-age=31536000; includeSubDomains’
msrest.http_logger :     ‘Date’: ‘Tue, 11 Apr 2017 20:10:02 GMT’
msrest.http_logger :     ‘Content-Length’: ‘305’
msrest.http_logger : Response content:
msrest.http_logger : b’{“value”:[{“id”:“/subscriptions/<snip>/resourceGroups/myResourceGroup/providers/Microsoft.KeyVault/vaults/myKeyVault18265",“name”:“myKeyVault18265",“type”:“Microsoft.KeyVault/vaults”,“location”:“westus”,“tags”:{}},{“id”:“/subscriptions/<snip>/resourceGroups/myResourceGroup/providers/Microsoft.KeyVault/vaults/myKeyVault9780”,“name”:“myKeyVault9780”,“type”:“Microsoft.KeyVault/vaults”,“location”:“westus”,“tags”:{}}]}'
msrest.exceptions : Parameter ‘resource_group_name’ can not be None.
Parameter ‘resource_group_name’ can not be None.

This behavior also replicates on another machine that was installed from latest interactive script rather than Docker image:

'User-Agent': 'python/2.7.6 (Linux-3.4.0+-x86_64-with-Ubuntu-14.04-trusty) requests/2.13.0 msrest/0.4.7 msrest_azure/0.4.7 keyvaultmanagementclient/0.30.0 Azure-SDK-For-Python AZURECLI/2.0.2'

[tjprescott]

@devigned can you take a look? The "regular" keyvault commands look up the resource group from the keyvault name. It may be that one of the code paths ends up not doing this?

[devigned]

@tjprescott just ran the same thing against master and it worked... Hmm, what's going on?

[squillace]

Furthermore: shall I file an issue for this? The help does not say what is done with whatever is passed here. THEN in addition, there's a vm format-secret that does something FOR This argument, but WHAT is done is not described.

    --secrets                          : One or many Key Vault secrets as JSON strings or files via
                                         '@<file path>' containing '[{ "sourceVault": { "id":
                                         "value" }, "vaultCertificates": [{ "certificateUrl":
                                         "value", "certificateStore": "cert store name (only on
                                         windows)"}] }]'.

[iainfoulds]

Okay, there's something in capitalization in resources. If I just go with straight all-lowercase names, this works (I appended my initials to the vault name to make it unique):

az keyvault create --resource-group resourcegroupname --name vaultnameikf --enabled-for-deployment

az keyvault certificate create --vault-name vaultnameikf -n cert1 \
   -p “$(az keyvault certificate get-default-policy)”

secrets=$(az keyvault secret list-versions --vault-name vaultnameikf \
   -n cert1 --query “[?attributes.enabled].id” -o tsv)

vm_secrets=$(az vm format-secret -s “$secrets”)

By twist of fate, we have naming conventions in our docs that typically uses camel-casing. As such, the following fails with the error initially noted:

az keyvault create --resource-group myResourceGroup --name myVaultIKF --enabled-for-deployment

az keyvault certificate create --vault-name myVaultIKF -n cert1 \
   -p “$(az keyvault certificate get-default-policy)”

secrets=$(az keyvault secret list-versions --vault-name myVaultIKF \
   -n cert1 --query “[?attributes.enabled].id” -o tsv)

vm_secrets=$(az vm format-secret -s “$secrets”)

So yes, it does work, but it seems case sensitive. Does this help narrow things down at all?

[squillace]

case-sensitive is considered BAD, as nothing else in the resource system is case-sensitive: resource groups, image names, vm names, whatever casing you pass is preserved for usage, but whatever you ASK for is given to you insensitively.

BTW, @iainfoulds, what do you mean, "work"? What happens? Where do these values end up on the VM?

[iainfoulds]

@squillace Secrets are dropped in to /var/lib/waagent on the VM. For this example, you get a .crt and .prv placed there.

[squillace]

to what end? what happens?

[squillace]

I'm just going on what the help says: nothing. one, doesn't say what's going to happen. Two, doesn't say what you should do with them there. Three, doesn't say anything about any other kind of secrets.

[iainfoulds]

Beyond placing the secrets on the VM? Nothing.

One scenario I'm looking to build out is drop a cert on the VM during deployment, then apply that cert to NGINX site. So, rather than baking certs in to the image, you would pull them from Key Vault. If the cert needs to be renewed or rolled, you don't have to update an entire image to accommodate.

But, back to original issue - yes, case sensitivity isn't great. Hopefully that gives some pointers as to where to look for a fix.

[devigned]

@squillace I like to use this flow to provision an identity onto the machine so that I can access Azure resources like DocumentDb keys or Storage keys via config management tools.

[devigned]

@iainfoulds perhaps, we should be run a to_lower on any incoming string to find the resource group.

Thank you for the details!

[devigned]

@squillace It would be really nice if you could provide a name to the secret, so you could reference the cert by name instead of by fingerprint. It would make rolling the certificate so much easier.

[squillace]

So, is the idea here that it drops "something" on the machine, and cloud-init (in the Linux case) does something with it? What? How do you process these "things" once they get there? I'm waiting to understand how this is not merely customscript, with which I can also pull things.... (ignoring that with that I have to pull the secret in the cli first and then push it with customscript to do something....)

[devigned]

@squillace the point is that you'd have to push this in a custom script. When you do that you have to access to the secret and pull it down to semi secure environment (your dev or deployment machine). In the vm-secrets model, fabric puts the secret onto the machine, so it never touches the outside world.

[JetPac33]

Still no fix available for this ? I ran into this problem with Azure CLI version 2.0.20.

[yungezz]

close stable issue. feel free to create new issue if any error met. thanks.