cloudformation deploy --parameter-overrides parses parameters with spaces in a wrong way

[seeebiii]

Description: I want to hand over parameters to the deployment of my cloudformation template using cloudformation deploy --parameter-overrides. Some parameters might contain spaces which are parsed in a wrong way. Even by using escape characters an error is thrown.

For example: cloudformation deploy [...] --parameter-overrides Foo=B a r or Foo=B\ a\ r or Foo="B a r" do not work.

Expected Behaviour: The parameters are parsed as they would not contain any space, e.g. like Foo=Bar.

Actual Behaviour: An error is thrown: ['a'] value passed to --parameter-overrides must be of format Key=Value

Debug infos:

• aws --version returns: aws-cli/1.15.0 Python/3.6.5 Darwin/17.4.0 botocore/1.10.0

Here is the --debug output (I have removed some params as I had no time to prepare a simple example yet).

2018-04-19 11:26:00,017 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.15.0 Python/3.6.5 Darwin/17.4.0 botocore/1.10.0
2018-04-19 11:26:00,018 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['cloudformation', 'deploy', '--debug', '--template-file', 'cfn.packaged.yml', 'Foo=B', 'a', 'r']

...

2018-04-19 11:26:00,063 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "/usr/local/Cellar/awscli/1.15.0/libexec/lib/python3.6/site-packages/awscli/clidriver.py", line 208, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "/usr/local/Cellar/awscli/1.15.0/libexec/lib/python3.6/site-packages/awscli/clidriver.py", line 345, in __call__
    return command_table[parsed_args.operation](remaining, parsed_globals)
  File "/usr/local/Cellar/awscli/1.15.0/libexec/lib/python3.6/site-packages/awscli/customizations/commands.py", line 187, in __call__
    return self._run_main(parsed_args, parsed_globals)
  File "/usr/local/Cellar/awscli/1.15.0/libexec/lib/python3.6/site-packages/awscli/customizations/cloudformation/deploy.py", line 259, in _run_main
    self.PARAMETER_OVERRIDE_CMD)
  File "/usr/local/Cellar/awscli/1.15.0/libexec/lib/python3.6/site-packages/awscli/customizations/cloudformation/deploy.py", line 381, in parse_key_value_arg
    value=key_value_pair)
awscli.customizations.cloudformation.exceptions.InvalidKeyValuePairArgumentError: ['a'] value passed to --parameter-overrides must be of format Key=Value
2018-04-19 11:26:00,066 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

If you need more information to reproduce it, let me know.

[dstufft]

This is going to come down to the rules of your shell, you can see what values the CLI thinks it recieved from the shell on the second line of your ouput above.

What I would suggest trying is:

$ cloudformation deploy [...] --parameter-overrides "Foo=B a r"

Notice that the " is quoting the Entire Key=Value and not just the value. In most shells that will have it pass that entire string in as one argument to the CLI.

[seeebiii]

@dstufft Indeed! Sorry, I didn't thought about quoting the whole key value pair and also searched a lot for docs about it, but could not find them.. Thanks for the hint! I'm closing it ;-)

[ispyinternet]

Having problems with this when trying to parse a json file with values that have whitespace

E.G:

$ array=$(jq -r '.[] | [.ParameterKey, .ParameterValue] | "'\''\(.[0])=\(.[1])'\''"' parameters.json)
$ aws cloudformation deploy --template-file deploy.yaml --parameter-overrides "${array}" --stack-name TEST

$ echo "${array}"
//
    'param1=val1'
    'param2=value with space'
//

$ [u'with'] value passed to --parameter-overrides must be of format ParameterKey=ParameterValue

This isn't the only iteration of my efforts. I have tried many combinations / approaches all in vain.

I can't for the life of me get bash or aws cli to keep the ParameterKey=ParameterValue parts quoted.

Very frustrating as, if you want to parse a large parameter file programmatically it appears impossible.

Can anybody offer a solution?

[ispyinternet]

Ok, should anybody stumble on this and want a solution, this is what worked for me:

// package.json...

.
scripts: {
   "deploy": "array=( \"aws\" \"cloudformation\" \"deploy\" \"--template-file\" \"deploy.yaml\" \"--capabilities\" \"CAPABILITY_NAMED_IAM\" \"--parameter-overrides\" \"$(jq -r '.[] | [.ParameterKey, .ParameterValue] | \"'\\''\\(.[0])=\\(.[1])'\\'' \"' parameters.json)\" \"--stack-name\") && eval $(echo ${array[@]} $*)"
}
.
// usage:
$ npm run deploy MY_STACK_NAME

handles parameters with spaces.

[PiyushMittl]

@ispyinternet thanks for your solution. i am using shell and have used below command: $ array=$(jq -r '.[] | [.ParameterKey, .ParameterValue] | "'\''(.[0])=(.[1])'\''"' parameters.json) $ aws cloudformation deploy --template-file deploy.yaml --parameter-overrides "${array}" --stack-name TEST

and getting same error. you have given solution in which using npm. is there any way to achieve the same in shell.

[ispyinternet]

I think in your case you need to do something like:

$ array=(aws cloudformation deploy --template-file deploy.yaml --parameter-overrides $(jq -r '.[] | [.ParameterKey, .ParameterValue] | "\(.[0])=\(.[1])"' parameters.json) --stack-name TEST)
$ eval $(echo ${array[@]})

You could probably 1 line that, but Im not sure off the top of my head

[cmaxwellau]

Try this first:

export IFS=$'\n'

It will stop bash from expanding on anything other than a line return.

4055

[default50]

This worked for me:

#!/bin/bash

STACK_NAME="mystack"
TEMPLATE="mytemplate.yaml"
PARAMETERS_FILE="parameters.json"
PARAMS=($(jq -r '.Parameters[] | [.ParameterKey, .ParameterValue] | "\(.[0])=\(.[1])"' ${PARAMETERS_FILE}))

aws cloudformation deploy \
    --template-file "${TEMPLATE}" \
    --stack-name "${STACK_NAME}" \
    --parameter-overrides ${PARAMS[@]}

With parameters.json having this structure:

{
  "Parameters": [
    {
      "ParameterKey": "MyParameter",
      "ParameterValue": "SomeValue"
    }
 ]
}

[szmulder]

{
  "Parameters": [
    {
      "ParameterKey": "MyParameter",
      "ParameterValue": "SomeValue"
    }
 ]
}

but when the value has space it will not work{ "Parameters": [ { "ParameterKey": "MyParameter", "ParameterValue": "Some Value ha space" } ] }

[Fectorez]

--parameter-overrides "key=my\ value" Works on PowerShell

[Ochaun]

Reopen this. I'm having the same problem even when quoting the key value pair. "Key=Value" "Key=Value with space"

[elreydetoda]

so, instead of escaping spaces and quotes there is a built in function for bash called mapfile which will handle parsing of the jq command properly and quote things when necessary.

here is an example on my blog where we got it working for my work: https://blog.elreydetoda.site/cool-shell-tricks/#bash-aws-cloudformation-parameter-overrides specifically these are the commands, but checkout the blog to know what they do.

# pull secrets from AWS secrets manager service by the name
infra_params=$( aws secretsmanager get-secret-value --secret-id "${secret_name}" )

# parse our secrets format (json) and add it to paramz array
mapfile -t paramz < <(jq -r '.SecretString' <<< "${infra_params}" | jq -r '.params[] | "\(.ParameterKey)=\(.ParameterValue)"')

# deploy cloudformation stack w/expanding the paramz array
aws cloudformation deploy --stack-name "${stack_name}" --template <template_name> --parameter-overrides "${paramz[@]}"

let me know if you have anymore questions around the commands (probably comment on my blog to not take over this issue)

[vfrank66]

The solution proposed by @elreydetoda and @ispyinternet work but will this be corrected in aws cli?

[kdaily]

@vfrank66,

Thanks for your question. I don't think there's anything else we can do here. As previously noted, quoting behavior is shell-dependent, and when you specifically consider Windows, it's really tricky, and even moreso when you throw whitespace or escaping quote characters into it. We updated the quoting documentation pretty recently:

https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-parameters-quoting-strings.html#cli-usage-parameters-quoting-strings-around

It references this Powershell documentation page that details all (or, at least many) of the ways quoting works there:

https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_quoting_rules?view=powershell-7

[vfrank66]

@kdaily thanks for the response! I think that link should be clarified for linux/mac because I believe you have to set IFS to something other than ' \t\n' and use [@] in the bash script simple single quote in insufficient. I understand quoting is difficult and I don't work on a windows anymore so I don't know the added complexity, but I do think this is something that should be partially responsible for the aws cli. Here is some research I did after posting my short comment so maybe I can persuade you with a longer one?

I currently have this error on --tags but it functions the same for --parameter-overrides on this command. Even though this error is related to how the shell handles spaces/arrays/string it sends in the --parameters-overrides one long string in the cases above, but I feel like the cli should be handling this. For cloudformation deploy there is already a method parse_key_value_arg() that parses both the parameters and tags (my error is the tags). In this function there could be check for single parameter with multiple = and it could reparse this out since it has specific formatting for parameters and overrides.

The aws cli code assumption is by the time you get to parse_key_value_arg() the --parameters-overrides (and --tags) are an "Array of strings, where each string is of form Key=Value". But depending on how you load that string in shell and how python cli accepts sys.argv, plus we cant use a file, it is typically with jq and possibly sed, sometimes the shell will send in to the aws cli a "string "in the form Key=Value.

Method location: https://github.com/aws/aws-cli/blob/develop/awscli/customizations/cloudformation/deploy.py#L397

An example of bash sending in the wrong sys.argv into python:

My example, I have to handle spacing in these values + arrays so my jq is different but

# LOAD PARAMS: aws cli - correctly identifies this as a array of strings 
params=$(cat blah-parameters-${environment}.json \
    | jq '.[] | .ParameterKey + "=" + if .ParameterValue|type=="array" then .ParameterValue | join(",") else .ParameterValue end ' \
    | sed -e 's/"//g' \
    | sed -e $'s/\r//g' | tr '\n' ' ')

# LOAD TAGS TWO FILES: aws cli -incorrectly treats this as a single string
tags=$(cat tags-common.json | \
    jq '.[] | (.Key + "=" +.Value)' | \
    sed -n -e 'H;${x;s/\n/ /g;s/^ //;p;}' | \
    tr '\n' ' ')
tags=$tags$(cat tags-${environment}.json | \
    jq '.[] | (.Key + "=" +.Value)' | \
    sed -n -e 'H;${x;s/\n/ /g;s/^ //;p;}')

   ...

    aws cloudformation deploy \
    --template-file "stack-name-transform.yaml" \
    --stack-name "stack-name" \
    --parameter-overrides "${params}" \
    --tags "${tags}"

This is the output echo'ed out, as you can see no difference between the parameters and tags, mine is failing the the bash script on the $tags

   aws cloudformation deploy \
    --region us-east-1 \
    --no-fail-on-empty-changeset \
    --template-file ./cfn/stack-name-transform.yaml \
    --stack-name stack-name-sand \
    --capabilities CAPABILITY_NAMED_IAM \
    --parameter-overrides Environment=sand CertificateArn=arn:aws:acm:us-east-1:::: DomainName=test.api.io \
    --tags TAG1=stack-name TAG2=Yes CostCenter=78-900 TAG3='a string' Environment=Sandbox

If I take the echo output and then copy that to the clipboard and then paste that without any variable interpolation it will run successfully (so the problem is how the shell loads file in correlation to how python executable accepts inputs which is why answers by @elreydetoda and @ispyinternet work, they standardize all the variables in shell instead of the aws cli executable, another working example is my question and answer on SO).

If it is in the bash script it will fail because $tags is interpreted as a single string within the cli, in my example this is because of IFS and spaces in --tags but not the --parameter-overrides. For use of aws cli in cicd the parameters/tags will change xcross environnments. To me this should be handled by the cli, either by allowing a consistent interface in accept these list of values such as loading a defined file, already suggested, or by knowing that the shell can insert either a string or list into these defined (list of strings)[https://github.com/aws/aws-cli/blob/develop/awscli/customizations/cloudformation/deploy.py#L106].

[PalituxD]

I resolved this scenario using options below:

"scripts": { "invoke": "sam ... --parameter-overrides \"$(jq -j 'to_entries[] | \"\\(.key)='\\\\\\\"'\\(.value)'\\\\\\\"''\\ '\"' params.json)\"" }

Or

sam ... --parameter-overrides "$(jq -j 'to_entries[] | "\(.key)='\\\"'\(.value)'\\\"''\ '"' params.json)"

[markokole]

Here is what works for me in powershell

$Env:VAR_NAME = 'Bed & Breakfest'

--parameter-overrides `
    "VarName='$Env:VAR_NAME'" `

The whole parameter key-value is in double quotes and then the value itself is in single quotes.

[AlejandroFNadal]

Just a random fact that I learnt and that I think it was not so clear: The parameters cannot have underscores in the name. This was causing my issue with --parameter-overrides, and yet the template was being accepted with no errors.