ec2.RunInstances output syntax errors

[jbananas]

Describe the bug Python and CLI output for ec2.RunInstances contains syntax errors.

Related Mapping ec2.RunInstances

Related Language CLI and Python tested

To Reproduce Steps to reproduce the behavior:

1. Start recorder
2. From EC2 Dashboard click Launch Instance
3. Select default amazon linux 2 ami
4. Select default t2.micro instance type
5. Change default VPC/subnet to anything else.
6. Next, Next, Next
7. On security group setup page, choose any existing security group.
8. Next next finish through wizard

Expected behavior Expectation is generated commands can be run without modification. This is not the case. For example, using above repro steps generates the following CLI command:

ec2 run-instances --image-id "ami-009d6802948d06e52" --count 1 --key-name "mykey" --instance-type "t2.micro" --placement '{"Tenancy":"default"}' --monitoring '{"Enabled":false}' --enable-api-termination --instance-initiated-shutdown-behavior "stop" --credit-specification '{"CpuCredits":"standard"}' --no-ebs-optimized --block-device-mappings '[{"DeviceName":"/dev/xvda","Ebs":{"VolumeSize":8,"DeleteOnTermination":true,"VolumeType":"gp2"}}]' --region us-east-1

Attempting to run this command results in:

Error parsing parameter '--block-device-mappings': Expected: '=', received: ''' for input:
'[{DeviceName:/dev/xvda,Ebs:{VolumeSize:8,DeleteOnTermination:true,VolumeType:gp2}}]'

The correctly formatted command is:

aws ec2 run-instances --image-id "ami-009d6802948d06e52" --count 1 --key-name "mykey" --instance-type "t2.micro" --placement Tenancy="default" --monitoring Enabled=false --enable-api-termination --instance-initiated-shutdown-behavior "stop" --credit-specification CpuCredits="standard" --no-ebs-optimized --block-device-mappings DeviceName="/dev/xvda",Ebs={VolumeSize=8,DeleteOnTermination=true,VolumeType="gp2"} --region us-east-1

Similar incorrect syntax is seen with python. Recorder output:

import boto3
ec2_client = boto3.client('ec2', region_name='us-east-1')
response = ec2_client.run_instances(
    ImageId='ami-009d6802948d06e52',
    MaxCount=1,
    MinCount=1,
    KeyName='mykey',
    InstanceType='t2.micro',
    Placement={
        Tenancy='default'
    },
    Monitoring={
        Enabled=False
    },
    DisableApiTermination=False,
    InstanceInitiatedShutdownBehavior='stop',
    CreditSpecification={
        CpuCredits='standard'
    },
    EbsOptimized=False,
    BlockDeviceMapping=[
        {
            DeviceName='/dev/xvda',
            Ebs={
                VolumeSize=8,
                DeleteOnTermination=True,
                VolumeType='gp2'
            }
        }
    ]
)

Running this results in:

python test.py
  File "test.py", line 10
    Tenancy='default'
           ^
SyntaxError: invalid syntax

...due to improperly formatted parameter section. The correct formatting is:

response = ec2_client.run_instances(
    ImageId='ami-009d6802948d06e52',
    MaxCount=1,
    MinCount=1,
    KeyName='mykey',
    InstanceType='t2.micro',
    Placement={
        'Tenancy':'default'
    },
    Monitoring={
        'Enabled':False
    },
    DisableApiTermination=False,
    InstanceInitiatedShutdownBehavior='stop',
    CreditSpecification={
        'CpuCredits':'standard'
    },
    EbsOptimized=False,
    BlockDeviceMappings=[
        {
            'DeviceName':'/dev/xvda',
            'Ebs':{
                'VolumeSize':8,
                'DeleteOnTermination':True,
                'VolumeType':'gp2'
            }
        }
    ]
)

[iann0036]

Fixed the boto syntax, but can't replicate your issue with the CLI (the array syntax works fine on my Mac).

Could I get you to report your OS and CLI version, and possibly run the command with --debug so we can see how the param is being passed in?

For reference, my command looks like this:

aws ec2 run-instances --image-id "ami-01e24be29428c15b2" --count 1 --key-name "imckay-oregon" --security-group-ids '["sg-0315b6f68ba8dd9f5"]' --instance-type "t2.micro" --placement '{"Tenancy":"default"}' --monitoring '{"Enabled":false}' --enable-api-termination --instance-initiated-shutdown-behavior "stop" --credit-specification '{"CpuCredits":"standard"}' --no-ebs-optimized --block-device-mappings '[{"DeviceName":"/dev/xvda","Ebs":{"VolumeSize":8,"DeleteOnTermination":true,"VolumeType":"gp2"}}]' --region us-west-2 --debug

[jbananas]

Sorry for the delay, I thought I had responded.

λ aws --version
aws-cli/1.16.80 Python/3.6.4 Windows/10 botocore/1.12.70

λ aws ec2 run-instances --image-id "ami-009d6802948d06e52" --count 1 --key-name "jbananas" --instance-type "t2.micro" --placement '{"Tenancy":"default"}' --monitoring '{"Enabled":false}' --enable-api-termination --instance-initiated-shutdown-behavior "stop" --credit-specification '{"CpuCredits":"standard"}' --no-ebs-optimized --block-device-mappings '[{"DeviceName":"/dev/xvda","Ebs":{"VolumeSize":8,"DeleteOnTermination":true,"VolumeType":"gp2"}}]' --region us-east-1 --debug

2019-01-09 11:09:19,961 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.16.80 Python/3.6.4 Windows/10 botocore/1.12.70 2019-01-09 11:09:19,962 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['ec2', 'run-instances', '--image-id', 'ami-009d6802948d06e52', '--count', '1', '--key-name', 'jbananas', '--instance-type', 't2.micro', '--placement', "'{Tenancy:default}'", '--monitoring', "'{Enabled:false}'", '--enable-api-termination', '--instance-initiated-shutdown-behavior', 'stop', '--credit-specification', "'{CpuCredits:standard}'", '--no-ebs-optimized', '--block-device-mappings', "'[{DeviceName:/dev/xvda,Ebs:{VolumeSize:8,DeleteOnTermination:true,VolumeType:gp2}}]'", '--region', 'us-east-1', '--debug']
2019-01-09 11:09:19,963 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x065AA348>
2019-01-09 11:09:19,964 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x061D6A98>
2019-01-09 11:09:19,965 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x061F5AE0>
2019-01-09 11:09:20,237 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x064BC228>
2019-01-09 11:09:20,247 - MainThread - botocore.loaders - DEBUG - Loading JSON file: c:\users\jbananas\.aws\models\ec2\2016-11-15\service-2.json
2019-01-09 11:09:20,286 - MainThread - botocore.hooks - DEBUG - Event service-data-loaded.ec2: calling handler <function register_retries_for_service at 0x05FC2CD8>
2019-01-09 11:09:20,287 - MainThread - botocore.handlers - DEBUG - Registering retry handlers for service: ec2
2019-01-09 11:09:20,299 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ec2: calling handler functools.partial(<function _remove_commands at 0x06554AE0>, commands_to_remove=['import-instance', 'import-volume'])
2019-01-09 11:09:20,299 - MainThread - awscli.customizations.removals - DEBUG - Removing operation: import-instance
2019-01-09 11:09:20,300 - MainThread - awscli.customizations.removals - DEBUG - Removing operation: import-volume
2019-01-09 11:09:20,300 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ec2: calling handler <function add_waiters at 0x065B3108>
2019-01-09 11:09:20,396 - MainThread - botocore.loaders - DEBUG - Loading JSON file: C:\Users\jbananas\Python\Python36-32\lib\site-packages\botocore\data\ec2\2016-11-15\waiters-2.json
2019-01-09 11:09:20,401 - MainThread - awscli.clidriver - DEBUG - OrderedDict([('block-device-mappings', <awscli.arguments.ListArgument object at 0x06D592F0>), ('image-id', <awscli.arguments.CLIArgument object at 0x06D59A90>), ('instance-type', <awscli.arguments.CLIArgument object at 0x06D59AD0>), ('ipv6-address-count', <awscli.arguments.CLIArgument object at 0x06D59AB0>), ('ipv6-addresses', <awscli.arguments.ListArgument object at 0x06D59AF0>), ('kernel-id', <awscli.arguments.CLIArgument object at 0x06D59B30>), ('key-name', <awscli.arguments.CLIArgument object at 0x06D593D0>), ('max-count', <awscli.arguments.CLIArgument object at 0x06D59B10>), ('min-count', <awscli.arguments.CLIArgument object at 0x06D59B50>), ('monitoring', <awscli.arguments.CLIArgument object at 0x06D59B70>), ('placement', <awscli.arguments.CLIArgument object at 0x06D59B90>), ('ramdisk-id', <awscli.arguments.CLIArgument object at 0x06D59BB0>), ('security-group-ids', <awscli.arguments.ListArgument object at 0x06D59BD0>), ('security-groups', <awscli.arguments.ListArgument object at 0x06D59BF0>), ('subnet-id', <awscli.arguments.CLIArgument object at 0x06D59C10>), ('user-data', <awscli.arguments.CLIArgument object at 0x06D59C50>), ('additional-info', <awscli.arguments.CLIArgument object at 0x06D59C30>), ('client-token', <awscli.arguments.CLIArgument object at 0x06D59C70>), ('disable-api-termination', <awscli.arguments.BooleanArgument object at 0x06D59C90>), ('no-disable-api-termination', <awscli.arguments.BooleanArgument object at 0x06D59CB0>), ('dry-run', <awscli.arguments.BooleanArgument object at 0x06D59D30>), ('no-dry-run', <awscli.arguments.BooleanArgument object at 0x06D59D10>), ('ebs-optimized', <awscli.arguments.BooleanArgument object at 0x06D59D50>), ('no-ebs-optimized', <awscli.arguments.BooleanArgument object at 0x06D59D70>), ('iam-instance-profile', <awscli.arguments.CLIArgument object at 0x06D59D90>), ('instance-initiated-shutdown-behavior', <awscli.arguments.CLIArgument object at 0x06D59DB0>), ('network-interfaces', <awscli.arguments.ListArgument object at 0x06D59DD0>), ('private-ip-address', <awscli.arguments.CLIArgument object at 0x06D59DF0>), ('elastic-gpu-specification', <awscli.arguments.ListArgument object at 0x06D59E10>), ('tag-specifications', <awscli.arguments.ListArgument object at 0x06D59E30>), ('launch-template', <awscli.arguments.CLIArgument object at 0x06D59E50>), ('instance-market-options', <awscli.arguments.CLIArgument object at 0x06D59E70>), ('credit-specification', <awscli.arguments.CLIArgument object at 0x06D59E90>), ('cpu-options', <awscli.arguments.CLIArgument object at 0x06D59ED0>)])
2019-01-09 11:09:20,403 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.run-instances: calling handler <function add_streaming_output_arg at 0x065AA660>
2019-01-09 11:09:20,404 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.run-instances: calling handler <function rename_arg.<locals>._rename_arg at 0x065C3390>
2019-01-09 11:09:20,405 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.run-instances: calling handler <function rename_arg.<locals>._rename_arg at 0x065C3420>
2019-01-09 11:09:20,406 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.run-instances: calling handler functools.partial(<function pull_up_bool at 0x065AABB8>, event_handler=<botocore.hooks.EventAliaser object at 0x04F34130>)
2019-01-09 11:09:20,408 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.run-instances: calling handler <function ec2_add_count at 0x064BC738>
2019-01-09 11:09:20,409 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.run-instances: calling handler <function _add_params at 0x064C3270>
2019-01-09 11:09:20,412 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.run-instances: calling handler <function add_cli_input_json at 0x061FC2B8>
2019-01-09 11:09:20,413 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.run-instances: calling handler <function unify_paging_params at 0x0654A588>
2019-01-09 11:09:20,504 - MainThread - botocore.loaders - DEBUG - Loading JSON file: C:\Users\jbananas\Python\Python36-32\lib\site-packages\botocore\data\ec2\2016-11-15\paginators-1.json
2019-01-09 11:09:20,506 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.run-instances: calling handler <function add_generate_skeleton at 0x06530780>
2019-01-09 11:09:20,507 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.ec2.run-instances: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x06D6F250>>
2019-01-09 11:09:20,508 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.ec2.run-instances: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x06D6F2B0>>
2019-01-09 11:09:20,512 - MainThread - botocore.hooks - DEBUG - Event operation-args-parsed.ec2.run-instances: calling handler functools.partial(<function validate_boolean_mutex_groups at 0x065AAC00>, boolean_pairs=[])
2019-01-09 11:09:20,512 - MainThread - botocore.hooks - DEBUG - Event operation-args-parsed.ec2.run-instances: calling handler <function _check_args at 0x064C32B8>
2019-01-09 11:09:20,513 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ec2.run-instances.block-device-mappings: calling handler <awscli.paramfile.URIArgumentHandler object at 0x05DE7410>
2019-01-09 11:09:20,514 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.ec2.run-instances: calling handler <awscli.argprocess.ParamShorthandParser object at 0x04ED4E70>
2019-01-09 11:09:20,515 - MainThread - awscli.argprocess - DEBUG - Parsing param --block-device-mappings as shorthand
2019-01-09 11:09:20,515 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "C:\Users\jbananas\Python\Python36-32\lib\site-packages\awscli\argprocess.py", line 331, in _parse_as_shorthand
    parsed = [self._parser.parse(v) for v in value]
  File "C:\Users\jbananas\Python\Python36-32\lib\site-packages\awscli\argprocess.py", line 331, in <listcomp>
    parsed = [self._parser.parse(v) for v in value]
  File "C:\Users\jbananas\Python\Python36-32\lib\site-packages\awscli\shorthand.py", line 146, in parse
    return self._parameter()
  File "C:\Users\jbananas\Python\Python36-32\lib\site-packages\awscli\shorthand.py", line 151, in _parameter
    params.update(self._keyval())
  File "C:\Users\jbananas\Python\Python36-32\lib\site-packages\awscli\shorthand.py", line 160, in _keyval
    self._expect('=', consume_whitespace=True)
  File "C:\Users\jbananas\Python\Python36-32\lib\site-packages\awscli\shorthand.py", line 320, in _expect
    actual, self._index)
awscli.shorthand.ShorthandParseError: Expected: '=', received: ''' for input:
'[{DeviceName:/dev/xvda,Ebs:{VolumeSize:8,DeleteOnTermination:true,VolumeType:gp2}}]'
^

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\jbananas\Python\Python36-32\lib\site-packages\awscli\clidriver.py", line 207, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "C:\Users\jbananas\Python\Python36-32\lib\site-packages\awscli\clidriver.py", line 348, in __call__
    return command_table[parsed_args.operation](remaining, parsed_globals)
  File "C:\Users\jbananas\Python\Python36-32\lib\site-packages\awscli\clidriver.py", line 491, in __call__
    parsed_args, self.arg_table)
  File "C:\Users\jbananas\Python\Python36-32\lib\site-packages\awscli\clidriver.py", line 547, in _build_call_parameters
    arg_object.add_to_params(service_params, value)
  File "C:\Users\jbananas\Python\Python36-32\lib\site-packages\awscli\arguments.py", line 453, in add_to_params
    unpacked = self._unpack_argument(value)
  File "C:\Users\jbananas\Python\Python36-32\lib\site-packages\awscli\arguments.py", line 463, in _unpack_argument
    cli_argument=self, value=value)
  File "C:\Users\jbananas\Python\Python36-32\lib\site-packages\awscli\arguments.py", line 476, in _emit_first_response
    responses = self._emit(name, **kwargs)
  File "C:\Users\jbananas\Python\Python36-32\lib\site-packages\awscli\arguments.py", line 473, in _emit
    return self._event_emitter.emit(name, **kwargs)
  File "C:\Users\jbananas\Python\Python36-32\lib\site-packages\botocore\hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "C:\Users\jbananas\Python\Python36-32\lib\site-packages\botocore\hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "C:\Users\jbananas\Python\Python36-32\lib\site-packages\botocore\hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "C:\Users\jbananas\Python\Python36-32\lib\site-packages\awscli\argprocess.py", line 314, in __call__
    cli_argument, value, service_id, operation_name)
  File "C:\Users\jbananas\Python\Python36-32\lib\site-packages\awscli\argprocess.py", line 338, in _parse_as_shorthand
    raise ParamError(cli_argument.cli_name, str(e))
awscli.argprocess.ParamError: Error parsing parameter '--block-device-mappings': Expected: '=', received: ''' for input:
'[{DeviceName:/dev/xvda,Ebs:{VolumeSize:8,DeleteOnTermination:true,VolumeType:gp2}}]'
^
2019-01-09 11:09:20,527 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

Error parsing parameter '--block-device-mappings': Expected: '=', received: ''' for input:
'[{DeviceName:/dev/xvda,Ebs:{VolumeSize:8,DeleteOnTermination:true,VolumeType:gp2}}]'
^

[iann0036]

Hi,

Thanks for getting back to me. Looks like the CLI may need to use double quotes in Windows. Could you try:

aws ec2 run-instances --image-id "ami-009d6802948d06e52" --count 1 --key-name "jbananas" --instance-type "t2.micro" --placement "{\"Tenancy\":\"default\"}" --monitoring "{\"Enabled\":false}" --enable-api-termination --instance-initiated-shutdown-behavior "stop" --credit-specification "{\"CpuCredits\":\"standard\"}" --no-ebs-optimized --block-device-mappings "[{\"DeviceName\":\"/dev/xvda\",\"Ebs\":{\"VolumeSize\":8,\"DeleteOnTermination\":true,\"VolumeType\":\"gp2\"}}]" --region us-east-1 --debug

[jbananas]

Yes, confirmed. That worked as expected.

[iann0036]

Thanks soo much for that - I appreciate the help. I'll have the fix published in version 0.3.3+, due out in a couple hours.