New-EC2LaunchTemplate/New-EC2LaunchTemplateVersion SourceTemplateData parameter issue on gp2 EBS disk

[hsuantio]

The SourceTemplateDataparameter always sets Amazon.EC2.Model.LaunchTemplateEbsBlockDeviceRequest IOPS value even though the original one is not. This causes launching EC2 instance with gp2 disk from the launch template to fail with The parameter iops is not supported for gp2 volumes error as the IOPS value is set.

Expected Behavior

The SourceTemplateData parameter should not set Amazon.EC2.Model.LaunchTemplateEbsBlockDeviceRequest IOPS value if the original isn't when mapping Amazon.EC2.Model.ResponseLaunchTemplateData to Amazon.EC2.Model.RequestLaunchTemplateData.

Current Behavior The SourceTemplateData parameter always sets Amazon.EC2.Model.LaunchTemplateEbsBlockDeviceRequest IOPS value even though the original one is not. This causes launching EC2 instance with gp2 disk from the launch template to fail with The parameter iops is not supported for gp2 volumes error as the IOPS value is set.

Possible Solution -SourceTemplateData should not set the IOPS value if the original template data is not. The workaround is to manually map Amazon.EC2.Model.ResponseLaunchTemplateData to Amazon.EC2.Model.RequestLaunchTemplateData and use -LaunchTemplateData parameter instead.

Steps to Reproduce (for bugs) Create LaunchTemplateEbsBlockDeviceRequest:

$LaunchTemplateName = "TestIssue2"
$LaunchTemplateEbsBlockDeviceRequest = New-Object -TypeName Amazon.EC2.Model.LaunchTemplateEbsBlockDeviceRequest
$LaunchTemplateEbsBlockDeviceRequest.DeleteOnTermination = $true
$LaunchTemplateEbsBlockDeviceRequest.SnapshotId = "snap-0a1194e6c78114ead" #AMI ami-0bbdf9279190cdd33 root device snapshot ID
$LaunchTemplateEbsBlockDeviceRequest.VolumeSize = 30
$LaunchTemplateEbsBlockDeviceRequest.VolumeType = "gp2" 

Create LaunchTemplateBlockDeviceMappingRequest:

$LaunchTemplateBlockDeviceMappingRequest = New-Object -TypeName Amazon.EC2.Model.LaunchTemplateBlockDeviceMappingRequest
$LaunchTemplateBlockDeviceMappingRequest.Ebs = $LaunchTemplateEbsBlockDeviceRequest
$LaunchTemplateBlockDeviceMappingRequest.DeviceName = "/dev/sda1"

Create RequestLaunchTemplateData:

$RequestLaunchTemplateData = New-Object -TypeName Amazon.EC2.Model.RequestLaunchTemplateData
$RequestLaunchTemplateData.BlockDeviceMappings = $LaunchTemplateBlockDeviceMappingRequest
$RequestLaunchTemplateData.ImageId = "ami-0bbdf9279190cdd33"

Create EC2 launch template

New-EC2LaunchTemplate -LaunchTemplateData $RequestLaunchTemplateData -LaunchTemplateName $LaunchTemplateName
#Launch template is created successfully.

Check if v1 launch template __iops is set using IsSetIops()method:

$IsSetIopsValue = ((Get-EC2TemplateVersion -LaunchTemplateName $LaunchTemplateName -Version 1).LaunchTemplateData.BlockDeviceMappings | Where-Object {$_.DeviceName -Match "/dev/sda1"}).Ebs
Write-Host "Is v1 IOPS value set: $(($IsSetIopsValue.GetType().GetMethods('NonPublic,Instance') | Where-Object Name -eq IsSetIops).Invoke($IsSetIopsValue, @()))"
#Returns false, all good.

Create LaunchTemplateSpecification:

$LaunchTemplateSpec = New-Object -TypeName Amazon.EC2.Model.LaunchTemplateSpecification
$LaunchTemplateSpec.LaunchTemplateName = $LaunchTemplateName
$LaunchTemplateSpec.Version = 1

Launch new EC2 instance from created launch template:

New-EC2Instance -LaunchTemplate $LaunchTemplateSpec -SubnetId subnet-0924382f5e3e147df
#Instance launched successfully.

Create new launch template version from original TestIssue2 as-is. This is launch template version 2.

New-EC2LaunchTemplateVersion -SourceTemplateData (Get-EC2TemplateVersion -LaunchTemplateName $LaunchTemplateName -Version 1).LaunchTemplateData -LaunchTemplateName $LaunchTemplateName

Check if v2 launch template __iops is set using IsSetIops() method

$IsSetIopsValue = ((Get-EC2TemplateVersion -LaunchTemplateName $LaunchTemplateName -Version 2).LaunchTemplateData.BlockDeviceMappings | Where-Object {$_.DeviceName -Match "/dev/sda1"}).Ebs
Write-Host "Is v2 IOPS value set: $(($IsSetIopsValue.GetType().GetMethods('NonPublic,Instance') | Where-Object Name -eq IsSetIops).Invoke($IsSetIopsValue, @()))"
#Returns true, the value is now set for some reason.

Let's see if I can apply a workaround

$LatestLaunchTemplateData = (Get-EC2TemplateVersion -LaunchTemplateName $LaunchTemplateName -Version (Get-EC2TemplateVersion -LaunchTemplateName $LaunchTemplateName | Sort-Object VersionNumber -Descending | Select-Object -First 1).VersionNumber).LaunchTemplateData
$LaunchTemplateEbsBlockDevice = New-Object -TypeName Amazon.EC2.Model.LaunchTemplateEbsBlockDevice 
$LaunchTemplateEbsBlockDevice.DeleteOnTermination = $LatestLaunchTemplateData.BlockDeviceMappings.Ebs.DeleteOnTermination
$LaunchTemplateEbsBlockDevice.SnapshotId = $LatestLaunchTemplateData.BlockDeviceMappings.Ebs.SnapshotId
$LaunchTemplateEbsBlockDevice.VolumeSize = $LatestLaunchTemplateData.BlockDeviceMappings.Ebs.VolumeSize
$LaunchTemplateEbsBlockDevice.VolumeType = $LatestLaunchTemplateData.BlockDeviceMappings.Ebs.VolumeType

The new template data EBS IOPS should not be set now

$ModifiedTemplateData = $LatestLaunchTemplateData
$ModifiedTemplateData.BlockDeviceMappings[0].Ebs = $LaunchTemplateEbsBlockDevice
Write-Host "Is ModifiedTemplateData IOPS value set: $(($ModifiedTemplateData.BlockDeviceMappings.Ebs.GetType().GetMethods('NonPublic,Instance') | Where-Object Name -eq IsSetIops).Invoke($ModifiedTemplateData.BlockDeviceMappings.Ebs, @()))"
#Returns false, new LaunchTemplateEbsBlockDevice IOPS looks good.

So we now have a LaunchTemplateData which IOPS value is unset. Let's create a new launch template version using this one as v3.

New-EC2LaunchTemplateVersion -SourceTemplateData $ModifiedTemplateData -LaunchTemplateName $LaunchTemplateName
$IsSetIopsValue = ((Get-EC2TemplateVersion -LaunchTemplateName $LaunchTemplateName -Version 3).LaunchTemplateData.BlockDeviceMappings | Where-Object {$_.DeviceName -Match "/dev/sda1"}).Ebs
Write-Host "Is v3 IOPS value set: $(($IsSetIopsValue.GetType().GetMethods('NonPublic,Instance') | Where-Object Name -eq IsSetIops).Invoke($IsSetIopsValue, @()))"
#Returns true

For some reason, now the value is set to true when -SourceTemplateData maps values from ResponseLaunchTemplateDatato RequestLaunchTemplateData

Let's try to launch an EC2 instance using v3 launch template

$LaunchTemplateSpec.Version = 3
New-EC2Instance -LaunchTemplate $LaunchTemplateSpec -SubnetId subnet-0924382f5e3e147df
<# Exception caught
The parameter iops is not supported for gp2 volumes.
At MapLaunchTemplate1.ps1:47 char:1
+ New-EC2Instance -LaunchTemplate $LaunchTemplateSpec -SubnetId subnet- ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (Amazon.PowerShe...2InstanceCmdlet:NewEC2InstanceCmdlet) [New-EC2Instance], InvalidOperationException
    + FullyQualifiedErrorId : Amazon.EC2.AmazonEC2Exception,Amazon.PowerShell.Cmdlets.EC2.NewEC2InstanceCmdlet #>

Context Unable to implement launch template/versions automation using Powershell.

Your Environment

• AWSPowershell used: 3.3.522.0
• Service assembly and version used: .NET 4.7.2
• Operating System and version: Windows 10 (10.0.17763)
• Visual Studio version: 2017
• Targeted .NET platform: .NET 4.7.2

[matteo-prosperi]

Depends on feature request https://github.com/aws/aws-sdk-net/issues/1331.

[matteo-prosperi]

The root cause of this is that the New-EC2LaunchTemplateVersion cmdlet copies the value of SourceTemplateData into LaunchTemplateData performing the necessary conversion from type ResponseLaunchTemplateData to type RequestLaunchTemplateData. Because such conversion is performed by copying each field and optional integer fields return 0 when the field is not set, the resulting RequestLaunchTemplateData always has Iops set. In order to address this, we need the AWS SDK for .NET to implement a way to distinguish unset value-type response fields (https://github.com/aws/aws-sdk-net/issues/1331 is tracking this request).

I cannot think of a good workaround for this issue, except for manually creating the RequestLaunchTemplateData object and passing it to the LaunchTemplateData parameter of New-EC2LaunchTemplateVersion.

[matteo-prosperi]

Hello, this issue is fixed with the latest release of AWSPowerShell 3.3.553.0.

We have also added a method in the SDK to check if a field is set in a service response (see https://github.com/aws/aws-sdk-net/issues/1331#issuecomment-513275696). Please use that instead of the reflection-based workaround (... .GetMethods('NonPublic,Instance') | Where-Object Name -eq IsSetIops).Invoke ...). The PowerShell syntax would be:

[Amazon.Util.AWSSDKUtils]::IsPropertySet($blockDeviceMapping.Ebs, 'Iops')