ec2 delete-launch-template-versions doesn't have any effect but doesn't give any errors

RealityCtrl commented 3 weeks ago

Describe the bug

When executing command ec2 delete-launch-template-versions it doesn't apply any effects and it seems from --debug flag it doesn't call the APIs needed to delete the versions.

This seems to occur when an an autoscaling group is using multiple versions of the same template and the non-default template is still in use by an instance and you try to delete that version.

It made no difference if supplying versions in quotes or if a launch template ID was used instead of a name. The version to be deleted isn't the default version.

DMCCAND-J4VKQK5:refresh-instances david.mccandless$ aws ec2 delete-launch-template-versions --launch-template-name "$TemplateName" --version 1 --region us-east-1 --profile mc-qa --debug
2024-09-02 17:04:16,690 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.17.42 Python/3.11.9 Darwin/22.6.0 source/arm64
2024-09-02 17:04:16,690 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['ec2', 'delete-launch-template-versions', '--launch-template-name', 'cert-mc-es-cluster-20240904-MasterEligibleNodeLaunchTemplate', '--version', '1', '--region', 'us-east-1', '--profile', 'mc-qa', '--debug']
2024-09-02 17:04:16,730 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x1045f3100>
2024-09-02 17:04:16,730 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x104412660>
2024-09-02 17:04:16,730 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2024-09-02 17:04:16,730 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x103c7f100>
2024-09-02 17:04:16,730 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x103ca47c0>
2024-09-02 17:04:16,730 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x1046031a0>
2024-09-02 17:04:16,730 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x10446d6c0>
2024-09-02 17:04:16,730 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
2024-09-02 17:04:16,730 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x104602160>
2024-09-02 17:04:16,730 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x1046dba50>>
2024-09-02 17:04:16,731 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/homebrew/Cellar/awscli/2.17.42/libexec/lib/python3.11/site-packages/awscli/data/cli.json
aws-cli/2.17.42 Python/3.11.9 Darwin/22.6.0 source/arm64

One instance is using version 1 and two are using version 2.

aws  autoscaling describe-auto-scaling-instances --instance-ids "i-03948afb6f323e59a" "i-09e62ce791ffe4292" "i-0f79bb6c595b4e9dd"  --profile mc-qa --region us-east-1
{
    "AutoScalingInstances": [
        {
            "InstanceId": "i-03948afb6f323e59a",
            "InstanceType": "m6a.large",
            "AutoScalingGroupName": "cert-mc-es-cluster-20240904-MasterEligibleNodeAutoScalingGroup-Qb7cYNvtYtOR",
            "AvailabilityZone": "us-east-1a",
            "LifecycleState": "InService",
            "HealthStatus": "HEALTHY",
            "LaunchTemplate": {
                "LaunchTemplateId": "lt-0702decbdc390f5cc",
                "LaunchTemplateName": "cert-mc-es-cluster-20240904-MasterEligibleNodeLaunchTemplate",
                "Version": "2"
            },
            "ProtectedFromScaleIn": false
        },
        {
            "InstanceId": "i-09e62ce791ffe4292",
            "InstanceType": "m6a.large",
            "AutoScalingGroupName": "cert-mc-es-cluster-20240904-MasterEligibleNodeAutoScalingGroup-Qb7cYNvtYtOR",
            "AvailabilityZone": "us-east-1b",
            "LifecycleState": "InService",
            "HealthStatus": "HEALTHY",
            "LaunchTemplate": {
                "LaunchTemplateId": "lt-0702decbdc390f5cc",
                "LaunchTemplateName": "cert-mc-es-cluster-20240904-MasterEligibleNodeLaunchTemplate",
                "Version": "1"
            },
            "ProtectedFromScaleIn": false
        },
        {
            "InstanceId": "i-0f79bb6c595b4e9dd",
            "InstanceType": "m6a.large",
            "AutoScalingGroupName": "cert-mc-es-cluster-20240904-MasterEligibleNodeAutoScalingGroup-Qb7cYNvtYtOR",
            "AvailabilityZone": "us-east-1c",
            "LifecycleState": "InService",
            "HealthStatus": "HEALTHY",
            "LaunchTemplate": {
                "LaunchTemplateId": "lt-0702decbdc390f5cc",
                "LaunchTemplateName": "cert-mc-es-cluster-20240904-MasterEligibleNodeLaunchTemplate",
                "Version": "2"
            },
            "ProtectedFromScaleIn": false
        }
    ]
}

Version 2 is the default version Screenshot 2024-09-02 at 17 13 27

Expected Behavior

Either the API call is made to delete the launch template version and the output from documentation is returned or a validation error message is received.

Current Behavior

The CLI version is output and no actions take place or errors are received.

Reproduction Steps

Create a AutoScaling Group via cloudformation with a launch template and a single instance. Update the AutoScaling Group via cloudformation to add additional instances and a user data. This creates multiple versions of the same launch template with different user data. The auto scaling group has both versions in place with one instance using the old version. Set the version 2 to be the default via the CLI. Try to delete version 1 that isn't the default with the command ec2 delete-launch-template-versions.

Possible Solution

No response

Additional Information/Context

No response

CLI version used

2.17.42

Environment details (OS name and version, etc.)

Python/3.11.9 Darwin/22.6.0 source/arm64

adev-code commented 2 weeks ago

Hello RealityCtrl@ thanks for reaching out.

I have tried replicating the steps provided and ran the command: $ aws ec2 delete-launch-template-versions --launch-template-id lt-xxxx --versions 4 --region us-east-1 --debug

The AWS CLI gave a successful output response and the EC2 launch template was deleted.

The steps replicated:

Created anAuto Scaling Group with a launch template and had an instance.
Updated the Auto Scaling Group to add additional instances.
Different launch template versions were created and used on different instances (with launch templates having different user data).
Changed Default template versions.
Deleted versions using the CLI command (versions used or not used by Instances) and gave a response and the templates were deleted.

For further look, please include the full --debug response from the CLI command that you have entered and redacting any sensitive information.

Thank you.

RealityCtrl commented 2 weeks ago

Hi, thanks for looking at this. The output I provided in the issue is the full debug output from the command.

I can see the exact same issue when the command is run on a github action runner, the version of the cli is output instead of any action being taken or an error message returned. So I don't think its down to using homebrew to install the cli.

This is the script that runs in the action runner to set the latest version of the template to the default and then terminate the instance using version 1 before trying to delete version 1.

VERSIONS=$(aws ec2 describe-launch-template-versions --launch-template-name "$TemplateName" --query LaunchTemplateVersions[*].VersionNumber --region "$Region")

echo "Versions found $VERSIONS"

LATEST_VERSION=1
VERSION_ONE=false
for VERSION in $(jq .[] <<< "$VERSIONS");do
    if [[ "$VERSION" -gt "$LATEST_VERSION" ]]
    then
      LATEST_VERSION="$VERSION"
      echo "LATEST_VERSION $LATEST_VERSION"
    fi
    if [[ "$VERSION" -eq "1" ]]; then
        VERSION_ONE=true
         echo "Version one found"
    fi
done

if [[ "$LATEST_VERSION" -gt "1" ]]
  then
    echo "Setting $LATEST_VERSION as the default version for $TemplateName"
    aws ec2 modify-launch-template --launch-template-name "$TemplateName" --default-version "$LATEST_VERSION" --region "$Region"
fi

if [[ "$VERSION_ONE" == "true" ]]
  then
    echo "Version 1 of $TemplateName is in use"

    MasterNodes=$(aws ec2 describe-instances --filters "Name=tag:Role,Values=MasterEligibleNode" "Name=tag:bv:system,Values=manage-content" "Name=tag:DiscoveryTag, Values=$DiscoveryTag" "Name=instance-state-name, Values=running" --query 'Reservations[].Instances[].InstanceId' --region "$Region" )

    for NODE in $(jq .[] <<< "$MasterNodes" | tr -d '"');do
      echo "checking $NODE for template version 1 in use"
      TEMPLATE_VERSION=$(aws autoscaling describe-auto-scaling-instances --instance-ids "$NODE" --query AutoScalingInstances[].LaunchTemplate.Version --region "$Region" | jq '.[0]' | tr -d '"')
      echo "$NODE using template version $TEMPLATE_VERSION"
      if [[ "$TEMPLATE_VERSION" -eq "1" ]]; then
        aws ec2 terminate-instances --instance-ids "$NODE" --region "$Region"
        aws ec2 wait instance-terminated --instance-ids "$NODE" --region "$Region"
      fi
    done
    echo "Deleting version 1 of $TemplateName"
    aws ec2 delete-launch-template-versions --launch-template-name "$TemplateName" --version "1" --region us-east-1
fi

Output of the script

Versions found [
    2,
    1
]
LATEST_VERSION 2
Version one found
Setting 2 as the default version for cert-mc-es-cluster-20240903-MasterEligibleNodeLaunchTemplate
{
    "LaunchTemplate": {
        "LaunchTemplateId": "lt-0f1b804c94b657db8",
        "LaunchTemplateName": "cert-mc-es-cluster-20240903-MasterEligibleNodeLaunchTemplate",
        "CreateTime": "[19](https://github.com/bvengineering/mc-elasticsearch/actions/runs/10684404679/job/29614827357#step:18:20)70-01-01T00:00:00+00:00",
        "CreatedBy": "arn:aws:sts::***:assumed-role/mc-elasticsearch-delete-cluster-runner-qa/kiam-kiam",
        "DefaultVersionNumber": 2,
        "LatestVersionNumber": 2
    }
}
Version 1 of cert-mc-es-cluster-20240903-MasterEligibleNodeLaunchTemplate is in use
checking i-0ab564114a17d92c2 for template version 1 in use
i-0ab564114a17d92c2 using template version 1
{
    "TerminatingInstances": [
        {
            "CurrentState": {
                "Code": 32,
                "Name": "shutting-down"
            },
            "InstanceId": "i-0ab564114a17d92c2",
            "PreviousState": {
                "Code": 16,
                "Name": "running"
            }
        }
    ]
}
checking i-0ee68ee224b3e4915 for template version 1 in use
i-0ee68ee224b3e4915 using template version 2
checking i-04ec2798f2ea84a2f for template version 1 in use
i-04ec2798f2ea84a2f using template version 2
Deleting version 1 of cert-mc-es-cluster-20240903-MasterEligibleNodeLaunchTemplate
aws-cli/2.11.2 Python/3.11.2 Linux/5.15.0-1063-aws exe/x86_64.ubuntu.20 prompt/off

adev-code commented 2 weeks ago

Hello RealityCtrl@ , thanks for the follow up. When running the command : $ aws ec2 delete-launch-template-versions --launch-template-id "lt-xxxx" --version 1 --region us-east-1

The reason it returns the AWS Version is because of the flag "--version". The correct parameter is "--versions". Using "--versions", I was able to delete a launch template version as intended.

Please let me know if you have any follow-up questions.

RealityCtrl commented 2 weeks ago

Thanks, problem entirely between keyboard and chair on my end!

github-actions[bot] commented 2 weeks ago

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.

aws / aws-cli