Install Script fails in EC2 Image Builder

[brogers-propstream]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• [x] I've gone though the User Guide and the API reference
• [x] I've searched for previous similar issues and didn't find any solution

Describe the bug A clear and concise description of what the bug is.

Invoking the install script on a fresh ubuntu 20.04 base image returns error:

CmdExecution: Stderr: /aws_cli_version_2/aws/install: 64: shift: can't shift that many

For some reason. I cannot find this script anywhere in this repository. Which isn't cool.

So here it is in full:

#!/bin/sh
# Copyright 2012-2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
#     http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

usage() {
  cat 1>&2 <<EOF
Installs the AWS CLI v2

USAGE:
    install [FLAGS] [OPTIONS]

FLAGS:
    -u, --update              Updates the AWS CLI v2 if a different version
                              is previously installed. By default, this script
                              will not update the AWS CLI if a previous
                              installation is detected.

    -h, --help                Prints help information

OPTIONS:
    -i, --install-dir <path>  The directory to install the AWS CLI v2. By
                              default, this directory is: /usr/local/aws-cli

    -b, --bin-dir <path>      The directory to store symlinks to executables
                              for the AWS CLI v2. By default, the directory
                              used is: /usr/local/bin
EOF
}

parse_commandline() {
  while test $# -gt 0
  do
    key="$1"
    case "$key" in
      -i|--install-dir)
        PARSED_INSTALL_DIR="$2"
        shift
       ;;
      -b|--bin-dir)
        PARSED_BIN_DIR="$2"
        shift
       ;;
      -u|--update)
        PARSED_UPGRADE="yes"
      ;;
      -h|--help)
        usage
        exit 0
      ;;
      *)
       die "Got an unexpected argument: $1"
      ;;
    esac
    shift
  done
}

set_global_vars() {
  ROOT_INSTALL_DIR=${PARSED_INSTALL_DIR:-/usr/local/aws-cli}
  BIN_DIR=${PARSED_BIN_DIR:-/usr/local/bin}
  UPGRADE=${PARSED_UPGRADE:-no}

  EXE_NAME="aws"
  COMPLETER_EXE_NAME="aws_completer"
  INSTALLER_DIR="$( cd "$( dirname "$0" )" >/dev/null 2>&1 && pwd )"
  INSTALLER_DIST_DIR="$INSTALLER_DIR/dist"
  INSTALLER_EXE="$INSTALLER_DIST_DIR/$EXE_NAME"
  AWS_EXE_VERSION=$($INSTALLER_EXE --version | cut -d ' ' -f 1 | cut -d '/' -f 2)

  INSTALL_DIR="$ROOT_INSTALL_DIR/v2/$AWS_EXE_VERSION"
  INSTALL_DIR="$INSTALL_DIR"
  INSTALL_DIST_DIR="$INSTALL_DIR/dist"
  INSTALL_BIN_DIR="$INSTALL_DIR/bin"
  INSTALL_AWS_EXE="$INSTALL_BIN_DIR/$EXE_NAME"
  INSTALL_AWS_COMPLETER_EXE="$INSTALL_BIN_DIR/$COMPLETER_EXE_NAME"

  CURRENT_INSTALL_DIR="$ROOT_INSTALL_DIR/v2/current"
  CURRENT_AWS_EXE="$CURRENT_INSTALL_DIR/bin/$EXE_NAME"
  CURRENT_AWS_COMPLETER_EXE="$CURRENT_INSTALL_DIR/bin/$COMPLETER_EXE_NAME"

  BIN_AWS_EXE="$BIN_DIR/$EXE_NAME"
  BIN_AWS_COMPLETER_EXE="$BIN_DIR/$COMPLETER_EXE_NAME"
}

create_install_dir() {
  mkdir -p "$INSTALL_DIR" || exit 1
  {
    setup_install_dist &&
    setup_install_bin &&
    create_current_symlink
  } || {
    rm -rf "$INSTALL_DIR"
    exit 1
  }
}

check_preexisting_install() {
  if [ -L "$CURRENT_INSTALL_DIR" ] && [ "$UPGRADE" = "no" ]
  then
    die "Found preexisting AWS CLI installation: $CURRENT_INSTALL_DIR. Please rerun install script with --update flag."
  fi
  if [ -d "$INSTALL_DIR" ]
  then
    echo "Found same AWS CLI version: $INSTALL_DIR. Skipping install."
    exit 0
  fi
}

setup_install_dist() {
  cp -r "$INSTALLER_DIST_DIR" "$INSTALL_DIST_DIR"
}

setup_install_bin() {
  mkdir -p "$INSTALL_BIN_DIR"
  ln -s "../dist/$EXE_NAME" "$INSTALL_AWS_EXE"
  ln -s "../dist/$COMPLETER_EXE_NAME" "$INSTALL_AWS_COMPLETER_EXE"
}

create_current_symlink() {
  ln -snf "$INSTALL_DIR" "$CURRENT_INSTALL_DIR"
}

create_bin_symlinks() {
  mkdir -p "$BIN_DIR"
  ln -sf "$CURRENT_AWS_EXE" "$BIN_AWS_EXE"
  ln -sf "$CURRENT_AWS_COMPLETER_EXE" "$BIN_AWS_COMPLETER_EXE"
}

die() {
    err_msg="$1"
    echo "$err_msg" >&2
    exit 1
}

main() {
  parse_commandline "$@"
  set_global_vars
  check_preexisting_install
  create_install_dir
  create_bin_symlinks
  echo "You can now run: $BIN_AWS_EXE --version"
  exit 0
}

main "$@" || exit 1

Note - I should not have to get your install script from your already-compile package. It should be HERE.

SDK version number v2 Platform/OS/Hardware/Device Ubuntu 20.04 BASE AMI

To Reproduce (observed behavior)

1. Create an EC2 Image Builder configuration starting with the Ubuntu 20.04 base image, then add the AWS CLI component.
2. Create an Image Builder pipeline with the configuration.
3. Run the pipeline.

Expected behavior I expected the pipeline to install AWSCLI v2.

Logs/output N/A - Unable to install - -debug flag will NOT function. I do have bash logs from cloudwatch though:

1629406547328,ExecuteBash: STARTED EXECUTION
1629406547328,ExecuteBash: Created temporary directory: /tmp/AWSTOE231426636
1629406547328,ExecuteBash: Created temporary script file at: /tmp/AWSTOE231426636/script-652914491.sh
1629406547328,CmdExecution: Preparing object for command bash with arguments [/tmp/AWSTOE231426636/script-652914491.sh]
1629406547328,CmdExecution: Starting execution of command with arguments [bash /tmp/AWSTOE231426636/script-652914491.sh]
1629406547330,Stdout: /aws_cli_version_2
1629406547330,CmdExecution: Terminating read operation on STDOUT pipe - EOF
1629406547330,CmdExecution: Command execution has been completed
1629406547330,CmdExecution: Command execution was completed successfully
1629406547330,CmdExecution: Stderr: 
1629406547330,CmdExecution: ExitCode 0
1629406547330,ExecuteBash: Removing temporary directory: /tmp/AWSTOE231426636
1629406547330,ExecuteBash: Successfully removed temporary directory: /tmp/AWSTOE231426636
1629406547330,ExecuteBash: FINISHED EXECUTION
1629406547337,Step ExtractZipFile
1629406547338,ExecuteBash: STARTED EXECUTION
1629406547338,ExecuteBash: Created temporary directory: /tmp/AWSTOE946424670
1629406547339,ExecuteBash: Created temporary script file at: /tmp/AWSTOE946424670/script-152237349.sh
1629406547339,CmdExecution: Preparing object for command bash with arguments [/tmp/AWSTOE946424670/script-152237349.sh]
1629406547339,CmdExecution: Starting execution of command with arguments [bash /tmp/AWSTOE946424670/script-152237349.sh]
1629406547347,Stdout: Extracting '/awscliv2.zip' to '/aws_cli_version_2'
1629406548585,CmdExecution: Terminating read operation on STDOUT pipe - EOF
1629406548585,CmdExecution: Command execution has been completed
1629406548585,CmdExecution: Command execution was completed successfully
1629406548585,CmdExecution: Stderr: 
1629406548585,CmdExecution: ExitCode 0
1629406548585,ExecuteBash: Removing temporary directory: /tmp/AWSTOE946424670
1629406548585,ExecuteBash: Successfully removed temporary directory: /tmp/AWSTOE946424670
1629406548585,ExecuteBash: FINISHED EXECUTION
1629406548601,Step Install
1629406548603,ExecuteBash: STARTED EXECUTION
1629406548603,ExecuteBash: Created temporary directory: /tmp/AWSTOE651285056
1629406548603,ExecuteBash: Created temporary script file at: /tmp/AWSTOE651285056/script-835495071.sh
1629406548603,CmdExecution: Preparing object for command bash with arguments [/tmp/AWSTOE651285056/script-835495071.sh]
1629406548604,CmdExecution: Starting execution of command with arguments [bash /tmp/AWSTOE651285056/script-835495071.sh]
1629406548606,Stdout: Invoking '/aws_cli_version_2/aws/install'
1629406548612,CmdExecution: Terminating read operation on STDOUT pipe - EOF
1629406548612,CmdExecution: Command execution has been completed
1629406548612,CmdExecution: [ ERROR ] Command execution has resulted in an error
1629406548612,"CmdExecution: Stderr: /aws_cli_version_2/aws/install: 64: shift: can't shift that many
"
1629406548612,CmdExecution: ExitCode 2
1629406548612,ExecuteBash: Removing temporary directory: /tmp/AWSTOE651285056
1629406548613,ExecuteBash: Successfully removed temporary directory: /tmp/AWSTOE651285056
1629406548613,ExecuteBash: FINISHED EXECUTION
1629406549627,TOE has completed execution with failure - Execution failed!

Additional context Add any other context about the problem here.

[brogers-propstream]

Posted this in the forum as well - https://forums.aws.amazon.com/thread.jspa?threadID=344876

[brogers-propstream]

Update:

I found a workaround by creating a NEW component from the Amazon owned one and removing the -i $HOME from the install command.

This is a really really really silly default install directory to begin with, but it appears that $HOME isn't even resolvable... C'mon Amazon! This is YOUR STUFF. Take care of it.

Right now nothing works as advertised here and it's not nearly as easy as the documentation suggests.

[kdaily]

Hi @brogers-propstream,

Sorry to hear you're having trouble, I'm looking into this.

The installation file you're looking for is in this repository, on the v2 branch here: https://github.com/aws/aws-cli/blob/v2/exe/assets/install

~I think this is a AWS CLI v2 specific thing based on how PyInstaller (the tool used to build the bundle) works and it's access to directories that are read only.~ I was able to successfully install directly on an Ubuntu 20.04 image, I'm trying to reproduce with EC2 Image Builder now.

Update: I reproduced with EC2 Image Builder as well, and see the same error, and agree with your assessment of the cause. The default directory, if -i is not supplied, is /usr/local/aws-cli, as documented here:

https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html#cliv2-linux-install

I will immediately raise this with the EC2 Image Builder team.

[kdaily]

@brogers-propstream,

One clarifying question to followup separately. Using the AWS CLI v2 component out of the box did not work for me initially - I had to apt-get install unzip as well in my user data. Did you also need to do this?

[kdaily]

Hi @brogers-propstream,

It looks like this component was just updated (2021-08-18). The EC2 team is aware of the issue and is working on a fix.

[easimon]

There are two versions of the Image builder component, 1.0.0 and 1.0.1.

1.0.0 seems to work for me, only 1.0.1 fails.

I did not analyze this completely, but 1.0.1 uses $HOME in some places, which evaluates to nothing during execution of the component. 1.0.0 does not. So until this is fixed, try using 1.0.0

Both component versions require unzip, and you need to install it yourself before executing the AWS cli installer component.

[kdaily]

Thanks @easimon, I confirm that 1.0.0 works for me as well. Waiting for an update on the fix still.

[tim-finnigan]

I believe the fix has now been implemented and we haven't received any additional reports of this so I am going to close the issue. If anyone's still encountering this error please let us know here or create a new issue. I'll also link this troubleshooting guide on EC2 Image Builder in case that helps anyone. Thanks!

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.