search

gruntwork-io / fetch

Download files, folders, and release assets from a specific git commit, branch, or tag of public and private GitHub repos.
https://www.gruntwork.io/
MIT License
589 stars 90 forks source link

New version of Fetch fails with Gruntwork Installer Fetch version #59

Closed Merlz closed 4 years ago

Merlz commented 4 years ago

When using the latest version of gruntwork installer v0.0.24, it installs Fetch v0.3.2 as part of its install.

    ubuntu-ami: Installing gruntwork-install...
    ubuntu-ami: Installing fetch version v0.3.2 to /usr/local/bin/fetch
    ubuntu-ami: Downloading https://github.com/gruntwork-io/fetch/releases/download/v0.3.2/fetch_linux_amd64 to /tmp/gruntwork-bootstrap-download-9HOtyS
    ubuntu-ami: Got expected status code 200
    ubuntu-ami: Moving /tmp/gruntwork-bootstrap-download-9HOtyS to /usr/local/bin/fetch
    ubuntu-ami: Installing gruntwork-install version v0.0.24 to /usr/local/bin/gruntwork-install
    ubuntu-ami: Downloading https://raw.githubusercontent.com/gruntwork-io/gruntwork-installer/v0.0.24/gruntwork-install to /tmp/gruntwork-bootstrap-download-mz7gtj
    ubuntu-ami: Got expected status code 200
    ubuntu-ami: Moving /tmp/gruntwork-bootstrap-download-mz7gtj to /usr/local/bin/gruntwork-install
    ubuntu-ami: Creating /etc/user-data as a place to store scripts intended to be run in the User Data of an EC2 instance during boot
    ubuntu-ami: Success!

If I proceed to install the latest version of Fetch v0.3.7 and have additional gruntwork-install --module-name XYZ --repo https://github.com/gruntwork-io/xxxx --tag vN.N.N commands after installing Fetch, then it will fail with the error:

==> ubuntu-ami: ERROR: You specified the --branch flag but did not provide any value.

The above only happens when using --module-name and not --binary-name, binaries like gruntkms, ssh-grunt install fine and then it will fail on installing a module.

As a workaround, I installed Fetch v0.3.7 last in the list of GW modules and then everything works as expected in building the packer image. I require that the latest fetch is installed on the AMI, which is why I'm not leaving v0.3.2 in place.

This is my install-gruntwork-modules.sh that is called from the packer json.

#!/usr/bin/env bash
# Install the Gruntwork modules

set -e

readonly GRUNTWORK_INSTALL_VERSION="v0.0.24"
readonly BASH_COMMONS_VERSION="v0.1.2"
readonly GRUNTKMS_VERSION="v0.0.8"
readonly MODULE_SECURITY_VERSION="v0.27.1"
readonly MODULE_AWS_MONITORING_VERSION="v0.19.0"
readonly MODULE_STATEFUL_SERVER_VERSION="v0.8.1"
readonly METRICS_MODULES="--disk-path=/ --mem-util --mem-used --mem-avail --disk-space-util --auto-scaling"
readonly FETCH_VERSION="v0.3.7"

function get_aws_region {
    curl --silent http://169.254.169.254/latest/dynamic/instance-identity/document | grep region | awk -F\" '{print $4}'
}

function install_bash_commons {
  local -r bash_commons_version="$1"

  echo "Installing bash-commons version $bash_commons_version"
  gruntwork-install --module-name 'bash-commons' --repo https://github.com/gruntwork-io/bash-commons --tag ${bash_commons_version}
}

function install_security_packages {
  local -r kms_version="$1"
  local -r module_security_version="$2"

  echo "Installing Gruntwork Security Modules"
  gruntwork-install --binary-name 'gruntkms' --repo https://github.com/gruntwork-io/gruntkms --tag ${kms_version}
  gruntwork-install --binary-name 'ssh-grunt' --repo https://github.com/gruntwork-io/module-security --tag ${module_security_version}
  gruntwork-install --module-name 'auto-update' --repo https://github.com/gruntwork-io/module-security --tag ${module_security_version}
  gruntwork-install --module-name 'ntp' --repo https://github.com/gruntwork-io/module-security --tag ${module_security_version}
  gruntwork-install --module-name 'ip-lockdown' --repo https://github.com/gruntwork-io/module-security --tag ${module_security_version}
}

function install_monitoring_packages {
  local -r module_aws_monitoring_version="$1"
  local -r metrics_modules="$2"
  local -r aws_region=$(get_aws_region)

  echo "Installing Gruntwork Monitoring Modules"
  gruntwork-install --module-name 'logs/cloudwatch-log-aggregation-scripts' --repo https://github.com/gruntwork-io/module-aws-monitoring --tag ${module_aws_monitoring_version} --module-param aws-region="$aws_region"
  gruntwork-install --module-name 'metrics/cloudwatch-memory-disk-metrics-scripts' --repo https://github.com/gruntwork-io/module-aws-monitoring --tag ${module_aws_monitoring_version}  --module-param "metrics-to-monitor=$metrics_modules"
  gruntwork-install --module-name 'logs/syslog' --repo https://github.com/gruntwork-io/module-aws-monitoring --tag ${module_aws_monitoring_version}
}

function install_stateful_server_packages {
  local -r module_server_version="$1"

  echo "Installing Gruntwork Stateful Server Modules"
  gruntwork-install --module-name 'persistent-ebs-volume' --repo 'https://github.com/gruntwork-io/module-server' --tag ${module_server_version}
  gruntwork-install --module-name 'route53-helpers' --repo 'https://github.com/gruntwork-io/module-server' --tag ${module_server_version}
}

function install_fetch {
  local -r fetch_version="$1"

  echo "Installing Fetch"
  gruntwork-install --binary-name 'fetch' --repo https://github.com/gruntwork-io/fetch --tag ${fetch_version}

}
function assert_env_var_not_empty {
  local -r var_name="$1"
  local -r var_value="${!var_name}"

  if [[ -z "$var_value" ]]; then
    echo "ERROR: Required environment variable $var_name not set."
    print_usage
    exit 1
  fi
}

function install_gruntwork_modules {
  assert_env_var_not_empty "GITHUB_OAUTH_TOKEN"

  curl -LsS https://raw.githubusercontent.com/gruntwork-io/gruntwork-installer/master/bootstrap-gruntwork-installer.sh | bash /dev/stdin --version ${GRUNTWORK_INSTALL_VERSION}

  install_bash_commons ${BASH_COMMONS_VERSION}
  install_fetch ${FETCH_VERSION}
  install_security_packages ${GRUNTKMS_VERSION} ${MODULE_SECURITY_VERSION}
  install_monitoring_packages ${MODULE_AWS_MONITORING_VERSION} ${METRICS_MODULES}
  install_stateful_server_packages ${MODULE_STATEFUL_SERVER_VERSION}
}

install_gruntwork_modules

If the install_fetch ${FETCH_VERSION} is called directly after install_bash_commons ${BASH_COMMONS_VERSION} and prior to other modules, it will fail to install any subsequent modules, but if the install function is called last, after install_stateful_server_packages ${MODULE_STATEFUL_SERVER_VERSION} then it works.

Here is a full error output:

==> ubuntu-ami: Provisioning with shell script: /Users/john/code/infra-live-modules/terraform/aws/services/monitor-pm2/packer/install-gruntwork-modules.sh
    ubuntu-ami: Installing gruntwork-install...
    ubuntu-ami: Installing fetch version v0.3.2 to /usr/local/bin/fetch
    ubuntu-ami: Downloading https://github.com/gruntwork-io/fetch/releases/download/v0.3.2/fetch_linux_amd64 to /tmp/gruntwork-bootstrap-download-9HOtyS
    ubuntu-ami: Got expected status code 200
    ubuntu-ami: Moving /tmp/gruntwork-bootstrap-download-9HOtyS to /usr/local/bin/fetch
    ubuntu-ami: Installing gruntwork-install version v0.0.24 to /usr/local/bin/gruntwork-install
    ubuntu-ami: Downloading https://raw.githubusercontent.com/gruntwork-io/gruntwork-installer/v0.0.24/gruntwork-install to /tmp/gruntwork-bootstrap-download-mz7gtj
    ubuntu-ami: Got expected status code 200
    ubuntu-ami: Moving /tmp/gruntwork-bootstrap-download-mz7gtj to /usr/local/bin/gruntwork-install
    ubuntu-ami: Creating /etc/user-data as a place to store scripts intended to be run in the User Data of an EC2 instance during boot
    ubuntu-ami: Success!
    ubuntu-ami: Installing bash-commons version v0.1.2
==> ubuntu-ami: 2020-03-27 07:46:17 [INFO] [gruntwork-install] Installing from bash-commons...
    ubuntu-ami: Downloading tag "v0.1.2" of https://github.com/gruntwork-io/bash-commons ...
==> ubuntu-ami: 2020-03-27 07:46:17 [INFO] [gruntwork-install] Downloading module bash-commons from https://github.com/gruntwork-io/bash-commons
==> ubuntu-ami: 2020-03-27 07:46:18 [INFO] [gruntwork-install] Executing /tmp/gruntwork-script-modules/bash-commons/install.sh
    ubuntu-ami: Extracting files from <repo>/modules/bash-commons to /tmp/gruntwork-script-modules/bash-commons ...
==> ubuntu-ami: 2020-03-27 07:46:18 [INFO] [install.sh] Starting install of bash-commons...
    ubuntu-ami: Download and file extraction complete.
==> ubuntu-ami: 2020-03-27 07:46:18 [INFO] [install.sh] Successfully installed bash-commons!
    ubuntu-ami: Installing Fetch
==> ubuntu-ami: 2020-03-27 07:46:18 [INFO] [gruntwork-install] Success!
==> ubuntu-ami: 2020-03-27 07:46:18 [INFO] [gruntwork-install] Installing fetch...
    ubuntu-ami: Downloading release asset fetch_linux_amd64 to /tmp/gruntwork-script-modules/fetch_linux_amd64
    ubuntu-ami: Download of release assets complete.
==> ubuntu-ami: 2020-03-27 07:46:18 [INFO] [gruntwork-install] Moving /tmp/gruntwork-script-modules/fetch_linux_amd64 to /usr/local/bin/fetch and setting execute permissions
==> ubuntu-ami: 2020-03-27 07:46:19 [INFO] [gruntwork-install] Success!
    ubuntu-ami: Installing Gruntwork Security Modules
==> ubuntu-ami: 2020-03-27 07:46:19 [INFO] [gruntwork-install] Repository is not public. GITHUB_OAUTH_TOKEN environment variable is required.
==> ubuntu-ami: 2020-03-27 07:46:19 [INFO] [gruntwork-install] Installing gruntkms...
    ubuntu-ami: Downloading release asset gruntkms_linux_amd64 to /tmp/gruntwork-script-modules/gruntkms_linux_amd64
    ubuntu-ami: Downloaded /tmp/gruntwork-script-modules/gruntkms_linux_amd64
    ubuntu-ami: Download of release assets complete
==> ubuntu-ami: 2020-03-27 07:46:19 [INFO] [gruntwork-install] Moving /tmp/gruntwork-script-modules/gruntkms_linux_amd64 to /usr/local/bin/gruntkms and setting execute permissions
==> ubuntu-ami: 2020-03-27 07:46:19 [INFO] [gruntwork-install] Success!
==> ubuntu-ami: 2020-03-27 07:46:19 [INFO] [gruntwork-install] Repository is not public. GITHUB_OAUTH_TOKEN environment variable is required.
==> ubuntu-ami: 2020-03-27 07:46:19 [INFO] [gruntwork-install] Installing ssh-grunt...
    ubuntu-ami: Downloading release asset ssh-grunt_linux_amd64 to /tmp/gruntwork-script-modules/ssh-grunt_linux_amd64
    ubuntu-ami: Downloaded /tmp/gruntwork-script-modules/ssh-grunt_linux_amd64
    ubuntu-ami: Download of release assets complete
==> ubuntu-ami: 2020-03-27 07:46:20 [INFO] [gruntwork-install] Moving /tmp/gruntwork-script-modules/ssh-grunt_linux_amd64 to /usr/local/bin/ssh-grunt and setting execute permissions
==> ubuntu-ami: 2020-03-27 07:46:20 [INFO] [gruntwork-install] Success!
==> ubuntu-ami: 2020-03-27 07:46:20 [INFO] [gruntwork-install] Repository is not public. GITHUB_OAUTH_TOKEN environment variable is required.
==> ubuntu-ami: 2020-03-27 07:46:20 [INFO] [gruntwork-install] Installing from auto-update...
==> ubuntu-ami: 2020-03-27 07:46:20 [INFO] [gruntwork-install] Downloading module auto-update from https://github.com/gruntwork-io/module-security
==> ubuntu-ami: ERROR: You specified the --branch flag but did not provide any value.
==> ubuntu-ami: Terminating the source AWS instance...
brikis98 commented 4 years ago

Perhaps this is some unintended side effect of https://github.com/gruntwork-io/fetch/releases/tag/v0.3.6? Out of curiosity, does it work with v0.3.5?

Merlz commented 4 years ago

Confirmed it works on v0.3.5

brikis98 commented 4 years ago

Thx for checking! Def sounds like some bug was introduced in #55. @vedala Any chance you recognize what the issue is off the top of your head? If not, we'll have to dig in ourselves.

@Merlz Out of curiosity, was there something in 0.3.7 that you needed?

Merlz commented 4 years ago

The --source-path flag is required for pulling certain files from our own internal repos.

vedala commented 4 years ago

@brikis98 The error encountered by @Merlz is definitely due to changes I made in #55.

Based on a comment in gruntwork-install code, looks like gruntwork-install expects fetch to accept empty value for --branch option:

https://github.com/gruntwork-io/gruntwork-installer/blob/d7f6de01d7def76cdb21fc508a43b43e5e3511bc/gruntwork-install#L118-L119

I feel the fix I made in PR #55 is not really needed. Working backwards, I feel the bug reported in issue #31, was not really a bug. If this is needed urgently, I can submit a PR that rolls back the changes. But, to be completely sure, I think we should review the fetch's expected and actual behavior and then decide whether issue #31 should be considered a bug or not.

brikis98 commented 4 years ago

@vedala Thanks for looking into it! Yes, gruntwork-install does expect empty values to work. That way, you can set --branch, --tag, etc to variables that might be empty, and so long as one isn't empty, it should still accept it. Otherwise, we'd have to include lots of conditional logic in the gruntwork-install calls, which would prevent them from being simple one-liners. Perhaps the intended fix for #31 was that if all the values are empty, the error message should disambiguate between the arguments not being set, versus the arguments being set to empty values? @josh-padnick Is that what you had in mind?

vedala commented 4 years ago

@brikis98 It is likely I interpreted the bug incorrectly. Your suggestion sounds right, I will wait for confirmation @josh-padnick and submit a PR with fix.

brikis98 commented 4 years ago

@josh-padnick Could you chime in here?

josh-padnick commented 4 years ago

You know, when I was looking through #31 myself, I had the same thought as @vedala: this isn't really an error. I'm sorry to create extra work for everyone, but I would be in favor of rolling back #55, especially if it's causing regressions.

vedala commented 4 years ago

@josh-padnick No problem. Thank you for confirming. I will roll back my changes made in #55 and send in a PR soon.

brikis98 commented 4 years ago

https://github.com/gruntwork-io/fetch/releases/tag/v0.3.9