Enable automatic checks for the correctness of our shell scripts

[arielsvn]

Context

Since we have started to add automatic checks for our python code I think it would also make sense to check the script files as well.

Problem or idea

I found shellcheck which looks like it's widely used. I ran it on all the scripts and got a few warnings that look relevant, example:

In ./infrastructure/deploy.sh line 272:
       $DOCKERHUB_REPO/$FOREMAN_DOCKER_IMAGE python3 manage.py migrate
       ^-------------^ SC2086: Double quote to prevent globbing and word splitting.
                       ^-------------------^ SC2086: Double quote to prevent globbing and word splitting.

In ./infrastructure/nomad-configuration/client-instance-user-data.tpl.sh line 50:
        if echo "$EBS_VOLUME_INDEX" | egrep -q '^\-?[0-9]+$'; then
                                      ^---^ SC2196: egrep is non-standard and deprecated. Use grep -E instead.

In ./infrastructure/nomad-configuration/server-instance-user-data.tpl.sh line 18:
cd /home/ubuntu
^-------------^ SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

In ./workers/tester.sh line 90:
export AWS_SECRET_ACCESS_KEY=`~/bin/aws configure get default.aws_secret_access_key`
       ^-------------------^ SC2155: Declare and assign separately to avoid masking return values.
                             ^-- SC2006: Use $(...) notation instead of legacy backticked `...`.

Full output from `shellcheck ./**/*.sh -e SC2154`

``` In ./infrastructure/api-configuration/api-server-instance-user-data.tpl.sh line 13: cd /home/ubuntu ^-------------^ SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails. Did you mean: cd /home/ubuntu || exit In ./infrastructure/api-configuration/api-server-instance-user-data.tpl.sh line 72: python ./awslogs-agent-setup.py --region ${region} --non-interactive --configfile awslogs.conf ^-------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: python ./awslogs-agent-setup.py --region "${region}" --non-interactive --configfile awslogs.conf In ./infrastructure/api-configuration/api-server-instance-user-data.tpl.sh line 105: docker pull ${dockerhub_repo}/${api_docker_image} ^---------------^ SC2086: Double quote to prevent globbing and word splitting. ^-----------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: docker pull "${dockerhub_repo}"/"${api_docker_image}" In ./infrastructure/api-configuration/api-server-instance-user-data.tpl.sh line 111: -e DATABASE_HOST=${database_host} \ ^--------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: -e DATABASE_HOST="${database_host}" \ In ./infrastructure/api-configuration/api-server-instance-user-data.tpl.sh line 112: -e DATABASE_NAME=${database_name} \ ^--------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: -e DATABASE_NAME="${database_name}" \ In ./infrastructure/api-configuration/api-server-instance-user-data.tpl.sh line 113: -e DATABASE_USER=${database_user} \ ^--------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: -e DATABASE_USER="${database_user}" \ In ./infrastructure/api-configuration/api-server-instance-user-data.tpl.sh line 114: -e DATABASE_PASSWORD=${database_password} \ ^------------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: -e DATABASE_PASSWORD="${database_password}" \ In ./infrastructure/api-configuration/api-server-instance-user-data.tpl.sh line 115: -e ELASTICSEARCH_HOST=${elasticsearch_host} \ ^-------------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: -e ELASTICSEARCH_HOST="${elasticsearch_host}" \ In ./infrastructure/api-configuration/api-server-instance-user-data.tpl.sh line 116: -e ELASTICSEARCH_PORT=${elasticsearch_port} \ ^-------------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: -e ELASTICSEARCH_PORT="${elasticsearch_port}" \ In ./infrastructure/api-configuration/api-server-instance-user-data.tpl.sh line 119: --log-opt awslogs-region=${region} \ ^-------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: --log-opt awslogs-region="${region}" \ In ./infrastructure/api-configuration/api-server-instance-user-data.tpl.sh line 120: --log-opt awslogs-group=${log_group} \ ^----------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: --log-opt awslogs-group="${log_group}" \ In ./infrastructure/api-configuration/api-server-instance-user-data.tpl.sh line 121: --log-opt awslogs-stream=${log_stream} \ ^-----------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: --log-opt awslogs-stream="${log_stream}" \ In ./infrastructure/api-configuration/api-server-instance-user-data.tpl.sh line 124: -it -d ${dockerhub_repo}/${api_docker_image} /bin/sh -c "/home/user/collect_and_run_uwsgi.sh" ^---------------^ SC2086: Double quote to prevent globbing and word splitting. ^-----------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: -it -d "${dockerhub_repo}"/"${api_docker_image}" /bin/sh -c "/home/user/collect_and_run_uwsgi.sh" In ./infrastructure/cluster_connect.sh line 6: if [ $(cssh -v > /dev/null 2>&1; echo $?) == 0 ]; then ^-- SC2046: Quote this to prevent word splitting. In ./infrastructure/cluster_connect.sh line 7: cssh --options "-i data-refinery-key.pem" --username ubuntu $( /dev/null 2>&1; echo $?) == 2 ]; then # MacOS, csshX has an exit code of 2 for -v... ^-- SC2046: Quote this to prevent word splitting. In ./infrastructure/cluster_connect.sh line 12: echo 'cssh can be installed with: `sudo apt install clusterssh`' ^-- SC2016: Expressions don't expand in single quotes, use double quotes for that. In ./infrastructure/cluster_connect.sh line 13: echo 'csshX can be installed with: `brew install csshX`' ^-- SC2016: Expressions don't expand in single quotes, use double quotes for that. In ./infrastructure/deploy.sh line 4: echo 'This script can be used to deploy and update a `refine.bio` instance stack.' ^-- SC2016: Expressions don't expand in single quotes, use double quotes for that. In ./infrastructure/deploy.sh line 98: echo $(curl --write-out %{http_code} \ ^-- SC2046: Quote this to prevent word splitting. ^-- SC2005: Useless echo? Instead of 'echo $(cmd)', just use 'cmd'. ^-- SC1083: This { is literal. Check expression (missing ;/\n?) or quote it. ^-- SC1083: This } is literal. Check expression (missing ;/\n?) or quote it. In ./infrastructure/deploy.sh line 101: $NOMAD_ADDR/v1/status/leader) ^---------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: "$NOMAD_ADDR"/v1/status/leader) In ./infrastructure/deploy.sh line 110: env_var_assignment=$(echo $row | jq -r ".name")=$(echo $row | jq -r ".value") ^--^ SC2086: Double quote to prevent globbing and word splitting. ^--^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: env_var_assignment=$(echo "$row" | jq -r ".name")=$(echo "$row" | jq -r ".value") In ./infrastructure/deploy.sh line 111: export $env_var_assignment ^-----------------^ SC2163: This does not export 'env_var_assignment'. Remove $/${} for that, or use ${var?} to quiet. ^-----------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: export "$env_var_assignment" In ./infrastructure/deploy.sh line 112: echo $env_var_assignment >> prod_env ^-----------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: echo "$env_var_assignment" >> prod_env In ./infrastructure/deploy.sh line 117: source ../scripts/common.sh ^------------------^ SC1091: Not following: ../scripts/common.sh was not specified as input (see shellcheck -x). In ./infrastructure/deploy.sh line 119: export TF_VAR_host_ip=`dig +short myip.opendns.com @resolver1.opendns.com` ^------------^ SC2155: Declare and assign separately to avoid masking return values. ^-- SC2006: Use $(...) notation instead of legacy backticked `...`. Did you mean: export TF_VAR_host_ip=$(dig +short myip.opendns.com @resolver1.opendns.com) In ./infrastructure/deploy.sh line 125: IMAGE_UPPER=$IMAGE | tr a-z A-Z ^---------^ SC2036: If you wanted to assign the output of the pipeline, use a=$(b | c) . ^---------^ SC2030: Modification of IMAGE_UPPER is local (to subshell caused by pipeline). ^-^ SC2018: Use '[:lower:]' to support accents and foreign alphabets. ^-^ SC2019: Use '[:upper:]' to support accents and foreign alphabets. In ./infrastructure/deploy.sh line 126: export ${IMAGE_UPPER}_DOCKER_IMAGE=dr_$IMAGE:$SYSTEM_VERSION ^------------^ SC2031: IMAGE_UPPER was modified in a subshell. That change might be lost. ^------------^ SC2086: Double quote to prevent globbing and word splitting. ^----^ SC2086: Double quote to prevent globbing and word splitting. ^-------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: export "${IMAGE_UPPER}"_DOCKER_IMAGE=dr_"$IMAGE":"$SYSTEM_VERSION" In ./infrastructure/deploy.sh line 127: export TF_VAR_${IMAGE}_docker_image=dr_$IMAGE:$SYSTEM_VERSION ^------^ SC2086: Double quote to prevent globbing and word splitting. ^----^ SC2086: Double quote to prevent globbing and word splitting. ^-------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: export TF_VAR_"${IMAGE}"_docker_image=dr_"$IMAGE":"$SYSTEM_VERSION" In ./infrastructure/deploy.sh line 150: if [[ ! -z $CIRCLE_BUILD_NUM ]]; then ^-- SC2236: Use -n instead of ! -z. In ./infrastructure/deploy.sh line 152: terraform apply -var-file=environments/$env.tfvars -auto-approve > /dev/null ^--^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: terraform apply -var-file=environments/"$env".tfvars -auto-approve > /dev/null In ./infrastructure/deploy.sh line 154: terraform apply -var-file=environments/$env.tfvars -auto-approve ^--^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: terraform apply -var-file=environments/"$env".tfvars -auto-approve In ./infrastructure/deploy.sh line 162: ../scripts/format_nomad_with_env.sh -p api -e $env -o $(pwd)/api-configuration/ ^--^ SC2086: Double quote to prevent globbing and word splitting. ^----^ SC2046: Quote this to prevent word splitting. Did you mean: ../scripts/format_nomad_with_env.sh -p api -e "$env" -o $(pwd)/api-configuration/ In ./infrastructure/deploy.sh line 163: ../scripts/format_nomad_with_env.sh -p foreman -e $env -o $(pwd)/foreman-configuration/ ^--^ SC2086: Double quote to prevent globbing and word splitting. ^----^ SC2046: Quote this to prevent word splitting. Did you mean: ../scripts/format_nomad_with_env.sh -p foreman -e "$env" -o $(pwd)/foreman-configuration/ In ./infrastructure/deploy.sh line 173: if [[ ! -z $CIRCLE_BUILD_NUM ]]; then ^-- SC2236: Use -n instead of ! -z. In ./infrastructure/deploy.sh line 175: terraform apply -var-file=environments/$env.tfvars -auto-approve > /dev/null ^--^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: terraform apply -var-file=environments/"$env".tfvars -auto-approve > /dev/null In ./infrastructure/deploy.sh line 177: terraform apply -var-file=environments/$env.tfvars -auto-approve ^--^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: terraform apply -var-file=environments/"$env".tfvars -auto-approve In ./infrastructure/deploy.sh line 182: export NOMAD_LEAD_SERVER_IP=`terraform output nomad_server_1_ip` ^------------------^ SC2155: Declare and assign separately to avoid masking return values. ^-- SC2006: Use $(...) notation instead of legacy backticked `...`. Did you mean: export NOMAD_LEAD_SERVER_IP=$(terraform output nomad_server_1_ip) In ./infrastructure/deploy.sh line 192: while [ "$diff" -lt "900" -a $nomad_status != "200" ]; do ^-- SC2166: Prefer [ p ] && [ q ] as [ p -a q ] is not well defined. ^-----------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: while [ "$diff" -lt "900" -a "$nomad_status" != "200" ]; do In ./infrastructure/deploy.sh line 195: let "diff = $(date +%s) - $start_time" ^-- SC2219: Instead of 'let expr', prefer (( expr )) . In ./infrastructure/deploy.sh line 207: for job in $(nomad status | grep running | awk {'print $1'} || grep --invert-match /) ^-- SC1083: This { is literal. Check expression (missing ;/\n?) or quote it. ^-- SC1083: This } is literal. Check expression (missing ;/\n?) or quote it. In ./infrastructure/deploy.sh line 211: nomad stop -purge -detach $job > /dev/null || true & ^--^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: nomad stop -purge -detach "$job" > /dev/null || true & In ./infrastructure/deploy.sh line 217: wait $(jobs -p) ^--------^ SC2046: Quote this to prevent word splitting. In ./infrastructure/deploy.sh line 224: for job in $(nomad status | awk {'print $1'} || grep /) ^-- SC1083: This { is literal. Check expression (missing ;/\n?) or quote it. ^-- SC1083: This } is literal. Check expression (missing ;/\n?) or quote it. In ./infrastructure/deploy.sh line 227: if [ $job != "ID" ]; then ^--^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: if [ "$job" != "ID" ]; then In ./infrastructure/deploy.sh line 230: nomad stop -purge -detach $job > /dev/null || true & ^--^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: nomad stop -purge -detach "$job" > /dev/null || true & In ./infrastructure/deploy.sh line 237: wait $(jobs -p) ^--------^ SC2046: Quote this to prevent word splitting. In ./infrastructure/deploy.sh line 244: wait $(jobs -p) ^--------^ SC2046: Quote this to prevent word splitting. In ./infrastructure/deploy.sh line 255: docker pull $DOCKERHUB_REPO/$FOREMAN_DOCKER_IMAGE ^-------------^ SC2086: Double quote to prevent globbing and word splitting. ^-------------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: docker pull "$DOCKERHUB_REPO"/"$FOREMAN_DOCKER_IMAGE" In ./infrastructure/deploy.sh line 261: --env DATABASE_HOST=$RDS_HOST \ ^-------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: --env DATABASE_HOST="$RDS_HOST" \ In ./infrastructure/deploy.sh line 262: --env DATABASE_PORT=$DATABASE_HIDDEN_PORT \ ^-------------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: --env DATABASE_PORT="$DATABASE_HIDDEN_PORT" \ In ./infrastructure/deploy.sh line 264: $DOCKERHUB_REPO/$FOREMAN_DOCKER_IMAGE python3 manage.py migrate auth ^-------------^ SC2086: Double quote to prevent globbing and word splitting. ^-------------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: "$DOCKERHUB_REPO"/"$FOREMAN_DOCKER_IMAGE" python3 manage.py migrate auth In ./infrastructure/deploy.sh line 269: --env DATABASE_HOST=$RDS_HOST \ ^-------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: --env DATABASE_HOST="$RDS_HOST" \ In ./infrastructure/deploy.sh line 270: --env DATABASE_PORT=$DATABASE_HIDDEN_PORT \ ^-------------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: --env DATABASE_PORT="$DATABASE_HIDDEN_PORT" \ In ./infrastructure/deploy.sh line 272: $DOCKERHUB_REPO/$FOREMAN_DOCKER_IMAGE python3 manage.py migrate ^-------------^ SC2086: Double quote to prevent globbing and word splitting. ^-------------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: "$DOCKERHUB_REPO"/"$FOREMAN_DOCKER_IMAGE" python3 manage.py migrate In ./infrastructure/deploy.sh line 277: --env DATABASE_HOST=$RDS_HOST \ ^-------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: --env DATABASE_HOST="$RDS_HOST" \ In ./infrastructure/deploy.sh line 278: --env DATABASE_PORT=$DATABASE_HIDDEN_PORT \ ^-------------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: --env DATABASE_PORT="$DATABASE_HIDDEN_PORT" \ In ./infrastructure/deploy.sh line 280: $DOCKERHUB_REPO/$FOREMAN_DOCKER_IMAGE python3 manage.py createcachetable ^-------------^ SC2086: Double quote to prevent globbing and word splitting. ^-------------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: "$DOCKERHUB_REPO"/"$FOREMAN_DOCKER_IMAGE" python3 manage.py createcachetable In ./infrastructure/deploy.sh line 289: ../scripts/format_nomad_with_env.sh -p workers -e $env -o $(pwd)/nomad-job-specs ^--^ SC2086: Double quote to prevent globbing and word splitting. ^----^ SC2046: Quote this to prevent word splitting. Did you mean: ../scripts/format_nomad_with_env.sh -p workers -e "$env" -o $(pwd)/nomad-job-specs In ./infrastructure/deploy.sh line 290: ../scripts/format_nomad_with_env.sh -p surveyor -e $env -o $(pwd)/nomad-job-specs ^--^ SC2086: Double quote to prevent globbing and word splitting. ^----^ SC2046: Quote this to prevent word splitting. Did you mean: ../scripts/format_nomad_with_env.sh -p surveyor -e "$env" -o $(pwd)/nomad-job-specs In ./infrastructure/deploy.sh line 293: ../scripts/format_nomad_with_env.sh -p foreman -e $env -o $(pwd)/foreman-configuration ^--^ SC2086: Double quote to prevent globbing and word splitting. ^----^ SC2046: Quote this to prevent word splitting. Did you mean: ../scripts/format_nomad_with_env.sh -p foreman -e "$env" -o $(pwd)/foreman-configuration In ./infrastructure/deploy.sh line 294: ../scripts/format_nomad_with_env.sh -p api -e $env -o $(pwd)/api-configuration/ ^--^ SC2086: Double quote to prevent globbing and word splitting. ^----^ SC2046: Quote this to prevent word splitting. Did you mean: ../scripts/format_nomad_with_env.sh -p api -e "$env" -o $(pwd)/api-configuration/ In ./infrastructure/deploy.sh line 302: nomad_job_specs=nomad-job-specs/* ^---------------^ SC2125: Brace expansions and globs are literal in assignments. Quote it or use an array. In ./infrastructure/deploy.sh line 304: nomad run $nomad_job_spec & ^-------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: nomad run "$nomad_job_spec" & In ./infrastructure/deploy.sh line 316: if [[ ! -z $CIRCLE_BUILD_NUM ]]; then ^-- SC2236: Use -n instead of ! -z. In ./infrastructure/deploy.sh line 318: terraform apply -var-file=environments/$env.tfvars -auto-approve > /dev/null ^--^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: terraform apply -var-file=environments/"$env".tfvars -auto-approve > /dev/null In ./infrastructure/deploy.sh line 320: terraform apply -var-file=environments/$env.tfvars -auto-approve ^--^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: terraform apply -var-file=environments/"$env".tfvars -auto-approve In ./infrastructure/deploy.sh line 337: ubuntu@$API_IP_ADDRESS "docker ps -a" | grep dr_api || echo "") ^-------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: ubuntu@"$API_IP_ADDRESS" "docker ps -a" | grep dr_api || echo "") In ./infrastructure/deploy.sh line 344: if [[ ! -z $container_running ]]; then ^-- SC2236: Use -n instead of ! -z. In ./infrastructure/deploy.sh line 351: ubuntu@$API_IP_ADDRESS "docker pull $DOCKERHUB_REPO/$API_DOCKER_IMAGE" ^-------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: ubuntu@"$API_IP_ADDRESS" "docker pull $DOCKERHUB_REPO/$API_DOCKER_IMAGE" In ./infrastructure/deploy.sh line 357: ubuntu@$API_IP_ADDRESS "docker rm -f dr_api" ^-------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: ubuntu@"$API_IP_ADDRESS" "docker rm -f dr_api" In ./infrastructure/deploy.sh line 363: api-configuration/environment ubuntu@$API_IP_ADDRESS:/home/ubuntu/environment ^-------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: api-configuration/environment ubuntu@"$API_IP_ADDRESS":/home/ubuntu/environment In ./infrastructure/deploy.sh line 369: ubuntu@$API_IP_ADDRESS "docker run \ ^-------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: ubuntu@"$API_IP_ADDRESS" "docker run \ In ./infrastructure/deploy.sh line 390: ubuntu@$API_IP_ADDRESS "docker run \ ^-------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: ubuntu@"$API_IP_ADDRESS" "docker run \ In ./infrastructure/deploy.sh line 411: ubuntu@$API_IP_ADDRESS "rm -f environment" ^-------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: ubuntu@"$API_IP_ADDRESS" "rm -f environment" In ./infrastructure/foreman-configuration/foreman-server-instance-user-data.tpl.sh line 13: cd /home/ubuntu ^-------------^ SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails. Did you mean: cd /home/ubuntu || exit In ./infrastructure/foreman-configuration/foreman-server-instance-user-data.tpl.sh line 74: echo ' ^-- SC2016: Expressions don't expand in single quotes, use double quotes for that. In ./infrastructure/foreman-configuration/foreman-server-instance-user-data.tpl.sh line 98: -e DATABASE_HOST=${database_host} \ ^--------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: -e DATABASE_HOST="${database_host}" \ In ./infrastructure/foreman-configuration/foreman-server-instance-user-data.tpl.sh line 99: -e DATABASE_NAME=${database_name} \ ^--------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: -e DATABASE_NAME="${database_name}" \ In ./infrastructure/foreman-configuration/foreman-server-instance-user-data.tpl.sh line 100: -e DATABASE_USER=${database_user} \ ^--------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: -e DATABASE_USER="${database_user}" \ In ./infrastructure/foreman-configuration/foreman-server-instance-user-data.tpl.sh line 101: -e DATABASE_PASSWORD=${database_password} \ ^------------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: -e DATABASE_PASSWORD="${database_password}" \ In ./infrastructure/foreman-configuration/foreman-server-instance-user-data.tpl.sh line 103: --add-host=nomad:${nomad_lead_server_ip} \ ^---------------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: --add-host=nomad:"${nomad_lead_server_ip}" \ In ./infrastructure/foreman-configuration/foreman-server-instance-user-data.tpl.sh line 105: --log-opt awslogs-region=${region} \ ^-------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: --log-opt awslogs-region="${region}" \ In ./infrastructure/foreman-configuration/foreman-server-instance-user-data.tpl.sh line 106: --log-opt awslogs-group=${log_group} \ ^----------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: --log-opt awslogs-group="${log_group}" \ In ./infrastructure/foreman-configuration/foreman-server-instance-user-data.tpl.sh line 107: --log-opt awslogs-stream=log-stream-foreman-${user}-${stage} \ ^-----^ SC2086: Double quote to prevent globbing and word splitting. ^------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: --log-opt awslogs-stream=log-stream-foreman-"${user}"-"${stage}" \ In ./infrastructure/foreman-configuration/foreman-server-instance-user-data.tpl.sh line 109: -it -d ${dockerhub_repo}/${foreman_docker_image} python3 manage.py create_missing_processor_jobs ^---------------^ SC2086: Double quote to prevent globbing and word splitting. ^---------------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: -it -d "${dockerhub_repo}"/"${foreman_docker_image}" python3 manage.py create_missing_processor_jobs In ./infrastructure/init_terraform.sh line 3: if [ -z $TF_VAR_user ]; then ^----------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: if [ -z "$TF_VAR_user" ]; then In ./infrastructure/init_terraform.sh line 4: echo 'You must set the $TF_VAR_user environment variable!' >&2 ^-- SC2016: Expressions don't expand in single quotes, use double quotes for that. In ./infrastructure/init_terraform.sh line 8: if [ -z $TF_VAR_stage ]; then ^-----------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: if [ -z "$TF_VAR_stage" ]; then In ./infrastructure/nomad-configuration/client-instance-user-data.tpl.sh line 32: EBS_VOLUME_ID=`aws ec2 describe-volumes --filters "Name=tag:User,Values=$1" "Name=tag:Stage,Values=$2" "Name=tag:IsBig,Values=True" "Name=tag:Index,Values=0" "Name=status,Values=available" "Name=availability-zone,Values=us-east-1a" --region us-east-1 | jq '.Volumes[0].VolumeId' | tr -d '"'` ^-- SC2006: Use $(...) notation instead of legacy backticked `...`. Did you mean: EBS_VOLUME_ID=$(aws ec2 describe-volumes --filters "Name=tag:User,Values=$1" "Name=tag:Stage,Values=$2" "Name=tag:IsBig,Values=True" "Name=tag:Index,Values=0" "Name=status,Values=available" "Name=availability-zone,Values=us-east-1a" --region us-east-1 | jq '.Volumes[0].VolumeId' | tr -d '"') In ./infrastructure/nomad-configuration/client-instance-user-data.tpl.sh line 34: EBS_VOLUME_ID=`aws ec2 describe-volumes --filters "Name=tag:User,Values=$1" "Name=tag:Stage,Values=$2" "Name=tag:IsBig,Values=True" "Name=status,Values=available" "Name=availability-zone,Values=us-east-1a" --region us-east-1 | jq '.Volumes[0].VolumeId' | tr -d '"'` ^-- SC2006: Use $(...) notation instead of legacy backticked `...`. Did you mean: EBS_VOLUME_ID=$(aws ec2 describe-volumes --filters "Name=tag:User,Values=$1" "Name=tag:Stage,Values=$2" "Name=tag:IsBig,Values=True" "Name=status,Values=available" "Name=availability-zone,Values=us-east-1a" --region us-east-1 | jq '.Volumes[0].VolumeId' | tr -d '"') In ./infrastructure/nomad-configuration/client-instance-user-data.tpl.sh line 37: aws ec2 attach-volume --volume-id $EBS_VOLUME_ID --instance-id $INSTANCE_ID --device "/dev/sdf" --region ${region} ^------------^ SC2086: Double quote to prevent globbing and word splitting. ^----------^ SC2086: Double quote to prevent globbing and word splitting. ^-------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: aws ec2 attach-volume --volume-id "$EBS_VOLUME_ID" --instance-id "$INSTANCE_ID" --device "/dev/sdf" --region "${region}" In ./infrastructure/nomad-configuration/client-instance-user-data.tpl.sh line 49: EBS_VOLUME_INDEX=`aws ec2 describe-volumes --filters "Name=tag:Index,Values=*" "Name=volume-id,Values=$EBS_VOLUME_ID" --query "Volumes[*].{ID:VolumeId,Tag:Tags}" --region ${region} | jq ".[0].Tag[$COUNTER].Value" | tr -d '"'` ^-- SC2006: Use $(...) notation instead of legacy backticked `...`. ^-------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: EBS_VOLUME_INDEX=$(aws ec2 describe-volumes --filters "Name=tag:Index,Values=*" "Name=volume-id,Values=$EBS_VOLUME_ID" --query "Volumes[*].{ID:VolumeId,Tag:Tags}" --region "${region}" | jq ".[0].Tag[$COUNTER].Value" | tr -d '"') In ./infrastructure/nomad-configuration/client-instance-user-data.tpl.sh line 50: if echo "$EBS_VOLUME_INDEX" | egrep -q '^\-?[0-9]+$'; then ^---^ SC2196: egrep is non-standard and deprecated. Use grep -E instead. In ./infrastructure/nomad-configuration/client-instance-user-data.tpl.sh line 56: let COUNTER=COUNTER+1 ^-------------------^ SC2219: Instead of 'let expr', prefer (( expr )) . In ./infrastructure/nomad-configuration/client-instance-user-data.tpl.sh line 60: ATTACHED_AS=`lsblk -n | grep 8.8T | cut -d' ' -f1` ^-- SC2006: Use $(...) notation instead of legacy backticked `...`. Did you mean: ATTACHED_AS=$(lsblk -n | grep 8.8T | cut -d' ' -f1) In ./infrastructure/nomad-configuration/client-instance-user-data.tpl.sh line 61: FILE_RESULT=`file -s /dev/$ATTACHED_AS` ^---------^ SC2034: FILE_RESULT appears unused. Verify use (or export if used externally). ^-------------------------^ SC2006: Use $(...) notation instead of legacy backticked `...`. ^----------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: FILE_RESULT=$(file -s /dev/"$ATTACHED_AS") In ./infrastructure/nomad-configuration/client-instance-user-data.tpl.sh line 64: if file -s /dev/$ATTACHED_AS | grep data | grep -v ext4; then ^----------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: if file -s /dev/"$ATTACHED_AS" | grep data | grep -v ext4; then In ./infrastructure/nomad-configuration/client-instance-user-data.tpl.sh line 65: mkfs -t ext4 /dev/$ATTACHED_AS # This is slow ^----------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: mkfs -t ext4 /dev/"$ATTACHED_AS" # This is slow In ./infrastructure/nomad-configuration/client-instance-user-data.tpl.sh line 67: mount /dev/$ATTACHED_AS /var/ebs/ ^----------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: mount /dev/"$ATTACHED_AS" /var/ebs/ In ./infrastructure/nomad-configuration/client-instance-user-data.tpl.sh line 70: echo $EBS_VOLUME_INDEX > /var/ebs/VOLUME_INDEX ^---------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: echo "$EBS_VOLUME_INDEX" > /var/ebs/VOLUME_INDEX In ./infrastructure/nomad-configuration/client-instance-user-data.tpl.sh line 76: PGPASSWORD=${database_password} psql -c 'CREATE EXTENSION IF NOT EXISTS hstore;' -h ${database_host} -p 5432 -U ${database_user} -d ${database_name} ^--------------^ SC2086: Double quote to prevent globbing and word splitting. ^--------------^ SC2086: Double quote to prevent globbing and word splitting. ^--------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: PGPASSWORD=${database_password} psql -c 'CREATE EXTENSION IF NOT EXISTS hstore;' -h "${database_host}" -p 5432 -U "${database_user}" -d "${database_name}" In ./infrastructure/nomad-configuration/client-instance-user-data.tpl.sh line 79: cd /home/ubuntu ^-------------^ SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails. Did you mean: cd /home/ubuntu || exit In ./infrastructure/nomad-configuration/client-instance-user-data.tpl.sh line 89: python ./awslogs-agent-setup.py --region ${region} --non-interactive --configfile awslogs.conf ^-------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: python ./awslogs-agent-setup.py --region "${region}" --non-interactive --configfile awslogs.conf In ./infrastructure/nomad-configuration/client-smasher-instance-user-data.tpl.sh line 30: PGPASSWORD=${database_password} psql -c 'CREATE EXTENSION IF NOT EXISTS hstore;' -h ${database_host} -p 5432 -U ${database_user} -d ${database_name} ^--------------^ SC2086: Double quote to prevent globbing and word splitting. ^--------------^ SC2086: Double quote to prevent globbing and word splitting. ^--------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: PGPASSWORD=${database_password} psql -c 'CREATE EXTENSION IF NOT EXISTS hstore;' -h "${database_host}" -p 5432 -U "${database_user}" -d "${database_name}" In ./infrastructure/nomad-configuration/client-smasher-instance-user-data.tpl.sh line 33: cd /home/ubuntu ^-------------^ SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails. Did you mean: cd /home/ubuntu || exit In ./infrastructure/nomad-configuration/client-smasher-instance-user-data.tpl.sh line 43: python ./awslogs-agent-setup.py --region ${region} --non-interactive --configfile awslogs.conf ^-------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: python ./awslogs-agent-setup.py --region "${region}" --non-interactive --configfile awslogs.conf In ./infrastructure/nomad-configuration/lead-server-instance-user-data.tpl.sh line 26: cd /home/ubuntu ^-------------^ SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails. Did you mean: cd /home/ubuntu || exit In ./infrastructure/nomad-configuration/lead-server-instance-user-data.tpl.sh line 36: python ./awslogs-agent-setup.py --region ${region} --non-interactive --configfile awslogs.conf ^-------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: python ./awslogs-agent-setup.py --region "${region}" --non-interactive --configfile awslogs.conf In ./infrastructure/nomad-configuration/lead-server-instance-user-data.tpl.sh line 124: echo -e '#!/bin/bash\naws cloudwatch put-metric-data --metric-name NomadQueueLength --namespace ${user}-${stage} --value `nomad status | grep dispatch | grep -e pending -e running | wc -l` --region ${region}' >> update_metric.sh ^-- SC2016: Expressions don't expand in single quotes, use double quotes for that. In ./infrastructure/nomad-configuration/pg-bouncer-instance-user-data.tpl.sh line 18: cd /home/ubuntu ^-------------^ SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails. Did you mean: cd /home/ubuntu || exit In ./infrastructure/nomad-configuration/server-instance-user-data.tpl.sh line 18: cd /home/ubuntu ^-------------^ SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails. Did you mean: cd /home/ubuntu || exit In ./infrastructure/nomad-configuration/server-instance-user-data.tpl.sh line 28: python ./awslogs-agent-setup.py --region ${region} --non-interactive --configfile awslogs.conf ^-------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: python ./awslogs-agent-setup.py --region "${region}" --non-interactive --configfile awslogs.conf In ./infrastructure/nomad-configuration/server-instance-user-data.tpl.sh line 95: nomad_job_specs=nomad-job-specs/* ^---------------^ SC2125: Brace expansions and globs are literal in assignments. Quote it or use an array. In ./infrastructure/nomad-configuration/server-instance-user-data.tpl.sh line 98: nomad run -address http://$IP_ADDRESS:4646 $nomad_job_spec ^---------^ SC2086: Double quote to prevent globbing and word splitting. ^-------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: nomad run -address http://"$IP_ADDRESS":4646 "$nomad_job_spec" In ./scripts/create_virtualenv.sh line 20: . dr_env/bin/activate ^-----------------^ SC1091: Not following: dr_env/bin/activate was not specified as input (see shellcheck -x). In ./scripts/hooks/autoformat.sh line 5: files=`git diff --staged --name-only --diff-filter=d -- "*.py"` ^-- SC2006: Use $(...) notation instead of legacy backticked `...`. Did you mean: files=$(git diff --staged --name-only --diff-filter=d -- "*.py") In ./scripts/hooks/autoformat.sh line 8: black --line-length 100 $file ^---^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: black --line-length 100 "$file" In ./scripts/hooks/autoformat.sh line 9: git add $file ^---^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: git add "$file" In ./scripts/install_all.sh line 184: if ! test -e ../.git/hooks/pre-commit > /dev/null; then ^---------^ SC2065: This is interpreted as a shell file redirection, not a comparison. In ./scripts/install_all.sh line 208: . ../dr_env/bin/activate ^--------------------^ SC1091: Not following: ../dr_env/bin/activate was not specified as input (see shellcheck -x). In ./scripts/make_migrations.sh line 18: . ./common.sh ^---------^ SC1091: Not following: ./common.sh was not specified as input (see shellcheck -x). In ./scripts/prepare_image.sh line 11: . ./common.sh ^---------^ SC1091: Not following: ./common.sh was not specified as input (see shellcheck -x). In ./scripts/run_manage.sh line 15: . ./common.sh ^---------^ SC1091: Not following: ./common.sh was not specified as input (see shellcheck -x). In ./scripts/run_nomad.sh line 64: . ./common.sh ^---------^ SC1091: Not following: ./common.sh was not specified as input (see shellcheck -x). In ./scripts/run_shell.sh line 21: . ./common.sh ^---------^ SC1091: Not following: ./common.sh was not specified as input (see shellcheck -x). In ./workers/run_tests.sh line 77: if [ ! -e $volume_directory/salmon_tests ] || [ -e $volume_directory/salmon_tests/newer ]; then ^---------------^ SC2086: Double quote to prevent globbing and word splitting. ^---------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: if [ ! -e "$volume_directory"/salmon_tests ] || [ -e "$volume_directory"/salmon_tests/newer ]; then In ./workers/run_tests.sh line 79: rm -rf $volume_directory/salmon_tests ^---------------^ SC2086: Double quote to prevent globbing and word splitting. Did you mean: rm -rf "$volume_directory"/salmon_tests In ./workers/tester.sh line 64: script_directory=`perl -e 'use File::Basename; ^-- SC2006: Use $(...) notation instead of legacy backticked `...`. Did you mean: script_directory=$(perl -e 'use File::Basename; use Cwd "abs_path"; print dirname(abs_path(@ARGV[0]));' -- "$0") In ./workers/tester.sh line 67: cd "$script_directory" ^--------------------^ SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails. Did you mean: cd "$script_directory" || exit In ./workers/tester.sh line 71: cd .. ^---^ SC2103: Use a ( subshell ) to avoid having to cd back. In ./workers/tester.sh line 89: export AWS_ACCESS_KEY_ID=`~/bin/aws configure get default.aws_access_key_id` ^---------------^ SC2155: Declare and assign separately to avoid masking return values. ^-- SC2006: Use $(...) notation instead of legacy backticked `...`. Did you mean: export AWS_ACCESS_KEY_ID=$(~/bin/aws configure get default.aws_access_key_id) In ./workers/tester.sh line 90: export AWS_SECRET_ACCESS_KEY=`~/bin/aws configure get default.aws_secret_access_key` ^-------------------^ SC2155: Declare and assign separately to avoid masking return values. ^-- SC2006: Use $(...) notation instead of legacy backticked `...`. Did you mean: export AWS_SECRET_ACCESS_KEY=$(~/bin/aws configure get default.aws_secret_access_key) For more information: https://www.shellcheck.net/wiki/SC1083 -- This { is literal. Check expressi... https://www.shellcheck.net/wiki/SC2034 -- FILE_RESULT appears unused. Verif... https://www.shellcheck.net/wiki/SC2036 -- If you wanted to assign the outpu... ```

Solution or next step

We should look into these errors and fix the ones that might cause problems for us. shellcheck can be configured to skip the error codes.

[arielsvn]

I'm working to configure pre-commit on the project, once that's implemented the configuration to add spellcheck to the pre-commit is:

  - repo: git://github.com/detailyang/pre-commit-shell
    rev: 1.0.5
    hooks:
      - id: shell-lint
        args: [--format=json]

[kurtwheeler]

Sweet! I like this a lot! This is the linter that @wvauclain used to cleanup/standardize all of our bash over the summer, which is probably why there's so few warnings now. (He fixed a lot of things back then.)

[wvauclain]

This was done in https://github.com/AlexsLemonade/refinebio/pull/2389