AWS CLI JSON Handling Guide

[hoangquochung1110]

AWS CLI JSON Handling Guide

Common Challenges

• Escaping quotes in JSON strings
• Variable interpolation within JSON
• Maintaining proper multi-line formatting
• Preventing shell interpretation of special characters

Solution 1: Heredoc Syntax

aws s3api put-bucket-policy --bucket my-bucket --policy <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::$ACCOUNT_ID:role/MyRole"},
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::my-bucket/*"
    }
  ]
}
EOF

Best for: Scripts where you want inline JSON with variable interpolation

Solution 2: JSON Files + Variable Substitution

# Create policy-template.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::${ACCOUNT_ID}:role/MyRole"},
      "Action": "s3:GetObject", 
      "Resource": "arn:aws:s3:::${BUCKET_NAME}/*"
    }
  ]
}

# In shell script:
export ACCOUNT_ID="123456789012"
export BUCKET_NAME="my-bucket"
envsubst < policy-template.json > policy.json
aws s3api put-bucket-policy --bucket "$BUCKET_NAME" --policy file://policy.json

Best for: Complex policies, team environments with code review

Solution 3: jq for Dynamic JSON

ACCOUNT_ID="123456789012"
BUCKET_NAME="my-bucket"

aws s3api put-bucket-policy \
  --bucket "$BUCKET_NAME" \
  --policy "$(jq -n \
    --arg account "$ACCOUNT_ID" \
    --arg bucket "$BUCKET_NAME" \
    '{
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {"AWS": "arn:aws:iam::" + $account + ":role/MyRole"},
          "Action": "s3:GetObject",
          "Resource": "arn:aws:s3:::" + $bucket + "/*"
        }
      ]
    }')"

Best for: Dynamic JSON generation with complex variable substitution

Solution 4: Shell Functions for AWS Parameters

function create_s3_readonly_policy() {
  local bucket="$1"
  local role_arn="$2"

  aws s3api put-bucket-policy \
    --bucket "$bucket" \
    --policy "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"$role_arn\"},\"Action\":\"s3:GetObject\",\"Resource\":\"arn:aws:s3:::$bucket/*\"}]}"
}

# Usage
create_s3_readonly_policy "my-bucket" "arn:aws:iam::123456789012:role/MyRole"

Best for: Frequently used operations that need to be parameterized

Solution 5: AWS CLI v2 Parameter Builder

# For simpler cases only - generally not recommended for bucket policies
aws s3api put-bucket-policy \
  --bucket my-bucket \
  --policy '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {"AWS": "arn:aws:iam::123456789012:role/MyRole"},
        "Action": "s3:GetObject",
        "Resource": "arn:aws:s3:::my-bucket/*"
      }
    ]
  }'

Best for: Interactive use, not recommended for scripts with S3 policies

Recommendations for S3 Bucket Policies

1. Simple policies: Use heredoc syntax
2. Complex policies: Use file-based approach with envsubst
3. Highly dynamic policies: Use jq
4. Reusable operations: Create shell functions
5. Avoid: Direct string interpolation within JSON

Alternative Tools

• yq: Like jq but for YAML (can convert between YAML/JSON)
• aws-shell: Interactive shell for AWS CLI with auto-completion
• AWS CDK: Infrastructure as code, avoids shell JSON entirely