cloudfront_distribution: Type error when validating distribution origins: "'list' object has no attribute 'get'"

life5ign commented 3 years ago

SUMMARY

I get an unexpected type error, for what I assume to be the "origins" argument to the community.aws.cloudfront_distribution module: Type error when validating distribution origins: "'list' object has no attribute 'get'"

Posted on StackExchange DevOps: https://devops.stackexchange.com/questions/13639/unexpected-type-error-in-ansible-community-aws-cloudfront-distribution-origins

ISSUE TYPE

Bug Report

COMPONENT NAME

cloudfront_distribution

ANSIBLE VERSION

ansible 2.10.7
  config file = /home/bryant/src/api_guys/ansible-deploy-vst/ansible.cfg
  configured module search path = ['/home/bryant/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/bryant/.pyenv/versions/vst-ansible-deployment/lib/python3.9/site-packages/ansible
  executable location = /home/bryant/.pyenv/versions/vst-ansible-deployment/bin/ansible
  python version = 3.9.1 (default, Feb 16 2021, 14:25:16) [GCC 9.3.0]

CONFIGURATION

DEFAULT_CALLBACK_WHITELIST(/home/bryant/src/api_guys/ansible-deploy-vst/ansible.cfg) = ['ansible.posix.timer']
DEFAULT_HOST_LIST(/home/bryant/src/api_guys/ansible-deploy-vst/ansible.cfg) = ['/home/bryant/src/api_guys/ansible-deploy-vst/inventory']
DEFAULT_PRIVATE_KEY_FILE(/home/bryant/src/api_guys/ansible-deploy-vst/ansible.cfg) = /home/bryant/.ssh/ansible-control-node_rsa
DEFAULT_REMOTE_USER(/home/bryant/src/api_guys/ansible-deploy-vst/ansible.cfg) = ubuntu
DEFAULT_VAULT_PASSWORD_FILE(/home/bryant/src/api_guys/ansible-deploy-vst/ansible.cfg) = /home/bryant/src/api_guys/ansible-deploy-vst/.vault_passwords
HOST_KEY_CHECKING(/home/bryant/src/api_guys/ansible-deploy-vst/ansible.cfg) = False

OS / ENVIRONMENT

control node: Ubuntu 20.04 LTS as PopOs, running ansible in a pyenv virtual environment managed node: localhost (since using aws modules); routes to an IAM user in our aws account with sufficient attached policies.

STEPS TO REPRODUCE

I run a role, containing the task below, from my main playbook; boto profiles are configured in ~/.aws/credentials with correct keys, secrets and regions in ~/.aws.config.

From the main playbook site.yml:

- name: Perform AWS tasks for a full deployment
  hosts: localhost
  tags: aws_full_deployment
  gather_facts: yes
  vars:
    boto_profile: "{{ vst_gov_boto_profile }}"
  tasks:
    - name: Conditionally run the AWS Full Deployment role
      import_role:
        name: aws_full_deployment
      when: deployment_type == "full_deployment"

The task in the role presenting this issue:

- name: Create Cloudfront distribution for the interface and api reverse proxy
  community.aws.cloudfront_distribution:
    profile: "{{ vst_nongov_boto_profile }}"
    state: present
    enabled: yes
    caller_reference: "{{ cloudfront_domain_alias }}" 
    alias: "{{ cloudfront_domain_alias }}" 
    aliases:
      - "{{ cloudfront_domain_alias }}"
    origins:
      # reverse proxy for api
      - id: "{{ base_domain }}"
        domain_name: "{{ base_domain }}"
        custom_origin_config:
          http_port: 80
          https_port: 443
          origin_keepalive_timeout: 5
          origin_protocol_policy: "https-only"
          origin_read_timeout: 10
          origin_ssl_protocols:
            - "TLSv1.2"
      # s3 bucket for interface
      - id: "{{ s3_bucket_www_domain_name }}"
        domain_name: "{{ s3_bucket_domain_name }}" 
        custom_origin_config:
          http_port: 80
          https_port: 443
          origin_keepalive_timeout: 5
          origin_protocol_policy: "http-only"
          origin_read_timeout: 10
          origin_ssl_protocols:
            - "TLSv1.2"
    default_cache_behavior:
      #   this contains as many settings as possible that are able to be
      # replicated from the UI option caching policy
      # "Managed-CachingOptimized"; some are guesses, which may need to be
      # changed if we notice performance issues
      target_origin_id: "{{ s3_bucket_domain_name }}" 
      forwarded_values:
        allowed_methods:
          - GET
          - HEAD
        items:
          - GET
          - HEAD
        compress: yes
        cookies:
          forward: none
        default_ttl: 86400
        max_ttl: 31536000
        min_ttl: 1
        query_string: no
        smooth_streaming: no
        trusted_signers:
          enabled: no
        viewer_protocol_policy: "redirect-to-https"
    cache_behaviors:
      - path_pattern: "api/*"
        target_origin_id: "{{ base_domain }}" 
        forwarded_values:
          allowed_methods:
            - GET
            - HEAD
          items:
            - GET
            - HEAD
            - OPTIONS
            - PUT
            - POST
            - PATCH
            - DELETE
          compress: no
          cookies:
            forward: all
          default_ttl: 0
          headers:
            - Authorization
            - Referer
            - User-Agent
          max_ttl: 0
          min_ttl: 0
          query_string: yes
          query_string_cache_keys:
            - '*'
          smooth_streaming: no
          trusted_signers:
            enabled: no
          viewer_protocol_policy: "redirect-to-https"
      - path_pattern: "testapi*"
        target_origin_id: "{{ base_domain }}" 
        forwarded_values:
          allowed_methods:
            - GET
            - HEAD
          items:
            - GET
            - HEAD
          compress: no
          cookies:
            forward: all
          default_ttl: 0
          headers:
            - Authorization
            - User-Agent
          max_ttl: 0
          min_ttl: 0
          query_string: yes
          query_string_cache_keys:
            - '*'
          smooth_streaming: no
          trusted_signers:
            enabled: no
          viewer_protocol_policy: "redirect-to-https"
    custom_error_responses:
      - error_caching_min_ttl: 10
        error_code: 403
        response_code: 200
        response_page_path: "/index.html"
      - error_caching_min_ttl: 10
        error_code: 404
        response_code: 200
        response_page_path: "/index.html"
    viewer_certificate:
      acm_certificate_arn: "{{ nongov_acm_cert_upload_result.certificate.arn }}"
      cloudfront_default_certificate: no
      minimum_protocol_version: "TLSv1.2_2019"
      ssl_support_method: "sni-only"
    http_version: http2
    ipv6_enabled: no
    tags:
      Name: "{{ generic_name_tag }}"
    wait: yes
    default_root_object: "index.html"
    price_class: "{{ cloudfront_price_class }}"
    comment: "Cloudfront distribution for {{ base_interface_domain }} interface and API reverse proxy"
  notify: primary_cloudfront_invalidation
  tags: cloudfront

EXPECTED RESULTS

I expect a cloudfront distribution to be created; I've review my syntax on the "origins" argument many times.

ACTUAL RESULTS

I get the type error explained above. It seems that perhaps ansible is expecting "origins" to be a dictionary, not a list, as described in the documentation?

TASK [aws_full_deployment : Create Cloudfront distribution for the interface and api reverse proxy] ***
task path: /home/bryant/src/api_guys/ansible-deploy-vst/roles/aws_full_deployment/tasks/main.yml:501
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: bryant
<127.0.0.1> EXEC /bin/sh -c 'echo ~bryant && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/bryant/.ansible/tmp `"&& mkdir "` echo /home/bryant/.ansible/tmp/ansible-tmp-1616796020.2343066-177081-217806658917877 `" && echo ansible-tmp-1616796020.2343066-177081-217806658917877="` echo /home/bryant/.ansible/tmp/ansible-tmp-1616796020.2343066-177081-217806658917877 `" ) && sleep 0'
Using module file /home/bryant/.ansible/collections/ansible_collections/community/aws/plugins/modules/cloudfront_distribution.py
<127.0.0.1> PUT /home/bryant/.ansible/tmp/ansible-local-175967cbvvbt8u/tmpamrug82t TO /home/bryant/.ansible/tmp/ansible-tmp-1616796020.2343066-177081-217806658917877/AnsiballZ_cloudfront_distribution.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/bryant/.ansible/tmp/ansible-tmp-1616796020.2343066-177081-217806658917877/ /home/bryant/.ansible/tmp/ansible-tmp-1616796020.2343066-177081-217806658917877/AnsiballZ_cloudfront_distribution.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/home/bryant/.pyenv/versions/vst-ansible-deployment/bin/python /home/bryant/.ansible/tmp/ansible-tmp-1616796020.2343066-177081-217806658917877/AnsiballZ_cloudfront_distribution.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/bryant/.ansible/tmp/ansible-tmp-1616796020.2343066-177081-217806658917877/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_community.aws.cloudfront_distribution_payload_vqd_hi9m/ansible_community.aws.cloudfront_distribution_payload.zip/ansible_collections/community/aws/plugins/modules/cloudfront_distribution.py", line 1665, in validate_origins
  File "/tmp/ansible_community.aws.cloudfront_distribution_payload_vqd_hi9m/ansible_community.aws.cloudfront_distribution_payload.zip/ansible_collections/community/aws/plugins/modules/cloudfront_distribution.py", line 1729, in validate_origin
AttributeError: 'list' object has no attribute 'get'
fatal: [localhost]: FAILED! => {
    "boto3_version": "1.16.25",
    "botocore_version": "1.19.25",
    "changed": false,
    "invocation": {
        "module_args": {
            "alias": "v6-full-deployment.icloudcms.com",
            "aliases": [
                "v6-full-deployment.icloudcms.com"
            ],
            "aws_access_key": null,
            "aws_ca_bundle": null,
            "aws_config": null,
            "aws_secret_key": null,
            "cache_behaviors": [
                {
                    "forwarded_values": {
                        "allowed_methods": [
                            "GET",
                            "HEAD"
                        ],
                        "compress": false,
                        "cookies": {
                            "forward": "all"
                        },
                        "default_ttl": 0,
                        "headers": [
                            "Authorization",
                            "Referer",
                            "User-Agent"
                        ],
                        "items": [
                            "GET",
                            "HEAD",
                            "OPTIONS",
                            "PUT",
                            "POST",
                            "PATCH",
                            "DELETE"
                        ],
                        "max_ttl": 0,
                        "min_ttl": 0,
                        "query_string": true,
                        "query_string_cache_keys": [
                            "*"
                        ],
                        "smooth_streaming": false,
                        "trusted_signers": {
                            "enabled": false
                        },
                        "viewer_protocol_policy": "redirect-to-https"
                    },
                    "path_pattern": "api/*",
                    "target_origin_id": "api.v6-full-deployment.icloudcms.com"
                },
                {
                    "forwarded_values": {
                        "allowed_methods": [
                            "GET",
                            "HEAD"
                        ],
                        "compress": false,
                        "cookies": {
                            "forward": "all"
                        },
                        "default_ttl": 0,
                        "headers": [
                            "Authorization",
                            "User-Agent"
                        ],
                        "items": [
                            "GET",
                            "HEAD"
                        ],
                        "max_ttl": 0,
                        "min_ttl": 0,
                        "query_string": true,
                        "query_string_cache_keys": [
                            "*"
                        ],
                        "smooth_streaming": false,
                        "trusted_signers": {
                            "enabled": false
                        },
                        "viewer_protocol_policy": "redirect-to-https"
                    },
                    "path_pattern": "testapi*",
                    "target_origin_id": "api.v6-full-deployment.icloudcms.com"
                }
            ],
            "caller_reference": "v6-full-deployment.icloudcms.com",
            "comment": "Cloudfront distribution for v6-full-deployment.icloudcms.com interface and API reverse proxy",
            "custom_error_responses": [
                {
                    "error_caching_min_ttl": 10,
                    "error_code": 403,
                    "response_code": 200,
                    "response_page_path": "/index.html"
                },
                {
                    "error_caching_min_ttl": 10,
                    "error_code": 404,
                    "response_code": 200,
                    "response_page_path": "/index.html"
                }
            ],
            "debug_botocore_endpoint_logs": false,
            "default_cache_behavior": {
                "forwarded_values": {
                    "allowed_methods": [
                        "GET",
                        "HEAD"
                    ],
                    "compress": true,
                    "cookies": {
                        "forward": "none"
                    },
                    "default_ttl": 86400,
                    "items": [
                        "GET",
                        "HEAD"
                    ],
                    "max_ttl": 31536000,
                    "min_ttl": 1,
                    "query_string": false,
                    "smooth_streaming": false,
                    "trusted_signers": {
                        "enabled": false
                    },
                    "viewer_protocol_policy": "redirect-to-https"
                },
                "target_origin_id": "v6-full-deployment-s3-bucket.s3-website-us-gov-west-1.amazonaws.com"
            },
            "default_origin_domain_name": null,
            "default_origin_path": null,
            "default_root_object": "index.html",
            "distribution_id": null,
            "e_tag": null,
            "ec2_url": null,
            "enabled": true,
            "http_version": "http2",
            "ipv6_enabled": false,
            "logging": null,
            "origins": [
                {
                    "custom_headers": {
                        "quantity": 0
                    },
                    "custom_origin_config": {
                        "h_t_t_p_port": 80,
                        "h_t_t_p_s_port": 443,
                        "origin_keepalive_timeout": 5,
                        "origin_protocol_policy": "https-only",
                        "origin_read_timeout": 10,
                        "origin_ssl_protocols": [
                            "TLSv1.2"
                        ]
                    },
                    "domain_name": "api.v6-full-deployment.icloudcms.com",
                    "id": "api.v6-full-deployment.icloudcms.com",
                    "origin_path": ""
                },
                {
                    "custom_origin_config": {
                        "http_port": 80,
                        "https_port": 443,
                        "origin_keepalive_timeout": 5,
                        "origin_protocol_policy": "http-only",
                        "origin_read_timeout": 10,
                        "origin_ssl_protocols": [
                            "TLSv1.2"
                        ]
                    },
                    "domain_name": "v6-full-deployment-s3-bucket.s3-website-us-gov-west-1.amazonaws.com",
                    "id": "v6-full-deployment-s3-bucket-www.s3-website-us-gov-west-1.amazonaws.com"
                }
            ],
            "price_class": "PriceClass_100",
            "profile": "vst_nongov",
            "purge_aliases": false,
            "purge_cache_behaviors": false,
            "purge_custom_error_responses": false,
            "purge_origins": false,
            "purge_tags": false,
            "region": null,
            "restrictions": null,
            "security_token": null,
            "state": "present",
            "tags": {
                "Name": "v6-full-deployment"
            },
            "validate_certs": true,
            "viewer_certificate": {
                "acm_certificate_arn": "xxxxxxxxxxOBFUSCATED",
                "cloudfront_default_certificate": false,
                "minimum_protocol_version": "TLSv1.2_2019",
                "ssl_support_method": "sni-only"
            },
            "wait": true,
            "wait_timeout": 1800,
            "web_acl_id": null
        }
    },
    "msg": "Error validating distribution origins: 'list' object has no attribute 'get'"
}

ansibullbot commented 3 years ago

Files identified in the description: None

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

ansibullbot commented 3 years ago

Files identified in the description:

plugins/modules/cloudfront_distribution.py

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

ansibullbot commented 3 years ago

cc @jillr @s-hertel @tremble @wilvk @wimnat click here for bot help

life5ign commented 3 years ago

Hello, can someone take a look at this? If I'm not making a syntax error, it seems like a rather significant issue (a list object being asked to "get" implies that it's a dictionary). Thank you

flowerysong commented 3 years ago

https://github.com/flowerysong/ansible-community.aws/commit/d48b9b9639e17de9bd50e6d9ac339f294b5e6952 might fix this, but tests for this module are disabled in CI and I probably won't get around to testing it manually any time soon.

tremble commented 3 years ago

@Life5ign I've had a quick test of @flowerysong's initial change. Building upon that I've submitted #540.

Since I'm still seeing idempotency issues (one of which looks like it mangles existing origins) the PR isn't ready to merge yet.

life5ign commented 3 years ago

Can I help with testing this?

tremble commented 3 years ago

You can try the change in #540 I believe there are still bugs (related to idempotency) but I'm not sure if they were there before the change or not.

life5ign commented 3 years ago

Hi guys, I'm trying to learn how to test, doing my best, but I'm still clunky at it. Is there any chance you can work on this so that the fix is merged? thank you

life5ign commented 3 years ago

Hello @tremble , In order to test this, I have forked community.aws (and general), cloned them, and put them in a virtual environment; do I follow the core ansible instructions for testing a PR, but for collections instead? https://docs.ansible.com/ansible/2.3/dev_guide/developing_test_pr.html

I'm also not clear, do I need a source checkout of ansible itself? Please excuse my inexperience, I just really want to get this bug fixed and help out. thank you,

life5ign commented 3 years ago

Or, @tremble , do I clone community.aws as in the instructions for community.general (https://docs.ansible.com/ansible/latest/dev_guide/developing_collections_contributing.html), switch to a new branch in the clone and checkout the PR, modify COLLECTIONS_PATHS in my project's ansible configuration to point to and prefer this version of the collection, and begin testing, continuing to use the version of ansible I have in my project's virtual environment, of course making sure to test on a VM or container?

ioah86 commented 1 year ago

I do think there is a different issue here. Looking at the documentation, it does not make sense to have "compression" e.g. under forwarded_values. Is it possible that the docs are erroneous, and the fix here would be to unindent anything that is not "forwarded_values?

ansible-collections / community.aws