cloudwatchlogs_log_group not handling pagination

soppenlander commented 2 years ago

Summary

cloudwatchlogs_log_group only returns the first 50 or so log groups from cloudwatch. Our environment has 67 so the complete list is not being returned.

Issue Type

Bug Report

Component Name

community.aws

Ansible Version

ansible [core 2.11.6] config file = /Users/soppenla/dev/nrs-ansible/ansible.cfg configured module search path = ['/Users/soppenla/dev/nrs-ansible/library/modules'] ansible python module location = /Users/soppenla/.pyenv/versions/anaconda3-2020.11/lib/python3.8/site-packages/ansible ansible collection location = /Users/soppenla/.ansible/collections:/usr/share/ansible/collections executable location = /Users/soppenla/.pyenv/versions/anaconda3-2020.11/bin/ansible python version = 3.8.5 (default, Sep 4 2020, 02:22:02) [Clang 10.0.0 ] jinja version = 2.11.2 libyaml = True

Collection Versions

Collection Version

amazon.aws 1.5.1 ansible.netcommon 2.4.0 ansible.posix 1.3.0 ansible.utils 2.4.0 ansible.windows 1.7.2 arista.eos 2.2.0 awx.awx 19.2.2 azure.azcollection 1.9.0 check_point.mgmt 2.0.0 chocolatey.chocolatey 1.1.0 cisco.aci 2.0.0 cisco.asa 2.0.3 cisco.intersight 1.0.17 cisco.ios 2.4.0 cisco.iosxr 2.4.0 cisco.meraki 2.4.2 cisco.mso 1.2.0 cisco.nso 1.0.3 cisco.nxos 2.5.1 cisco.ucs 1.6.0 cloudscale_ch.cloud 2.2.0 community.aws 1.5.0 community.azure 1.0.0 community.crypto 1.9.3 community.digitalocean 1.10.0 community.docker 1.9.1 community.fortios 1.0.0 community.general 3.7.0 community.google 1.0.0 community.grafana 1.2.3 community.hashi_vault 1.3.2 community.hrobot 1.1.1 community.kubernetes 1.2.1 community.kubevirt 1.0.0 community.libvirt 1.0.2 community.mongodb 1.3.1 community.mysql 2.1.1 community.network 3.0.0 community.okd 1.1.2 community.postgresql 1.4.0 community.proxysql 1.2.0 community.rabbitmq 1.1.0 community.routeros 1.2.0 community.skydive 1.0.0 community.sops 1.1.0 community.vmware 1.14.0 community.windows 1.6.0 community.zabbix 1.4.0 containers.podman 1.8.0 cyberark.conjur 1.1.0 cyberark.pas 1.0.7 dellemc.enterprise_sonic 1.1.0 dellemc.openmanage 3.6.0 dellemc.os10 1.1.1 dellemc.os6 1.0.7 dellemc.os9 1.0.4 f5networks.f5_modules 1.11.1 fortinet.fortimanager 2.1.3 fortinet.fortios 2.1.2 frr.frr 1.0.3 gluster.gluster 1.0.2 google.cloud 1.0.2 hetzner.hcloud 1.6.0 hpe.nimble 1.1.3 ibm.qradar 1.0.3 infinidat.infinibox 1.2.4 inspur.sm 1.3.0 junipernetworks.junos 2.5.0 kubernetes.core 1.2.1 mellanox.onyx 1.0.0 netapp.aws 21.6.0 netapp.azure 21.9.0 netapp.cloudmanager 21.10.0 netapp.elementsw 21.6.1 netapp.ontap 21.11.0 netapp.um_info 21.7.0 netapp_eseries.santricity 1.2.13 netbox.netbox 3.1.2 ngine_io.cloudstack 2.1.0 ngine_io.exoscale 1.0.0 ngine_io.vultr 1.1.0 openstack.cloud 1.5.1 openvswitch.openvswitch 2.0.0 ovirt.ovirt 1.6.3 purestorage.flasharray 1.10.0 purestorage.flashblade 1.6.0 sensu.sensu_go 1.12.0 servicenow.servicenow 1.0.6 splunk.es 1.0.2 t_systems_mms.icinga_director 1.22.0 theforeman.foreman 2.2.0 vyos.vyos 2.5.1 wti.remote 1.0.1

AWS SDK versions

Name: boto Version: 2.49.0 Summary: Amazon Web Services Library Home-page: https://github.com/boto/boto/ Author: Mitch Garnaat Author-email: mitch@garnaat.com License: MIT Location: /Users/soppenla/.pyenv/versions/anaconda3-2020.11/lib/python3.8/site-packages Requires: Required-by:

Name: boto3 Version: 1.18.41 Summary: The AWS SDK for Python Home-page: https://github.com/boto/boto3 Author: Amazon Web Services Author-email: None License: Apache License 2.0 Location: /Users/soppenla/.pyenv/versions/anaconda3-2020.11/lib/python3.8/site-packages Requires: botocore, jmespath, s3transfer Required-by:

Name: botocore Version: 1.21.41 Summary: Low-level, data-driven core of boto 3. Home-page: https://github.com/boto/botocore Author: Amazon Web Services Author-email: None License: Apache License 2.0 Location: /Users/soppenla/.pyenv/versions/anaconda3-2020.11/lib/python3.8/site-packages Requires: urllib3, jmespath, python-dateutil Required-by: s3transfer, boto3

Configuration

[DEPRECATION WARNING]: [defaults]callback_whitelist option, normalizing names to new standard, use callbacks_enabled instead. This feature will be removed from ansible-core in version 2.15. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. ANSIBLE_PIPELINING(/Users/soppenla/dev/nrs-ansible/ansible.cfg) = True BECOME_PLUGIN_PATH(/Users/soppenla/dev/nrs-ansible/ansible.cfg) = ['/Users/soppenla/dev/nrs-ansible/library/plugins/become'] CALLBACKS_ENABLED(/Users/soppenla/dev/nrs-ansible/ansible.cfg) = ['junit'] DEFAULT_ACTION_PLUGIN_PATH(/Users/soppenla/dev/nrs-ansible/ansible.cfg) = ['/Users/soppenla/dev/nrs-ansible/library/plugins/action'] DEFAULT_CACHE_PLUGIN_PATH(/Users/soppenla/dev/nrs-ansible/ansible.cfg) = ['/Users/soppenla/dev/nrs-ansible/library/plugins/cache'] DEFAULT_CALLBACK_PLUGIN_PATH(/Users/soppenla/dev/nrs-ansible/ansible.cfg) = ['/Users/soppenla/dev/nrs-ansible/library/plugins/callback'] DEFAULT_CONNECTION_PLUGIN_PATH(/Users/soppenla/dev/nrs-ansible/ansible.cfg) = ['/Users/soppenla/dev/nrs-ansible/library/plugins/connection' DEFAULT_FILTER_PLUGIN_PATH(/Users/soppenla/dev/nrs-ansible/ansible.cfg) = ['/Users/soppenla/dev/nrs-ansible/library/plugins/filter'] DEFAULT_FORKS(/Users/soppenla/dev/nrs-ansible/ansible.cfg) = 16 DEFAULT_GATHERING(/Users/soppenla/dev/nrs-ansible/ansible.cfg) = explicit DEFAULT_HASH_BEHAVIOUR(/Users/soppenla/dev/nrs-ansible/ansible.cfg) = merge DEFAULT_HOST_LIST(/Users/soppenla/dev/nrs-ansible/ansible.cfg) = ['/Users/soppenla/dev/nrs-ansible/inventory/local'] DEFAULT_INVENTORY_PLUGIN_PATH(/Users/soppenla/dev/nrs-ansible/ansible.cfg) = ['/Users/soppenla/dev/nrs-ansible/library/plugins/inventory'] DEFAULT_LOCAL_TMP(/Users/soppenla/dev/nrs-ansible/ansible.cfg) = /tmp/ansible-local-59298y_utx09g DEFAULT_LOOKUP_PLUGIN_PATH(/Users/soppenla/dev/nrs-ansible/ansible.cfg) = ['/Users/soppenla/dev/nrs-ansible/library/plugins/lookup'] DEFAULT_MODULE_PATH(/Users/soppenla/dev/nrs-ansible/ansible.cfg) = ['/Users/soppenla/dev/nrs-ansible/library/modules'] DEFAULT_MODULE_UTILS_PATH(/Users/soppenla/dev/nrs-ansible/ansible.cfg) = ['/Users/soppenla/dev/nrs-ansible/library/module_utils'] DEFAULT_PRIVATE_KEY_FILE(/Users/soppenla/dev/nrs-ansible/ansible.cfg) = /Users/soppenla/.ssh/ansible_id_rsa DEFAULT_REMOTE_USER(/Users/soppenla/dev/nrs-ansible/ansible.cfg) = ansible DEFAULT_ROLES_PATH(/Users/soppenla/dev/nrs-ansible/ansible.cfg) = ['/Users/soppenla/dev/nrs-ansible/roles'] DEFAULT_STRATEGY_PLUGIN_PATH(/Users/soppenla/dev/nrs-ansible/ansible.cfg) = ['/Users/soppenla/dev/nrs-ansible/library/plugins/strategy'] DEFAULT_TERMINAL_PLUGIN_PATH(/Users/soppenla/dev/nrs-ansible/ansible.cfg) = ['/Users/soppenla/dev/nrs-ansible/library/plugins/terminal'] DEFAULT_TEST_PLUGIN_PATH(/Users/soppenla/dev/nrs-ansible/ansible.cfg) = ['/Users/soppenla/dev/nrs-ansible/library/plugins/test'] DEFAULT_VARS_PLUGIN_PATH(/Users/soppenla/dev/nrs-ansible/ansible.cfg) = ['/Users/soppenla/dev/nrs-ansible/library/plugins/vars'] DISPLAY_SKIPPED_HOSTS(/Users/soppenla/dev/nrs-ansible/ansible.cfg) = False HOST_KEY_CHECKING(/Users/soppenla/dev/nrs-ansible/ansible.cfg) = False

OS / Environment

MacOS 11.3.1 but issue observed on linux as well

Steps to Reproduce

Setup: have more than 50 cloudwatch log groups

block:

name: Ensure Log Group is available cloudwatchlogs_log_group: state: present log_group_name: "/ecs/{{ ecs_service_name }}" retention: 1 profile: "{{ ecs_aws_profile }}" region: "{{ ecs_aws_region }}" register: log_group_fact

Expected Results

I expected log_group_fact to have 67 entries but it only had 50

Actual Results

too much to list

Code of Conduct

[X] I agree to follow the Ansible Code of Conduct

markuman commented 2 years ago

@soppenlander cloudwatchlogs_log_group does not return all your cloudwatch log groups. It takes care about if single groups exists or not.

When you need a list of all your log groups, you must use cloudwatchlogs_log_group_info.

Example

---
- hosts: localhost
  connection: local

  tasks:
    - community.aws.cloudwatchlogs_log_group_info:
      register: log_groups

    - debug:
        var: log_groups.log_groups | count

results here in

PLAY [localhost] *****************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************************************
ok: [localhost]

TASK [community.aws.cloudwatchlogs_log_group_info] *******************************************************************************************************************************************************************
ok: [localhost]

TASK [debug] *********************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "log_groups.log_groups | count": "268"
}

PLAY RECAP ***********************************************************************************************************************************************************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

jillr commented 2 years ago

@soppenlander Using cloudwatchlogs_log_group_info should resolve this issue. If you still have a problem with the cloudwatchlogs_log_group module please open an issue with additional details.

ansible-collections / community.aws