ansible-collections / azure

Development area for Azure Collections
https://galaxy.ansible.com/azure/azcollection
GNU General Public License v3.0
245 stars 327 forks source link

azure_rm_adgroup Present members is not idempotent and does not follow ansible best practices #1519

Closed hematic closed 2 weeks ago

hematic commented 5 months ago
SUMMARY

azure_rm_adgroup allows you to add a list of members, potentially hundreds at a time by object id. However if any one of those hundreds of members is already present on the group the entire task fails and doesnt add the other users. This is not following the idempotent strategy that ansible playbooks normally follow.

This forces you to instead create a complex looping/failing/rescue structure to eliminate out any users that already exist on the group. This increases the actual run time of this process by hundreds of times potentially and makes the task mostly useless.

In its current state its better to use native PowerShell than this module.

ISSUE TYPE
COMPONENT NAME

azure_rm_adgroup

ANSIBLE VERSION
Not relevant to this issue
COLLECTION VERSION
ansible-galaxy collection list azure.azcollection
CONFIGURATION
This is not relevant to this issue.
OS / ENVIRONMENT

Using collection version 2.2.0 (latest)

STEPS TO REPRODUCE
- name: Create EntraAD Group
  hosts: localhost
  connection: local
  gather_facts: false

  vars: # We redeclare the variables here for simpler reference later in the code.
    user_object_ids: "{{ survey_user_object_ids.split(',') }}"
    owner_object_ids: "{{ survey_owner_object_ids.split(',') }}"
    group_display_name: "{{ survey_group_display_name }}"
    group_mail_nickname: "{{ survey_group_mail_nickname }}"

  tasks:

    - name: Create EntraID Group
      azure.azcollection.azure_rm_adgroup:
        tenant: "{{ azure_tenant_id }}"
        display_name: "{{ group_display_name }}"
        mail_nickname: "{{ group_mail_nickname }}"
        state: 'present'

    - name: Add members to EntraID Group
      azure.azcollection.azure_rm_adgroup:
        tenant: "{{ azure_tenant_id }}"
        display_name: "{{ group_display_name }}"
        mail_nickname: "{{ group_mail_nickname }}"
        state: 'present'
        present_members: "{{ user_object_ids }}"
      when: user_object_ids is defined and user_object_ids | length > 0

    - name: Add owners to EntraID Group
      azure.azcollection.azure_rm_adgroup:
        tenant: "{{ azure_tenant_id }}"
        display_name: "{{ group_display_name }}"
        mail_nickname: "{{ group_mail_nickname }}"
        state: 'present'
        present_owners: "{{ owner_object_ids }}"
      when: owner_object_ids is defined and owner_object_ids | length > 0
EXPECTED RESULTS

It is expected that if you pass a userlist of users that SHOULD be present on a group, if one of those is ALREADY present, this should not fail the task.

ACTUAL RESULTS

"One or more added object references already exist for the following modified properties: 'members'

{
  "module_stdout": "",
  "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_kw500gh4/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 302, in exec_module\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_kw500gh4/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 324, in update_members\n  File \"/usr/lib64/python3.9/asyncio/base_events.py\", line 647, in run_until_complete\n    return future.result()\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_kw500gh4/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 472, in add_group_member\n  File \"/usr/local/lib/python3.9/site-packages/msgraph/generated/groups/item/members/ref/ref_request_builder.py\", line 73, in post\n    return await self.request_adapter.send_no_response_content_async(request_info, error_mapping)\n  File \"/usr/local/lib/python3.9/site-packages/kiota_http/httpx_request_adapter.py\", line 377, in send_no_response_content_async\n    await self.throw_failed_responses(response, error_map, parent_span, parent_span)\n  File \"/usr/local/lib/python3.9/site-packages/kiota_http/httpx_request_adapter.py\", line 503, in throw_failed_responses\n    raise exc\nmsgraph.generated.models.o_data_errors.o_data_error.ODataError: \n        APIError\n        Code: 400\n        message: None\n        error: MainError(additional_data={}, code='Request_BadRequest', details=None, inner_error=InnerError(additional_data={'date': DateTime(2024, 3, 28, 13, 23, 43, tzinfo=Timezone('UTC'))}, client_request_id='236e30f3-3c69-4fc4-b945-d81bc0d20320', date=None, odata_type=None, request_id='99194263-b32e-489b-8a05-49ea3cba1d71'), message=\"One or more added object references already exist for the following modified properties: 'members'.\", target=None)\n        \n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"<stdin>\", line 107, in <module>\n  File \"<stdin>\", line 99, in _ansiballz_main\n  File \"<stdin>\", line 47, in invoke_module\n  File \"/usr/lib64/python3.9/runpy.py\", line 225, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib64/python3.9/runpy.py\", line 97, in _run_module_code\n    _run_code(code, mod_globals, init_globals,\n  File \"/usr/lib64/python3.9/runpy.py\", line 87, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_kw500gh4/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 501, in <module>\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_kw500gh4/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 497, in main\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_kw500gh4/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 249, in __init__\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_kw500gh4/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py\", line 469, in __init__\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_kw500gh4/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 307, in exec_module\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_kw500gh4/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py\", line 502, in fail\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_kw500gh4/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible/module_utils/basic.py\", line 1553, in fail_json\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_kw500gh4/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible/module_utils/basic.py\", line 1522, in _return_formatted\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_kw500gh4/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible/module_utils/common/parameters.py\", line 927, in remove_values\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_kw500gh4/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible/module_utils/common/parameters.py\", line 470, in _remove_values_conditions\nTypeError: Value of unknown type: <class 'msgraph.generated.models.o_data_errors.o_data_error.ODataError'>, \n        APIError\n        Code: 400\n        message: None\n        error: MainError(additional_data={}, code='Request_BadRequest', details=None, inner_error=InnerError(additional_data={'date': DateTime(2024, 3, 28, 13, 23, 43, tzinfo=Timezone('UTC'))}, client_request_id='236e30f3-3c69-4fc4-b945-d81bc0d20320', date=None, odata_type=None, request_id='99194263-b32e-489b-8a05-49ea3cba1d71'), message=\"One or more added object references already exist for the following modified properties: 'members'.\", target=None)\n        \n",
  "exception": "Traceback (most recent call last):\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_kw500gh4/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 302, in exec_module\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_kw500gh4/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 324, in update_members\n  File \"/usr/lib64/python3.9/asyncio/base_events.py\", line 647, in run_until_complete\n    return future.result()\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_kw500gh4/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 472, in add_group_member\n  File \"/usr/local/lib/python3.9/site-packages/msgraph/generated/groups/item/members/ref/ref_request_builder.py\", line 73, in post\n    return await self.request_adapter.send_no_response_content_async(request_info, error_mapping)\n  File \"/usr/local/lib/python3.9/site-packages/kiota_http/httpx_request_adapter.py\", line 377, in send_no_response_content_async\n    await self.throw_failed_responses(response, error_map, parent_span, parent_span)\n  File \"/usr/local/lib/python3.9/site-packages/kiota_http/httpx_request_adapter.py\", line 503, in throw_failed_responses\n    raise exc\nmsgraph.generated.models.o_data_errors.o_data_error.ODataError: \n        APIError\n        Code: 400\n        message: None\n        error: MainError(additional_data={}, code='Request_BadRequest', details=None, inner_error=InnerError(additional_data={'date': DateTime(2024, 3, 28, 13, 23, 43, tzinfo=Timezone('UTC'))}, client_request_id='236e30f3-3c69-4fc4-b945-d81bc0d20320', date=None, odata_type=None, request_id='99194263-b32e-489b-8a05-49ea3cba1d71'), message=\"One or more added object references already exist for the following modified properties: 'members'.\", target=None)\n        \n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"<stdin>\", line 107, in <module>\n  File \"<stdin>\", line 99, in _ansiballz_main\n  File \"<stdin>\", line 47, in invoke_module\n  File \"/usr/lib64/python3.9/runpy.py\", line 225, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib64/python3.9/runpy.py\", line 97, in _run_module_code\n    _run_code(code, mod_globals, init_globals,\n  File \"/usr/lib64/python3.9/runpy.py\", line 87, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_kw500gh4/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 501, in <module>\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_kw500gh4/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 497, in main\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_kw500gh4/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 249, in __init__\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_kw500gh4/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py\", line 469, in __init__\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_kw500gh4/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 307, in exec_module\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_kw500gh4/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py\", line 502, in fail\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_kw500gh4/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible/module_utils/basic.py\", line 1553, in fail_json\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_kw500gh4/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible/module_utils/basic.py\", line 1522, in _return_formatted\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_kw500gh4/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible/module_utils/common/parameters.py\", line 927, in remove_values\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_kw500gh4/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible/module_utils/common/parameters.py\", line 470, in _remove_values_conditions\nTypeError: Value of unknown type: <class 'msgraph.generated.models.o_data_errors.o_data_error.ODataError'>, \n        APIError\n        Code: 400\n        message: None\n        error: MainError(additional_data={}, code='Request_BadRequest', details=None, inner_error=InnerError(additional_data={'date': DateTime(2024, 3, 28, 13, 23, 43, tzinfo=Timezone('UTC'))}, client_request_id='236e30f3-3c69-4fc4-b945-d81bc0d20320', date=None, odata_type=None, request_id='99194263-b32e-489b-8a05-49ea3cba1d71'), message=\"One or more added object references already exist for the following modified properties: 'members'.\", target=None)\n        \n",
  "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
  "rc": 1,
  "_ansible_no_log": false,
  "changed": false
}
hematic commented 5 months ago

To be fair to however you guys are handling this. I get the exact same error when trying the bulk upload from CSV inside the Azure UI itself. If there are users in the group currently in the CSV the whole process fails.

This leads me to believe it is important for the ansible task to first get a list of all existing users and perform a diff against the ones being passed in.

Fred-sun commented 5 months ago

@hematic I tested it locally and didn't have any problems. The following is my use case and the return result. First , I can add Group members normally, after adding, idempotent also holds. Second, I also checked the logic code, which only handles the newly added Group Member. Thank you!

- name: Create Group Root
  azure_rm_adgroup:
    display_name: "{{ resource_prefix }}-Group-Root"
    mail_nickname: "{{ resource_prefix }}-Group-Root"
    state: 'present'
  register: group_create_changed_shouldpass

- name: Create Group Member 1
  azure_rm_adgroup:
    display_name: "{{ resource_prefix }}-Group-Member-1"
    mail_nickname: "{{ resource_prefix }}-Group-Member-1"
    state: 'present'
  register: create_pass_first

- name: Create Group Member 2
  azure_rm_adgroup:
    display_name: "{{ resource_prefix }}-Group-Member-2"
    mail_nickname: "{{ resource_prefix }}-Group-Member-2"
    state: 'present'
  register: create_pass_second

- name: Add first Group Members
  azure_rm_adgroup:
    display_name: "{{ resource_prefix }}-Group-Root"
    mail_nickname: "{{ resource_prefix }}-Group-Root"
    state: 'present'
    present_members:
      - "{{ create_pass_first.object_id }}"

- name: Add secondary Group Members
  azure_rm_adgroup:
    display_name: "{{ resource_prefix }}-Group-Root"
    mail_nickname: "{{ resource_prefix }}-Group-Root"
    state: 'present'
    present_members:
      - "{{ create_pass_second.object_id }}"
  register: add_pass

- name: Ensure member is in group that is already present using object_id (Idempotent)
  azure_rm_adgroup:
    object_id: "{{ group_create_changed_shouldpass.object_id }}"
    state: 'present'
    present_members:
      - "{{ create_pass_second.object_id }}"
      - "{{ create_pass_first.object_id }}"
  register: add_already_present_member_to_group_shouldpass

- name: Return a specific group using object_id
  azure_rm_adgroup_info:
    object_id: "{{ group_create_changed_shouldpass.object_id }}"
    return_group_members: true
  register: object_id_shouldpass

- debug:
    var: object_id_shouldpass

Excute result:
TASK [Create Group Root] ********************************************************************************************
changed: [localhost]

TASK [Create Group Member 1] ****************************************************************************************
changed: [localhost]

TASK [Create Group Member 2] ****************************************************************************************
changed: [localhost]

TASK [Add first Group Members] **************************************************************************************
changed: [localhost]

TASK [Add secondary Group Members] **********************************************************************************
changed: [localhost]

TASK [Ensure member is in group that is already present using object_id (Idempotent)] *******************************
ok: [localhost]

TASK [Return a specific group using object_id] **********************************************************************
ok: [localhost]

TASK [debug] ********************************************************************************************************
ok: [localhost] => {
    "object_id_shouldpass": {
        "ad_groups": [
            {
                "description": null,
                "display_name": "fredtest001teest06-Group-Root",
                "group_members": [
                    {
                        "description": null,
                        "display_name": "fredtest001teest06-Group-Member-2",
                        "mail": null,
                        "mail_enabled": null,
                        "mail_nickname": "fredtest001teest06-Group-Member-2",
                        "object_id": "xxxxxxxxxxxxxxxxxx",
                        "security_enabled": null
                    },
                    {
                        "description": null,
                        "display_name": "fredtest001teest06-Group-Member-1",
                        "mail": null,
                        "mail_enabled": null,
                        "mail_nickname": "fredtest001teest06-Group-Member-1",
                        "object_id": "xxxxxxxxxxxxxxxxxx",
                        "security_enabled": null
                    }
                ],
                "mail": null,
                "mail_enabled": false,
                "mail_nickname": "fredtest001teest06-Group-Root",
                "object_id": "xxxxxxxxxx",
                "security_enabled": true
            }
        ],
        "changed": false,
        "failed": false
    }
}
hematic commented 5 months ago

You are 100% correct. I had done something somehow wrong in the playbook above, i'm not sure what it was. However, in validating your code and my code, i have come across something else with this that i am certain is not working.

Owners.

I ran the following playbook.

# This playbook creates a group in Azure AD and manages its members based on provided user IDs.
- name: Create EntraAD Group
  hosts: localhost
  connection: local
  gather_facts: false

  vars:
    user_object_ids: "{{ survey_user_object_ids.split(',') }}" # Split user object IDs from a comma-separated string
    owner_object_ids: "{{ survey_owner_object_ids.split(',')}}" # Split owner object IDs
    group_display_name: "{{ survey_group_display_name }}" # Set the display name of the group
    group_mail_nickname: "{{ group_display_name.replace('_', '') }}" # Set the mail nickname of the group
    matching_user_object_ids: [] # Initialize an empty list for users that exist in Azure AD
    non_matching_user_object_ids: [] # Initialize an empty list for users not found in Azure AD
    matching_owner_object_ids: [] # Initialize an empty list for owner that exist in Azure AD
    non_matching_owner_object_ids: [] # Initialize an empty list for owners not found in Azure AD

  tasks:
    - name: Create EntraID Group
      azure.azcollection.azure_rm_adgroup:
        tenant: "{{ azure_tenant_id }}"
        display_name: "{{ group_display_name }}"
        mail_nickname: "{{ group_mail_nickname }}"
        state: 'present'

    # This task loops through all of the user object ids and checks to see if they exist in Entra ID. It ignores failures.
    # Outputs these found user object ids to the variable user_info
    - name: Retrieve Entra Data for Users
      azure.azcollection.azure_rm_aduser_info:
        object_id: "{{ item }}"
      register: user_info
      with_items: "{{ user_object_ids }}"
      when: user_object_ids is defined
      ignore_errors: true

    # This task loops through all of the owner object ids and checks to see if they exist in Entra ID. It ignores failures.
    # Outputs these found owner object ids to the variable owner_info
    - name: Retrieve Entra Data for Owners
      azure.azcollection.azure_rm_aduser_info:
        object_id: "{{ item }}"
      register: owner_info
      with_items: "{{ owner_object_ids }}"
      when: owner_object_ids is defined
      ignore_errors: true

    # This task loops through the results returned from looking up the user ids in entra.
    # It only runs if results were found.
    # It creates a new list called matching_user_object_ids that only contains the matched ids.
    - name: Create List of Matching ObjectIDs for Users
      ansible.builtin.set_fact:
        matching_user_object_ids: "{{ matching_user_object_ids + [item.item] }}"
      loop: "{{ user_info.results }}"
      when:
        - user_info.results is defined
        - item.ad_users is defined and item.ad_users | length > 0

    # Here we output any user object ids that were matched.
    # This only runs if there were matched object ids.
    - name: Debug Output of Matched User ObjectIDs
      ansible.builtin.debug:
        msg: "These User Object IDs were found in Entra ID: {{ matching_user_object_ids }}"
      when: matching_user_object_ids | length > 0

    # This task loops through the results returned from looking up the owner ids in entra.
    # It only runs if results were found.
    # It creates a new list called matching_owner_object_ids that only contains the matched ids.
    - name: Create List of Matching ObjectIDs for Owners
      ansible.builtin.set_fact:
        matching_owner_object_ids: "{{ matching_owner_object_ids + [item.item] }}"
      loop: "{{ owner_info.results }}"
      when:
        - owner_info.results is defined
        - item.ad_users is defined and item.ad_users | length > 0

    # Here we output any owner object ids that were matched.
    # This only runs if there were matched owner object ids.
    - name: Debug Output of Matched Owner ObjectIDs
      ansible.builtin.debug:
        msg: "These User Object IDs were found in Entra ID: {{ matching_owner_object_ids }}"
      when: matching_owner_object_ids | length > 0

    # This task adds members to the group. We split members and owners separate for ease of control.
    - name: Add members to EntraID Group
      azure.azcollection.azure_rm_adgroup:
        tenant: "{{ azure_tenant_id }}"
        display_name: "{{ group_display_name }}"
        mail_nickname: "{{ group_mail_nickname }}"
        state: 'present'
        present_members: "{{ matching_user_object_ids }}"
      when: matching_user_object_ids is defined and matching_user_object_ids | length > 0

    # This task adds owners to the group. We split members and owners separate for ease of control.
    - name: Add owners to EntraID Group
      azure.azcollection.azure_rm_adgroup:
        tenant: "{{ azure_tenant_id }}"
        display_name: "{{ group_display_name }}"
        mail_nickname: "{{ group_mail_nickname }}"
        state: 'present'
        present_owners: "{{ matching_owner_object_ids }}"
      when: matching_owner_object_ids is defined and matching_owner_object_ids | length > 0

I ran this 4 times with different setups and the results are below.

When creating just the group, and passing no other vars, this is successful. When creating the group, and passing in good user object ids, this is successful. When creating the group, and passing in good user object ids and bad object ids, this is successful.

However any time i try to add an owner i get the following error.

{ "module_stdout": "", "module_stderr": "Traceback (most recent call last):\n File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_f6g9gplq/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 303, in exec_module\n File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_f6g9gplq/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 340, in update_owners\n File \"/usr/lib64/python3.9/asyncio/base_events.py\", line 647, in run_until_complete\n return future.result()\n File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_f6g9gplq/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 484, in get_group_owners\n File \"/usr/local/lib/python3.9/site-packages/msgraph/generated/groups/item/owners/owners_request_builder.py\", line 59, in get\n request_info = self.to_get_request_information(\n File \"/usr/local/lib/python3.9/site-packages/msgraph/generated/groups/item/owners/owners_request_builder.py\", line 82, in to_get_request_information\n request_info.headers.add_all(request_configuration.headers)\n File \"/usr/local/lib/python3.9/site-packages/kiota_abstractions/headers_collection.py\", line 83, in add_all\n for key, values in headers.get_all().items():\nAttributeError: 'dict' object has no attribute 'get_all'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<stdin>\", line 107, in <module>\n File \"<stdin>\", line 99, in _ansiballz_main\n File \"<stdin>\", line 47, in invoke_module\n File \"/usr/lib64/python3.9/runpy.py\", line 225, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.9/runpy.py\", line 97, in _run_module_code\n _run_code(code, mod_globals, init_globals,\n File \"/usr/lib64/python3.9/runpy.py\", line 87, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_f6g9gplq/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 501, in <module>\n File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_f6g9gplq/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 497, in main\n File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_f6g9gplq/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 249, in __init__\n File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_f6g9gplq/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py\", line 469, in __init__\n File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_f6g9gplq/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 307, in exec_module\n File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_f6g9gplq/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py\", line 502, in fail\n File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_f6g9gplq/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible/module_utils/basic.py\", line 1553, in fail_json\n File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_f6g9gplq/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible/module_utils/basic.py\", line 1522, in _return_formatted\n File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_f6g9gplq/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible/module_utils/common/parameters.py\", line 927, in remove_values\n File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_f6g9gplq/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible/module_utils/common/parameters.py\", line 470, in _remove_values_conditions\nTypeError: Value of unknown type: <class 'AttributeError'>, 'dict' object has no attribute 'get_all'\n", "exception": "Traceback (most recent call last):\n File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_f6g9gplq/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 303, in exec_module\n File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_f6g9gplq/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 340, in update_owners\n File \"/usr/lib64/python3.9/asyncio/base_events.py\", line 647, in run_until_complete\n return future.result()\n File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_f6g9gplq/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 484, in get_group_owners\n File \"/usr/local/lib/python3.9/site-packages/msgraph/generated/groups/item/owners/owners_request_builder.py\", line 59, in get\n request_info = self.to_get_request_information(\n File \"/usr/local/lib/python3.9/site-packages/msgraph/generated/groups/item/owners/owners_request_builder.py\", line 82, in to_get_request_information\n request_info.headers.add_all(request_configuration.headers)\n File \"/usr/local/lib/python3.9/site-packages/kiota_abstractions/headers_collection.py\", line 83, in add_all\n for key, values in headers.get_all().items():\nAttributeError: 'dict' object has no attribute 'get_all'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<stdin>\", line 107, in <module>\n File \"<stdin>\", line 99, in _ansiballz_main\n File \"<stdin>\", line 47, in invoke_module\n File \"/usr/lib64/python3.9/runpy.py\", line 225, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.9/runpy.py\", line 97, in _run_module_code\n _run_code(code, mod_globals, init_globals,\n File \"/usr/lib64/python3.9/runpy.py\", line 87, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_f6g9gplq/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 501, in <module>\n File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_f6g9gplq/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 497, in main\n File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_f6g9gplq/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 249, in __init__\n File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_f6g9gplq/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py\", line 469, in __init__\n File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_f6g9gplq/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 307, in exec_module\n File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_f6g9gplq/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py\", line 502, in fail\n File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_f6g9gplq/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible/module_utils/basic.py\", line 1553, in fail_json\n File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_f6g9gplq/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible/module_utils/basic.py\", line 1522, in _return_formatted\n File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_f6g9gplq/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible/module_utils/common/parameters.py\", line 927, in remove_values\n File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_f6g9gplq/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible/module_utils/common/parameters.py\", line 470, in _remove_values_conditions\nTypeError: Value of unknown type: <class 'AttributeError'>, 'dict' object has no attribute 'get_all'\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1, "_ansible_no_log": false, "changed": false }

The way user and owner lists are created and handled are identical in the above playbook. users works, owners does not.

Fred-sun commented 5 months ago

@hematic I executed your script locally for many times, and did not encounter the error you mentioned above, and I can pass the test normally (the results are as follows). Thank you!

From your use case, you are still using the version before v2.1.0(https://github.com/ansible-collections/azure/blob/v2.1.0/plugins/modules/azure_rm_adgroup.py), and the tenant parameter for ad resource has been removed from the version after v2.1.0. It is recommended that you update to the latest version. Thank you!


TASK [Create EntraID Group] ************************************************************************************************
changed: [localhost]

TASK [Retrieve Entra Data for Users] ***************************************************************************************
ok: [localhost] => (item=86XXXXXXXXXXXXXXXXXXXXXXXXXXXX63)
ok: [localhost] => (item=c4XXXXXXXXXXXXXXXXXXXXXXXXXXXXfb)

TASK [Retrieve Entra Data for Owners] **************************************************************************************
ok: [localhost] => (item=f7XXXXXXXXXXXXXXXXXXXXXXXXXXXX38)
ok: [localhost] => (item=41XXXXXXXXXXXXXXXXXXXXXXXXXXXX28)

TASK [Create List of Matching ObjectIDs for Users] *************************************************************************
ok: [localhost] => (item={'changed': False, 'ad_users': [{'object_id': '86XXXXXXXXXXXXXXXXXXXXXXXXXXXX63', 'display_name': 'test17', 'user_principal_name': 'test17@824736848qq.onmicrosoft.com', 'mail_nickname': 'test17', 'mail': None, 'account_enabled': True, 'user_type': 'Member', 'company_name': None, 'on_premises_extension_attributes': {}}], 'invocation': {'module_args': {'object_id': '86XXXXXXXXXXXXXXXXXXXXXXXXXXXX63', 'auth_source': 'auto', 'cloud_environment': 'AzureCloud', 'api_profile': 'latest', 'disable_instance_discovery': False, 'profile': None, 'subscription_id': None, 'client_id': None, 'secret': None, 'tenant': None, 'ad_user': None, 'password': None, 'cert_validation_mode': None, 'adfs_authority_url': None, 'log_mode': None, 'log_path': None, 'x509_certificate_path': None, 'thumbprint': None, 'user_principal_name': None, 'attribute_name': None, 'attribute_value': None, 'odata_filter': None, 'all': None}}, 'failed': False, 'item': '86XXXXXXXXXXXXXXXXXXXXXXXXXXXX63', 'ansible_loop_var': 'item'})
ok: [localhost] => (item={'changed': False, 'ad_users': [{'object_id': 'c4XXXXXXXXXXXXXXXXXXXXXXXXXXXXfb', 'display_name': 'test18', 'user_principal_name': 'test18@824736848qq.onmicrosoft.com', 'mail_nickname': 'test18', 'mail': None, 'account_enabled': True, 'user_type': 'Member', 'company_name': None, 'on_premises_extension_attributes': {}}], 'invocation': {'module_args': {'object_id': 'c4XXXXXXXXXXXXXXXXXXXXXXXXXXXXfb', 'auth_source': 'auto', 'cloud_environment': 'AzureCloud', 'api_profile': 'latest', 'disable_instance_discovery': False, 'profile': None, 'subscription_id': None, 'client_id': None, 'secret': None, 'tenant': None, 'ad_user': None, 'password': None, 'cert_validation_mode': None, 'adfs_authority_url': None, 'log_mode': None, 'log_path': None, 'x509_certificate_path': None, 'thumbprint': None, 'user_principal_name': None, 'attribute_name': None, 'attribute_value': None, 'odata_filter': None, 'all': None}}, 'failed': False, 'item': 'c4XXXXXXXXXXXXXXXXXXXXXXXXXXXXfb', 'ansible_loop_var': 'item'})

TASK [Debug Output of Matched User ObjectIDs] ******************************************************************************
ok: [localhost] => {
    "msg": "These User Object IDs were found in Entra ID: ['86XXXXXXXXXXXXXXXXXXXXXXXXXXXX63', 'c4XXXXXXXXXXXXXXXXXXXXXXXXXXXXfb']"
}

TASK [Create List of Matching ObjectIDs for Owners] ************************************************************************
ok: [localhost] => (item={'changed': False, 'ad_users': [{'object_id': 'f7XXXXXXXXXXXXXXXXXXXXXXXXXXXX38', 'display_name': 'test15', 'user_principal_name': 'test15@824736848qq.onmicrosoft.com', 'mail_nickname': 'test15', 'mail': None, 'account_enabled': True, 'user_type': 'Member', 'company_name': None, 'on_premises_extension_attributes': {}}], 'invocation': {'module_args': {'object_id': 'f7XXXXXXXXXXXXXXXXXXXXXXXXXXXX38', 'auth_source': 'auto', 'cloud_environment': 'AzureCloud', 'api_profile': 'latest', 'disable_instance_discovery': False, 'profile': None, 'subscription_id': None, 'client_id': None, 'secret': None, 'tenant': None, 'ad_user': None, 'password': None, 'cert_validation_mode': None, 'adfs_authority_url': None, 'log_mode': None, 'log_path': None, 'x509_certificate_path': None, 'thumbprint': None, 'user_principal_name': None, 'attribute_name': None, 'attribute_value': None, 'odata_filter': None, 'all': None}}, 'failed': False, 'item': 'f7XXXXXXXXXXXXXXXXXXXXXXXXXXXX38', 'ansible_loop_var': 'item'})
ok: [localhost] => (item={'changed': False, 'ad_users': [{'object_id': '41XXXXXXXXXXXXXXXXXXXXXXXXXXXX28', 'display_name': 'test16', 'user_principal_name': 'test16@824736848qq.onmicrosoft.com', 'mail_nickname': 'test16', 'mail': None, 'account_enabled': True, 'user_type': 'Member', 'company_name': None, 'on_premises_extension_attributes': {}}], 'invocation': {'module_args': {'object_id': '41XXXXXXXXXXXXXXXXXXXXXXXXXXXX28', 'auth_source': 'auto', 'cloud_environment': 'AzureCloud', 'api_profile': 'latest', 'disable_instance_discovery': False, 'profile': None, 'subscription_id': None, 'client_id': None, 'secret': None, 'tenant': None, 'ad_user': None, 'password': None, 'cert_validation_mode': None, 'adfs_authority_url': None, 'log_mode': None, 'log_path': None, 'x509_certificate_path': None, 'thumbprint': None, 'user_principal_name': None, 'attribute_name': None, 'attribute_value': None, 'odata_filter': None, 'all': None}}, 'failed': False, 'item': '41XXXXXXXXXXXXXXXXXXXXXXXXXXXX28', 'ansible_loop_var': 'item'})

TASK [Debug Output of Matched Owner ObjectIDs] *****************************************************************************
ok: [localhost] => {
    "msg": "These User Object IDs were found in Entra ID: ['f7d6e8a5-637f-4ad0-9b67-547ea0446938', '41b907de-474b-425c-89ee-de2afe170928']"
}

TASK [Add members to EntraID Group] ****************************************************************************************
changed: [localhost]

TASK [Add owners to EntraID Group] *****************************************************************************************
changed: [localhost]

PLAY RECAP *****************************************************************************************************************
localhost                  : ok=10   changed=3    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
howardjones commented 1 month ago

I am also seeing this issue with 2.6.0 (in my case it's re-running the same playbook to ensure the same users is in the same group as part of a much larger playbook, but this single task isolated will fail with the same error:

One or more added object references already exist for the following modified properties: 'members'
Fred-sun commented 3 weeks ago

@hematic Could you please upgrade to v2.6.0 and try again, because I haven't encountered the problem you mentioned in many local tests, thank you!

howardjones commented 2 weeks ago
---
- hosts: localhost
  connection: local
  vars:
    state: present
    user_obj_id: aaaaaaaa-bbbb-cccc-dddd-a2c785dc0dc8
    group_obj_id: aaaaaaaa-bbbb-cccc-dddd-6de8cbd4e560
  tasks:
  - name: Check version of azure.azcollection
    ansible.builtin.debug:
      msg: "azure.azcollection version {{ lookup('community.general.collection_version', 'azure.azcollection') }}"
  - name: Ensure Users are Members of a Group using object_id
    azure.azcollection.azure_rm_adgroup:
      object_id: "{{ group_obj_id }}"
      state: 'present'
      present_members:        
        # https://github.com/ansible-collections/azure/issues/1387
        # https://github.com/ansible-collections/azure/issues/1519
        - "{{ user_obj_id }}"
TASK [Check version of azure.azcollection] ************************************************************************************************************
ok: [localhost] => {
    "msg": "azure.azcollection version 2.6.0"
}

TASK [Ensure Users are Members of a Group using object_id] ********************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was:         error: MainError(additional_data={}, code='Request_BadRequest', details=None, inner_error=InnerError(additional_data={'date': DateTime(2024, 8, 22, 13, 28, 38, tzinfo=Timezone('UTC'))}, client_request_id='d71b5bc7-699d-46cd-901b-deb3a6dc208a', date=None, odata_type=None, request_id='cccf7778-83e2-442a-9ed9-5c8af545e46d'), message="One or more added object references already exist for the following modified properties: 'members'.", target=None)
howardjones commented 2 weeks ago

Debugging a bit more, it seems that the get_group_members() returns an empty list

CURRENT
[]

PRESENT_BY_OBJ
{'aaaaaaaa-bbbb-cccc-9de2-a2c785dc0dc8': 'aaaaaaaa-bbbb-cccc-dddd-a2c785dc0dc8'}

MEMBERS_TO_ADD
['aaaaaaaa-bbbb-cccc-dddd-a2c785dc0dc8']
hematic commented 2 weeks ago

@hematic Could you please upgrade to v2.6.0 and try again, because I haven't encountered the problem you mentioned in many local tests, thank you!

I can confirm that this works absolutely great in 2.6.0

Fred-sun commented 2 weeks ago

@howardjones Thank you very much for your feedback, I will close it for the time being.