Route table resourceID is using the subscription_id from credentials profile instead of subscription_id provided to the route_table parameter

davejdeemer commented 2 years ago

SUMMARY

When building/updating subnets, the route_table parameter is not using the value given for the subscription_id. Instead, the subscription_id provided in the credentials file is used (as observed by the state change after apply).

ISSUE TYPE

Bug Report

COMPONENT NAME

azure.azcollection.azure_rm_subnet

ANSIBLE VERSION

ansible [core 2.13.0]
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.10/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.10.5 (main, Jul 25 2022, 15:52:08) [GCC 11.2.1 20220219]
  jinja version = 3.0.3
  libyaml = True

COLLECTION VERSION

Collection         Version
------------------ -------
azure.azcollection 1.12.0

# /root/.ansible/collections/ansible_collections
Collection         Version
------------------ -------
azure.azcollection 1.13.0

CONFIGURATION

n/a

OS / ENVIRONMENT

Micorsoft Azure

STEPS TO REPRODUCE

Azure resources already created (resource groups, vNet, subnet, route-table and routes)
Credentials file with environment variable of AZURE_PROFILE=default
Run playbook with -vvv, observe invocation output:
- The resourceID for the subnet route_table parameter is as expected, using the supplied subscription_id from group_vars
  1. Observe state output:
- The resourceID for the subnet route_table parameter reflects the subscription_id defined within the credentials file
- Flagged as a change because of the route_table resourceID difference

Resource groups sample group vars

---
resource_groups:
  - name: rg-subnet
    subscription: MY-SUBCRIPTION
    region: northcentralus

vNet sample group vars

---
vnets:
  - name: vnet-northcentralus-1
    region: northcentralus
    subscription: MY-SUBCRIPTION
    resource_group: rg-subnet
    address_space:
      - "xx.xxx.xx.x/25"
    subnets:
      - name: subnet-northcentralus-1
        address_space: "xx.xxx.xx.x/25"
        private_endpoint_network_policies: Disabled
        route_table:
          name: rt-northcentralus-1
          resource_group: rg-subnet

Inventory hosts

[nonprod]
azure-non-prod

Playbook

---
- name: Build Azure Networking
  hosts: all
  connection: local
  vars:
    subscription_map: {}
  tasks:
    - name: Get subscription facts for all rg subscriptions
      azure.azcollection.azure_rm_subscription_info:
        name: "{{ item['subscription'] }}"
      loop: "{{ resource_groups }}"
      register: rg_subscription_facts

    - name: Get subscription facts for all vnet subscriptions
      azure.azcollection.azure_rm_subscription_info:
        name: "{{ item['subscription'] }}"
      loop: "{{ vnets }}"
      register: vnet_subscription_facts

    - name: Populate subscription map
      ansible.builtin.set_fact:
        subscription_map: "{{ subscription_map | combine({item.subscriptions[0].display_name: item.subscriptions[0].subscription_id}) }}"
      loop: "{{ vnet_subscription_facts.results + rg_subscription_facts.results }}"

    - name: Build subnets
      azure.azcollection.azure_rm_subnet:
        address_prefix_cidr: "{{ item.1['address_space'] }}"
        name: "{{ item.1['name'] }}"
        private_endpoint_network_policies: "{{ item.1['private_endpoint_network_policies'] }}"
        resource_group: "{{ item.0['resource_group'] }}"
        # Upon running this Ansible task, the state change for the route_table reference uses the subscription_id defined within
        #  the credentials file rather than what is provided here
        route_table: "/subscriptions/{{ subscription_map[item.0['subscription']] }}/resourceGroups/{{ item.1['route_table']['resource_group'] }}/providers/Microsoft.Network/routeTables/{{ item.1['route_table']['name'] }}"
        subscription_id: "{{ subscription_map[item.0['subscription']] }}"
        virtual_network_name: "{{ item.0['name'] }}"
      loop: "{{ vnets | subelements('subnets') }}"
      tags:
        - subnets

Stub credentials file

[default]
subscription_id=c123********
client_id=**removed**
secret=**removed**
tenant=**removed**

Run command

Note: Using check mode during testing

ansible-playbook -C site.yaml --limit nonprod -i hosts -vvv

EXPECTED RESULTS

Given that the subnet exists in our Azure tenant, expected result is no change when applying the Ansible playbook.

ACTUAL RESULTS

Apply of ansible playbook flags a change for the "Build subnets" task. Review of the -vvv output, the invocation shows that the route_table resourceID, specifically the subscription_id, is as expected. Yet review of the state shows that the route_table reference is using the subscription_id defined within the credentials file and not what was gathered in the first task of the playbook.

Subscription info from the first playbook task: "Get subscription facts for all rg subscriptions"

Sensitive output removed from actual run in my environment

    "subscriptions": [
        {
            "display_name": "MY-SUBSCRIPTION",
            "fqid": "/subscriptions/dde8*********",
            "state": "Enabled",
            "subscription_id": "dde8**********",
            "tags": null,
            "tenant_id": "**removed**"
        }
    ]

Subnet task invocation and state (-vvv output)

Sensitive output removed from actual run in my environment / some values changed to align with example above

changed: [azure-non-prod] => (item=[{'name': 'vnet-northcentralus-1', 'region': 'northcentralus', 'subscription': 'MY-SUBSCRIPTION, 'resource_group': 'rg-subnet', 'address_space': ['xx.xxx.xx.x/25'], 'subnets': [{'name': 'subnet-northcentralus-1', 'address_space': 'xx.xxx.xx.x/25', 'private_endpoint_network_policies': 'Disabled', 'route_table': {'name': 'rt-northcentralus-1', 'resource_group': 'rg-subnet'}}]}, {'name': 'subnet-northcentralus-1', 'address_space': 'xx.xxx.xx.x/25', 'private_endpoint_network_policies': 'Disabled', 'route_table': {'name': 'rt-northcentralus-1', 'resource_group': 'rg-subnet'}}]) => {
    "ansible_loop_var": "item",
    "changed": true,
    "invocation": {
        "module_args": {
            "ad_user": null,
            "address_prefix_cidr": "xx.xxx.xx.x/25",
            "address_prefixes_cidr": null,
            "adfs_authority_url": null,
            "api_profile": "latest",
            "auth_source": "auto",
            "cert_validation_mode": null,
            "client_id": null,
            "cloud_environment": "AzureCloud",
            "delegations": null,
            "log_mode": null,
            "log_path": null,
            "name": "subnet-northcentralus-1",
            "password": null,
            "private_endpoint_network_policies": "Disabled",
            "private_link_service_network_policies": "Enabled",
            "profile": "default",
            "resource_group": "rg-subnet",
            "route_table": "/subscriptions/dde8**********/resourceGroups/rg-subnet/providers/Microsoft.Network/routeTables/rt-northcentralus-1",
            "secret": null,
            "security_group": null,
            "service_endpoints": null,
            "state": "present",
            "subscription_id": "dde8********",
            "tenant": null,
            "virtual_network_name": "vnet-northcentralus-1"
        }
    },
    "item": [
        {
            "address_space": [
                "xx.xxx.xx.x/25"
            ],
            "name": "vnet-northcentralus-1",
            "region": "northcentralus",
            "resource_group": "rg-subnet",
            "subnets": [
                {
                    "address_space": "xx.xxx.xx.x/25",
                    "name": "subnet-northcentralus-1",
                    "private_endpoint_network_policies": "Disabled",
                    "route_table": {
                        "name": "rt-northcentralus-1",
                        "resource_group": "rg-subnet"
                    }
                }
            ],
            "subscription": "MY-SUBSCRIPTION"
        },
        {
            "address_space": "xx.xxx.xx.x/25",
            "name": "subnet-northcentralus-1",
            "private_endpoint_network_policies": "Disabled",
            "route_table": {
                "name": "rt-northcentralus-1",
                "resource_group": "rg-subnet"
            }
        }
    ],
    "state": {
        "address_prefix": "xx.xxx.xx.x/25",
        "address_prefixes": null,
        "id": "/subscriptions/dde8*******/resourceGroups/rg-subnet/providers/Microsoft.Network/virtualNetworks/vnet-northcentralus-1/subnets/subnet-northcentralus-1",
        "name": "subnet-northcentralus-1",
        "network_security_group": {},
        "private_endpoint_network_policies": "Disabled",
        "private_link_service_network_policies": "Enabled",
        "provisioning_state": "Succeeded",
        "route_table": {
            "id": "/subscriptions/c123********/resourceGroups/rg-subnet/providers/Microsoft.Network/routeTables/rt-northcentralus-npd-1",
            "name": "rt-northcentralus-1",
            "resource_group": "rg-subnet"
        }
    }
}

Unexpected result is here where the ID should be dde8**** for the target subscription but instead is c123**** from the credentials file:

        "route_table": {
            "id": "/subscriptions/c123********/resourceGroups/rg-subnet/providers/Microsoft.Network/routeTables/rt-northcentralus-npd-1",
            "name": "rt-northcentralus-1",
            "resource_group": "rg-subnet"
        }

As a test, update of the credentials file specifying the target subscription yields the expected results in the example above.

davejdeemer commented 2 years ago

Any thoughts about this? Is there additional information that I can provide to help describe what I'm running into?

Fred-sun commented 2 years ago

@davejdeemer I'm sorry, but I'm really confused by your above information. The value defined by your Playbook is “"route_table": "/ subscriptions/dde8 / resourceGroups/rg-subnet/providers/Microsoft.Net work/routeTables/rt - northcentralus - 1", But the result of the update is "id": "/ subscriptions/c123 / resourceGroups/rg-subnet/providers/Microsoft.Net work/routeTables/rt - northcentralus - NPD - 1", Have you mixed up the information? Thank you very much!

davejdeemer commented 2 years ago

@Fred-sun Appreciate you taking a look at this, thank you!

That is the challenge I've bumped into. The invocation uses the expected subscription id (dde8**) yet the state is actually using the subscription id defined from within my credentials file (c123). As a test, changing my credentials file to specify the subscription id of dde8 will result in the state reflecting subscription id dde8**. This is a challenge because we are operating across multiple subscriptions. Does that help clarify?

Fred-sun commented 2 years ago

@davejdeemer I tested it again locally, but still could not reproduce your problem. The most important thing is that the routing table associated with the subnet must be from the same subscription, thank you!

davejdeemer commented 2 years ago

I'll dig around more on my side. Both a co-worker and I have bumped into this but maybe I've got something not quite right with the code, the example or both.

davejdeemer commented 2 years ago

@Fred-sun An interesting item that I observed is that setting the subscription_id for the route table dict in the example vNet group vars, then building the subnet yields the expected behavior in using the expected subscription_id rather then using what is defined in the credentials file. Here is an update to the example:

---
vnets:
  - name: vnet-northcentralus-1
    region: northcentralus
    subscription: MY-SUBCRIPTION
    resource_group: rg-subnet
    address_space:
      - "xx.xxx.xx.x/25"
    subnets:
      - name: subnet-northcentralus-1
        address_space: "xx.xxx.xx.x/25"
        private_endpoint_network_policies: Disabled
        route_table:
          name: rt-northcentralus-1
          resource_group: rg-subnet
          subscription_id: dde8**** <<--- New; added as part of the routing table config

In the collection documentation it doesn't mention the dict can (perhaps should?) contain the subscription_id. When supplied, the invocation results in the expected output. Otherwise, the route table will reference the subscription_id defined within the credentials file.

Here's the output from running the example with that modification:

ok: [azure-sbx] => (item=[{'name': 'vnet-northcentralus-1', 'region': 'northcentralus', 'subscription': 'MY-SUBSCRIPTION', 'resource_group': 'rg-subnet', 'address_space': ['xx.xxx.xx.x/26'], 'peered_to_core': True, 'subnets': [{'name': 'subnet-northcentralus-1', 'address_space': 'xx.xxx.xx.x/26', 'private_endpoint_network_policies': 'Disabled', 'route_table': {'name': 'rt-northcentralus-1', 'resource_group': 'rg-subnet', 'subscription_id': 'dde8****'}}]}, {'name': 'subnet-northcentralus-1', 'address_space': 'xx.xxx.xx.x/26', 'private_endpoint_network_policies': 'Disabled', 'route_table': {'name': 'rt-northcentralus-1', 'resource_group': 'rg-subnet', 'subscription_id': 'dde8****'}}]) => {
    "ansible_loop_var": "item",
    "changed": false,
    "invocation": {
        "module_args": {
            "ad_user": null,
            "address_prefix_cidr": "xx.xxx.xx.x/26",
            "address_prefixes_cidr": null,
            "adfs_authority_url": null,
            "api_profile": "latest",
            "auth_source": "auto",
            "cert_validation_mode": null,
            "client_id": null,
            "cloud_environment": "AzureCloud",
            "delegations": null,
            "log_mode": null,
            "log_path": null,
            "name": "subnet-northcentralus-1",
            "password": null,
            "private_endpoint_network_policies": "Disabled",
            "private_link_service_network_policies": "Enabled",
            "profile": null,
            "resource_group": "rg-subnet",
            "route_table": {
                "name": "rt-northcentralus-1",
                "resource_group": "rg-subnetworking",
                "subscription_id": "dde8****"
            },
            "secret": null,
            "security_group": null,
            "service_endpoints": null,
            "state": "present",
            "subscription_id": "dde8****",
            "tenant": null,
            "virtual_network_name": "vnet-northcentralus-1"
        }
    },
    "item": [
        {
            "address_space": [
                "xx.xxx.xx.x/26"
            ],
            "name": "vnet-northcentralus-1",
            "peered_to_core": true,
            "region": "northcentralus",
            "resource_group": "rg-subnet",
            "subnets": [
                {
                    "address_space": "xx.xxx.xx.x/26",
                    "name": "subnet-northcentralus-1",
                    "private_endpoint_network_policies": "Disabled",
                    "route_table": {
                        "name": "rt-northcentralus-1",
                        "resource_group": "rg-subnet",
                        "subscription_id": "dde8****"
                    }
                }
            ],
            "subscription": "MY-SUBSCRPTION"
        },
        {
            "address_space": "xx.xxx.xx.x/26",
            "name": "subnet-northcentralus-1",
            "private_endpoint_network_policies": "Disabled",
            "route_table": {
                "name": "rt-northcentralus-1",
                "resource_group": "rg-subnet",
                "subscription_id": "dde8****"
            }
        }
    ],
    "state": {
        "address_prefix": "xx.xxx.xx.x/26",
        "address_prefixes": null,
        "id": "/subscriptions/dde8****/resourceGroups/rg-subnet/providers/Microsoft.Network/virtualNetworks/vnet-northcentralus-1/subnets/subnet-northcentralus-1",
        "name": "subnet-northcentralus1",
        "network_security_group": {},
        "private_endpoint_network_policies": "Disabled",
        "private_link_service_network_policies": "Enabled",
        "provisioning_state": "Succeeded",
        "route_table": {
            "id": "/subscriptions/dde8****/resourceGroups/rg-subnet/providers/Microsoft.Network/routeTables/rt-northcentralus-1",
            "name": "rt-northcentralus-1",
            "resource_group": "rg-subnet"
        }
    }
}

Note the change in result for invocation; specifically the route_table -> id. Now it correctly references the expected subscription_id of dde8****.

I understand you are not seeing the same behavior in your tests. A co-worker of mine runs into the same thing I am describing, perhaps it's something within our environment or a nuance with the test case as described. Hopefully the observation noted above will help.

Clarification Update:

Updated "build subnets" task (the original example was altered during testing of this case):

  - name: Build subnets
    azure.azcollection.azure_rm_subnet:
      address_prefix_cidr: "{{ item.1['address_space'] }}"
      name: "{{ item.1['name'] }}"
      private_endpoint_network_policies: "{{ item.1['private_endpoint_network_policies'] }}"
      resource_group: "{{ item.0['resource_group'] }}"
      route_table: "{{ item.1['route_table'] }}"
      subscription_id: "{{ subscription_map[item.0['subscription']] }}"
      virtual_network_name: "{{ item.0['name'] }}"
    loop: "{{ vnets|subelements('subnets') }}"

It appears that when no subscription_id is provided for for the route_table dict, the module will use the subscription_id provided in the credential file.

Fred-sun commented 2 years ago

@davejdeemer Although you can specify subscription_id, subnets under different subs cannot be directly associated with routing tables. Thanks!


 "Error creating or updating subnet foobar - (LinkedAuthorizationFailed) The client has permission to perform action 'Microsoft.Network/routeTables/join/action' on scope '/subscriptions/xxxxxxxx/resourceGroups/v-xisuRG01/providers/Microsoft.Network/virtualNetworks/My_Virtual_Network07/subnets/foobar', however the current tenant 'yyyyyyyy' is not authorized to access linked subscription 'zzzzzzzzzzz'.\nCode: LinkedAuthorizationFailed\nMessage: The client has permission to perform action 'Microsoft.Network/routeTables/join/action' on scope '/subscriptions/xxxxxxxxxxxx/resourceGroups/v-xisuRG01/providers/Microsoft.Network/virtualNetworks/My_Virtual_Network07/subnets/foobar7', however the current tenant 'yyyyyyyyyyy' is not authorized to access linked subscription 'zzzzzzzzzzzzzz'.

davejdeemer commented 2 years ago

@Fred-sun Understood. The interesting thing is that the subscription_id defined for the subnet isn't being used for the route table reference in this case. Instead, it is using the subscription_id defined within the credentials file. When explicitly setting the subscription_id in the route-table dict (duplicating the subscription set for the subnet), the route-table reference is as expected.

davejdeemer commented 2 years ago

Here are a few observations from the code:

Here is where the route table dict is inspected
In the code that parses the input is where I believe I'm bumping into this problem. If no subscription_id is passed in, then whatever subscription_id is set elsewhere, potentially within this function, is what is returned (which is the value from my credentials file). Instead, perhaps the code should use the subscription_id set for the subnet if supplied (which I am providing).

ghost commented 3 months ago

Hi,

Do you know if this bug has been fixed? I'm runnig into the same problem:

Thanks for your help.

fatal: [localhost]: FAILED! => { "changed": false, "invocation": { "module_args": { "ad_user": null, "address_prefix": "192.168.100.0/24", "address_prefix_cidr": "192.168.100.0/24", "address_prefixes_cidr": null, "adfs_authority_url": null, "api_profile": "latest", "auth_source": "auto", "cert_validation_mode": null, "client_id": null, "cloud_environment": "AzureCloud", "delegations": null, "disable_instance_discovery": false, "log_mode": null, "log_path": null, "name": "subProdeastus-Test-Networking01", "nat_gateway": null, "password": null, "private_endpoint_network_policies": "Enabled", "private_link_service_network_policies": "Enabled", "profile": null, "resource_group": "rgProdeastus-Test-Networking01", "route_table": "/subscriptions/ef94xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/resourceGroups/rgProdeastus-Test-Networking01/providers/Microsoft.Network/routeTables/rtProdeastus-Test-Networking01", "secret": null, "security_group": null, "service_endpoints": null, "state": "present", "subscription_id": "ef94xxxxxxxxxxxxxxxxxxxxxxxxxxx", "tenant": null, "thumbprint": null, "virtual_network": "vnProdeastus-Test-Networking01", "virtual_network_name": "vnProdeastus-Test-Networking01", "x509_certificate_path": null } }, "msg": "Error creating or updating subnet subProdeastus-Test-Networking01 - (InvalidResourceReference) Resource /subscriptions/ca81180xxxxxxxxxxxxxxxxxxxxxxxxxxx/resourceGroups/RGPRODEASTUS-TEST-NETWORKING01/providers/Microsoft.Network/routeTables/RTPRODEASTUS-TEST-NETWORKING01 referenced by resource /subscriptions/ef94bbd9-34e4-4b85-b4bc-b40d07d3791f/resourceGroups/rgProdeastus-Test-Networking01/providers/Microsoft.Network/virtualNetworks/vnProdeastus-Test-Networking01/subnets/subProdeastus-Test-Networking01 was not found. Please make sure that the referenced resource exists, and that both resources are in the same region.\nCode: InvalidResourceReference\nMessage: Resource /subscriptions/ca81180xxxxxxxxxxxxxxxxxxxxxxxxxxx/resourceGroups/RGPRODEASTUS-TEST-NETWORKING01/providers/Microsoft.Network/routeTables/RTPRODEASTUS-TEST-NETWORKING01 referenced by resource /subscriptions/ef94bbd9-34e4-4b85-b4bc-b40d07d3791f/resourceGroups/rgProdeastus-Test-Networking01/providers/Microsoft.Network/virtualNetworks/vnProdeastus-Test-Networking01/subnets/subProdeastus-Test-Networking01 was not found. Please make sure that the referenced resource exists, and that both resources are in the same region.\nException Details:\t(NotFound) Resource /subscriptions/ca81180xxxxxxxxxxxxxxxxxxxxxxxxxxx/resourceGroups/RGPRODEASTUS-TEST-NETWORKING01/providers/Microsoft.Network/routeTables/RTPRODEASTUS-TEST-NETWORKING01 not found.\n\tCode: NotFound\n\tMessage: Resource /subscriptions/ca81180xxxxxxxxxxxxxxxxxxxxxxxxxxx/resourceGroups/RGPRODEASTUS-TEST-NETWORKING01/providers/Microsoft.Network/routeTables/RTPRODEASTUS-TEST-NETWORKING01 not found." }

Thanks

ansible-collections / azure