NotImplementedError: APIVersion 2019-06-01 is not available for azure_rm_virtualnetwork and related

[mark100net]

SUMMARY

ISSUE TYPE

• Bug Report

As noted in the coment in the code below, I do not have a problem if I use the default tasks from the cloud/ansible module. But if I use the most recent code from the collection, create virtual network and several related tasks break in the same way.

COMPONENT NAME

ansible-collection version 0.1.3 installed via:

ansible-galaxy collection install azure.azcollection --force

ANSIBLE VERSION

2.9.9 (installed via `pip3 install ansible`)

CONFIGURATION

No output except DEFAULT_VALUE_PASSWORD_FILE setting

OS / ENVIRONMENT

MacOS 10.15.4

STEPS TO REPRODUCE

Configure Azure CLI with credentials then run following tasks:

    - name: Create resource group 
      azure.azcollection.azure_rm_resourcegroup: # this works
      name: "{{azure_rg_name}}"
      location: "{{azure_region}}"

    - name: Create virtual network
      azure.azcollection.azure_rm_virtualnetwork: # works if I remove `azure.azcollection.`
        resource_group: "{{azure_rg_name}}"
        name: "{{azure_vnet_name}}"
        address_prefixes: "10.0.0.0/16"

EXPECTED RESULTS

Virtual network is created in azure.

ACTUAL RESULTS

TASK [create-azure-vm : Create virtual network] *********************************************************************************************************************************************************************************************************************
task path: /Users/mark/development/rubyforgood/babywearing/deploy/ansible/create-azure-vm/tasks/main.yml:12
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: mark
<127.0.0.1> EXEC /bin/sh -c 'echo ~mark && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/mark/.ansible/tmp `"&& mkdir /Users/mark/.ansible/tmp/ansible-tmp-1589607011.755611-53797-34652261434357 && echo ansible-tmp-1589607011.755611-53797-34652261434357="` echo /Users/mark/.ansible/tmp/ansible-tmp-1589607011.755611-53797-34652261434357 `" ) && sleep 0'
Using module file /Users/mark/.ansible/collections/ansible_collections/azure/azcollection/plugins/modules/azure_rm_virtualnetwork.py
<127.0.0.1> PUT /Users/mark/.ansible/tmp/ansible-local-53600_6fm55ho/tmp7ivwdgzr TO /Users/mark/.ansible/tmp/ansible-tmp-1589607011.755611-53797-34652261434357/AnsiballZ_azure_rm_virtualnetwork.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /Users/mark/.ansible/tmp/ansible-tmp-1589607011.755611-53797-34652261434357/ /Users/mark/.ansible/tmp/ansible-tmp-1589607011.755611-53797-34652261434357/AnsiballZ_azure_rm_virtualnetwork.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/local/opt/python/bin/python3.7 /Users/mark/.ansible/tmp/ansible-tmp-1589607011.755611-53797-34652261434357/AnsiballZ_azure_rm_virtualnetwork.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /Users/mark/.ansible/tmp/ansible-tmp-1589607011.755611-53797-34652261434357/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/Users/mark/.ansible/tmp/ansible-tmp-1589607011.755611-53797-34652261434357/AnsiballZ_azure_rm_virtualnetwork.py", line 102, in <module>
    _ansiballz_main()
  File "/Users/mark/.ansible/tmp/ansible-tmp-1589607011.755611-53797-34652261434357/AnsiballZ_azure_rm_virtualnetwork.py", line 94, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/Users/mark/.ansible/tmp/ansible-tmp-1589607011.755611-53797-34652261434357/AnsiballZ_azure_rm_virtualnetwork.py", line 40, in invoke_module
    runpy.run_module(mod_name='ansible_collections.azure.azcollection.plugins.modules.azure_rm_virtualnetwork', init_globals=None, run_name='__main__', alter_sys=True)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 205, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/var/folders/dh/qjj1yjqd2yz20zjh6n8r3w380000gn/T/ansible_azure.azcollection.azure_rm_virtualnetwork_payload_p2ddo37m/ansible_azure.azcollection.azure_rm_virtualnetwork_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_virtualnetwork.py", line 394, in <module>
  File "/var/folders/dh/qjj1yjqd2yz20zjh6n8r3w380000gn/T/ansible_azure.azcollection.azure_rm_virtualnetwork_payload_p2ddo37m/ansible_azure.azcollection.azure_rm_virtualnetwork_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_virtualnetwork.py", line 390, in main
  File "/var/folders/dh/qjj1yjqd2yz20zjh6n8r3w380000gn/T/ansible_azure.azcollection.azure_rm_virtualnetwork_payload_p2ddo37m/ansible_azure.azcollection.azure_rm_virtualnetwork_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_virtualnetwork.py", line 245, in __init__
  File "/var/folders/dh/qjj1yjqd2yz20zjh6n8r3w380000gn/T/ansible_azure.azcollection.azure_rm_virtualnetwork_payload_p2ddo37m/ansible_azure.azcollection.azure_rm_virtualnetwork_payload.zip/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py", line 418, in __init__
  File "/var/folders/dh/qjj1yjqd2yz20zjh6n8r3w380000gn/T/ansible_azure.azcollection.azure_rm_virtualnetwork_payload_p2ddo37m/ansible_azure.azcollection.azure_rm_virtualnetwork_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_virtualnetwork.py", line 272, in exec_module
  File "/usr/local/lib/python3.7/site-packages/azure/mgmt/network/network_management_client.py", line 2449, in virtual_networks
    raise NotImplementedError("APIVersion {} is not available".format(api_version))
NotImplementedError: APIVersion 2019-06-01 is not available
fatal: [localhost]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/Users/mark/.ansible/tmp/ansible-tmp-1589607011.755611-53797-34652261434357/AnsiballZ_azure_rm_virtualnetwork.py\", line 102, in <module>\n    _ansiballz_main()\n  File \"/Users/mark/.ansible/tmp/ansible-tmp-1589607011.755611-53797-34652261434357/AnsiballZ_azure_rm_virtualnetwork.py\", line 94, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/Users/mark/.ansible/tmp/ansible-tmp-1589607011.755611-53797-34652261434357/AnsiballZ_azure_rm_virtualnetwork.py\", line 40, in invoke_module\n    runpy.run_module(mod_name='ansible_collections.azure.azcollection.plugins.modules.azure_rm_virtualnetwork', init_globals=None, run_name='__main__', alter_sys=True)\n  File \"/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py\", line 205, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py\", line 96, in _run_module_code\n    mod_name, mod_spec, pkg_name, script_name)\n  File \"/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/var/folders/dh/qjj1yjqd2yz20zjh6n8r3w380000gn/T/ansible_azure.azcollection.azure_rm_virtualnetwork_payload_p2ddo37m/ansible_azure.azcollection.azure_rm_virtualnetwork_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_virtualnetwork.py\", line 394, in <module>\n  File \"/var/folders/dh/qjj1yjqd2yz20zjh6n8r3w380000gn/T/ansible_azure.azcollection.azure_rm_virtualnetwork_payload_p2ddo37m/ansible_azure.azcollection.azure_rm_virtualnetwork_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_virtualnetwork.py\", line 390, in main\n  File \"/var/folders/dh/qjj1yjqd2yz20zjh6n8r3w380000gn/T/ansible_azure.azcollection.azure_rm_virtualnetwork_payload_p2ddo37m/ansible_azure.azcollection.azure_rm_virtualnetwork_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_virtualnetwork.py\", line 245, in __init__\n  File \"/var/folders/dh/qjj1yjqd2yz20zjh6n8r3w380000gn/T/ansible_azure.azcollection.azure_rm_virtualnetwork_payload_p2ddo37m/ansible_azure.azcollection.azure_rm_virtualnetwork_payload.zip/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py\", line 418, in __init__\n  File \"/var/folders/dh/qjj1yjqd2yz20zjh6n8r3w380000gn/T/ansible_azure.azcollection.azure_rm_virtualnetwork_payload_p2ddo37m/ansible_azure.azcollection.azure_rm_virtualnetwork_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_virtualnetwork.py\", line 272, in exec_module\n  File \"/usr/local/lib/python3.7/site-packages/azure/mgmt/network/network_management_client.py\", line 2449, in virtual_networks\n    raise NotImplementedError(\"APIVersion {} is not available\".format(api_version))\nNotImplementedError: APIVersion 2019-06-01 is not available\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}

[Fred-sun]

@mark100net I'm glad you are interested in ansilbe-collections. I tested your playbook and it was successful. The test results are as follows:

TASK [create resource group] ****************************************************************************************************************************************************ok: [localhost]

TASK [Create virtual network] ***************************************************************************************************************************************************changed: [localhost]

TASK [debug] ********************************************************************************************************************************************************************ok: [localhost] => {
    "output": {
        "changed": true,
        "check_mode": false,
        "failed": false,
        "state": {
            "address_prefixes": [
                "10.0.0.0/16"
            ],
            "etag": "W/\"24d39e33-4824-43b4-b7a0-d71c22fe7714\"",
            "id": "/subscriptions/xxxxxxxxxxxx/resourceGroups/v-xisuRG/providers/Microsoft.Network/virtualNetworks/fredvnet003",
            "location": "eastus",
            "name": "fredvnet003",
            "provisioning_state": "Succeeded",
            "tags": null,
            "type": "Microsoft.Network/virtualNetworks"
        }
    }
}

PLAY RECAP **********************************************************************************************************************************************************************localhost                  : ok=4    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

[Fred-sun]

@mark100net I suspect that your ansible and ansible-collection refer to azure-mgmt-network differently. Can you help confirm this? Thank you very much! ansible-collection: /usr/local/lib/python3.7/site-packages/azure/mgmt/network/network_management_client.py ansible: ?

[mark100net]

@Fred-sun Thanks for trying to reproduce my issue.

I am not completely sure what you are asking. I do have network_management_client.py in the location you specified.

My azcollection is in ~/.ansible/collections/ansible_collections/azure/azcollection

My ansible --version output is (sorry, I should have put that in the original issue):

 config file = None
 configured module search path = ['/Users/mark/.ansible/plugins/modules',        '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.7/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.7.7 (default, Apr 25 2020, 11:17:28) [Clang 11.0.3 (clang-1103.0.32.29)]

If that does not answer your question please let me know what I need to do to answer your question.

Thanks again.

[Fred-sun]

@mark100net Can you do such a verification, edit "/usr/local/lib/python3.7/site-packages/ansible/module_utils/azure_rm_common.py". Then test with default tasks and share the fail log to me! Thank you very much!

    @property
    def network_client(self):
        self.log('Getting network client')
        if not self._network_client:
            self._network_client = self.get_mgmt_svc_client(NetworkManagementClient,
                                                            base_url=self._cloud_environment.endpoints.resource_manager,
                                                            api_version='2019-07-01')
        return self._network_client

    @property
    def network_models(self):
        self.log("Getting network models...")
        return NetworkManagementClient.models("2018-09-01")

[mark100net]

Hello @Fred-sun , thanks for getting back to me.

I was not sure what you meant by "default tasks". But I made the edits you suggested and ran the playbook with the tasks above and got this fail log:

ansible-azure.txt

[Fred-sun]

@mark100net Sorry for misunderstand to you. default tasks from the cloud/ansible module. Thank you very much!

[mark100net]

@Fred-sun I did (also) run the task as the default (i.e. as azure_rm_virtualnetwork without the collection qualifier). I do not get a failure that way (with or without the edits you gave me). In case it helps, here is the output when I run it with the default with the edits:

azure.txt

[Fred-sun]

@mark100net Thank you very much for your feedback. First of all, you should definitely report an error after changing the API version, because the network API does not have these two versions. So your two task executions use different APIs. Second , that I have encountered such a similar problem, ansible and ansible-collection use azure-mgmt-network in different locations. Could you help remove ansible and ansible-collection and reinstall to retry? Thank you very much!

[mark100net]

@Fred-sun I'm sorry, I realized the file you told me to modify was not properly saved and I was running against the original. I made the edits again and actually am getting a failure log (using the default tasks), attached below, so I hope this helps:

azure.txt

Thanks again.

[Fred-sun]

@mark100net Thanks for your feedback! I see!

[mark100net]

@Fred-sun Do you have any other suggestions then?

[Fred-sun]

@Fred-sun Do you have any other suggestions then?

As you can see from your log, ansible and ansible-collections use the same network_management_client.py, the two test results should be the same, and you should not go wrong here. I am still thinking about how to clarify, thank you!

[Fred-sun]

@mark100net Please delete “/usr/local/lib/python3.7/site-packages/azure/mgmt/network/”, reinstall azure-mgmt-network and try again, thank you!

sudo pip3 install azure-mgmt-network -I

[mark100net]

@Fred-sun Thanks for the suggestion. I was able to get past the original error after doing your suggestion, but I got another error later on in my playbook when creating a virtual network. I will get you the details on that at a later date.

In the meantime, did you want me to do that with or without the edits to the azure_rm_common.py file you gave me before? I get errors either way but the error is different depending on which one.

Thanks again.

[Fred-sun]

without the edits to the azure_rm_common.py

@mark100net Without the edits to the azure_rm_common.py, Thank you very much!

[mark100net]

@Fred-sun After deleting the network directory and reinstalling azure-mgmt-network and running with the original azure_rm_common.py as you suggested I am still getting the same error as originally reported with the azure.azcollection.azure_rm_vurtualnetwork task. Output below:

azure.txt

Thanks again.

[Fred-sun]

@mark100net Please help provide "pip3 list |grep azure-mgmt-network" information and send me these two files. Thank you!

/Users/mark/Library/Python/3.7/lib/python/site-packages/azure/mgmt/network/network_management_client.py
/Users/mark/Library/Python/3.7/lib/python/site-packages/azure/mgmt/network/_network_management_client.py

[mark100net]

@Fred-sun here you go:

> pip3 list | grep azure-mgmt-network
azure-mgmt-network             2.3.0

I do not have a _network_management_client.py in that directory. I have network_management_client.pay included below (with .txt extension added to I could upload it here):

network_management_client.py.txt

Thanks

[Fred-sun]

@mark100net You are using an old version of azure-mgmt-network, please delete ”/Users/mark/Library/Python/3.7/lib/python/site-packages/azure/mgmt/network“ folder and reinstall azure-mgmt-network (sudo pip3 install azure-mgmt-network==10.1.0 -I). Thank you!

[mark100net]

@Fred-sun I am going to close this issue because clearly the problem relates to my setup. But if you have any further suggestions please let me know.

When I deleted the 'network' folder as you instructed and then ran the azure-mgmt-network install it did not install it to the location under my home directory. It installed it to /usr/local/lib/python3.7, and the playbook failed because it was still looking for the azure-mgmt-network under my home directory.

Then I removed ~/Library/Python altogether and the problem went away.

Clearly this applies to me:

https://xkcd.com/1987/

:(

Thanks again for all your time and help with this.