na_ontap_volume - FlexGroup expand not working with space_guarantee: none

[AwesomeNameMan]

Summary

When I create a FlexGroup with space_guarantee: none and want to expand the FlexGroup with sizing_method: add_new_resources (even using wait_for_completion: yes), I run in the following error:

"msg": "Error resizing volume testvolume: job reported error: Timeout error: Process still running, received {'job': {'uuid': '84322a53-ba66-11ec-98ac-d039ea1c5e05', '_links': {'self': {'href': '/api/cluster/jobs/84322a53-ba66-11ec-98ac-d039ea1c5e05'}}}}."

Component Name

na_ontap_volume

Ansible Version

$ ansible --version

ansible [core 2.11.10]
  config file = None
  configured module search path = ['/home/awesome/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/awesome/.local/lib/python3.6/site-packages/ansible
  ansible collection location = /home/awesome/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/awesome/.local/bin/ansible
  python version = 3.6.8 (default, Nov  9 2021, 14:44:26) [GCC 8.5.0 20210514 (Red Hat 8.5.0-3)]
  jinja version = 3.0.3
  libyaml = True

ONTAP Collection Version

$ ansible-galaxy collection list

# /home/awesome/.local/lib/python3.6/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    1.5.1
ansible.netcommon             2.5.0
ansible.posix                 1.3.0
ansible.utils                 2.4.3
ansible.windows               1.8.0
arista.eos                    2.2.0
awx.awx                       19.4.0
azure.azcollection            1.10.0
check_point.mgmt              2.2.0
chocolatey.chocolatey         1.1.0
cisco.aci                     2.1.0
cisco.asa                     2.1.0
cisco.intersight              1.0.18
cisco.ios                     2.6.0
cisco.iosxr                   2.6.0
cisco.meraki                  2.5.0
cisco.mso                     1.2.0
cisco.nso                     1.0.3
cisco.nxos                    2.8.2
cisco.ucs                     1.6.0
cloudscale_ch.cloud           2.2.0
community.aws                 1.5.0
community.azure               1.1.0
community.crypto              1.9.8
community.digitalocean        1.13.0
community.docker              1.10.2
community.fortios             1.0.0
community.general             3.8.3
community.google              1.0.0
community.grafana             1.3.0
community.hashi_vault         1.5.0
community.hrobot              1.2.1
community.kubernetes          1.2.1
community.kubevirt            1.0.0
community.libvirt             1.0.2
community.mongodb             1.3.2
community.mysql               2.3.2
community.network             3.0.0
community.okd                 1.1.2
community.postgresql          1.6.0
community.proxysql            1.3.0
community.rabbitmq            1.1.0
community.routeros            1.2.0
community.skydive             1.0.0
community.sops                1.2.0
community.vmware              1.17.0
community.windows             1.8.0
community.zabbix              1.5.1
containers.podman             1.9.0
cyberark.conjur               1.1.0
cyberark.pas                  1.0.13
dellemc.enterprise_sonic      1.1.0
dellemc.openmanage            3.6.0
dellemc.os10                  1.1.1
dellemc.os6                   1.0.7
dellemc.os9                   1.0.4
f5networks.f5_modules         1.13.0
fortinet.fortimanager         2.1.4
fortinet.fortios              2.1.3
frr.frr                       1.0.3
gluster.gluster               1.0.2
google.cloud                  1.0.2
hetzner.hcloud                1.6.0
hpe.nimble                    1.1.4
ibm.qradar                    1.0.3
infinidat.infinibox           1.3.0
inspur.sm                     1.3.0
junipernetworks.junos         2.8.0
kubernetes.core               1.2.1
mellanox.onyx                 1.0.0
netapp.aws                    21.7.0
netapp.azure                  21.10.0
netapp.cloudmanager           21.12.1
netapp.elementsw              21.7.0
netapp.ontap                  21.14.1
netapp.um_info                21.8.0
netapp_eseries.santricity     1.2.13
netbox.netbox                 3.4.0
ngine_io.cloudstack           2.2.2
ngine_io.exoscale             1.0.0
ngine_io.vultr                1.1.0
openstack.cloud               1.5.3
openvswitch.openvswitch       2.1.0
ovirt.ovirt                   1.6.6
purestorage.flasharray        1.11.0
purestorage.flashblade        1.8.1
sensu.sensu_go                1.12.0
servicenow.servicenow         1.0.6
splunk.es                     1.0.2
t_systems_mms.icinga_director 1.26.0
theforeman.foreman            2.2.0
vyos.vyos                     2.6.0
wti.remote                    1.0.3

# /home/awesome/.ansible/collections/ansible_collections
Collection   Version
------------ -------
netapp.ontap 21.17.3

ONTAP Version

awesome-cluster::> version
NetApp Release 9.10.1: Sat Jan 15 15:04:44 UTC 2022

Playbook

- name: Expand FlexGroup
  na_ontap_volume:
    <<: *login
    state: present
    wait_for_completion: yes
    name: "{{ item.name }}"
    size: "{{ item.size }}"
    size_unit: "{{ item.size_unit  }}"
    sizing_method: add_new_resources
    vserver: "{{ item.svm }}"
    aggr_list: "{{ aggregates | random }}"
    aggr_list_multiplier: 1
    use_rest: always
    space_guarantee: none
  loop: "{{ volume }}"

Steps to Reproduce

        - name: Create initial FlexGroup
          na_ontap_volume:
            <<: *login
            state: present
            wait_for_completion: yes
            percent_snapshot_space: 0
            space_guarantee: none
            language: C.UTF-8
            volume_security_style: unix
            name: "{{ item.name }}"
            junction_path: "/{{ item.name }}"
            export_policy: "{{ item.export_policy }}"
            snapshot_policy: "{{ item.snapshot_policy }}"
            aggr_list : "{{ aggregates | random }}"
            aggr_list_multiplier: 1
            size: "{{ item.constituent_size }}"
            size_unit: "{{ item.size_unit }}"
            vserver: "{{ item.svm }}"
          loop: "{{ volume }}"

        - name: Expand FlexGroup
          na_ontap_volume:
            <<: *login
            state: present
            wait_for_completion: yes
            name: "{{ item.name }}"
            size: "{{ item.size }}"
            size_unit: "{{ item.size_unit  }}"
            sizing_method: add_new_resources
            vserver: "{{ item.svm }}"
            aggr_list: "{{ aggregates | random }}"
            aggr_list_multiplier: 1
            use_rest: always
            space_guarantee: none
          loop: "{{ volume }}"

############
#    Volume    #
############

volume:
  - name: testvolume
    svm: testing
    size: 600
    constituent_size: 200
    size_unit: gb
    export_policy: default
    snapshot_policy: none

Expected Results

After creating the initial FlexGroup with 1 constituent and constituent_size: 200GB the next task should expand the FlexGroup with 2 new constituents á 200GB to a FlexGroup size of 600GB.

This works well, when space_guarantee: volume, but not with none.

Actual Results

<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: awesome
<127.0.0.1> EXEC /bin/sh -c 'echo ~awesome && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/awesome/.ansible/tmp `"&& mkdir "` echo /home/awesome/.ansible/tmp/ansible-tmp-1649770995.9501147-80544-160394095426396 `" && echo ansible-tmp-1649770995.9501147-80544-160394095426396="` echo /home/awesome/.ansible/tmp/ansible-tmp-1649770995.9501147-80544-160394095426396 `" ) && sleep 0'
Including module_utils file ansible/__init__.py
Including module_utils file ansible/module_utils/__init__.py
Including module_utils file ansible/module_utils/_text.py
Including module_utils file ansible/module_utils/basic.py
Including module_utils file ansible/module_utils/common/_collections_compat.py
Including module_utils file ansible/module_utils/common/__init__.py
Including module_utils file ansible/module_utils/common/_json_compat.py
Including module_utils file ansible/module_utils/common/_utils.py
Including module_utils file ansible/module_utils/common/arg_spec.py
Including module_utils file ansible/module_utils/common/file.py
Including module_utils file ansible/module_utils/common/parameters.py
Including module_utils file ansible/module_utils/common/collections.py
Including module_utils file ansible/module_utils/common/process.py
Including module_utils file ansible/module_utils/common/sys_info.py
Including module_utils file ansible/module_utils/common/text/converters.py
Including module_utils file ansible/module_utils/common/text/__init__.py
Including module_utils file ansible/module_utils/common/text/formatters.py
Including module_utils file ansible/module_utils/common/validation.py
Including module_utils file ansible/module_utils/common/warnings.py
Including module_utils file ansible/module_utils/compat/selectors.py
Including module_utils file ansible/module_utils/compat/__init__.py
Including module_utils file ansible/module_utils/compat/_selectors2.py
Including module_utils file ansible/module_utils/compat/selinux.py
Including module_utils file ansible/module_utils/distro/__init__.py
Including module_utils file ansible/module_utils/distro/_distro.py
Including module_utils file ansible/module_utils/errors.py
Including module_utils file ansible/module_utils/parsing/convert_bool.py
Including module_utils file ansible/module_utils/parsing/__init__.py
Including module_utils file ansible/module_utils/pycompat24.py
Including module_utils file ansible/module_utils/six/__init__.py
Including module_utils file ansible_collections/netapp/ontap/plugins/module_utils/netapp.py
Including module_utils file ansible/module_utils/ansible_release.py
Including module_utils file ansible_collections/__init__.py
Including module_utils file ansible_collections/netapp/__init__.py
Including module_utils file ansible_collections/netapp/ontap/__init__.py
Including module_utils file ansible_collections/netapp/ontap/plugins/__init__.py
Including module_utils file ansible_collections/netapp/ontap/plugins/module_utils/__init__.py
Including module_utils file ansible_collections/netapp/ontap/plugins/module_utils/netapp_module.py
Including module_utils file ansible_collections/netapp/ontap/plugins/module_utils/rest_application.py
Including module_utils file ansible_collections/netapp/ontap/plugins/module_utils/rest_generic.py
Including module_utils file ansible_collections/netapp/ontap/plugins/module_utils/rest_response_helpers.py
Including module_utils file ansible_collections/netapp/ontap/plugins/module_utils/rest_volume.py
Using module file /home/awesome/.ansible/collections/ansible_collections/netapp/ontap/plugins/modules/na_ontap_volume.py
<127.0.0.1> PUT /home/awesome/.ansible/tmp/ansible-local-80493t512zpgy/tmpmetbjz7h TO /home/awesome/.ansible/tmp/ansible-tmp-1649770995.9501147-80544-160394095426396/AnsiballZ_na_ontap_volume.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/awesome/.ansible/tmp/ansible-tmp-1649770995.9501147-80544-160394095426396/ /home/awesome/.ansible/tmp/ansible-tmp-1649770995.9501147-80544-160394095426396/AnsiballZ_na_ontap_volume.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python3.6 /home/awesome/.ansible/tmp/ansible-tmp-1649770995.9501147-80544-160394095426396/AnsiballZ_na_ontap_volume.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/awesome/.ansible/tmp/ansible-tmp-1649770995.9501147-80544-160394095426396/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
NoneType: None
failed: [localhost] (item={'name': 'testvolume', 'svm': 'testing', 'size': 600, 'constituent_size': 200, 'size_unit': 'gb', 'export_policy': 'default', 'snapshot_policy': 'none'}) => {
    "ansible_loop_var": "item",
    "changed": false,
    "invocation": {
        "module_args": {
            "aggr_list": [
                "awesome_cluster_01_aggr1"
            ],
            "aggr_list_multiplier": 1,
            "aggregate_name": null,
            "atime_update": null,
            "auto_provision_as": null,
            "auto_remap_luns": null,
            "cert_filepath": null,
            "check_interval": 30,
            "comment": null,
            "compression": null,
            "cutover_action": null,
            "efficiency_policy": null,
            "encrypt": null,
            "export_policy": null,
            "feature_flags": {},
            "force_restore": null,
            "force_unmap_luns": null,
            "from_name": null,
            "from_vserver": null,
            "group_id": null,
            "hostname": "**********",
            "http_port": null,
            "https": true,
            "inline_compression": null,
            "is_infinite": false,
            "is_online": true,
            "junction_path": null,
            "key_filepath": null,
            "language": null,
            "logical_space_enforcement": null,
            "logical_space_reporting": null,
            "name": "testvolume",
            "nas_application_template": null,
            "nvfail_enabled": null,
            "ontapi": null,
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "percent_snapshot_space": null,
            "preserve_lun_ids": null,
            "qos_adaptive_policy_group": null,
            "qos_policy_group": null,
            "size": 600,
            "size_change_threshold": 10,
            "size_unit": "gb",
            "sizing_method": "add_new_resources",
            "snapdir_access": null,
            "snapshot_auto_delete": null,
            "snapshot_policy": null,
            "snapshot_restore": null,
            "space_guarantee": "none",
            "space_slo": null,
            "state": "present",
            "tiering_minimum_cooling_days": null,
            "tiering_policy": null,
            "time_out": 180,
            "type": null,
            "unix_permissions": null,
            "use_rest": "always",
            "user_id": null,
            "username": "vsadmin",
            "validate_certs": false,
            "volume_security_style": null,
            "vserver": "testing",
            "vserver_dr_protection": null,
            "wait_for_completion": true
        }
    },
    "item": {
        "constituent_size": 200,
        "export_policy": "default",
        "name": "testvolume",
        "size": 600,
        "size_unit": "gb",
        "snapshot_policy": "none",
        "svm": "testing"
    },
    "msg": "Error resizing volume testvolume: job reported error: Timeout error: Process still running, received {'job': {'uuid': '84322a53-ba66-11ec-98ac-d039ea1c5e05', '_links': {'self': {'href': '/api/cluster/jobs/84322a53-ba66-11ec-98ac-d039ea1c5e05'}}}}."
}

[lonico]

We are looping to get the job results until it completes or a failure is reported. But the loop is also limited by: job_timeout=120

This is a bug as the value is hardcoded.

The solution will be to use the time_out value so that this is configurable

  time_out:
    description:
    - time to wait for Flexgroup creation, modification, or deletion in seconds.
    - Error out if task is not completed in defined time.
    - if 0, the request is asynchronous.
    - default is set to 3 minutes.
    default: 180
    type: int
    version_added: 2.8.0

We will address this issue in the May release.

[AwesomeNameMan]

Thanks for your quick reply. Will look forward to the May release.

[AwesomeNameMan]

Hi Team,

could you address the issue in the May release? Couldn't find something in the release notes. Hope you had the time, the fix would be crucial for us.

Best

[lonico]

Sorry this was fixed in 21.19.0 released early May. We forgot to update this issue.

na_ontap_volume - use time_out value when creating/modifying/deleting volumes with REST rathar than hardcoded value.