Module na_ontap_export_policy fails in case of applying to IP Metrocluster via REST API

sto228 commented 3 years ago

Summary

Symptom:

Creation of the export-policy on the IP Metrocluster running ONTAP 9.8P6 on platform AFF-A300 fails with message: Error on creating export policy: {'message': 'This operation is not permitted on a SVM that is configured as the destination of a MetroCluster SVM relationship.', 'code': '2621574'} although the mentioned vserver is not destination SVM(replica with suffix "-mc") of the metrocluster configuration.

Workaround:

Falling back to the ZAPI by using use_rest: never resolves that issue

Description:

Module na_ontap_export_policy works fine on the single-site(HA) configuration with the same ONTAP version 9.8P2. When we are applying the same playbook including the na_ontap_export_policy module to the IP Metrocluster it fails with the message mentioned above although we are applying it to the right one(primary) vserver SVM_NAME, not to the inactive replica with SVM_NAME-mc suffix on the opposite cluster of the metrocluster deployment.

Used versions:

netapp.ontap collection 21.7.0. ansible 2.9.17 python 3.6.8

Component Name

na_ontap_export_policy

Ansible Version

ansible 2.9.17

ONTAP Collection Version

netapp.ontap collection 21.7.0.

ONTAP Version

NetApp Release 9.8P6: Tue Aug 03 16:21:11 UTC 2021

Playbook

- name: Create export-policy for respective volume
  hosts: localhost
  gather_facts: no
  collections:
    - netapp.ontap

  vars:
    ontap_admin_username: admin
    ontap_admin_password: strong_ontap_password

  tasks:

    - name: Create EP
      na_ontap_export_policy:
        state: present
        name: "{{ ['ep_', volume_name ]|join('')}}"
        vserver: "{{ svm }}"
        https: yes
        validate_certs: no
        hostname: "{{ cluster }}"
        username: "{{ ontap_admin_username }}"
        password: "{{ ontap_admin_password }}"
      vars:
        volume_name: "{{ [[svm, volume.value.name]|join('_')]|join('') }}"
      loop: "{{ volumes | dict2items }}"
      loop_control:
         loop_var: volume
         index_var: loop_index
         extended: yes

Steps to Reproduce

Run the playbook below against to IP Metrocluster(currently running ONTAP 9.8P6 on AFF-A300)

- name: Create export-policy for respective volume
  hosts: localhost
  gather_facts: no
  collections:
    - netapp.ontap

  vars:
    ontap_admin_username: admin
    ontap_admin_password: strong_ontap_password

  tasks:

    - name: Create EP
      na_ontap_export_policy:
        state: present
        name: "{{ ['ep_', volume_name ]|join('')}}"
        vserver: "{{ svm }}"
        https: yes
        validate_certs: no
        hostname: "{{ cluster }}"
        username: "{{ ontap_admin_username }}"
        password: "{{ ontap_admin_password }}"
      vars:
        volume_name: "{{ [[svm, volume.value.name]|join('_')]|join('') }}"
      loop: "{{ volumes | dict2items }}"
      loop_control:
         loop_var: volume
         index_var: loop_index
         extended: yes

Expected Results

{
  "changed": true,
  "invocation": {
    "module_args": {
      "state": "present",
      "name": "ep_svm_name_dummy",
      "vserver": "svm_name",
      "https": true,
      "validate_certs": false,
      "use_rest": "never",
      "hostname": "cluster1",
      "username": "admin",
      "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
      "feature_flags": {},
      "http_port": null,
      "ontapi": null,
      "cert_filepath": null,
      "key_filepath": null,
      "from_name": null
    }
  },
  "_ansible_no_log": false,
  "volume": {
    "key": "volume_1",
    "value": {
      "backup": true,
      "exports": "0.0.0.0",
      "name": "dummy",
      "performance": "high",
      "protocol": "NFS",
      "size": 10,
      "worm": false
    }
  },
  "ansible_loop_var": "volume",
  "loop_index": 0,
  "ansible_index_var": "loop_index",
  "ansible_loop": {
    "allitems": [
      {
        "key": "volume_1",
        "value": {
          "backup": true,
          "exports": "0.0.0.0",
          "name": "dummy",
          "performance": "high",
          "protocol": "NFS",
          "size": 10,
          "worm": false
        }
      }
    ],
    "index": 1,
    "index0": 0,
    "first": true,
    "last": true,
    "length": 1,
    "revindex": 1,
    "revindex0": 0
  },
  "_ansible_item_label": {
    "key": "volume_1",
    "value": {
      "backup": true,
      "exports": "0.0.0.0",
      "name": "dummy",
      "performance": "high",
      "protocol": "NFS",
      "size": 10,
      "worm": false
    }
  }
}

Actual Results

{
  "msg": "Error on creating export policy: {'message': 'This operation is not permitted on a SVM that is configured as the destination of a MetroCluster SVM relationship.', 'code': '2621574'}",
  "invocation": {
    "module_args": {
      "state": "present",
      "name": "ep_svm_name_dummy",
      "vserver": "svm_name",
      "https": true,
      "validate_certs": false,
      "hostname": "cluster1",
      "username": "admin",
      "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
      "use_rest": "auto",
      "feature_flags": {},
      "http_port": null,
      "ontapi": null,
      "cert_filepath": null,
      "key_filepath": null,
      "from_name": null
    }
  },
  "_ansible_no_log": false,
  "changed": false,
  "volume": {
    "key": "volume_1",
    "value": {
      "backup": true,
      "exports": "0.0.0.0",
      "name": "dummy",
      "performance": "high",
      "protocol": "NFS",
      "size": 10,
      "worm": false
    }
  },
  "ansible_loop_var": "volume",
  "loop_index": 0,
  "ansible_index_var": "loop_index",
  "ansible_loop": {
    "allitems": [
      {
        "key": "volume_1",
        "value": {
          "backup": true,
          "exports": "0.0.0.0",
          "name": "dummy",
          "performance": "high",
          "protocol": "NFS",
          "size": 10,
          "worm": false
        }
      }
    ],
    "index": 1,
    "index0": 0,
    "first": true,
    "last": true,
    "length": 1,
    "revindex": 1,
    "revindex0": 0
  },
  "_ansible_item_label": {
    "key": "volume_1",
    "value": {
      "backup": true,
      "exports": "0.0.0.0",
      "name": "dummy",
      "performance": "high",
      "protocol": "NFS",
      "size": 10,
      "worm": false
    }
  }
}

lonico commented 3 years ago

we found an issue where we could pick up the wrong SVM. Both both SVMs would need to be in the same cluster. Is this the case here?

sto228 commented 3 years ago

Hi, yes most likely that's the issue because in the case of metrocluster are SVMs sitting on the different clusters, basically primary SVM named svm1 is sitting on the cluster1 and the DR replica named svm1-mc is sitting on the cluster2 and between clusters is metrocluster relationship(synchronous mirror of the data and SVM configurations). So, if mentioned module expects that both SVMs are located on the same cluster, then it's wrong.

lonico commented 2 years ago

Do you still see the issue with 21.14.0 or 21.14.1?

We fixed an issue in a similar area, though I cannot say for sure it is the same issue.

sto228 commented 2 years ago

Hi, I've tested with netapp.ontap collection version 21.14.1 and it seems to be working properly also in the case of Metrocluster setup. I used the use_rest: auto option in the playbook and there was no message in the output about falling back to ZAPI, so I guess that issue has been fixed in mentioned collection version.

lonico commented 2 years ago

Thank you for confirming.

sto228 commented 2 years ago

Hi, we are facing this issue again - the same error code in the case of Metrocluser, only a bit expanded message error. I have tested it a couple of times before confirming this issue seemed to be fixed but now it has occurred again and there is no evident difference compared to my test tasks. I'll do more tests and get back to you with the details.

{ "msg": "Error on creating export policy: calling: protocols/nfs/export-policies: got {'message': 'This operation is not permitted on a SVM that is configured as the destination of a MetroCluster SVM relationship.', 'code': '2621574'}.",

iStep2Step commented 2 years ago

We are affected by this Bug too on our Metrocluster. Same error Message as sto228. Is there another workaround other than use ZAPI?

I could be wrong, but could this be related to?

https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP_OS/REST_API%3A_unable_to_manage_export_policy_on_sync-source_SVM

lonico commented 2 years ago

Thank you for the pointer.

It's an ONTAP bug, so the solution is to:

Upgrade to a version of ONTAP with a fix for Bug ID 1351013.
9.10.1, 9.10.1P1, 9.10.1RC1, 9.10.1RC1P1, 9.10.1RC2, 9.8P10, 9.8P8, 9.8P9, 9.9.1P4, 9.9.1P5, 9.9.1P6, 9.9.1P7

(the link requires to be authenticated before clicking on it).

One work-around is to

Recreate the vserver and verify the first digit of the UUID is a letter. The UUID is in hex so I understand you'd want to keep recreating until you find something that start with A to F.

On our side, we can look at the second work-around

Use REST API to pass the CLI command for modifying the export policy.

ansible-collections / netapp.ontap