remove online node - ERROR: 'PyrakoonClient' object has no attribute 'identifier'

[yongshengma]

I got an error when I removed an online node. It didn't abort but continued until successful message returned. Everything also looks OK on GUI and that node is gone except another node has warning for a few minutes.

+++ Demoting node +++

Leaving Arakoon ovsdb cluster
Leaving Arakoon config cluster
ERROR: 
Failed to leave configuration cluster
ERROR: 'PyrakoonClient' object has no attribute 'identifier'
Update configurations
Restarting master node services
Removing/unconfiguring RabbitMQ

2018-09-25 10:30:08 78300 +0800 - Node-22 - 29579/140535695824640 - lib/node-type - 265 - INFO - Demoting node
2018-09-25 10:30:08 81000 +0800 - Node-22 - 29579/140535695824640 - lib/node-type - 266 - INFO - Leaving Arakoon ovsdb cluster
2018-09-25 10:30:19 60500 +0800 - Node-22 - 29579/140535695824640 - lib/node-type - 300 - INFO - Leaving Arakoon config cluster
2018-09-25 10:30:41 25200 +0800 - Node-22 - 29579/140535695824640 - lib/node-type - 318 - ERROR -
Failed to leave configuration cluster
Traceback (most recent call last):
  File "ovs/lib/nodetype.py", line 394, in demote_node
    ip=master_node_ips[0])
  File "ovs/extensions/db/arakoon/arakooninstaller.py", line 924, in restart_cluster_after_shrinking
    ArakoonInstaller._wait_for_cluster(config=config)
  File "ovs/extensions/db/arakoon/arakooninstaller.py", line 1002, in _wait_for_cluster
    arakoon_client.nop()
  File "ovs/extensions/db/arakoon/pyrakoon/client.py", line 131, in nop
    return self._try(self._client.nop)
  File "ovs/extensions/db/arakoon/pyrakoon/client.py", line 197, in _try
    method.__name__, os.getpid(), current_thread().ident, self.identifier
AttributeError: 'PyrakoonClient' object has no attribute 'identifier'
2018-09-25 10:30:41 25300 +0800 - Node-22 - 29579/140535695824640 - lib/node-type - 319 - ERROR - 'PyrakoonClient' object has no attribute 'identifier'
Traceback (most recent call last):
  File "ovs/lib/nodetype.py", line 394, in demote_node
    ip=master_node_ips[0])
  File "ovs/extensions/db/arakoon/arakooninstaller.py", line 924, in restart_cluster_after_shrinking
    ArakoonInstaller._wait_for_cluster(config=config)
  File "ovs/extensions/db/arakoon/arakooninstaller.py", line 1002, in _wait_for_cluster
    arakoon_client.nop()
  File "ovs/extensions/db/arakoon/pyrakoon/client.py", line 131, in nop
    return self._try(self._client.nop)
  File "ovs/extensions/db/arakoon/pyrakoon/client.py", line 197, in _try
    method.__name__, os.getpid(), current_thread().ident, self.identifier
AttributeError: 'PyrakoonClient' object has no attribute 'identifier'
2018-09-25 10:30:41 25300 +0800 - Node-22 - 29579/140535695824640 - lib/node-type - 320 - INFO - Update configurations
2018-09-25 10:30:41 26300 +0800 - Node-22 - 29579/140535695824640 - lib/node-type - 326 - INFO - Restarting master node services
2018-09-25 10:30:41 28100 +0800 - Node-22 - 29579/140535695824640 - lib/node-type - 327 - INFO - Removing/unconfiguring RabbitMQ

[yongshengma]

BTW, i conducted a test on removing an offline node by using remove node <ip> command. It works fine!

However, I found DTL states of some vdisks' are in red. I also found the preset didn't switch to an usable policy for two nodes (only one alba-asd per node). After i manually switched the policy order by moving 2-dup ahead, the DTL states became green. The policies I defined : (1, 1, 1, 2), (1, 2, 1, 3). Not sure if this operation is related to DTL state.

[JeffreyDevloo]

Hi @yongshengma Which Framework version are you using? I can't seem to find any code pointing to self.identifier within the try of the pyrakoon You can patch this using self._identifer instead of self.identifer in that part of the code as identifier is a 'private' property Best regards

[yongshengma]

Hi @JeffreyDevloo

This is Fargo version.

ii  blktap-openvstorage-utils            2.0.90-2ubuntu5                       amd64        utilities to work with VHD disk images files
ii  libblktapctl0-openvstorage           2.0.90-2ubuntu5                       amd64        Xen API blktapctl shared library (shared library)
ii  libvhd0-openvstorage                 2.0.90-2ubuntu5                       amd64        VHD file format access library
ii  libvhdio-2.0.90-openvstorage         2.0.90-2ubuntu5                       amd64        Xen API blktap shared library (shared library)
ii  openvstorage                         2.9.5-1                               amd64        openvStorage
ii  openvstorage-backend                 1.9.1-1                               amd64        openvStorage Backend plugin
ii  openvstorage-backend-core            1.9.1-1                               amd64        openvStorage Backend plugin core
ii  openvstorage-backend-webapps         1.9.1-1                               amd64        openvStorage Backend plugin Web Applications
ii  openvstorage-core                    2.9.5-1                               amd64        openvStorage core
ii  openvstorage-hc                      1.9.1-1                               amd64        openvStorage Backend plugin HyperConverged
ii  openvstorage-sdm                     1.9.0-1                               amd64        Open vStorage Backend ASD Manager
ii  openvstorage-webapps                 2.9.5-1                               amd64        openvStorage Web Applications
ii  tgt-openvstorage                     99:1.0.63-0ovs1.4-1ubuntu1.1          amd64        Linux SCSI target user-space daemon and too

I will take a try on your suggestion later.

[yongshengma]

Yes, it's obviously a bug in code. _identifer is defined instead of identifer.

    def __init__(self, cluster, nodes):
        """
        Initializes the client
        """
        cleaned_nodes = {}
        for node, info in nodes.iteritems():
            cleaned_nodes[str(node)] = ([str(entry) for entry in info[0]], int(info[1]))
        self._config = ArakoonClientConfig(str(cluster), cleaned_nodes)
        self._client = ArakoonClient(self._config)
        self._identifier = int(round(random.random() * 10000000))
        self._lock = Lock()
        self._batch_size = 500
        self._sequences = {}