gluster / gluster-block

A framework for gluster block storage
GNU General Public License v2.0
74 stars 32 forks source link

create: fails directly when the StorageObject is already exist with the same NAME #243

Closed lxbsz closed 5 years ago

lxbsz commented 5 years ago

What does this PR achieve? Why do we need it?

create: split the creation into 2 phases

This will fix the following issues:

1, When the StorageObject creation failed the iqn will still be created and won't be deleted in case of:

When creating BV with the same NAME, the second time it will fail due to the targetcli cache db only allows one [user/$NAME] exist, but will leave the iqn not deleted correctly.

That means there will be two different Targets will both mapped to the same StorageObject.

2, For the case above we will also find that after the second creation failing the /etc/target/saveconfig.json will be one StorageObject with the second Target in pairs and exist.

In theory, there should be one StorageObject with 2 Targets.

This patch will split the creation into 2 phases: CREATE_SO_SRV and CREATE_TG_SRV, since in the targetcli it will build the cache by using the key = [usr/$NAME], so only the StorageObject is successfully created the second phase will make sense.

Does this PR fix issues?

Fixes: BZ#1725009

lxbsz commented 5 years ago

The 2 issues will be:

Create 2 blocks with the same name "block0" and in different gluster volumes, "dht" and "dht1", there will be 2 Targets with only one StorageObject:

]# gluster-block create dht1/block0 ha 1 192.168.195.162 21M; gluster-block create dht/block0 ha 1 192.168.195.162 21M; targetcli ls
IQN: iqn.2016-12.org.gluster-block:12d49c06-78d0-447b-974e-6bd37ee15507
PORTAL(S):  192.168.195.162:3260
RESULT: SUCCESS
IQN: -
PORTAL(S): -
ROLLBACK FAILED ON: 192.168.195.162 
RESULT: FAIL
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 0]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  | o- user:glfs .............................................................................................. [Storage Objects: 1]
  |   o- block0 ........................ [dht1@192.168.195.162/block-store/12d49c06-78d0-447b-974e-6bd37ee15507 (21.0MiB) activated]
  |     o- alua ................................................................................................... [ALUA Groups: 3]
  |       o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  |       o- glfs_tg_pt_gp_ano .................................................................. [ALUA state: Active/non-optimized]
  |       o- glfs_tg_pt_gp_ao ....................................................................... [ALUA state: Active/optimized]
  o- iscsi ............................................................................................................ [Targets: 2]
  | o- iqn.2016-12.org.gluster-block:12d49c06-78d0-447b-974e-6bd37ee15507 ................................................ [TPGs: 1]
  | | o- tpg1 .................................................................................................. [gen-acls, no-auth]
  | |   o- acls .......................................................................................................... [ACLs: 0]
  | |   o- luns .......................................................................................................... [LUNs: 1]
  | |   | o- lun0 ................................................................................. [user/block0 (glfs_tg_pt_gp_ao)]
  | |   o- portals .................................................................................................... [Portals: 1]
  | |     o- 192.168.195.162:3260 ............................................................................................. [OK]
  | o- iqn.2016-12.org.gluster-block:42b295bc-cdf0-457c-adda-fd1b3839575a ................................................ [TPGs: 1]
  |   o- tpg1 .................................................................................................. [gen-acls, no-auth]
  |     o- acls .......................................................................................................... [ACLs: 0]
  |     o- luns .......................................................................................................... [LUNs: 1]
  |     | o- lun0 ................................................................................. [user/block0 (glfs_tg_pt_gp_ao)]
  |     o- portals .................................................................................................... [Portals: 1]
  |       o- 192.168.195.162:3260 ............................................................................................. [OK]
  o- loopback ......................................................................................................... [Targets: 0]
  o- vhost ............................................................................................................ [Targets: 0]
  o- xen-pvscsi ....................................................................................................... [Targets: 0]
]# 

In the /etc/target/saveconfig.json we can see the Target's gbid is not the same with StorageObject's:

]# cat /etc/target/saveconfig.json 
{
  "storage_objects": [
    {
      "alua_tpgs": [
        {
          "alua_access_state": 1, 
          "alua_access_status": 2, 
          "alua_access_type": 1, 
          "alua_support_active_nonoptimized": 1, 
          "alua_support_active_optimized": 1, 
          "alua_support_offline": 1, 
          "alua_support_standby": 1, 
          "alua_support_transitioning": 1, 
          "alua_support_unavailable": 1, 
          "alua_write_metadata": 0, 
          "implicit_trans_secs": 0, 
          "name": "glfs_tg_pt_gp_ano", 
          "nonop_delay_msecs": 100, 
          "preferred": 0, 
          "tg_pt_gp_id": 2, 
          "trans_delay_msecs": 0
        }, 
        {
          "alua_access_state": 0, 
          "alua_access_status": 0, 
          "alua_access_type": 1, 
          "alua_support_active_nonoptimized": 1, 
          "alua_support_active_optimized": 1, 
          "alua_support_offline": 1, 
          "alua_support_standby": 1, 
          "alua_support_transitioning": 1, 
          "alua_support_unavailable": 1, 
          "alua_write_metadata": 0, 
          "implicit_trans_secs": 0, 
          "name": "glfs_tg_pt_gp_ao", 
          "nonop_delay_msecs": 100, 
          "preferred": 0, 
          "tg_pt_gp_id": 1, 
          "trans_delay_msecs": 0
        }, 
        {
          "alua_access_state": 0, 
          "alua_access_status": 0, 
          "alua_access_type": 3, 
          "alua_support_active_nonoptimized": 1, 
          "alua_support_active_optimized": 1, 
          "alua_support_offline": 1, 
          "alua_support_standby": 1, 
          "alua_support_transitioning": 1, 
          "alua_support_unavailable": 1, 
          "alua_write_metadata": 0, 
          "implicit_trans_secs": 0, 
          "name": "default_tg_pt_gp", 
          "nonop_delay_msecs": 100, 
          "preferred": 0, 
          "tg_pt_gp_id": 0, 
          "trans_delay_msecs": 0
        }
      ], 
      "attributes": {
        "cmd_time_out": 130, 
        "dev_size": 22020096, 
        "qfull_time_out": -1
      }, 
      "config": "glfs/dht1@192.168.195.162/block-store/12d49c06-78d0-447b-974e-6bd37ee15507", 
      "control": "max_data_area_mb=8,hw_block_size=512", 
      "hw_max_sectors": 128, 
      "name": "block0", 
      "plugin": "user", 
      "size": 22020096, 
      "wwn": "12d49c06-78d0-447b-974e-6bd37ee15507"
    }
  ], 
  "targets": [
    {
      "fabric": "iscsi", 
      "tpgs": [
        {
          "attributes": {
            "authentication": 0, 
            "cache_dynamic_acls": 1, 
            "default_cmdsn_depth": 64, 
            "default_erl": 0, 
            "demo_mode_discovery": 1, 
            "demo_mode_write_protect": 0, 
            "fabric_prot_type": 0, 
            "generate_node_acls": 1, 
            "login_timeout": 15, 
            "netif_timeout": 2, 
            "prod_mode_write_protect": 0, 
            "t10_pi": 0, 
            "tpg_enabled_sendtargets": 1
          }, 
          "enable": true, 
          "luns": [
            {
              "alias": "81f9fa86eb", 
              "alua_tg_pt_gp_name": "glfs_tg_pt_gp_ao", 
              "index": 0, 
              "storage_object": "/backstores/user/block0"
            }
          ], 
          "node_acls": [], 
          "parameters": {
            "AuthMethod": "CHAP,None", 
            "DataDigest": "CRC32C,None", 
            "DataPDUInOrder": "Yes", 
            "DataSequenceInOrder": "Yes", 
            "DefaultTime2Retain": "20", 
            "DefaultTime2Wait": "2", 
            "ErrorRecoveryLevel": "0", 
            "FirstBurstLength": "65536", 
            "HeaderDigest": "CRC32C,None", 
            "IFMarkInt": "Reject", 
            "IFMarker": "No", 
            "ImmediateData": "Yes", 
            "InitialR2T": "Yes", 
            "MaxBurstLength": "262144", 
            "MaxConnections": "1", 
            "MaxOutstandingR2T": "1", 
            "MaxRecvDataSegmentLength": "8192", 
            "MaxXmitDataSegmentLength": "262144", 
            "OFMarkInt": "Reject", 
            "OFMarker": "No", 
            "TargetAlias": "LIO Target"
          }, 
          "portals": [
            {
              "ip_address": "192.168.195.162", 
              "iser": false, 
              "offload": false, 
              "port": 3260
            }
          ], 
          "tag": 1
        }
      ], 
      "wwn": "iqn.2016-12.org.gluster-block:42b295bc-cdf0-457c-adda-fd1b3839575a"
    }
  ]
}
]# 
pkalever commented 5 years ago

@lxbsz splitting create into two phases will give some flexibility! I agree!

But this effects the overall time taken to create the block volume, especially at scale.

Hence I'm bit reluctant with this approach.

Why cannot we solve it like create target and SO in one phase. Delete target and SO if reply->exit !=0 ?

Thanks!

lxbsz commented 5 years ago

@lxbsz splitting create into two phases will give some flexibility! I agree!

But this effects the overall time taken to create the block volume, especially at scale.

Do you mean the cache loading in the targetcli/rtslib_fb when running the create operations ?

If so, I think this will be almost the same, because only when creating/deleting the StorageObject it will do the bs_cache loading. And for the others will won't use this bs_cache.

]# vim rtslib-fb/rtslib/tcm.py
[...]
 955 bs_cache = {}
 956   
 957 class _Backstore(CFSNode):
 958     """
 959     Backstore is needed as a level in the configfs hierarchy, but otherwise useless.
 960     1:1 so:backstore.
 961     Created by storageobject ctor before SO configfs entry.
 962     """
 963   
 964     def __init__(self, name, storage_object_cls, mode, index=None):
 965         super(_Backstore, self).__init__()  
 966         self._so_cls = storage_object_cls   
 967         self._plugin = bs_params[self._so_cls]['name']
 968   
 969         dirp = bs_params[self._so_cls].get("alt_dirprefix", self._plugin)
 970   
 971         # if the caller knows the index then skip the cache
 972         global bs_cache       
 973         if index is None and not bs_cache:  
 974             for dir in glob.iglob("%s/core/*_*/*/" % self.configfs_dir):
 975                 parts = dir.split("/")              
 976                 bs_name = parts[-2]
 977                 bs_dirp, bs_index = parts[-3].rsplit("_", 1) 
 978                 current_key = "%s/%s" % (bs_dirp, bs_name)
 979                 bs_cache[current_key] = int(bs_index)
 980   
[...]
1012         except:
1013             if self._lookup_key in bs_cache:
1014                 del bs_cache[self._lookup_key]
1015             raise
1016 
1017     def delete(self):
1018         super(_Backstore, self).delete()
1019         if self._lookup_key in bs_cache:
1020             del bs_cache[self._lookup_key]
[...]

Hence I'm bit reluctant with this approach.

Why cannot we solve it like create target and SO in one phase. Delete target and SO if reply->exit !=0 ?

This is mainly to resolve the issue 2 above, or the code will be very complicated.

Thanks. BRs

lxbsz commented 5 years ago

@pkalever Updated done. Thanks.

lxbsz commented 5 years ago

@pkalever Updated it. Thanks.

pkalever commented 5 years ago

@lxbsz tested this and works as expected.

[root@server-1 gluster-block]# gluster-block create hosting-volume1/block-volume1 ha 3 prealloc full block-size 1024 192.168.124.59,192.168.124.78,192.168.124.44 1GiB --json-pretty
{                                                                                                                                                                                                                                             
  "IQN":"iqn.2016-12.org.gluster-block:84cb6694-add9-420f-9092-37b03bfe9086",                                                                                                                                                                 
  "PORTAL(S)":[                                                                                                                                                                                                                               
    "192.168.124.59:3260",                                                                                                                                                                                                                    
    "192.168.124.78:3260",                                                                                                                                                                                                                    
    "192.168.124.44:3260"                                                                                                                                                                                                                     
  ],                                                                                                                                                                                                                                          
  "RESULT":"SUCCESS"
}
[root@server-1 gluster-block]# gluster-block create hosting-volume2/block-volume1 ha 3 prealloc full block-size 1024 192.168.124.59,192.168.124.78,192.168.124.44 1GiB --json-pretty
{
  "RESULT":"FAIL",
  "errCode":255,
  "errMsg":"failed to configure on 192.168.124.59 block 'block-volume1' is already exist\nfailed to configure on 192.168.124.78 block 'block-volume1' is already exist\nfailed to configure on 192.168.124.44 block 'block-volume1' is already
 exist"
}

Two observations:

  1. Can we tweak the user msg a bit, like:

block 'block-volume1' is already exist

Hint: block with name 'block-volume1' already exist (may be hosted on a different block-hosting volume)

  1. I see rollback is called when we fail in the create check, IMO this is not needed, can we fix it ?
2019-07-15 09:48:03.721765] INFO: create request, volume=hosting-volume2 volserver=localhost blockname=block-volume1 blockhosts=192.168.124.59,192.168.124.78,192.168.124.44 filename=f16f72e3-72a1-45cc-9d4a-e27a9859157b authmode=0 passwd= size=1073741824 [at block_svc_routines.c+4368 :<block_create_common>]
[2019-07-15 09:48:04.432346] ERROR: failed in remote create for block block-volume1 on host 192.168.124.78 volume hosting-volume2 [at block_svc_routines.c+971 :<glusterBlockCreateRemote>]
[2019-07-15 09:48:04.547346] ERROR: failed in remote create for block block-volume1 on host 192.168.124.44 volume hosting-volume2 [at block_svc_routines.c+971 :<glusterBlockCreateRemote>]
[2019-07-15 09:48:04.603144] ERROR: failed in remote create for block block-volume1 on host 192.168.124.59 volume hosting-volume2 [at block_svc_routines.c+971 :<glusterBlockCreateRemote>]
[2019-07-15 09:48:04.604859] WARNING: glusterBlockCreateRemoteAsync: return -1 failed in remote async create for block block-volume1 on volume hosting-volume2 with hosts 192.168.124.59,192.168.124.78,192.168.124.44 [at block_svc_routines.c+4072 :<block_create_cli_1_svc_st>]
[2019-07-15 09:48:04.638107] WARNING: No Spare nodes to create (block-volume1): rollingback creation of target on volume hosting-volume2 with given hosts 192.168.124.59,192.168.124.78,192.168.124.44 [at block_svc_routines.c+3103 :<glusterBlockAuditRequest>]
[2019-07-15 09:48:04.786051] INFO: delete request, blockname=block-volume1 filename=f16f72e3-72a1-45cc-9d4a-e27a9859157b [at block_svc_routines.c+4868 :<block_delete_1_svc_st>]
[2019-07-15 09:48:05.840416] ERROR: failed in remote delete for block block-volume1 on host 192.168.124.44 volume hosting-volume2 [at block_svc_routines.c+1094 :<glusterBlockDeleteRemote>]
[2019-07-15 09:48:05.897243] ERROR: Block 'block-volume1' may be not loaded. [at block_svc_routines.c+289 :<blockCheckBlockLoadedStatus>]
[2019-07-15 09:48:05.915958] ERROR: Block 'block-volume1' already deleted. [at block_svc_routines.c+314 :<blockCheckBlockLoadedStatus>]
[2019-07-15 09:48:05.973519] ERROR: failed in remote delete for block block-volume1 on host 192.168.124.78 volume hosting-volume2 [at block_svc_routines.c+1094 :<glusterBlockDeleteRemote>]
[2019-07-15 09:48:06.039381] ERROR: failed in remote delete for block block-volume1 on host 192.168.124.59 volume hosting-volume2 [at block_svc_routines.c+1094 :<glusterBlockDeleteRemote>]
[2019-07-15 09:48:06.406260] ERROR: glusterBlockAuditRequest: return -1volume: hosting-volume2 hosts: 192.168.124.59,192.168.124.78,192.168.124.44 blockname block-volume1 [at block_svc_routines.c+4081 :<block_create_cli_1_svc_st>]
[2019-07-15 09:48:06.406339] INFO: create cli returns success, block volume: hosting-volume2/block-volume1 [at block_svc_routines.c+4088 :<block_create_cli_1_svc_st>]

Thanks!

lxbsz commented 5 years ago

@lxbsz tested this and works as expected.

[root@server-1 gluster-block]# gluster-block create hosting-volume1/block-volume1 ha 3 prealloc full block-size 1024 192.168.124.59,192.168.124.78,192.168.124.44 1GiB --json-pretty
{                                                                                                                                                                                                                                             
  "IQN":"iqn.2016-12.org.gluster-block:84cb6694-add9-420f-9092-37b03bfe9086",                                                                                                                                                                 
  "PORTAL(S)":[                                                                                                                                                                                                                               
    "192.168.124.59:3260",                                                                                                                                                                                                                    
    "192.168.124.78:3260",                                                                                                                                                                                                                    
    "192.168.124.44:3260"                                                                                                                                                                                                                     
  ],                                                                                                                                                                                                                                          
  "RESULT":"SUCCESS"
}
[root@server-1 gluster-block]# gluster-block create hosting-volume2/block-volume1 ha 3 prealloc full block-size 1024 192.168.124.59,192.168.124.78,192.168.124.44 1GiB --json-pretty
{
  "RESULT":"FAIL",
  "errCode":255,
  "errMsg":"failed to configure on 192.168.124.59 block 'block-volume1' is already exist\nfailed to configure on 192.168.124.78 block 'block-volume1' is already exist\nfailed to configure on 192.168.124.44 block 'block-volume1' is already
 exist"
}

Two observations:

  1. Can we tweak the user msg a bit, like:

block 'block-volume1' is already exist

Hint: block with name 'block-volume1' already exist (may be hosted on a different block-hosting volume)

Not sure whether is the case that only 2/3 nodes have the SO for some reason, like user have 4 nodes, users are trying to create hosting-volume1/block1 on node-1, node-2 and node-3, then create hosting-volume2/block1 on node-2, node-3 and node-4 ?

So the current error should be more in detail. Will it make sense ?

  1. I see rollback is called when we fail in the create check, IMO this is not needed, can we fix it ?
2019-07-15 09:48:03.721765] INFO: create request, volume=hosting-volume2 volserver=localhost blockname=block-volume1 blockhosts=192.168.124.59,192.168.124.78,192.168.124.44 filename=f16f72e3-72a1-45cc-9d4a-e27a9859157b authmode=0 passwd= size=1073741824 [at block_svc_routines.c+4368 :<block_create_common>]
[2019-07-15 09:48:04.432346] ERROR: failed in remote create for block block-volume1 on host 192.168.124.78 volume hosting-volume2 [at block_svc_routines.c+971 :<glusterBlockCreateRemote>]
[2019-07-15 09:48:04.547346] ERROR: failed in remote create for block block-volume1 on host 192.168.124.44 volume hosting-volume2 [at block_svc_routines.c+971 :<glusterBlockCreateRemote>]
[2019-07-15 09:48:04.603144] ERROR: failed in remote create for block block-volume1 on host 192.168.124.59 volume hosting-volume2 [at block_svc_routines.c+971 :<glusterBlockCreateRemote>]
[2019-07-15 09:48:04.604859] WARNING: glusterBlockCreateRemoteAsync: return -1 failed in remote async create for block block-volume1 on volume hosting-volume2 with hosts 192.168.124.59,192.168.124.78,192.168.124.44 [at block_svc_routines.c+4072 :<block_create_cli_1_svc_st>]
[2019-07-15 09:48:04.638107] WARNING: No Spare nodes to create (block-volume1): rollingback creation of target on volume hosting-volume2 with given hosts 192.168.124.59,192.168.124.78,192.168.124.44 [at block_svc_routines.c+3103 :<glusterBlockAuditRequest>]
[2019-07-15 09:48:04.786051] INFO: delete request, blockname=block-volume1 filename=f16f72e3-72a1-45cc-9d4a-e27a9859157b [at block_svc_routines.c+4868 :<block_delete_1_svc_st>]
[2019-07-15 09:48:05.840416] ERROR: failed in remote delete for block block-volume1 on host 192.168.124.44 volume hosting-volume2 [at block_svc_routines.c+1094 :<glusterBlockDeleteRemote>]
[2019-07-15 09:48:05.897243] ERROR: Block 'block-volume1' may be not loaded. [at block_svc_routines.c+289 :<blockCheckBlockLoadedStatus>]
[2019-07-15 09:48:05.915958] ERROR: Block 'block-volume1' already deleted. [at block_svc_routines.c+314 :<blockCheckBlockLoadedStatus>]
[2019-07-15 09:48:05.973519] ERROR: failed in remote delete for block block-volume1 on host 192.168.124.78 volume hosting-volume2 [at block_svc_routines.c+1094 :<glusterBlockDeleteRemote>]
[2019-07-15 09:48:06.039381] ERROR: failed in remote delete for block block-volume1 on host 192.168.124.59 volume hosting-volume2 [at block_svc_routines.c+1094 :<glusterBlockDeleteRemote>]
[2019-07-15 09:48:06.406260] ERROR: glusterBlockAuditRequest: return -1volume: hosting-volume2 hosts: 192.168.124.59,192.168.124.78,192.168.124.44 blockname block-volume1 [at block_svc_routines.c+4081 :<block_create_cli_1_svc_st>]
[2019-07-15 09:48:06.406339] INFO: create cli returns success, block volume: hosting-volume2/block-volume1 [at block_svc_routines.c+4088 :<block_create_cli_1_svc_st>]

Yeah, this make sense. Will fix it. Thanks.

pkalever commented 5 years ago

Not sure whether is the case that only 2/3 nodes have the SO for some reason, like user have 4 nodes, users are trying to create hosting-volume1/block1 on node-1, node-2 and node-3, then create hosting-volume2/block1 on node-2, node-3 and node-4 ?

So the current error should be more in detail. Will it make sense ?

@lxbsz I'm okay to print the same msg from HA number of gluster-blockd's. I want you to fix the rpc/block_svc_routines.c line 4389 as:

- snprintf(reply->out, 8192, "block '%s' is already exist", blk->block_name);
+ snprintf(reply->out, 8192, "block with name '%s' already exist (Hint: may be hosted on a different block-hosting volume)", blk->block_name);

Thanks!

lxbsz commented 5 years ago

Not sure whether is the case that only 2/3 nodes have the SO for some reason, like user have 4 nodes, users are trying to create hosting-volume1/block1 on node-1, node-2 and node-3, then create hosting-volume2/block1 on node-2, node-3 and node-4 ? So the current error should be more in detail. Will it make sense ?

@lxbsz I'm okay to print the same msg from HA number of gluster-blockd's. I want you to fix the rpc/block_svc_routines.c line 4389 as:

- snprintf(reply->out, 8192, "block '%s' is already exist", blk->block_name);
+ snprintf(reply->out, 8192, "block with name '%s' already exist (Hint: may be hosted on a different block-hosting volume)", blk->block_name);

Thanks!

Sure. Thanks

lxbsz commented 5 years ago

@pkalever

Updated it and fallback to the old version.

Jut one NOTE:

Currently, I can still hit the old issue: https://github.com/gluster/gluster-block/issues/204, which due to the meta data couldn't be flushed to the disk in time, especially in muti nodes case, like:

rhel3 --> 192.168.195.164 rhel1 --> 192.168.195.162 Two block-hosting volumes: rep and rep1.

I will create the block14 BV in the rep volume with node rhel3 only, then create the block14 BV in the rep1 volume with nodes rhel3 and rhel1

The following are the normal output as expected, when create the block14 BV more than once, it will fail with exist error on node rhel3, and on node rhel1 it will create/delete success:

[root@rhel3 ~]# gluster-block create rep/block14 ha 1 192.168.195.164 20M
IQN: iqn.2016-12.org.gluster-block:f4a59a9a-6a31-46d0-b6bb-ba831ac95731
PORTAL(S):  192.168.195.164:3260
RESULT: SUCCESS
[root@rhel3 ~]# gluster-block create rep1/block14 ha 2 192.168.195.164,192.168.195.162 20M
failed to configure on 192.168.195.164 block with name 'block14' already exist (Hint: may be hosted on a different block-hosting volume)
RESULT:FAIL
[root@rhel3 ~]# cat /tmp/rep1/block-meta/
meta.lock  prio.info  
[root@rhel3 ~]# cat /tmp/rep1/block-meta/block14
cat: /tmp/dht1/block-meta/block14: No such file or directory
[root@rhel3 ~]# gluster-block create rep1/block14 ha 2 192.168.195.164,192.168.195.162 20M
failed to configure on 192.168.195.164 block with name 'block14' already exist (Hint: may be hosted on a different block-hosting volume)
RESULT:FAIL
[root@rhel3 ~]# gluster-block create rep1/block14 ha 2 192.168.195.164,192.168.195.162 20M
failed to configure on 192.168.195.164 block with name 'block14' already exist (Hint: may be hosted on a different block-hosting volume)
RESULT:FAIL
[root@rhel3 ~]# gluster-block create rep1/block14 ha 2 192.168.195.164,192.168.195.162 20M
failed to configure on 192.168.195.164 block with name 'block14' already exist (Hint: may be hosted on a different block-hosting volume)
RESULT:FAIL
[root@rhel3 ~]# gluster-block create rep1/block14 ha 2 192.168.195.164,192.168.195.162 20M
failed to configure on 192.168.195.164 block with name 'block14' already exist (Hint: may be hosted on a different block-hosting volume)
RESULT:FAIL
[root@rhel3 ~]# gluster-block create rep1/block14 ha 2 192.168.195.164,192.168.195.162 20M
failed to configure on 192.168.195.164 block with name 'block14' already exist (Hint: may be hosted on a different block-hosting volume)
RESULT:FAIL

But sometimes I can hit the following output in my setup:

[root@rhel3 ~]# gluster-block create rep1/block14 ha 2 192.168.195.164,192.168.195.162 20M
IQN: iqn.2016-12.org.gluster-block:3a6550f5-5dc1-482b-84eb-2ca54a5bda93
PORTAL(S):  192.168.195.164:3260 192.168.195.162:3260
ROLLBACK SUCCESS ON: 192.168.195.164 192.168.195.162 
RESULT: FAIL

This is because in block_create_cli_1_svc_st()--> glusterBlockAuditRequest() --> glusterBlockCleanUp() --> glusterBlockDeleteRemoteAsync() , in Line#1287, it will read the meta info when the Node 192.168.195.162's newest status hasn't be flushed to the disk, so cleanupsuccess != info->nhosts, then glusterBlockDeleteRemoteAsync() will return -1.

If I just wait for 2 seconds before Line#1287, this issue is disappeared.

Then the rep1/block-meta/block14 won't be deleted, the code:

1287   ret = blockGetMetaInfo(glfs, blockname, info_new, NULL);
1288   if (ret) {
1289     goto out;
1290   }
1291   ret = -1;
1292 
1293   for (i = 0; i < info_new->nhosts; i++) {
1294     switch (blockMetaStatusEnumParse(info_new->list[i]->status)) {
1295       case GB_CONFIG_INPROGRESS:  /* un touched */
1296       case GB_CLEANUP_SUCCESS:
1297         cleanupsuccess++;
1298         break;
1299     }
1300   }
1301 
1302   if (cleanupsuccess == info->nhosts) {
1303     ret = 0;
1304   }
1305   *savereply = local;

Just after this, I can see that the rep1/block-meta/block14:

[root@rhel3 ~]# cat /tmp/rep1/block-meta/block14
VOLUME: rep1
GBID: 3a6550f5-5dc1-482b-84eb-2ca54a5bda93
HA: 2
ENTRYCREATE: INPROGRESS
PRIOPATH: 192.168.195.164
SIZE: 20971520
RINGBUFFER: 0
BLKSIZE: 0
ENTRYCREATE: SUCCESS
192.168.195.164: CONFIGINPROGRESS
192.168.195.162: CONFIGINPROGRESS
192.168.195.164: CONFIGFAIL
192.168.195.162: CONFIGSUCCESS
192.168.195.164: CLEANUPINPROGRESS
192.168.195.162: CLEANUPINPROGRESS
192.168.195.164: CLEANUPFAIL
192.168.195.162: CLEANUPSUCCESS

The Node 192.168.195.162's status is already in GB_CLEANUP_SUCCESS = "CLEANUPSUCCESS".

Maybe this can be reproduced only in my local VM setups which is lacking of enough disk space.

Thanks.

pkalever commented 5 years ago

Merged now. Thanks @lxbsz .