gluster / glusterfs

Gluster Filesystem : Build your distributed storage in minutes
https://www.gluster.org
GNU General Public License v2.0
4.65k stars 1.08k forks source link

Host is not in 'Peer in Cluster' state #4189

Open redhyper9 opened 1 year ago

redhyper9 commented 1 year ago

Description of problem: I'm getting Host <host> is not in 'Peer in Cluster' state when attempting to create a gluster volume.

The exact command to reproduce the issue: gluster volume create myvolume de-fra-01:/mnt/storage de-fra-02:/mnt/storage

The full output of the command that failed: volume create: myvolume: failed: Host de-fra-02 is not in 'Peer in Cluster' state

Expected results: I expected the volume to successfully create.

Mandatory info: - The output of the gluster volume info command: No volumes present

- The output of the gluster volume status command: No volumes present

- The output of the gluster volume heal command: Usage: volume heal <VOLNAME> [enable | disable | full |statistics [heal-count [replica <HOSTNAME:BRICKNAME>]] |info [summary | split-brain] |split-brain {bigger-file <FILE> | latest-mtime <FILE> |source-brick <HOSTNAME:BRICKNAME> [<FILE>]} |granular-entry-heal {enable | disable}]

**- Provide logs present on following locations of client and server nodes -

Server 1:

/var/log/glusterfs/cli.log:

[2023-07-03 20:04:36.946549 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:04:37.008399 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:04:37.008638 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:04:37.026166 +0000] I [cli-rpc-ops.c:162:gf_cli_probe_cbk] 0-cli: Received resp to probe
[2023-07-03 20:04:37.026416 +0000] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2023-07-03 20:04:50.907525 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:04:50.966762 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:04:50.966985 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:04:50.968003 +0000] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2023-07-03 20:05:19.999696 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:05:20.061981 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:05:20.062049 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:05:20.062558 +0000] E [cli-cmd-parser.c:880:cli_cmd_volume_create_parse] 0-cli: Unable to parse create volume CLI
[2023-07-03 20:05:20.062583 +0000] I [input.c:31:cli_batch] 0-: Exiting with: -1
[2023-07-03 20:05:26.210219 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:05:26.269568 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:05:26.269607 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:05:26.272337 +0000] I [cli-rpc-ops.c:1015:gf_cli_create_volume_cbk] 0-cli: Received resp to create volume
[2023-07-03 20:05:26.272421 +0000] I [input.c:31:cli_batch] 0-: Exiting with: -1
[2023-07-03 20:06:07.451295 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:06:07.510142 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:06:07.510373 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:06:07.510744 +0000] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2023-07-03 20:06:16.286512 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:06:16.345938 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:06:16.346169 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:06:16.346543 +0000] I [cli-rpc-ops.c:162:gf_cli_probe_cbk] 0-cli: Received resp to probe
[2023-07-03 20:06:16.346756 +0000] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2023-07-03 20:06:19.774906 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:06:19.831691 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:06:19.831908 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:06:19.836772 +0000] I [cli-rpc-ops.c:162:gf_cli_probe_cbk] 0-cli: Received resp to probe
[2023-07-03 20:06:19.836792 +0000] E [cli-rpc-ops.c:167:gf_cli_probe_cbk] 0-cli: de-fra-01 is either already part of another cluster or having volumes configured
[2023-07-03 20:06:19.836857 +0000] I [input.c:31:cli_batch] 0-: Exiting with: -1
[2023-07-03 20:06:30.518396 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:06:30.577311 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:06:30.577523 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:06:30.577890 +0000] I [cli-rpc-ops.c:767:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2023-07-03 20:06:30.577944 +0000] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2023-07-03 20:07:38.750424 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:07:38.807493 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:07:38.807709 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:07:38.813631 +0000] I [cli-rpc-ops.c:162:gf_cli_probe_cbk] 0-cli: Received resp to probe
[2023-07-03 20:07:38.813651 +0000] E [cli-rpc-ops.c:167:gf_cli_probe_cbk] 0-cli: de-fra-01 is either already part of another cluster or having volumes configured
[2023-07-03 20:07:38.813703 +0000] I [input.c:31:cli_batch] 0-: Exiting with: -1
[2023-07-03 20:07:44.261561 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:07:44.321664 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:07:44.321888 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:07:44.322164 +0000] I [cli-rpc-ops.c:767:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2023-07-03 20:07:44.322227 +0000] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2023-07-03 20:07:46.480383 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:07:46.539208 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:07:46.539433 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:07:46.540201 +0000] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2023-07-03 20:11:30.446361 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:11:30.504710 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:11:30.504754 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:11:30.505275 +0000] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2023-07-03 20:12:45.646342 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:12:45.704337 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:12:45.704579 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:12:45.709569 +0000] I [cli-rpc-ops.c:1015:gf_cli_create_volume_cbk] 0-cli: Received resp to create volume
[2023-07-03 20:12:45.709627 +0000] I [input.c:31:cli_batch] 0-: Exiting with: -1
[2023-07-03 20:12:53.645911 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:12:53.703346 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:12:53.703581 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:12:53.708165 +0000] I [cli-rpc-ops.c:1015:gf_cli_create_volume_cbk] 0-cli: Received resp to create volume
[2023-07-03 20:12:53.708248 +0000] I [input.c:31:cli_batch] 0-: Exiting with: -1
[2023-07-03 20:13:07.404687 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:13:07.464907 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:13:07.465146 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:13:09.003468 +0000] I [cli-rpc-ops.c:1015:gf_cli_create_volume_cbk] 0-cli: Received resp to create volume
[2023-07-03 20:13:09.003579 +0000] I [input.c:31:cli_batch] 0-: Exiting with: -1
[2023-07-03 20:15:12.080312 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:15:12.140181 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:15:12.140229 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:15:12.140672 +0000] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2023-07-03 20:15:39.826359 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:15:39.885890 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:15:39.885936 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:15:39.888338 +0000] I [cli-rpc-ops.c:1015:gf_cli_create_volume_cbk] 0-cli: Received resp to create volume
[2023-07-03 20:15:39.888432 +0000] I [input.c:31:cli_batch] 0-: Exiting with: -1
[2023-07-03 20:18:27.519024 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:18:27.578008 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:18:27.578051 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:18:27.580184 +0000] I [cli-rpc-ops.c:1015:gf_cli_create_volume_cbk] 0-cli: Received resp to create volume
[2023-07-03 20:18:27.580302 +0000] I [input.c:31:cli_batch] 0-: Exiting with: -1

/var/log/glusterfs/cmd_history.log

[2023-07-03 20:04:37.025802 +0000]  : peer probe de-fra-02 : SUCCESS    
[2023-07-03 20:05:26.272249 +0000]  : volume create vl de-fra-01:/mnt/storage de-fra-02:/mnt/storage : FAILED : Host de-fra-01 is not in 'Peer in Cluster' state
[2023-07-03 20:06:16.346470 +0000]  : peer probe de-fra-02 : SUCCESS : Host de-fra-02 port 24007 already in peer list
[2023-07-03 20:06:19.836701 +0000]  : peer probe de-fra-01 : FAILED : de-fra-01 is either already part of another cluster or having volumes configured
[2023-07-03 20:07:38.813562 +0000]  : peer probe de-fra-01 : FAILED : de-fra-01 is either already part of another cluster or having volumes configured
[2023-07-03 20:13:09.003337 +0000]  : volume create gv0 replica 2 de-fra-02:/data/brick1/gv0 de-fra-01:/data/brick1/gv0 : FAILED : Host de-fra-01 is not in 'Peer in Cluster' state
[2023-07-03 20:15:39.888242 +0000]  : volume create myvolume de-fra-01:/mnt/storage de-fra-02:/mnt/storage : FAILED : Host de-fra-01 is not in 'Peer in Cluster' state
[2023-07-03 20:18:27.580063 +0000]  : volume create myvolume de-fra-01:/mnt/storage de-fra-02:/mnt/storage : FAILED : Host de-fra-01 is not in 'Peer in Cluster' state

/var/log/glusterfs/glusterd.log:

[2023-07-03 20:02:23.314309 +0000] I [MSGID: 100030] [glusterfsd.c:2767:main] 0-/usr/sbin/glusterd: Started running version [{arg=/usr/sbin/glusterd}, {version=10.3}, {cmdlinestr=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO}] 
[2023-07-03 20:02:23.314843 +0000] I [glusterfsd.c:2447:daemonize] 0-glusterfs: Pid of current running process is 4236
[2023-07-03 20:02:23.362527 +0000] I [MSGID: 106478] [glusterd.c:1470:init] 0-management: Maximum allowed open file descriptors set to 65536 
[2023-07-03 20:02:23.362781 +0000] I [MSGID: 106479] [glusterd.c:1560:init] 0-management: Using /var/lib/glusterd as working directory 
[2023-07-03 20:02:23.362787 +0000] I [MSGID: 106479] [glusterd.c:1564:init] 0-management: Using /var/run/gluster as pid file working directory 
[2023-07-03 20:02:23.367685 +0000] I [socket.c:971:__socket_server_bind] 0-socket.management: process started listening on port (24007)
[2023-07-03 20:02:23.368762 +0000] I [socket.c:914:__socket_server_bind] 0-socket.management: closing (AF_UNIX) reuse check socket 12
[2023-07-03 20:02:23.369494 +0000] I [MSGID: 106059] [glusterd.c:1952:init] 0-management: max-port override: 60999 
[2023-07-03 20:02:23.446330 +0000] E [MSGID: 106061] [glusterd.c:590:glusterd_crt_georep_folders] 0-glusterd: Dict get failed [{Key=log-group}, {errno=2}, {error=No such file or directory}] 
[2023-07-03 20:02:24.818045 +0000] E [MSGID: 101032] [store.c:461:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2023-07-03 20:02:24.818504 +0000] E [MSGID: 101032] [store.c:461:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2023-07-03 20:02:24.818506 +0000] I [MSGID: 106514] [glusterd-store.c:2153:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 100000 
[2023-07-03 20:02:24.818524 +0000] E [MSGID: 101032] [store.c:461:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/options. [No such file or directory]
[2023-07-03 20:02:24.823310 +0000] I [MSGID: 106194] [glusterd-store.c:3980:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list. 
[2023-07-03 20:02:24.823334 +0000] E [MSGID: 101032] [store.c:461:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.upgrade. [No such file or directory]
[2023-07-03 20:02:24.823343 +0000] I [glusterd.c:2090:init] 0-management: Regenerating volfiles due to a max op-version mismatch or glusterd.upgrade file not being present, op_version retrieved:0, max op_version: 100000
Final graph:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option rpc-auth.auth-glusterfs on
  4:     option rpc-auth.auth-unix on
  5:     option rpc-auth.auth-null on
  6:     option rpc-auth-allow-insecure on
  7:     option transport.listen-backlog 1024
  8:     option max-port 60999
  9:     option event-threads 1
 10:     option ping-timeout 0
 11:     option transport.socket.listen-port 24007
 12:     option transport.socket.read-fail-log off
 13:     option transport.socket.keepalive-interval 2
 14:     option transport.socket.keepalive-time 10
 15:     option transport-type socket
 16:     option working-directory /var/lib/glusterd
 17: end-volume
 18:  
+------------------------------------------------------------------------------+
[2023-07-03 20:02:24.827339 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:02:26.206630 +0000] I [MSGID: 106163] [glusterd-handshake.c:1501:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 100000 
[2023-07-03 20:02:26.206676 +0000] E [MSGID: 101032] [store.c:461:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2023-07-03 20:02:26.206911 +0000] I [MSGID: 106477] [glusterd.c:180:glusterd_uuid_generate_save] 0-management: generated UUID: 08eae420-3aef-4604-9f45-4c518739706a 
[2023-07-03 20:02:26.212530 +0000] I [MSGID: 106490] [glusterd-handler.c:2546:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 5a34f977-9373-46d9-8bab-2801f00c86af 
[2023-07-03 20:02:26.251718 +0000] I [MSGID: 106493] [glusterd-handler.c:3819:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 130.162.232.185 (24007), ret: 0, op_ret: -1 
[2023-07-03 20:04:37.009453 +0000] I [MSGID: 106487] [glusterd-handler.c:1133:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req de-fra-02 24007 
[2023-07-03 20:04:37.009700 +0000] I [MSGID: 106128] [glusterd-handler.c:3651:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: de-fra-02 (24007) 
[2023-07-03 20:04:37.012644 +0000] W [MSGID: 106061] [glusterd-handler.c:3424:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout 
[2023-07-03 20:04:37.012677 +0000] I [rpc-clnt.c:1015:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2023-07-03 20:04:37.013190 +0000] I [MSGID: 106498] [glusterd-handler.c:3583:glusterd_friend_add] 0-management: connect returned 0 
[2023-07-03 20:04:37.022249 +0000] I [MSGID: 106511] [glusterd-rpc-ops.c:248:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: 0a43f531-dff3-42a8-a7b0-aba584cb1245, host: de-fra-02 
[2023-07-03 20:04:37.022274 +0000] I [MSGID: 106511] [glusterd-rpc-ops.c:404:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req 
[2023-07-03 20:04:37.025778 +0000] I [MSGID: 106493] [glusterd-rpc-ops.c:467:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 0a43f531-dff3-42a8-a7b0-aba584cb1245, host: de-fra-02, port: 0 
[2023-07-03 20:04:37.031409 +0000] I [MSGID: 106163] [glusterd-handshake.c:1501:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 100000 
[2023-07-03 20:04:37.035881 +0000] I [MSGID: 106490] [glusterd-handler.c:2902:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: 0a43f531-dff3-42a8-a7b0-aba584cb1245 
[2023-07-03 20:04:37.036004 +0000] I [MSGID: 106493] [glusterd-handler.c:2961:__glusterd_handle_probe_query] 0-glusterd: Responded to de-fra-02, op_ret: 0, op_errno: 0, ret: 0 
[2023-07-03 20:04:37.038134 +0000] I [MSGID: 106490] [glusterd-handler.c:2546:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 0a43f531-dff3-42a8-a7b0-aba584cb1245 
[2023-07-03 20:04:37.041224 +0000] I [MSGID: 106493] [glusterd-handler.c:3819:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to de-fra-02 (0), ret: 0, op_ret: 0 
[2023-07-03 20:04:37.045796 +0000] I [MSGID: 106492] [glusterd-handler.c:2729:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 0a43f531-dff3-42a8-a7b0-aba584cb1245 
[2023-07-03 20:04:37.045823 +0000] I [MSGID: 106502] [glusterd-handler.c:2776:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend 
[2023-07-03 20:04:37.045947 +0000] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 0a43f531-dff3-42a8-a7b0-aba584cb1245 
[2023-07-03 20:04:50.967615 +0000] I [MSGID: 106487] [glusterd-handler.c:1388:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req 
[2023-07-03 20:05:26.271723 +0000] E [MSGID: 106429] [glusterd-utils.c:1306:glusterd_brickinfo_new_from_brick] 0-management: Failed to convert hostname de-fra-01 to uuid 
[2023-07-03 20:05:26.271749 +0000] E [MSGID: 106301] [glusterd-syncop.c:1328:gd_stage_op_phase] 0-management: Staging of operation 'Volume Create' failed on localhost : Host de-fra-01 is not in 'Peer in Cluster' state 
[2023-07-03 20:06:07.510592 +0000] I [MSGID: 106487] [glusterd-handler.c:1388:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req 
[2023-07-03 20:06:16.346462 +0000] I [MSGID: 106487] [glusterd-handler.c:1133:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req de-fra-02 24007 
[2023-07-03 20:06:19.832189 +0000] I [MSGID: 106487] [glusterd-handler.c:1133:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req de-fra-01 24007 
[2023-07-03 20:06:19.832457 +0000] I [MSGID: 106128] [glusterd-handler.c:3651:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: de-fra-01 (24007) 
[2023-07-03 20:06:19.835328 +0000] I [rpc-clnt.c:1015:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2023-07-03 20:06:19.835669 +0000] I [MSGID: 106498] [glusterd-handler.c:3583:glusterd_friend_add] 0-management: connect returned 0 
[2023-07-03 20:06:19.836378 +0000] E [MSGID: 106170] [glusterd-handshake.c:1331:gd_validate_mgmt_hndsk_req] 0-management: Request from peer <XXX>:49150 has an entry in peerinfo, but uuid does not match 
[2023-07-03 20:06:19.836396 +0000] E [MSGID: 106170] [glusterd-handshake.c:1341:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management handshake request from unknown peer <XXX>:49150 
[2023-07-03 20:06:19.836602 +0000] E [MSGID: 106165] [glusterd-handshake.c:2160:__glusterd_mgmt_hndsk_version_cbk] 0-management: failed to get the 'versions' from peer (<XXX>:24007) [Invalid argument]
[2023-07-03 20:06:19.836626 +0000] I [socket.c:833:__socket_shutdown] 0-management: intentional socket shutdown(14)
[2023-07-03 20:06:19.836641 +0000] I [MSGID: 106004] [glusterd-handler.c:6301:__glusterd_peer_rpc_notify] 0-management: Peer <de-fra-01> (<00000000-0000-0000-0000-000000000000>), in state <Establishing Connection>, has disconnected from glusterd. 
[2023-07-03 20:06:19.836771 +0000] I [rpc-clnt.c:1015:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600
[2023-07-03 20:06:19.836852 +0000] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped 
[2023-07-03 20:06:19.836862 +0000] I [MSGID: 106568] [glusterd-svc-mgmt.c:265:glusterd_svc_stop] 0-management: quotad service is stopped 
[2023-07-03 20:06:19.837292 +0000] I [rpc-clnt.c:1015:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600
[2023-07-03 20:06:19.837359 +0000] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped 
[2023-07-03 20:06:19.837367 +0000] I [MSGID: 106568] [glusterd-svc-mgmt.c:265:glusterd_svc_stop] 0-management: bitd service is stopped 
[2023-07-03 20:06:19.837384 +0000] I [rpc-clnt.c:1015:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600
[2023-07-03 20:06:19.837437 +0000] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped 
[2023-07-03 20:06:19.837445 +0000] I [MSGID: 106568] [glusterd-svc-mgmt.c:265:glusterd_svc_stop] 0-management: scrub service is stopped 
[2023-07-03 20:06:19.835321 +0000] W [MSGID: 106061] [glusterd-handler.c:3424:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout 
[2023-07-03 20:07:38.808496 +0000] I [MSGID: 106487] [glusterd-handler.c:1133:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req de-fra-01 24007 
[2023-07-03 20:07:38.808742 +0000] I [MSGID: 106128] [glusterd-handler.c:3651:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: de-fra-01 (24007) 
[2023-07-03 20:07:38.811976 +0000] W [MSGID: 106061] [glusterd-handler.c:3424:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout 
[2023-07-03 20:07:38.811998 +0000] I [rpc-clnt.c:1015:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2023-07-03 20:07:38.812309 +0000] I [MSGID: 106498] [glusterd-handler.c:3583:glusterd_friend_add] 0-management: connect returned 0 
[2023-07-03 20:07:38.813233 +0000] E [MSGID: 106170] [glusterd-handshake.c:1331:gd_validate_mgmt_hndsk_req] 0-management: Request from peer <XXX>:49150 has an entry in peerinfo, but uuid does not match 
[2023-07-03 20:07:38.813251 +0000] E [MSGID: 106170] [glusterd-handshake.c:1341:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management handshake request from unknown peer <XXX>:49150 
[2023-07-03 20:07:38.813458 +0000] E [MSGID: 106165] [glusterd-handshake.c:2160:__glusterd_mgmt_hndsk_version_cbk] 0-management: failed to get the 'versions' from peer (<XXX>:24007) [Invalid argument]
[2023-07-03 20:07:38.813490 +0000] I [socket.c:833:__socket_shutdown] 0-management: intentional socket shutdown(14)
[2023-07-03 20:07:38.813505 +0000] I [MSGID: 106004] [glusterd-handler.c:6301:__glusterd_peer_rpc_notify] 0-management: Peer <de-fra-01> (<00000000-0000-0000-0000-000000000000>), in state <Establishing Connection>, has disconnected from glusterd. 
[2023-07-03 20:07:38.813614 +0000] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped 
[2023-07-03 20:07:38.813627 +0000] I [MSGID: 106568] [glusterd-svc-mgmt.c:265:glusterd_svc_stop] 0-management: quotad service is stopped 
[2023-07-03 20:07:38.813637 +0000] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped 
[2023-07-03 20:07:38.813644 +0000] I [MSGID: 106568] [glusterd-svc-mgmt.c:265:glusterd_svc_stop] 0-management: bitd service is stopped 
[2023-07-03 20:07:38.813652 +0000] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped 
[2023-07-03 20:07:38.813659 +0000] I [MSGID: 106568] [glusterd-svc-mgmt.c:265:glusterd_svc_stop] 0-management: scrub service is stopped 
[2023-07-03 20:07:46.539797 +0000] I [MSGID: 106487] [glusterd-handler.c:1388:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req 
[2023-07-03 20:11:30.505096 +0000] I [MSGID: 106487] [glusterd-handler.c:1388:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req 
[2023-07-03 20:12:45.707972 +0000] E [MSGID: 106290] [glusterd-utils.c:14808:glusterd_check_brick_order] 0-management: unable to resolve host name for addr server1 
[2023-07-03 20:12:45.708506 +0000] E [MSGID: 106408] [glusterd-peer-utils.c:214:glusterd_peerinfo_find_by_hostname] 0-management: error in getaddrinfo: Name or service not known
 [Unknown error -2]
[2023-07-03 20:12:45.708965 +0000] E [MSGID: 101075] [common-utils.c:3641:gf_is_local_addr] 0-management: error in getaddrinfo [{ret=Name or service not known}] 
[2023-07-03 20:12:45.708980 +0000] E [MSGID: 106429] [glusterd-utils.c:1306:glusterd_brickinfo_new_from_brick] 0-management: Failed to convert hostname server1 to uuid 
[2023-07-03 20:12:45.708988 +0000] E [MSGID: 106301] [glusterd-syncop.c:1328:gd_stage_op_phase] 0-management: Staging of operation 'Volume Create' failed on localhost : Host server1 is not in 'Peer in Cluster' state 
[2023-07-03 20:12:53.706610 +0000] E [MSGID: 106290] [glusterd-utils.c:14808:glusterd_check_brick_order] 0-management: unable to resolve host name for addr server2 
[2023-07-03 20:12:53.707618 +0000] E [MSGID: 101075] [common-utils.c:3641:gf_is_local_addr] 0-management: error in getaddrinfo [{ret=Name or service not known}] 
[2023-07-03 20:12:53.707620 +0000] E [MSGID: 106429] [glusterd-utils.c:1306:glusterd_brickinfo_new_from_brick] 0-management: Failed to convert hostname server2 to uuid 
[2023-07-03 20:12:53.707640 +0000] E [MSGID: 106301] [glusterd-syncop.c:1328:gd_stage_op_phase] 0-management: Staging of operation 'Volume Create' failed on localhost : Host server2 is not in 'Peer in Cluster' state 
[2023-07-03 20:13:09.002867 +0000] E [MSGID: 106429] [glusterd-utils.c:1306:glusterd_brickinfo_new_from_brick] 0-management: Failed to convert hostname de-fra-01 to uuid 
[2023-07-03 20:13:09.002901 +0000] E [MSGID: 106301] [glusterd-syncop.c:1328:gd_stage_op_phase] 0-management: Staging of operation 'Volume Create' failed on localhost : Host de-fra-01 is not in 'Peer in Cluster' state 
[2023-07-03 20:12:53.707244 +0000] E [MSGID: 106408] [glusterd-peer-utils.c:214:glusterd_peerinfo_find_by_hostname] 0-management: error in getaddrinfo: Name or service not known
 [Unknown error -2]
[2023-07-03 20:15:12.140502 +0000] I [MSGID: 106487] [glusterd-handler.c:1388:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req 
[2023-07-03 20:15:39.887680 +0000] E [MSGID: 106429] [glusterd-utils.c:1306:glusterd_brickinfo_new_from_brick] 0-management: Failed to convert hostname de-fra-01 to uuid 
[2023-07-03 20:15:39.887706 +0000] E [MSGID: 106301] [glusterd-syncop.c:1328:gd_stage_op_phase] 0-management: Staging of operation 'Volume Create' failed on localhost : Host de-fra-01 is not in 'Peer in Cluster' state 
[2023-07-03 20:18:27.579440 +0000] E [MSGID: 106429] [glusterd-utils.c:1306:glusterd_brickinfo_new_from_brick] 0-management: Failed to convert hostname de-fra-01 to uuid 
[2023-07-03 20:18:27.579469 +0000] E [MSGID: 106301] [glusterd-syncop.c:1328:gd_stage_op_phase] 0-management: Staging of operation 'Volume Create' failed on localhost : Host de-fra-01 is not in 'Peer in Cluster' state 

Server 2:

/var/log/glusterfs/cli.log

[2023-07-03 20:04:42.077466 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:04:42.134475 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:04:42.134565 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:04:42.136312 +0000] I [cli-rpc-ops.c:162:gf_cli_probe_cbk] 0-cli: Received resp to probe
[2023-07-03 20:04:42.136537 +0000] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2023-07-03 20:04:46.536275 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:04:46.592871 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:04:46.592909 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:04:46.594449 +0000] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2023-07-03 20:08:51.797737 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:08:51.853291 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:08:51.853299 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:08:51.856487 +0000] I [cli-rpc-ops.c:1015:gf_cli_create_volume_cbk] 0-cli: Received resp to create volume
[2023-07-03 20:08:51.856553 +0000] I [input.c:31:cli_batch] 0-: Exiting with: -1
[2023-07-03 20:08:54.305348 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:08:54.361815 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:08:54.361871 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:08:54.362227 +0000] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2023-07-03 20:08:58.058780 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:08:58.114404 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:08:58.114446 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:08:58.114824 +0000] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2023-07-03 20:09:16.957638 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:09:17.015409 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:09:17.015461 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:09:24.899188 +0000] I [cli-rpc-ops.c:1015:gf_cli_create_volume_cbk] 0-cli: Received resp to create volume
[2023-07-03 20:09:24.899275 +0000] I [input.c:31:cli_batch] 0-: Exiting with: -1
[2023-07-03 20:09:28.812714 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:09:28.870678 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:09:28.870722 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:09:28.872712 +0000] I [cli-rpc-ops.c:1015:gf_cli_create_volume_cbk] 0-cli: Received resp to create volume
[2023-07-03 20:09:28.872813 +0000] I [input.c:31:cli_batch] 0-: Exiting with: -1
[2023-07-03 20:10:15.603021 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:10:15.658410 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:10:15.658473 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:10:15.659198 +0000] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2023-07-03 20:11:33.827112 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:11:33.883245 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:11:33.883280 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:11:33.883677 +0000] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2023-07-03 20:15:18.062162 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:15:18.117809 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:15:18.117856 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:15:18.118989 +0000] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2023-07-03 20:18:49.822495 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:18:49.878593 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:18:49.878737 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:18:49.880543 +0000] I [cli-rpc-ops.c:1015:gf_cli_create_volume_cbk] 0-cli: Received resp to create volume
[2023-07-03 20:18:49.880605 +0000] I [input.c:31:cli_batch] 0-: Exiting with: -1
[2023-07-03 20:31:25.533480 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:31:25.589404 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:31:25.589447 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:31:25.591305 +0000] I [cli-rpc-ops.c:1015:gf_cli_create_volume_cbk] 0-cli: Received resp to create volume
[2023-07-03 20:31:25.591351 +0000] I [input.c:31:cli_batch] 0-: Exiting with: -1
[2023-07-03 20:32:45.454811 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:32:45.510354 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:32:45.510402 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:32:45.510690 +0000] I [cli-rpc-ops.c:767:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2023-07-03 20:32:45.510733 +0000] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2023-07-03 20:33:01.031028 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:33:01.086425 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:33:01.086475 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:33:01.088729 +0000] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2023-07-03 20:33:12.154987 +0000] I [cli.c:828:main] 0-cli: Started running gluster with version 10.3
[2023-07-03 20:33:12.211663 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:33:12.211957 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] 
[2023-07-03 20:33:12.212019 +0000] I [input.c:31:cli_batch] 0-: Exiting with: -1

/var/log/glusterfs/cmd_history.log

[2023-07-03 20:04:42.135719 +0000]  : peer probe de-fra-01 : SUCCESS : Host de-fra-01 port 24007 already in peer list
[2023-07-03 20:08:51.856414 +0000]  : volume create vl de-fra-01:/mnt/storage de-fra-02:/mnt/storage : FAILED : Host de-fra-02 is not in 'Peer in Cluster' state
[2023-07-03 20:09:24.899080 +0000]  : volume create vl replica 2 de-fra-01:/mnt/storage de-fra-02:/mnt/storage : FAILED : Host de-fra-02 is not in 'Peer in Cluster' state
[2023-07-03 20:09:28.872629 +0000]  : volume create vl de-fra-01:/mnt/storage de-fra-02:/mnt/storage : FAILED : Host de-fra-02 is not in 'Peer in Cluster' state
[2023-07-03 20:18:49.880467 +0000]  : volume create myvolume de-fra-01:/mnt/storage de-fra-02:/mnt/storage : FAILED : Host de-fra-02 is not in 'Peer in Cluster' state
[2023-07-03 20:31:25.591224 +0000]  : volume create myvolume de-fra-01:/mnt/storage de-fra-02:/mnt/storage : FAILED : Host de-fra-02 is not in 'Peer in Cluster' state
[2023-07-03 20:33:01.088093 +0000]  : volume status : SUCCESS

/var/log/glusterfs/glusterd.log

[2023-07-03 20:02:30.295471 +0000] I [MSGID: 100030] [glusterfsd.c:2767:main] 0-/usr/sbin/glusterd: Started running version [{arg=/usr/sbin/glusterd}, {version=10.3}, {cmdlinestr=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO}] 
[2023-07-03 20:02:30.296164 +0000] I [glusterfsd.c:2447:daemonize] 0-glusterfs: Pid of current running process is 4218
[2023-07-03 20:02:30.349343 +0000] I [MSGID: 106478] [glusterd.c:1470:init] 0-management: Maximum allowed open file descriptors set to 65536 
[2023-07-03 20:02:30.349559 +0000] I [MSGID: 106479] [glusterd.c:1560:init] 0-management: Using /var/lib/glusterd as working directory 
[2023-07-03 20:02:30.349565 +0000] I [MSGID: 106479] [glusterd.c:1564:init] 0-management: Using /var/run/gluster as pid file working directory 
[2023-07-03 20:02:30.352420 +0000] I [socket.c:971:__socket_server_bind] 0-socket.management: process started listening on port (24007)
[2023-07-03 20:02:30.353522 +0000] I [socket.c:914:__socket_server_bind] 0-socket.management: closing (AF_UNIX) reuse check socket 12
[2023-07-03 20:02:30.354107 +0000] I [MSGID: 106059] [glusterd.c:1952:init] 0-management: max-port override: 60999 
[2023-07-03 20:02:30.431887 +0000] E [MSGID: 106061] [glusterd.c:590:glusterd_crt_georep_folders] 0-glusterd: Dict get failed [{Key=log-group}, {errno=2}, {error=No such file or directory}] 
[2023-07-03 20:02:31.737291 +0000] E [MSGID: 101032] [store.c:461:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2023-07-03 20:02:31.737684 +0000] E [MSGID: 101032] [store.c:461:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2023-07-03 20:02:31.737685 +0000] I [MSGID: 106514] [glusterd-store.c:2153:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 100000 
[2023-07-03 20:02:31.737701 +0000] E [MSGID: 101032] [store.c:461:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/options. [No such file or directory]
[2023-07-03 20:02:31.742944 +0000] I [MSGID: 106194] [glusterd-store.c:3980:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list. 
[2023-07-03 20:02:31.742966 +0000] E [MSGID: 101032] [store.c:461:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.upgrade. [No such file or directory]
[2023-07-03 20:02:31.742974 +0000] I [glusterd.c:2090:init] 0-management: Regenerating volfiles due to a max op-version mismatch or glusterd.upgrade file not being present, op_version retrieved:0, max op_version: 100000
Final graph:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option rpc-auth.auth-glusterfs on
  4:     option rpc-auth.auth-unix on
  5:     option rpc-auth.auth-null on
  6:     option rpc-auth-allow-insecure on
  7:     option transport.listen-backlog 1024
  8:     option max-port 60999
  9:     option event-threads 1
 10:     option ping-timeout 0
 11:     option transport.socket.listen-port 24007
 12:     option transport.socket.read-fail-log off
 13:     option transport.socket.keepalive-interval 2
 14:     option transport.socket.keepalive-time 10
 15:     option transport-type socket
 16:     option working-directory /var/lib/glusterd
 17: end-volume
 18:  
+------------------------------------------------------------------------------+
[2023-07-03 20:02:31.747807 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] 
[2023-07-03 20:02:32.038474 +0000] I [MSGID: 106163] [glusterd-handshake.c:1501:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 100000 
[2023-07-03 20:02:32.038514 +0000] E [MSGID: 101032] [store.c:461:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2023-07-03 20:02:32.038714 +0000] I [MSGID: 106477] [glusterd.c:180:glusterd_uuid_generate_save] 0-management: generated UUID: 0a43f531-dff3-42a8-a7b0-aba584cb1245 
[2023-07-03 20:02:32.044961 +0000] I [MSGID: 106490] [glusterd-handler.c:2546:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 5a34f977-9373-46d9-8bab-2801f00c86af 
[2023-07-03 20:02:32.072060 +0000] I [MSGID: 106493] [glusterd-handler.c:3819:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to <XXX> (24007), ret: 0, op_ret: -1 
[2023-07-03 20:04:37.129759 +0000] I [MSGID: 106163] [glusterd-handshake.c:1501:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 100000 
[2023-07-03 20:04:37.132625 +0000] I [MSGID: 106490] [glusterd-handler.c:2902:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: 08eae420-3aef-4604-9f45-4c518739706a 
[2023-07-03 20:04:37.132723 +0000] I [MSGID: 106128] [glusterd-handler.c:2935:__glusterd_handle_probe_query] 0-glusterd: Unable to find peerinfo for host: de-fra-01 (24007) 
[2023-07-03 20:04:37.136341 +0000] W [MSGID: 106061] [glusterd-handler.c:3424:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout 
[2023-07-03 20:04:37.136360 +0000] I [rpc-clnt.c:1015:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2023-07-03 20:04:37.136949 +0000] I [MSGID: 106498] [glusterd-handler.c:3583:glusterd_friend_add] 0-management: connect returned 0 
[2023-07-03 20:04:37.137190 +0000] I [MSGID: 106493] [glusterd-handler.c:2961:__glusterd_handle_probe_query] 0-glusterd: Responded to de-fra-01, op_ret: 0, op_errno: 0, ret: 0 
[2023-07-03 20:04:37.137516 +0000] I [MSGID: 106490] [glusterd-handler.c:2546:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 08eae420-3aef-4604-9f45-4c518739706a 
[2023-07-03 20:04:37.140556 +0000] I [MSGID: 106493] [glusterd-handler.c:3819:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to de-fra-01 (0), ret: 0, op_ret: 0 
[2023-07-03 20:04:37.152741 +0000] I [MSGID: 106511] [glusterd-rpc-ops.c:248:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: 08eae420-3aef-4604-9f45-4c518739706a, host: de-fra-01 
[2023-07-03 20:04:37.152758 +0000] I [MSGID: 106511] [glusterd-rpc-ops.c:404:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req 
[2023-07-03 20:04:37.156189 +0000] I [MSGID: 106493] [glusterd-rpc-ops.c:467:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 08eae420-3aef-4604-9f45-4c518739706a, host: de-fra-01, port: 0 
[2023-07-03 20:04:37.158400 +0000] I [MSGID: 106492] [glusterd-handler.c:2729:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 08eae420-3aef-4604-9f45-4c518739706a 
[2023-07-03 20:04:37.158416 +0000] I [MSGID: 106502] [glusterd-handler.c:2776:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend 
[2023-07-03 20:04:37.160884 +0000] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 08eae420-3aef-4604-9f45-4c518739706a 
[2023-07-03 20:04:42.135305 +0000] I [MSGID: 106487] [glusterd-handler.c:1133:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req de-fra-01 24007 
[2023-07-03 20:04:46.593875 +0000] I [MSGID: 106487] [glusterd-handler.c:1388:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req 
[2023-07-03 20:08:51.855923 +0000] E [MSGID: 106429] [glusterd-utils.c:1306:glusterd_brickinfo_new_from_brick] 0-management: Failed to convert hostname de-fra-02 to uuid 
[2023-07-03 20:08:51.855948 +0000] E [MSGID: 106301] [glusterd-syncop.c:1328:gd_stage_op_phase] 0-management: Staging of operation 'Volume Create' failed on localhost : Host de-fra-02 is not in 'Peer in Cluster' state 
[2023-07-03 20:08:54.362086 +0000] I [MSGID: 106487] [glusterd-handler.c:1388:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req 
The message "E [MSGID: 106429] [glusterd-utils.c:1306:glusterd_brickinfo_new_from_brick] 0-management: Failed to convert hostname de-fra-02 to uuid" repeated 2 times between [2023-07-03 20:08:51.855923 +0000] and [2023-07-03 20:09:28.872057 +0000]
The message "E [MSGID: 106301] [glusterd-syncop.c:1328:gd_stage_op_phase] 0-management: Staging of operation 'Volume Create' failed on localhost : Host de-fra-02 is not in 'Peer in Cluster' state" repeated 2 times between [2023-07-03 20:08:51.855948 +0000] and [2023-07-03 20:09:28.872060 +0000]
The message "I [MSGID: 106487] [glusterd-handler.c:1388:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated 2 times between [2023-07-03 20:08:54.362086 +0000] and [2023-07-03 20:10:15.658696 +0000]
[2023-07-03 20:11:33.883523 +0000] I [MSGID: 106487] [glusterd-handler.c:1388:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req 
[2023-07-03 20:15:18.118096 +0000] I [MSGID: 106487] [glusterd-handler.c:1388:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req 
[2023-07-03 20:18:49.879953 +0000] E [MSGID: 106429] [glusterd-utils.c:1306:glusterd_brickinfo_new_from_brick] 0-management: Failed to convert hostname de-fra-02 to uuid 
[2023-07-03 20:18:49.879980 +0000] E [MSGID: 106301] [glusterd-syncop.c:1328:gd_stage_op_phase] 0-management: Staging of operation 'Volume Create' failed on localhost : Host de-fra-02 is not in 'Peer in Cluster' state 
[2023-07-03 20:31:25.590662 +0000] E [MSGID: 106429] [glusterd-utils.c:1306:glusterd_brickinfo_new_from_brick] 0-management: Failed to convert hostname de-fra-02 to uuid 
[2023-07-03 20:31:25.590688 +0000] E [MSGID: 106301] [glusterd-syncop.c:1328:gd_stage_op_phase] 0-management: Staging of operation 'Volume Create' failed on localhost : Host de-fra-02 is not in 'Peer in Cluster' state 
[2023-07-03 20:33:01.088064 +0000] I [MSGID: 106061] [glusterd-utils.c:10725:glusterd_volume_status_copy_to_op_ctx_dict] 0-management: Dict get failed [{Key=count}]

**- Is there any crash ? Provide the backtrace and coredump No.

Additional info:

The servers have no firewall activated, all ports are open.

I have the following in /etc/hosts on both servers:

<server1PublicIP>   de-fra-01
<server2PublicIP>   de-fra-02

Output of gluster peer status on Server 1:

Number of Peers: 1

Hostname: de-fra-02
Uuid: 0a43f531-dff3-42a8-a7b0-aba584cb1245
State: Peer in Cluster (Connected)

Output of gluster peer status on Server 2:

Number of Peers: 1

Hostname: de-fra-01
Uuid: 08eae420-3aef-4604-9f45-4c518739706a
State: Peer in Cluster (Connected)

- The operating system / glusterfs version: GlusterFS version 10.3 running on Debian 12 ARM, installed from main Debian 12 repo.

namirda commented 1 year ago

I have exactly the same problem - Glusterfs 10.3, Debian 12, ARM

namirda commented 1 year ago

In my case, the problem was that on one node in the cluster there were some old files remaining from previous glusterfs version. That node was not even in my trusted pool!

To fix, I had to delete all gluster volumes and then

sudo systemctl stop glusterd - on ALL nodes in the cluster, even those not in the peer group sudo rm -rf /var/lib/glusterd - on ALL nodes in the cluster, even those not in the peer group sudo systemctl start glusterd - on ALL nodes in the cluster, even those not in the peer group

Then build your trusted pool and create the volume.

@redhyper9 - hope that helps

NathanEid commented 8 months ago

i have the same problem: volume create: volumes: failed: Host manager is not in 'Peer in Cluster' state i tried some solution but it dosn't work GlusterFS version 10.5 running on Ubuntu 18.04.3 LTS how can i solve this problem? @redhyper9 @namirda @manveru @dev-zero

cs35-owncloud commented 8 months ago

Hello,

I had that kind of problem on Debian 12 aswell. To fix it, you have to peer the first node from one of the other nodes:

root@node2 # gluster peer probe node1

Then you will be able to create your volume. It seems that when you probe all the others nodes from the first node, the first node will be added but with the "wrong" hostname.

Hope this helps, it took me a while to figure this out :exploding_head: Cheers.

EDIT: some doc

aravindavk commented 8 months ago
[2023-07-03 20:13:09.002901 +0000] E [MSGID: 106301] [glusterd-syncop.c:1328:gd_stage_op_phase] 0-management: Staging of operation 'Volume Create' failed on localhost : Host de-fra-01 is not in 'Peer in Cluster' state 

Looking at the logs, Gluster is failing to understand de-fra-01 is the local address in de-fra-01 and similarly in de-fra-02. Update the /etc/hosts like below and try again.

Server 1:

127.0.0.1   de-fra-01
<server2PublicIP>   de-fra-02

Server 2:

<server1PublicIP> de-fra-01
127.0.0.1   de-fra-02
thebigbone commented 1 week ago

For some reason, the host names in /etc/hosts are not properly mapped with the IPs. Just specifying the server IP (instead of host names) solved that issue for me.