ksandha commented 6 years ago

1) Updated the rpms of the glusterd2

[root@dhcp35-181 ~]# yum update 
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
epel/x86_64/metalink                                                              | 5.7 kB  00:00:00     
 * base: ftp.iitm.ac.in
 * epel: del-repos.extreme-ix.org
 * extras: ftp.iitm.ac.in
 * updates: ftp.iitm.ac.in
base                                                                              | 3.6 kB  00:00:00     
epel                                                                              | 3.2 kB  00:00:00     
extras                                                                            | 3.4 kB  00:00:00     
gluster-nightly-master                                                            | 3.0 kB  00:00:00     
glusterd2-nightly-master                                                          | 3.0 kB  00:00:00     
updates                                                                           | 3.4 kB  00:00:00     
(1/5): glusterd2-nightly-master/7/x86_64/primary_db                               | 3.2 kB  00:00:00     
(2/5): extras/7/x86_64/primary_db                                                 | 174 kB  00:00:00     
(3/5): updates/7/x86_64/primary_db                                                | 5.0 MB  00:00:01     
(4/5): epel/x86_64/updateinfo                                                     | 933 kB  00:00:02     
(5/5): epel/x86_64/primary                                                        | 3.6 MB  00:00:12     
epel                                                                                         12647/12647
Resolving Dependencies
--> Running transaction check
---> Package centos-release.x86_64 0:7-5.1804.1.el7.centos will be updated
---> Package centos-release.x86_64 0:7-5.1804.4.el7.centos will be an update
---> Package glusterd2.x86_64 0:4.1.0-143.git3c9ca84.el7 will be updated
---> Package glusterd2.x86_64 0:4.1.0-147.git38d20ee.el7 will be an update
---> Package kernel.x86_64 0:3.10.0-862.11.6.el7 will be installed
---> Package kernel-tools.x86_64 0:3.10.0-862.9.1.el7 will be updated
---> Package kernel-tools.x86_64 0:3.10.0-862.11.6.el7 will be an update
---> Package kernel-tools-libs.x86_64 0:3.10.0-862.9.1.el7 will be updated
---> Package kernel-tools-libs.x86_64 0:3.10.0-862.11.6.el7 will be an update
---> Package python-perf.x86_64 0:3.10.0-862.9.1.el7 will be updated
---> Package python-perf.x86_64 0:3.10.0-862.11.6.el7 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

=========================================================================================================
 Package                Arch        Version                          Repository                     Size
=========================================================================================================
Installing:
 kernel                 x86_64      3.10.0-862.11.6.el7              updates                        46 M
Updating:
 centos-release         x86_64      7-5.1804.4.el7.centos            updates                        25 k
 glusterd2              x86_64      4.1.0-147.git38d20ee.el7         glusterd2-nightly-master      8.1 M
 kernel-tools           x86_64      3.10.0-862.11.6.el7              updates                       6.3 M
 kernel-tools-libs      x86_64      3.10.0-862.11.6.el7              updates                       6.2 M
 python-perf            x86_64      3.10.0-862.11.6.el7              updates                       6.3 M

Transaction Summary
=========================================================================================================
Install  1 Package
Upgrade  5 Packages

Total download size: 73 M
Is this ok [y/d/N]: y
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
(1/6): centos-release-7-5.1804.4.el7.centos.x86_64.rpm                            |  25 kB  00:00:01     
(2/6): python-perf-3.10.0-862.11.6.el7.x86_64.rpm                                 | 6.3 MB  00:00:17     
(3/6): kernel-tools-3.10.0-862.11.6.el7.x86_64.rpm                                | 6.3 MB  00:00:47     
(4/6): kernel-3.10.0-862.11.6.el7.x86_64.rpm                                      |  46 MB  00:01:36     
(5/6): kernel-tools-libs-3.10.0-862.11.6.el7.x86_64.rpm                           | 6.2 MB  00:03:23     
(6/6): glusterd2-4.1.0-147.git38d20ee.el7.x86_64.rpm                              | 8.1 MB  00:05:56     
---------------------------------------------------------------------------------------------------------
Total                                                                    210 kB/s |  73 MB  00:05:56     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : centos-release-7-5.1804.4.el7.centos.x86_64                                          1/11 
  Updating   : kernel-tools-libs-3.10.0-862.11.6.el7.x86_64                                         2/11 
  Updating   : kernel-tools-3.10.0-862.11.6.el7.x86_64                                              3/11 
  Installing : kernel-3.10.0-862.11.6.el7.x86_64                                                    4/11 
  Updating   : glusterd2-4.1.0-147.git38d20ee.el7.x86_64                                            5/11 
  Updating   : python-perf-3.10.0-862.11.6.el7.x86_64                                               6/11 
  Cleanup    : kernel-tools-3.10.0-862.9.1.el7.x86_64                                               7/11 
  Cleanup    : centos-release-7-5.1804.1.el7.centos.x86_64                                          8/11 
  Cleanup    : kernel-tools-libs-3.10.0-862.9.1.el7.x86_64                                          9/11 
  Cleanup    : glusterd2-4.1.0-143.git3c9ca84.el7.x86_64                                           10/11 
  Cleanup    : python-perf-3.10.0-862.9.1.el7.x86_64                                               11/11 
  Verifying  : kernel-tools-3.10.0-862.11.6.el7.x86_64                                              1/11 
  Verifying  : kernel-tools-libs-3.10.0-862.11.6.el7.x86_64                                         2/11 
  Verifying  : python-perf-3.10.0-862.11.6.el7.x86_64                                               3/11 
  Verifying  : glusterd2-4.1.0-147.git38d20ee.el7.x86_64                                            4/11 
  Verifying  : kernel-3.10.0-862.11.6.el7.x86_64                                                    5/11 
  Verifying  : centos-release-7-5.1804.4.el7.centos.x86_64                                          6/11 
  Verifying  : python-perf-3.10.0-862.9.1.el7.x86_64                                                7/11 
  Verifying  : kernel-tools-libs-3.10.0-862.9.1.el7.x86_64                                          8/11 
  Verifying  : glusterd2-4.1.0-143.git3c9ca84.el7.x86_64                                            9/11 
  Verifying  : kernel-tools-3.10.0-862.9.1.el7.x86_64                                              10/11 
  Verifying  : centos-release-7-5.1804.1.el7.centos.x86_64                                         11/11 

Installed:
  kernel.x86_64 0:3.10.0-862.11.6.el7                                                                    

Updated:
  centos-release.x86_64 0:7-5.1804.4.el7.centos      glusterd2.x86_64 0:4.1.0-147.git38d20ee.el7        
  kernel-tools.x86_64 0:3.10.0-862.11.6.el7          kernel-tools-libs.x86_64 0:3.10.0-862.11.6.el7     
  python-perf.x86_64 0:3.10.0-862.11.6.el7          

Complete!

2) Ran the systemctl status glusterd2

[root@dhcp35-181 ~]# systemctl status glusterd2
● glusterd2.service - GlusterD2, the management service for GlusterFS (pre-release)
   Loaded: loaded (/usr/lib/systemd/system/glusterd2.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2018-08-16 12:57:00 IST; 11min ago
 Main PID: 1237 (glusterd2)
   CGroup: /system.slice/glusterd2.service
           ├─1237 /usr/sbin/glusterd2 --config=/etc/glusterd2/glusterd2.toml
           ├─1628 /usr/sbin/glusterfsd --volfile-server 127.0.0.1 --volfile-server-port 24007 --volfil...
           ├─1683 /usr/sbin/glusterfsd --volfile-server 127.0.0.1 --volfile-server-port 24007 --volfil...
           └─1736 /usr/sbin/glusterfsd --volfile-server 127.0.0.1 --volfile-server-port 24007 --volfil...

Aug 16 12:57:00 dhcp35-181.lab.eng.blr.redhat.com systemd[1]: Started GlusterD2, the management serv...).
Aug 16 12:57:00 dhcp35-181.lab.eng.blr.redhat.com systemd[1]: Starting GlusterD2, the management ser.....
Warning: glusterd2.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Hint: Some lines were ellipsized, use -l to show in full.

3) Check for peer status on all nodes

[root@dhcp35-181 ~]# gluster peer status
Connection failed. Please check if gluster daemon is operational.
[root@dhcp35-181 ~]# glustercli peer status
+--------------------------------------+-----------------------------------+--------------------+-----------------------------------------+--------+------+
|                  ID                  |               NAME                |  CLIENT ADDRESSES  |             PEER ADDRESSES              | ONLINE | PID  |
+--------------------------------------+-----------------------------------+--------------------+-----------------------------------------+--------+------+
| bc8dcfaf-e848-427b-839b-2d4c7a0a9034 | dhcp35-26.lab.eng.blr.redhat.com  | 127.0.0.1:24007    | dhcp35-26.lab.eng.blr.redhat.com:24008  | yes    | 1238 |
|                                      |                                   | 10.70.35.26:24007  | 10.70.35.26:24008                       |        |      |
| ddf8af56-04cf-4e49-a4c7-99d90904bcb7 | dhcp35-88.lab.eng.blr.redhat.com  | 127.0.0.1:24007    | 10.70.35.88:24008                       | yes    | 1233 |
|                                      |                                   | 10.70.35.88:24007  |                                         |        |      |
| de9fab15-40ac-46c7-90f0-6a1659b0fb4b | dhcp35-181.lab.eng.blr.redhat.com | 127.0.0.1:24007    | dhcp35-181.lab.eng.blr.redhat.com:24008 | yes    | 1237 |
|                                      |                                   | 10.70.35.181:24007 | 10.70.35.181:24008                      |        |      |
+--------------------------------------+-----------------------------------+--------------------+-----------------------------------------+--------+------+
[root@dhcp35-181 ~]# 
[root@dhcp35-181 ~]# 

4) Check the gluster peer status

[root@dhcp35-181 ~]# glustercli volume status Volume : newvol Error getting Volume status

Failed to connect to glusterd. Please check if

Glusterd is running(http://127.0.0.1:24007) and reachable from this node.
Make sure Endpoints specified in the command is valid [root@dhcp35-181 ~]#

### Observed behavior
Glustercli volume status not able to check

### Expected/desired behavior
There shouldn't be any errors in the out put

### Details on how to reproduce (minimal and precise)
hit it as sson as i updated the rpms

### Information about the environment:

- Glusterd2 version used (e.g. v4.1.0 or master): 
[root@dhcp35-181 ~]# rpm -qa | grep glusterd2
glusterd2-4.1.0-147.git38d20ee.el7.x86_64

- Operating system used: 
[root@dhcp35-181 ~]# cat /etc/centos-release
CentOS Linux release 7.5.1804 (Core) 

- Glusterd2 compiled from sources, as a package (rpm/deb), or container: 
package
- Using External ETCD: (yes/no, if yes ETCD version):
yes, etcdmain: etcd Version: 3.3.8

- If container, which container image: 
- Using kubernetes, openshift, or direct install: 
direct install

- If kubernetes/openshift, is gluster running inside kubernetes/openshift or outside: 

### Other useful information

- glusterd2 config files from all nodes (default /etc/glusterd2/glusterd2.toml)

[root@dhcp35-88 ~]# cat /etc/glusterd2/glusterd2.toml

localstatedir = "/var/lib/glusterd2" logdir = "/var/log/glusterd2" logfile = "glusterd2.log" loglevel = "INFO" rundir = "/var/run/glusterd2"

defaultpeerport = "24008"

peeraddress = ":24008" clientaddress = ":24007"

restauth should be set to false to disable REST authentication in glusterd2

restauth = false etcdendpoints = "http://10.70.35.220:2379" noembed = true [root@dhcp35-88 ~]#

[root@dhcp35-181 ~]# cat /etc/glusterd2/glusterd2.toml

localstatedir = "/var/lib/glusterd2" logdir = "/var/log/glusterd2" logfile = "glusterd2.log" loglevel = "INFO" rundir = "/var/run/glusterd2" defaultpeerport = "24008" peeraddress = ":24008" clientaddress = ":24007"

restauth should be set to false to disable REST authentication in glusterd2

restauth = false etcdendpoints = "http://10.70.35.220:2379" noembed = true [root@dhcp35-181 ~]#

[root@dhcp35-26 ~]# cat /etc/glusterd2/glusterd2.toml

localstatedir = "/var/lib/glusterd2" logdir = "/var/log/glusterd2" logfile = "glusterd2.log" loglevel = "INFO" rundir = "/var/run/glusterd2" defaultpeerport = "24008" peeraddress = ":24008" clientaddress = ":24007"

restauth should be set to false to disable REST authentication in glusterd2

restauth = false etcdendpoints = "http://10.70.35.220:2379" noembed = true [root@dhcp35-26 ~]#


- glusterd2 log files from all nodes (default /var/log/glusterd2/glusterd2.log)

- ETCD configuration
 ./etcd  --listen-peer-urls http://10.70.35.220:2380  --listen-client-urls http://10.70.35.220:2379  --advertise-client-urls http://10.70.35.220:2379 --data-dir="/var/lib/etcd"

- Contents of `uuid.toml` from all nodes (default /var/lib/glusterd2/uuid.toml)

[root@dhcp35-88 ~]# cat /var/lib/glusterd2/uuid.toml cluster-id = "a193619e-ce9d-4c19-8e0b-5a82ab8655ff" peer-id = "ddf8af56-04cf-4e49-a4c7-99d90904bcb7" [root@dhcp35-88 ~]# [root@dhcp35-181 ~]# cat /var/lib/glusterd2/uuid.toml cluster-id = "a193619e-ce9d-4c19-8e0b-5a82ab8655ff" peer-id = "de9fab15-40ac-46c7-90f0-6a1659b0fb4b" [root@dhcp35-181 ~]# [root@dhcp35-26 ~]# cat /var/lib/glusterd2/uuid.toml cluster-id = "a193619e-ce9d-4c19-8e0b-5a82ab8655ff" peer-id = "bc8dcfaf-e848-427b-839b-2d4c7a0a9034" [root@dhcp35-26 ~]#

- Output of `statedump` from any one of the node

### Useful commands

- To get glusterd2 version

glusterd2 --version

[root@dhcp35-88 ~]# glusterd2 --version glusterd version: v4.1.0-147.git38d20ee git SHA: 38d20ee go version: go1.9.4 go OS/arch: linux/amd64 [root@dhcp35-88 ~]#

```

To get ETCD version
```
etcd --version
```

To get output of statedump

curl http://glusterd2-IP:glusterd2-Port/statedump

rishubhjain commented 6 years ago

@ksandha How did glustercli volume status worked in https://github.com/gluster/glusterd2/issues/1179 ?

ksandha commented 6 years ago

@rishubhjain , I turned off the gluster servers and restarted it again for #1179 .

rishubhjain commented 6 years ago

@ksandha so this issue is not reproducible with updated glusterd2 rpm, am I right?

ksandha commented 6 years ago

This Issue came when i updated the rpms to the mentioned ones, i.e. the latest ones

On Thu, Aug 16, 2018 at 4:22 PM Rishubh Jain notifications@github.com wrote:

@ksandha https://github.com/ksandha so this issue is not reproducible with updated glusterd2 rpm, am I right?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/gluster/glusterd2/issues/1177#issuecomment-413504863, or mute the thread https://github.com/notifications/unsubscribe-auth/ASY27rIqy5fBh4X38im0Y3IZdlQVqByzks5uRU8BgaJpZM4V_dX8 .

--

KARAN SANDHA

QUALITY ENGINEER

Red Hat Bangalore https://www.redhat.com/

ksandha@redhat.com M: 9888009555 <javascript:void(0);> IM: Karan on @irc https://red.ht/sig TRIED. TESTED. TRUSTED. https://redhat.com/trusted @redhatnews https://twitter.com/redhatnews Red Hat https://www.linkedin.com/company/red-hat Red Hat https://www.facebook.com/RedHatInc

Madhu-1 commented 6 years ago

I have a couple of points to check here

make sure ETCD is reachable from all the glusterd2 nodes (do curl http://{ETCD-IP}:2379)
make sure firewalld service is not running on all nodes (including the system where ETCD is running)
I think you need to reload systemctl after yum update (Warning: glusterd2.service changed on disk. Run 'systemctl daemon-reload' to reload units.

Madhu-1 commented 6 years ago

@ksandha is this issue still reproducible?, if not please close this one

ksandha commented 6 years ago

@Madhu-1 I am hitting different issues after updating the rpms. Don't close this issue. I will update the issue with setup once i update the rpms.

gluster / glusterd2

Unable to get the volume status after updating the glusterd2 rpms #1177

defaultpeerport = "24008"

restauth should be set to false to disable REST authentication in glusterd2

restauth should be set to false to disable REST authentication in glusterd2

restauth should be set to false to disable REST authentication in glusterd2