search

globus / globus-connect-server

Globus Connect Server
6 stars 10 forks source link

globus-gridftp-server service breaks after yum update from Globus-Toolkit-6-Stable #10

Closed omsai closed 7 years ago

omsai commented 7 years ago

On September 13 onwards, running our daily yum-cron updater breaks the globus-gridftp-server service. Using telnet per the troubleshooting suggested by the docs:

[root@dtn-transfer ~]# telnet 127.0.0.1 2811
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
Connection closed by foreign host.
[root@dtn-transfer ~]#

The yum update leaves behind a gridftp process running and globus-gridftp-server service fails. 

[root@dtn-transfer ~]# pgrep -a gridftp
11545 /usr/sbin/globus-gridftp-server -c /etc/gridftp.conf -C /etc/gridftp.d -pidfile /var/run/globus-gridftp-server.pid -no-detach -config-base-path /
[root@dtn-transfer ~]# systemctl status globus-gridftp-server
● globus-gridftp-server.service - LSB: Globus GridFTP Server
   Loaded: loaded (/etc/rc.d/init.d/globus-gridftp-server; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2017-09-14 06:45:31 EDT; 5h 12min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 13095 ExecStop=/etc/rc.d/init.d/globus-gridftp-server stop (code=exited, status=203/EXEC)
  Process: 13097 ExecStart=/etc/rc.d/init.d/globus-gridftp-server start (code=exited, status=203/EXEC)
 Main PID: 11295 (code=exited, status=203/EXEC)
   CGroup: /system.slice/globus-gridftp-server.service
           └─11545 /usr/sbin/globus-gridftp-server -c /etc/gridftp.conf -C /etc/gridftp.d -pidfile /var/run/globus-gridftp-server.pid -no-detach -config-base-path /

Sep 14 06:45:31 dtn-transfer.net.uconn.edu systemd[1]: Starting LSB: Globus GridFTP Server...
Sep 14 06:45:31 dtn-transfer.net.uconn.edu systemd[1]: globus-gridftp-server.service: control process exited, code=exited status=203
Sep 14 06:45:31 dtn-transfer.net.uconn.edu systemd[1]: Failed to start LSB: Globus GridFTP Server.
Sep 14 06:45:31 dtn-transfer.net.uconn.edu systemd[1]: Unit globus-gridftp-server.service entered failed state.
Sep 14 06:45:31 dtn-transfer.net.uconn.edu systemd[1]: globus-gridftp-server.service failed.
Warning: globus-gridftp-server.service changed on disk. Run 'systemctl daemon-reload' to reload units.
[root@dtn-transfer ~]# journalctl -u globus-gridftp-server -u crond -S today | grep -v CRON | grep -v anacron
-- Logs begin at Fri 2017-08-18 01:44:50 EDT, end at Thu 2017-09-14 11:51:13 EDT. --
Sep 14 06:45:30 dtn-transfer.net.uconn.edu yum[8524]: Updated: globus-xio.x86_64 5.16-1.osg33.el7
Sep 14 06:45:30 dtn-transfer.net.uconn.edu yum[8524]: Updated: globus-gsi-openssl-error.x86_64 3.8-1.osg33.el7
Sep 14 06:45:30 dtn-transfer.net.uconn.edu yum[8524]: Updated: globus-gsi-sysconfig.x86_64 6.11-1.osg33.el7
Sep 14 06:45:30 dtn-transfer.net.uconn.edu yum[8524]: Updated: globus-gsi-proxy-ssl.x86_64 5.10-1.osg33.el7
Sep 14 06:45:30 dtn-transfer.net.uconn.edu yum[8524]: Updated: globus-openssl-module.x86_64 4.8-1.osg33.el7
Sep 14 06:45:30 dtn-transfer.net.uconn.edu yum[8524]: Updated: globus-gsi-cert-utils.x86_64 9.16-1.osg33.el7
Sep 14 06:45:30 dtn-transfer.net.uconn.edu yum[8524]: Updated: globus-gsi-callback.x86_64 5.13-1.osg33.el7
Sep 14 06:45:30 dtn-transfer.net.uconn.edu yum[8524]: Updated: globus-gsi-credential.x86_64 7.11-1.osg33.el7
Sep 14 06:45:30 dtn-transfer.net.uconn.edu yum[8524]: Updated: globus-gsi-proxy-core.x86_64 8.6-1.osg33.el7
Sep 14 06:45:30 dtn-transfer.net.uconn.edu yum[8524]: Updated: globus-gssapi-gsi.x86_64 12.17-3.osg33.el7
Sep 14 06:45:30 dtn-transfer.net.uconn.edu yum[8524]: Updated: globus-gss-assist.x86_64 10.21-1.osg33.el7
Sep 14 06:45:30 dtn-transfer.net.uconn.edu yum[8524]: Updated: globus-xio-gsi-driver.x86_64 3.11-1.osg33.el7
Sep 14 06:45:30 dtn-transfer.net.uconn.edu yum[8524]: Updated: globus-io.x86_64 11.9-1.osg33.el7
Sep 14 06:45:30 dtn-transfer.net.uconn.edu yum[8524]: Updated: globus-ftp-control.x86_64 7.8-1.osg33.el7
Sep 14 06:45:30 dtn-transfer.net.uconn.edu yum[8524]: Updated: globus-ftp-client.x86_64 8.36-1.1.osg33.el7
Sep 14 06:45:30 dtn-transfer.net.uconn.edu yum[8524]: Updated: globus-gass-copy.x86_64 9.27-1.osg33.el7
Sep 14 06:45:30 dtn-transfer.net.uconn.edu yum[8524]: Updated: globus-authz.x86_64 3.15-1.osg33.el7
Sep 14 06:45:30 dtn-transfer.net.uconn.edu yum[8524]: Updated: globus-xio-udt-driver.x86_64 1.28-1.osg33.el7
Sep 14 06:45:30 dtn-transfer.net.uconn.edu yum[8524]: Updated: globus-gridftp-server.x86_64 12.2-1.1.osg33.el7
Sep 14 06:45:31 dtn-transfer.net.uconn.edu yum[8524]: Updated: globus-gridftp-server-progs.x86_64 12.2-1.1.osg33.el7
Sep 14 06:45:31 dtn-transfer.net.uconn.edu yum[8524]: Updated: globus-gass-copy-progs.x86_64 9.27-1.osg33.el7
Sep 14 06:45:31 dtn-transfer.net.uconn.edu yum[8524]: Updated: globus-proxy-utils.x86_64 6.19-1.osg33.el7
Sep 14 06:45:31 dtn-transfer.net.uconn.edu yum[8524]: Updated: globus-xio-pipe-driver.x86_64 3.10-1.osg33.el7
Sep 14 06:45:31 dtn-transfer.net.uconn.edu yum[8524]: Updated: globus-gsi-cert-utils-progs.noarch 9.16-1.osg33.el7
Sep 14 06:45:31 dtn-transfer.net.uconn.edu yum[8524]: Updated: globus-gss-assist-progs.noarch 10.21-1.osg33.el7
Sep 14 06:45:31 dtn-transfer.net.uconn.edu systemd[1]: Stopping LSB: Globus GridFTP Server...
Sep 14 06:45:31 dtn-transfer.net.uconn.edu systemd[1]: globus-gridftp-server.service: control process exited, code=exited status=203
Sep 14 06:45:31 dtn-transfer.net.uconn.edu systemd[1]: Unit globus-gridftp-server.service entered failed state.
Sep 14 06:45:31 dtn-transfer.net.uconn.edu systemd[1]: globus-gridftp-server.service failed.
Sep 14 06:45:31 dtn-transfer.net.uconn.edu systemd[1]: Starting LSB: Globus GridFTP Server...
Sep 14 06:45:31 dtn-transfer.net.uconn.edu systemd[1]: globus-gridftp-server.service: control process exited, code=exited status=203
Sep 14 06:45:31 dtn-transfer.net.uconn.edu systemd[1]: Failed to start LSB: Globus GridFTP Server.
Sep 14 06:45:31 dtn-transfer.net.uconn.edu systemd[1]: Unit globus-gridftp-server.service entered failed state.
Sep 14 06:45:31 dtn-transfer.net.uconn.edu systemd[1]: globus-gridftp-server.service failed.
[root@dtn-transfer ~]#

Yes, one should not be instead using globus-connect-server-setup to start the globus-gridftp-service, but after the packages are updated, that also fails:

[root@dtn-transfer ~]# pkill gridftp
[root@dtn-transfer ~]# globus-connect-server-setup 
Globus Id:  uconn
Password: 
Configured MyProxy server on dtn-transfer.net.uconn.edu:7512
CA DN: /C=US/O=Globus Consortium/OU=Globus Connect Service/CN=920cfc6e-3e02-11e6-80c2-22000b1701d1
Service DN: /C=US/O=Globus Consortium/OU=Globus Connect Service/CN=0b39aad2-9966-11e7-ac63-22000a92523b
Configured GridFTP server to run on dtn-transfer.net.uconn.edu
Server DN: /C=US/O=Globus Consortium/OU=Globus Connect Service/CN=0c655500-9966-11e7-ac63-22000a92523b
Using Authentication Method MyProxy
Configured Endpoint dtn-transfer
[root@dtn-transfer ~]# telnet 127.0.0.1 2811
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
Connection closed by foreign host.
[root@dtn-transfer ~]#

Rolling back the update of 25 packages is a workaround (still trying to track narrow down which package(s) need to be masked):

[root@dtn-transfer ~]# yum history undo 155
...
[root@dtn-transfer ~]# globus-connect-server-setup 
Globus Id:  uconn
Password: 
Configured MyProxy server on dtn-transfer.net.uconn.edu:7512
CA DN: /C=US/O=Globus Consortium/OU=Globus Connect Service/CN=920cfc6e-3e02-11e6-80c2-22000b1701d1
Service DN: /C=US/O=Globus Consortium/OU=Globus Connect Service/CN=46e86cb2-9966-11e7-ac63-22000a92523b
Configured GridFTP server to run on dtn-transfer.net.uconn.edu
Server DN: /C=US/O=Globus Consortium/OU=Globus Connect Service/CN=47c21a16-9966-11e7-ac63-22000a92523b
Using Authentication Method MyProxy
Configured Endpoint dtn-transfer
[root@dtn-transfer ~]# telnet 127.0.0.1 2811
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
220 dtn-transfer.net.uconn.edu GridFTP Server 12.2 (gcc64, 1497977252-85) [Globus Toolkit 6.0.1488563530 GCS-4.0.45] ready.
^]
telnet> quit
Connection closed.
[root@dtn-transfer ~]#

Original transaction:

[root@dtn-transfer ~]# yum history info 155
Loaded plugins: fastestmirror, langpacks, priorities
Transaction ID : 155
Begin time     : Thu Sep 14 06:45:29 2017
Begin rpmdb    : 1067:2d0ab7ac34e0481b1b7d9a64b2cbd78079fa6ead
End time       :            06:45:32 2017 (3 seconds)
End rpmdb      : 1067:ab83de5e00597fadf0dba102a3b94bbc0cfbcb4b
User           : root <root>
Return-Code    : Success
Transaction performed with:
    Installed     rpm-4.11.3-21.el7.x86_64                      @base
    Installed     yum-3.4.3-150.el7.centos.noarch               @base
    Installed     yum-plugin-fastestmirror-1.1.31-40.el7.noarch @base
Packages Altered:
    Updated globus-authz-3.15-1.el7+gt6.x86_64                    @Globus-Toolkit-6-Stable
    Update               3.15-1.osg33.el7.x86_64                  @osg
    Updated globus-ftp-client-8.36-1.el7+gt6.x86_64               @Globus-Toolkit-6-Stable
    Update                    8.36-1.1.osg33.el7.x86_64           @osg
    Updated globus-ftp-control-7.8-1.el7+gt6.x86_64               @Globus-Toolkit-6-Stable
    Update                     7.8-1.osg33.el7.x86_64             @osg
    Updated globus-gass-copy-9.27-1.el7+gt6.x86_64                @Globus-Toolkit-6-Stable
    Update                   9.27-1.osg33.el7.x86_64              @osg
    Updated globus-gass-copy-progs-9.27-1.el7+gt6.x86_64          @Globus-Toolkit-6-Stable
    Update                         9.27-1.osg33.el7.x86_64        @osg
    Updated globus-gridftp-server-12.2-1.el7+gt6.x86_64           @Globus-Toolkit-6-Stable
    Update                        12.2-1.1.osg33.el7.x86_64       @osg
    Updated globus-gridftp-server-progs-12.2-1.el7+gt6.x86_64     @Globus-Toolkit-6-Stable
    Update                              12.2-1.1.osg33.el7.x86_64 @osg
    Updated globus-gsi-callback-5.13-1.el7+gt6.x86_64             @Globus-Toolkit-6-Stable
    Update                      5.13-1.osg33.el7.x86_64           @osg
    Updated globus-gsi-cert-utils-9.16-1.el7+gt6.x86_64           @Globus-Toolkit-6-Stable
    Update                        9.16-1.osg33.el7.x86_64         @osg
    Updated globus-gsi-cert-utils-progs-9.16-1.el7+gt6.x86_64     @Globus-Toolkit-6-Stable
    Update                              9.16-1.osg33.el7.noarch   @osg
    Updated globus-gsi-credential-7.11-1.el7+gt6.x86_64           @Globus-Toolkit-6-Stable
    Update                        7.11-1.osg33.el7.x86_64         @osg
    Updated globus-gsi-openssl-error-3.8-1.el7+gt6.x86_64         @Globus-Toolkit-6-Stable
    Update                           3.8-1.osg33.el7.x86_64       @osg
    Updated globus-gsi-proxy-core-8.6-1.el7+gt6.x86_64            @Globus-Toolkit-6-Stable
    Update                        8.6-1.osg33.el7.x86_64          @osg
    Updated globus-gsi-proxy-ssl-5.10-1.el7+gt6.x86_64            @Globus-Toolkit-6-Stable
    Update                       5.10-1.osg33.el7.x86_64          @osg
    Updated globus-gsi-sysconfig-6.11-1.el7+gt6.x86_64            @Globus-Toolkit-6-Stable
    Update                       6.11-1.osg33.el7.x86_64          @osg
    Updated globus-gss-assist-10.21-1.el7+gt6.x86_64              @Globus-Toolkit-6-Stable
    Update                    10.21-1.osg33.el7.x86_64            @osg
    Updated globus-gss-assist-progs-10.21-1.el7+gt6.x86_64        @Globus-Toolkit-6-Stable
    Update                          10.21-1.osg33.el7.noarch      @osg
    Updated globus-gssapi-gsi-12.17-1.el7+gt6.x86_64              @Globus-Toolkit-6-Stable
    Update                    12.17-3.osg33.el7.x86_64            @osg
    Updated globus-io-11.9-1.el7+gt6.x86_64                       @Globus-Toolkit-6-Stable
    Update            11.9-1.osg33.el7.x86_64                     @osg
    Updated globus-openssl-module-4.8-1.el7+gt6.x86_64            @Globus-Toolkit-6-Stable
    Update                        4.8-1.osg33.el7.x86_64          @osg
    Updated globus-proxy-utils-6.19-1.el7+gt6.x86_64              @Globus-Toolkit-6-Stable
    Update                     6.19-1.osg33.el7.x86_64            @osg
    Updated globus-xio-5.16-1.el7+gt6.x86_64                      @Globus-Toolkit-6-Stable
    Update             5.16-1.osg33.el7.x86_64                    @osg
    Updated globus-xio-gsi-driver-3.11-1.el7+gt6.x86_64           @Globus-Toolkit-6-Stable
    Update                        3.11-1.osg33.el7.x86_64         @osg
    Updated globus-xio-pipe-driver-3.10-1.el7+gt6.x86_64          @Globus-Toolkit-6-Stable
    Update                         3.10-1.osg33.el7.x86_64        @osg
    Updated globus-xio-udt-driver-1.28-1.el7+gt6.x86_64           @Globus-Toolkit-6-Stable
    Update                        1.28-1.osg33.el7.x86_64         @osg
history info
[root@dtn-transfer ~]#
michaellink commented 7 years ago

The updated packages came from the osg repository. Can you try the update again with the osg repository disabled, and confirm that there are no issues with packages from the Globus-Toolkit-6-Stable repository?

To prevent this for now, you can edit the /etc/yum.repos.d/globus-toolkit-6-stable-fedora.repo config to change the 'priority' setting to have a lower value than the osg setting.

Mike

omsai commented 7 years ago

Sorry for my misunderstanding! Indeed the problem was the osg repository, and changing the globus priority I see the osg packages being excluded by priority per yum list updates -d3 | grep 'globus.*osg'