get GridFTP working on comt.sura.org

[rsignell-usgs]

We need to get GridFTP enabled on comt.sura.org.

GridFTP was built for moving large files reliably and securely, and several people used it on Testbed 1 (not sure why the rest didn't -- they frequently beat their heads against the wall when connections timed out before their regular stp sessions had completed).

@brianmckenna , did you make some progress on this?

[rsignell-usgs]

@brianmckenna, I would like to get this resolved before the testbed face-to-face meeting in College Park, Aug 5. I can look into it if you are not able to.

[rluettich]

If the software is not installed or working properly on the SURA server, can we ask Hortensia to do this?

Rick

From: Rich Signell [mailto:notifications@github.com] Sent: Friday, July 18, 2014 8:12 AM To: ioos/comt Subject: Re: [comt] get GridFTP working on comt.sura.org (#6)

@brianmckennahttps://github.com/brianmckenna, I would like to get this resolved before the testbed face-to-face meeting in College Park, Aug 5. I can look into it if you are not able to.

— Reply to this email directly or view it on GitHubhttps://github.com/ioos/comt/issues/6#issuecomment-49423829.

[brianmckenna]

I'll have more time soon to look into this. Are users having issues uploading currently, I don't see much activity. I can certainly bump this up the priority list, but if not going to be used for a bit I'll keep it near the middle.

[rsignell-usgs]

It's certainly not going to be used if it's not installed and working!

I'm going to be presenting to the testbed 2 folks "how to get your data served" at the face-to-face meeting, so I'd to be able to show them how it works. @rluettich, I haven't heard anything from @hortenvaldes lately, perhaps she is on vacation.

[rsignell-usgs]

@brianmckenna and @hortenvaldes , I explored a bit where we were with getting GridFTP going, and I see that Globus Connect Server is already installed on testbed2 (comt.sura.org), and that the configuration file /etc/globus-connect-server.conf has been edited.

But when I login into globus online, I do not see any endpoints called "ioos#testbed2" or "ioos#comt". I still see the old "ioos#testbed" endpoint on the old testbed server (testbedapps.sura.org):

Brian, do you know what remains to be done?

On the old testbed machine, we maintained a file called /etc/grid-security/grid-mapfile that looked like:

[root@testbedapps ~]# more /etc/grid-security/grid-mapfile
"/C=US/O=Globus Consortium/OU=Globus Connect User/CN=ioos" sura_ftp
"/C=US/O=Globus Consortium/OU=Globus Connect User/CN=howard" sura_ftp
"/C=US/O=Globus Consortium/OU=Globus Connect User/CN=rsignell" sura_ftp
"/C=US/O=Globus Consortium/OU=Globus Connect User/CN=drf5n" sura_ftp
"/C=US/O=Globus Consortium/OU=Globus Connect User/CN=adonahue" sura_ftp
"/C=US/O=Globus Consortium/OU=Globus Connect User/CN=restes" sura_ftp
"/C=US/O=Globus Consortium/OU=Globus Connect User/CN=lianyuanz" sura_ftp
"/C=US/O=Globus Consortium/OU=Globus Connect User/CN=pkerr" sura_ftp
"/C=US/O=Globus Consortium/OU=Globus Connect User/CN=acrosby" sura_ftp

is that no longer necessary?

[rsignell-usgs]

@Lizsmith , I'm hoping we get this done before the face-to-face so that I can demo the process by which to transfer files quickly and reliably to the server.

[rsignell-usgs]

Just to save a step, here's the installation instructions for Globus Connect Server: https://support.globus.org/entries/23857088

[brianmckenna]

Spent some time yesterday looking through GridFTP, to be honest, I am unable to follow how testbedapps was originally configured (eg. "ioos#testbed" or even "testbed" is not found anywhere in `/etc/globus-connect-server.conf' and http://toolkit.globus.org/toolkit/docs/latest-stable/gridftp/admin/#gridftp-config-overview leads to confusion as well as none of those files are present, nor are command line configuration options.

@rsignell-usgs had mentioned Globus had helped configure initially, are they still available?

[rsignell-usgs]

I think perhaps the server software configuration is now very different from 5 years ago. We shouldn't worry too much about the old server config - just get the new one going following the new instructions.

[rsignell-usgs]

We could probably get globus folks to help if we needed it. But we should try to follow the docs first. Are we stuck?

[brianmckenna]

Is anaconda (specifically the notebook running at PID 18218) needed on the testbed2 machine?

globus-gridftp-server was attempting to use port 5631 as a default (I have no idea where this port is specified, not seeing it anywhere in docs). Setting the port explicitly allows gridftp to start, but I will have to work through the same process for all of the services needed (myproxy, GRAM5) to make sure they start properly.

[rsignell-usgs]

Yes, we need the notebook server running, but it doesn't need to be on port 5631. I only used that port because it was the one Hortensia (@hortenvaldes) told me to use. Perhaps we need to ping her on e-mail to see if there is another port open or one that could be opened -- I don't see any activity here https://github.com/hortenvaldes?tab=activity to indicate she is tracking this conversation

[brianmckenna]

Latest version has been installed and configured (according to the online docs), we are waiting for two ports

7512 – Globus MyProxy
2811 – Globus GridFTP

to be opened in the firewall so I can configure the Globus Online and test

[hortenvaldes]

All,

I am out of the country with limited access to email so I just saw this email/conversation - I have put a request into the LSU security group to open these two ports for us. Will let you know when it has been done.

Hortensia

---

On 7/23/14 2:33 PM, brianmckenna wrote:

Latest version has been installed and configured (according to the online docs), we are waiting for two ports

7512 – Globus MyProxy 2811 – Globus GridFTP

to be opened in the firewall so I can configure the Globus Online and test

— Reply to this email directly or view it on GitHub https://github.com/ioos/comt/issues/6#issuecomment-49923539.

[rsignell-usgs]

@hortenvaldes, these ports are open, right?

7512 – Globus MyProxy
2811 – Globus GridFTP

[rsignell-usgs]

@hotenvaldes, I'm talking to the globus guy right now and he's telling me we need ports 50000-51000 open for gridftp transfer.

[hortenvaldes]

Yes

Sent from my iPhone

On Sep 23, 2014, at 2:33 PM, Rich Signell notifications@github.com wrote:

@hortenvaldes, these ports are open, right?

7512 – Globus MyProxy 2811 – Globus GridFTP — Reply to this email directly or view it on GitHub.

[hortenvaldes]

Will place the request to the LSU security group

Sent from my iPhone

On Sep 23, 2014, at 2:53 PM, Rich Signell notifications@github.com wrote:

@hotenvaldes, I'm talking to the globus guy right now and he's telling me we need ports 50000-51000 open for gridftp transfer.

— Reply to this email directly or view it on GitHub.

[hortenvaldes]

Rich,

Ports have been opened.

Hortensia

Sent from my iPhone

On Sep 23, 2014, at 2:53 PM, Rich Signell notifications@github.com wrote:

@hotenvaldes, I'm talking to the globus guy right now and he's telling me we need ports 50000-51000 open for gridftp transfer.

— Reply to this email directly or view it on GitHub.

[rsignell-usgs]

Thanks Hortensia!

[rsignell-usgs]

@hortenvaldes , I'm still trying to get globus gridftp going.

when it try to connect, I get:

Unable to connect to comt.sura.org:2811 globus_xio: System error in connect: Connection refused globus_xio: A system call failed: Connection refused (Screenshot at: http://screencast.com/t/cNdT8v1BaL)

I tried to see if port 2811 was indeed open , but didn't see anything running. Does this mean that the port is NOT open, or just that nothing is accessing that port?

[root@testbed2 testbed]# nmap -sT -O localhost

Starting Nmap 5.51 ( http://nmap.org ) at 2014-10-06 09:59 CDT
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00011s latency).
Other addresses for localhost (not scanned): 127.0.0.1
Not shown: 985 closed ports
PORT     STATE SERVICE
21/tcp   open  ftp
22/tcp   open  ssh
25/tcp   open  smtp
80/tcp   open  http
111/tcp  open  rpcbind
443/tcp  open  https
631/tcp  open  ipp
3283/tcp open  netassistant
3306/tcp open  mysql
5989/tcp open  wbem-https
8009/tcp open  ajp13
8010/tcp open  xmpp
8080/tcp open  http-proxy
8081/tcp open  blackice-icecap
8443/tcp open  https-alt
No exact OS matches for host (If you know what OS is running on it, see http://nmap.org/submit/ ).

Should I see something running on 2811?

[hortenvaldes]

Rich,

Nothing is running on port 2811 (lsof -i :2811)

Hortensia

---

On 10/6/14 10:26 AM, Rich Signell wrote:

@hortenvaldes https://github.com/hortenvaldes , I'm still trying to get globus gridftp going.

when it try to connect, I get:

Unable to connect to comt.sura.org:2811 globus_xio: System error in connect: Connection refused globus_xio: A system call failed: Connection refused (Screenshot at: http://screencast.com/t/cNdT8v1BaL)

I tried to see if port 2811 was indeed open , but didn't see anything running. Does this mean that the port is NOT open, or just that nothing is accessing that port?

|When I run nmap on the machine I installed Globus Connect Server on, I see this:

[root@testbed2 testbed]# nmap -sT -O localhost

Starting Nmap 5.51 ( http://nmap.org ) at 2014-10-06 09:59 CDT Nmap scan report for localhost (127.0.0.1) Host is up (0.00011s latency). Other addresses for localhost (not scanned): 127.0.0.1 Not shown: 985 closed ports PORT STATE SERVICE 21/tcp open ftp 22/tcp open ssh 25/tcp open smtp 80/tcp open http 111/tcp open rpcbind 443/tcp open https 631/tcp open ipp 3283/tcp open netassistant 3306/tcp open mysql 5989/tcp open wbem-https 8009/tcp open ajp13 8010/tcp open xmpp 8080/tcp open http-proxy 8081/tcp open blackice-icecap 8443/tcp open https-alt No exact OS matches for host (If you know what OS is running on it, see http://nmap.org/submit/ ). |

Should I see something running on 2811?

— Reply to this email directly or view it on GitHub https://github.com/ioos/comt/issues/6#issuecomment-58034229.

[rsignell-usgs]

But is the port open?

[hortenvaldes]

Yes, the port is open.

On 10/6/14 10:54 AM, Rich Signell wrote:

But is the port open?

— Reply to this email directly or view it on GitHub https://github.com/ioos/comt/issues/6#issuecomment-58038888.

[rsignell-usgs]

Grrr.. Okay, I finally found the problem.
/etc/gridftp.conf was set to a custom port (5632), and I needed to set that back to 2811.

GridFTP is now working on comt.sura.org!