Document requirements to support EPSV for multi-host Globus GridFTP servers

[fscheiner]

https://github.com/gridcf/UberFTP/commit/1f28d62f2e688d0a4bfc89a754dfdf3424ef9ed7 enables IPv6 compatibility for UberFTP but also introduced an "incompatibility" with Globus GridFTP servers in split-process configuration (i.e. separate processes for PI and DTP) where the DTP is located on a different host than the PI (also true for multiple DTPs and PIs).

This is due to the use of the EPSV command (see RFC 2428 for details), specifically because of:

[...] The EPSV command requests that a server listen on a data port and wait for a connection. The EPSV command takes an optional argument. The response to this command includes only the TCP port number of the listening connection. [...]

Example log for a directory listing tried on a Globus GridFTP server in split-process configuration with DTP(s) on different hosts than the PI:

$ uberftp -debug 2 -ls gsiftp://multi-host-gridftp.domain.tld/~/
[...]
220 multi-host-gridftp.domain.tld GridFTP Server 13.20 (gcc64, 1566483868-0) [Grid Community Toolkit 6.2] ready.
230 User user logged in.
---> Sending to multi-host-gridftp.domain.tld:
MLST ~
250-status of ~

Type=dir;Modify=20200723135421;Size=32;Perm=cfmpel;UNIX.mode=0700;UNIX.uid=1000;UNIX.gid=1000;Unique=fd02-10000266; ~
250 End.
---> Sending to multi-host-gridftp.domain.tld:
MODE S
200 Mode set to S.
---> Sending to multi-host-gridftp.domain.tld:
DCAU A
200 DCAU A.
---> Sending to multi-host-gridftp.domain.tld:
PBSZ 1135616
200 PBSZ=1135616
---> Sending to multi-host-gridftp.domain.tld:
PROT C
200 Protection level set to C.
---> Sending to multi-host-gridftp.domain.tld:
TYPE I
200 Type set to I.
---> Sending to multi-host-gridftp.domain.tld:
EPSV
229 Entering Passive Mode (|||22848|)
---> Sending to multi-host-gridftp.domain.tld:
LIST ~/
<CONNECTION HANGS FROM HERE ON>

Providing only the TCP port number of the listening connection can't work if the DTP is not located on the same host as the PI. Without also providing the IP address of the corresponding host it can only work for Globus GridFTP servers where DTP(s) and PI are located on a singular host, but then even in split-process configuration. Example log for such a configuration:

$ uberftp -debug 2 -ls gsiftp://single-host-gridftp.domain.tld/~/
[...]
220 single-host-gridftp.domain.tld GridFTP Server 13.20 (gcc64, 1566483868-0) [Grid Community Toolkit 6.2] ready.
230 User user logged in.
---> Sending to single-host-gridftp.domain.tld:
MLST ~
250-status of ~

Type=dir;Modify=20200730130830;Size=4096;Perm=cfmpel;UNIX.mode=0700;UNIX.owner=user;UNIX.uid=501;UNIX.group=user;UNIX.gid=501;Unique=fd00-230a4; ~
250 End.
---> Sending to single-host-gridftp.domain.tld:
MODE S
200 Mode set to S.
---> Sending to single-host-gridftp.domain.tld:
DCAU A
200 DCAU A.
---> Sending to single-host-gridftp.domain.tld:
PBSZ 1135616
200 PBSZ=1135616
---> Sending to single-host-gridftp.domain.tld:
PROT C
200 Protection level set to C.
---> Sending to single-host-gridftp.domain.tld:
TYPE I
200 Type set to I.
---> Sending to single-host-gridftp.domain.tld:
EPSV
229 Entering Passive Mode (|||23653|)
---> Sending to single-host-gridftp.domain.tld:
LIST ~/
drwx------   4     user     user           32 Jul 23 15:54 .
drwxr-xr-x   7     root     root           99 Jul 23 15:53 ..
-rw-r--r--   1     user     user            0 Jul 23 15:54 test1
-rw-r--r--   1     user     user            0 Jul 23 15:54 test2
150 Beginning transfer.
226 Transfer Complete.
---> Sending to single-host-gridftp.domain.tld:
QUIT
221 Goodbye.
221 Goodbye.

This problem can be solved by activating epsv_ip for a multi-host Globus GridFTP server. I also recommend to activate epsv_match at the same time so the address family for the DTP's address in the response to EPSV is in sync to what was used to make the connection to the PI (although I now assume this is only relevant for dual-stack servers).

This PR documents that new requirement for Globus GridFTP servers to continue working with the (upcoming) UberFTP 2.9. I want to document it also here in the README.md for UberFTP, because the described behaviour is new and might otherwise irritate users and lead them to false conclusions about the functionality of the new UberFTP version.

[fscheiner]

What do you think @msalle?

Also putting @matyasselmeci and @ellert in CC as they're not watching this repo.

[msalle]

Hi @fscheiner the layout is a bit odd, it now looks like you're quoting someone? I also think for a README we definitely should make the text quite a bit shorter. The crucial point is that with the new default EPSV (as required to support IPv6) it's necessary to have the service configured to enable both epsv_ip 1 and epsv_match 1 iff the server is split over multiple hosts. Not sure we need any more information here.

[fscheiner]

Hi @fscheiner the layout is a bit odd, it now looks like you're quoting someone?

I use the "quote" to visually emphasize the notice for a user (like in https://github.com/fr4nk5ch31n3r/gtransfer/blob/master/README.md).

I also think for a README we definitely should make the text quite a bit shorter. The crucial point is that with the new default EPSV (as required to support IPv6) it's necessary to have the service configured to enable both epsv_ip 1 and epsv_match 1 iff the server is split over multiple hosts. Not sure we need any more information here.

Ok, then maybe just the part above tl;dr with some added information about the use of EPSV in UberFTP. I'll force-push an update. The - then removed - longer text could be a candidate for the GCT or the GCT documentation.

[fscheiner]

@msalle What do you think of the new version of the EPSV compatibility notice? If the quote markup is still a problem, I can remove that, but for visual reasons I'd actually like to keep it.

A rendered version of the changed README.md can be seen at https://github.com/fscheiner/UberFTP/tree/document-epsv-requirements-for-multi-host-gridftp-servers.

[fscheiner]

@msalle

I still find the quote-style a bit confusing but other than that looks good to me.

~Even when rendered or especially when rendered?~

~My PR #10 will add a short description beforehand, which might change the overall look (again).~

Ah, ok, I overlooked that you already approved the PR.

I'll rebase my two PRs, when your changes in PR #8 have been merged.