search

gridcf / gct

Grid Community Toolkit
Apache License 2.0
46 stars 30 forks source link

myproxy FTBFS on IPV6-only build servers #160

Closed ellert closed 3 years ago

ellert commented 3 years ago

Forwarded from: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=992436

Anyone has any idea how to fix this?

fscheiner commented 3 years ago

CCing @jbasney

Strange. Comparing the Debian build log to my log on the OBS (though for MyProxy v6.2.6, but the changes in #146 and #147 don't look like they changed something relevant), it looks to me, like there's just no service listening on the IPv6 localhost address on the Debian build host, because the server seems to listen on the IPv4 localhost address:

[...]
Attempting to connect to ::1:45721 
Attempting to connect to 127.0.0.1:45721 
Successfully connected to localhost:45721
[...]

Maybe myproxy-server (commandline in https://github.com/gridcf/gct/blob/master/myproxy/source/myproxy-test#L269) doesn't listen on IPv6 addresses by default, when the env var MYPROXY_SERVER is configured to localhost only (done in https://github.com/gridcf/gct/blob/master/myproxy/source/myproxy-test#L243). On Ubuntu 20.04 for example the IPv6 localhost address has the following aliases:

Not sure if (1) we just would need to add one of these to MYPROXY_SERVER (@jbasney: Can this be done and what separator is needed here?) or (2) we need to add additional -l [...] arguments to the server command in https://github.com/gridcf/gct/blob/master/myproxy/source/myproxy-test#L269) to solve this.

fscheiner commented 3 years ago

Additonal info:

jbasney commented 3 years ago

MYPROXY_SERVER can be a comma-separated list of hostnames. That tells clients where to connect. https://github.com/gridcf/gct/blob/246f1c3f6b3f0fbff080c1d73c6a60dc418cf183/myproxy/source/myproxy.c#L387

The myproxy-server may only listen on IPv4 by default. https://github.com/gridcf/gct/blob/246f1c3f6b3f0fbff080c1d73c6a60dc418cf183/myproxy/source/myproxy_server.c#L1124

Otherwise, the myproxy-server --listen option can only specify a single hostname. https://github.com/gridcf/gct/blob/246f1c3f6b3f0fbff080c1d73c6a60dc418cf183/myproxy/source/myproxy_server.c#L1017

Since only MyProxy Test 3T fails, I think there's not a problem in general with the clients connecting to the server, but something specific to myproxy_bootstrap_trust() though that uses get_connected_myproxy_host_socket() just like the other connections, so I'm not sure what's causing a 'Connection refused' error only in that case.

Running myproxy-test -verbose may give additional clues.

fscheiner commented 3 years ago

MYPROXY_SERVER can be a comma-separated list of hostnames. That tells clients where to connect.

https://github.com/gridcf/gct/blob/246f1c3f6b3f0fbff080c1d73c6a60dc418cf183/myproxy/source/myproxy.c#L387

What I don't understand is, why the client tries both IPv4 and IPv6 localhost addresses with a single localhost in MYPROXY_SERVER. Aha, because of this in /etc/hosts (checked on Raspberry Pi OS/Raspbian):

127.0.0.1       localhost
::1             localhost ip6-localhost ip6-loopback
[...]

And here is why: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=686970

Here's what other distributions do:

special IPv6 addresses

::1 localhost ipv6-localhost ipv6-loopback


* Ubuntu 18.04.x and 20.04.x use:

127.0.0.1 localhost [...]

The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback



But IMHO I'd have expected `localhost` to resolve to a singular address.

> Otherwise, the `myproxy-server --listen` option can only specify a single hostname.

Can `--listen` then be used mutliple times? Possibly not: https://github.com/gridcf/gct/blob/246f1c3f6b3f0fbff080c1d73c6a60dc418cf183/myproxy/source/myproxy_server.c#L963..L965

> Since only MyProxy Test 3T fails, I think there's not a problem in general with the clients connecting to the server, but something specific to `myproxy_bootstrap_trust()` though that uses `get_connected_myproxy_host_socket()` just like the other connections, so I'm not sure what's causing a 'Connection refused' error only in that case.

Do the other tests try to connect to all `localhost` addresses the same way as the 3T test? If not, then I assume `myproxy-server` just creates IPv4 sockets for all (IPv4) addresses the host has (see https://github.com/gridcf/gct/blob/246f1c3f6b3f0fbff080c1d73c6a60dc418cf183/myproxy/source/myproxy_server.c#L937..L938).
ellert commented 3 years ago

Complete logs are available here: https://buildd.debian.org/status/logs.php?pkg=myproxy&ver=6.2.8-1&arch=amd64 Both attempts say:

MyProxy Tests Complete: 53 tests passed, 3 tests failed

All 3 failed test say: trust root bootstrap failed Unable to connect to ::1: ....... Connection refused

jbasney commented 3 years ago

Can --listen then be used multiple times?

No, but myproxy-server --listen localhost should bind to all localhost addresses just like MYPROXY_SERVER=localhost tries to connect to all localhost addresses. Adding --listen localhost to the myproxy-server command in the myproxy-test script may be a fix for this issue.

Do the other tests try to connect to all localhost addresses the same way as the 3T test?

I think so, which is why I'm struggling to identify why only the 3T test fails. There must be some difference that I'm missing.

ellert commented 3 years ago

Since only MyProxy Test 3T fails, I think there's not a problem in general with the clients connecting to the server, but something specific to myproxy_bootstrap_trust() though that uses get_connected_myproxy_host_socket() just like the other connections, so I'm not sure what's causing a 'Connection refused' error only in that case.

Could the problem be get_connected_myproxy_host_socket()? It tries to split a host:port string on a colon, but maybe it gets confused with an IPv6 host that has lots of colons in it (like e.g. ::1).

I still don't get why only 3 tests fail though...

fscheiner commented 3 years ago

@ellert:

Complete logs are available here: https://buildd.debian.org/status/logs.php?pkg=myproxy&ver=6.2.8-1&arch=amd64 Both attempts say:

MyProxy Tests Complete: 53 tests passed, 3 tests failed

All 3 failed test say: trust root bootstrap failed Unable to connect to ::1: ....... Connection refused

Thanks for clarification, I was too quick and didn't check the whole log but just the first occurance of an error, sorry. :-/ I'll update https://github.com/gridcf/gct/issues/160#issuecomment-901864606 accordingly.

@jbasney: So I assume all tests where a client tries to connect to the server using the address in MYPROXY_SERVER which points to both 127.0.0.1 and ::1 on Debian seem to fail.

Can --listen then be used multiple times?

No, but myproxy-server --listen localhost should bind to all localhost addresses just like MYPROXY_SERVER=localhost tries to connect to all localhost addresses. Adding --listen localhost to the myproxy-server command in the myproxy-test script may be a fix for this issue.

But it should already per default listen on all addresses - at least that's what I understand from https://github.com/gridcf/gct/blob/246f1c3f6b3f0fbff080c1d73c6a60dc418cf183/myproxy/source/myproxy_server.c#L937..L938 - and the myproxy-test script doesn't use --listen nor anything address related in the configuration file.

@ellert

Since only MyProxy Test 3T fails, I think there's not a problem in general with the clients connecting to the server, but something specific to myproxy_bootstrap_trust() though that uses get_connected_myproxy_host_socket() just like the other connections, so I'm not sure what's causing a 'Connection refused' error only in that case.

Could the problem be get_connected_myproxy_host_socket()? It tries to split a host:port string on a colon, but maybe it gets confused with an IPv6 host that has lots of colons in it (like e.g. ::1).

So you assume that the myproxy-server is actually listening on ::1, too?

fscheiner commented 3 years ago

@ellert: But:

Unable to connect to ::1:45721 error:0200206F:system library:connect:Connection refused error:2008A067:BIO routines:BIO_connect:connect error error:0200206F:system library:connect:Connection refused hostname=localhost service=45721 error:20073067:BIO routines:conn_state:connect error error:140E0197:SSL routines:SSL_shutdown:shutdown while in init 
Connection refused

...seems to correctly identify the port used (service=45721).

jbasney commented 3 years ago

There's a subtle difference between myproxy-server --listen localhost and myproxy-server without any --listen option. For better or worse, my thinking at the time when I added IPv6 support was not to enable it by default for fear of breaking things but to require an explicit --listen option for server admins who knew they wanted IPv6 enabled.

Without any --listen option, the myproxy-server binds with AF_INET (and not AF_INET6): https://github.com/gridcf/gct/blob/246f1c3f6b3f0fbff080c1d73c6a60dc418cf183/myproxy/source/myproxy_server.c#L1125

With a --listen option, the myproxy-server binds with AF_UNSPEC so that IPv4 and/or IPv6 will be enabled based on the hostname provided: https://github.com/gridcf/gct/blob/246f1c3f6b3f0fbff080c1d73c6a60dc418cf183/myproxy/source/myproxy_server.c#L1036

I still don't understand why only the 3 "bootstrap" test cases are failing, but I still think changing the myproxy-test script to explicitly pass a myproxy-server --listen localhost option may fix it.

fscheiner commented 3 years ago

@jbasney Sounds good. We're in a VC now, so please don't mind us not replying for some time.

fscheiner commented 3 years ago

Small update: For me make myproxy-check works through w/o an issue on openSUSE Leap 15.2 with the /etc/hosts configuration mentioned in https://github.com/gridcf/gct/issues/160#issuecomment-901926463 (gct at 246f1c3, configured using ./configure --disable-gridftp --disable-gram5 --enable-myproxy):

johndoe@gridftp-5:~/git-projects/gct> make myproxy-check
make[1]: Entering directory '/home/johndoe/git-projects/gct/myproxy/source'
Making check in web
make[2]: Entering directory '/home/johndoe/git-projects/gct/myproxy/source/web'
make[2]: Nothing to be done for 'check'.
make[2]: Leaving directory '/home/johndoe/git-projects/gct/myproxy/source/web'
Making check in systemd
make[2]: Entering directory '/home/johndoe/git-projects/gct/myproxy/source/systemd'
make[2]: Nothing to be done for 'check'.
make[2]: Leaving directory '/home/johndoe/git-projects/gct/myproxy/source/systemd'
Making check in man
make[2]: Entering directory '/home/johndoe/git-projects/gct/myproxy/source/man'
make[2]: Nothing to be done for 'check'.
make[2]: Leaving directory '/home/johndoe/git-projects/gct/myproxy/source/man'
make[2]: Entering directory '/home/johndoe/git-projects/gct/myproxy/source'
make  myproxy-test-wrapper
make[3]: Entering directory '/home/johndoe/git-projects/gct/myproxy/source'
make[3]: Nothing to be done for 'myproxy-test-wrapper'.
make[3]: Leaving directory '/home/johndoe/git-projects/gct/myproxy/source'
make  check-TESTS
make[3]: Entering directory '/home/johndoe/git-projects/gct/myproxy/source'
make[4]: Entering directory '/home/johndoe/git-projects/gct/myproxy/source'
PASS: myproxy-test-wrapper
============================================================================
Testsuite summary for myproxy 6.2.8
============================================================================
# TOTAL: 1
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
make[4]: Leaving directory '/home/johndoe/git-projects/gct/myproxy/source'
make[3]: Leaving directory '/home/johndoe/git-projects/gct/myproxy/source'
make[2]: Leaving directory '/home/johndoe/git-projects/gct/myproxy/source'
make[1]: Leaving directory '/home/johndoe/git-projects/gct/myproxy/source'

johndoe@gridftp-5:~/git-projects/gct> cat myproxy/source/myproxy-test-wrapper.log
[...]
MyProxy Test 3T (retrieve stored credential w/ trustroots): SUCCEEDED
[...]
MyProxy Test 20T (retrieve stored credential w/ trustroots): SUCCEEDED
[...]
MyProxy Test 41 (retrieve trustroots w/o authentication): SUCCEEDED
[...]
MyProxy Tests Complete: 56 tests passed, 0 tests failed
PASS myproxy-test-wrapper (exit status: 0)

But when checking network connections with watch -n 0.1 "netstat -plantu | grep myproxy" I only ever saw myproxy-server listening to 0.0.0.0 which matches INADDR_ANY from https://github.com/gridcf/gct/blob/246f1c3f6b3f0fbff080c1d73c6a60dc418cf183/myproxy/source/myproxy_server.c#L937..L938. Why the clients don't complain for me the same way as on Debian, although localhost points to both 127.0.0.1 and ::1 on that host is the question.

fscheiner commented 3 years ago

Fixed in GCT 6.2.20210826 maintenance release.