Fedora 29: fails to connect

[klmitch]

I'm running an SSH server on a non-standard port and trying to get NetworkManager-ssh to connect to the machine, but I'm not even seeing a connection come in on the server. I tried running nm-ssh-service --debug as root in a terminal and triggering the VPN, but I see no messages. The only possible hint I can see in /var/log/messages is an AVC denial, but I get that even if I configure to use the default port, and the SE troubleshooter isn't even showing a denial. Here's the output:

Feb  3 11:24:45 bernoulli NetworkManager[1273]: <info>  [1549214685.2123] audit: op="connection-activate" uuid="214e6fc4-08f3-4707-995e-c875a0cdde82" name="KevNet" pid=27897 uid=13381 result="success"
Feb  3 11:24:45 bernoulli NetworkManager[1273]: <info>  [1549214685.2345] vpn-connection[0x564ec21ca310,214e6fc4-08f3-4707-995e-c875a0cdde82,"KevNet",0]: Started the VPN service, PID 29305
Feb  3 11:24:45 bernoulli audit[1128]: USER_AVC pid=1128 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { acquire_svc } for service=org.freedesktop.NetworkManager.ssh.Connection_20 spid=29305 scontext=system_u:system_r:NetworkManager_ssh_t:s0 tcontext=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 tclass=dbus permissive=0#012 exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'
Feb  3 11:24:45 bernoulli NetworkManager[1273]: <info>  [1549214685.2123] audit: op="connection-activate" uuid="214e6fc4-08f3-4707-995e-c875a0cdde82" name="KevNet" pid=27897 uid=13381 result="success"
Feb  3 11:24:45 bernoulli NetworkManager[1273]: <info>  [1549214685.2345] vpn-connection[0x564ec21ca310,214e6fc4-08f3-4707-995e-c875a0cdde82,"KevNet",0]: Started the VPN service, PID 29305
Feb  3 11:24:45 bernoulli audit[1128]: USER_AVC pid=1128 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { acquire_svc } for service=org.freedesktop.NetworkManager.ssh.Connection_20 spid=29305 scontext=system_u:system_r:NetworkManager_ssh_t:s0 tcontext=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 tclass=dbus permissive=0
 exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'
Feb  3 11:24:50 bernoulli NetworkManager[1273]: <warn>  [1549214690.2336] vpn-connection[0x564ec21ca310,214e6fc4-08f3-4707-995e-c875a0cdde82,"KevNet",0]: Timed out waiting for the service to start
Feb  3 11:24:50 bernoulli NetworkManager[1273]: <warn>  [1549214690.2336] vpn-connection[0x564ec21ca310,214e6fc4-08f3-4707-995e-c875a0cdde82,"KevNet",0]: Timed out waiting for the service to start

I've been monitoring the /var/log/secure on the target host, and don't even see a connection. I've also tried other hosts, and see the same behavior: an eventual timeout waiting for the service to start. Any ideas?

[danfruehauf]

Was about to open a bug about that in the Fedora bugzilla. If you really want it to work, you could temporarily disable selinux and connect, then re-enable it again. I haven't got to the bottom of things, but it will be solved soon, I hope.

[danfruehauf]

Attaching the Fedora bugzilla bug: https://bugzilla.redhat.com/show_bug.cgi?id=1677484

[klmitch]

Yeah, after I logged this bug, I spent a lot of time fiddling with selinux settings. In the end, I think I cleared all the selinux problems, but still couldn't get the SSH VPN to work for some reason. Afraid it's been too long since I worked on that to include useful details for the selinux side either here or on the Fedora bug, but there were a lot of policy bits involved :/ If I get any time soon and happen to remember it, I'll try to reproduce that debugging and attach selinux details to the Fedora bug.

(It's probably worth noting that I run my SSH servers on a non-standard port for a little added obscurity and to avoid filling my logs with bots; I'm wondering if that could be related to why I couldn't get the SSH VPN to work after clearing up the selinux issues…)

[danfruehauf]

(It's probably worth noting that I run my SSH servers on a non-standard port for a little added obscurity and to avoid filling my logs with bots; I'm wondering if that could be related to why I couldn't get the SSH VPN to work after clearing up the selinux issues…)

I always do that too. And also against my test server, should work absolutely fine.

Yeah, after I logged this bug, I spent a lot of time fiddling with selinux settings. In the end, I think I cleared all the selinux problems, but still couldn't get the SSH VPN to work for some reason. Afraid it's been too long since I worked on that to include useful details for the selinux side either here or on the Fedora bug, but there were a lot of policy bits involved :/ If I get any time soon and happen to remember it, I'll try to reproduce that debugging and attach selinux details to the Fedora bug.

For that, I'll let the selinux experts of Fedora fix it. I'm very far from being even a selinux beginner. The trouble is it used to work in F27, so some introduced policy broke it. Hence, it should be fixed by the policy maintainer.

As for debugging, looking at /var/log/messages should give you more information about understanding how SSH connects behind the scenes. The remote host needs to be in your .ssh/known_hosts to avoid it from prompting. You can then also see how your ssh-agent socket is being probed (the connection itself runs as root, but the ssh-agent that you run as your local user is being used).

I hope this can get you going.

[klmitch]

Well, I already have the remote hosts I tried in .ssh/known_hosts and watched /var/log/messages; some of the selinux problems seemed to be related to the ssh-agent, but I believe I cleared those, and I still wasn't making any additional headway…

[danfruehauf]

but I believe I cleared those, and I still wasn't making any additional headway…

And with setenforce 0, can you connect?

[danfruehauf]

Looks like an update was pushed to f29. I'll give it a go soon, and if it works - close this one...

[danfruehauf]

See also: https://bodhi.fedoraproject.org/updates/FEDORA-2019-38a1de7619

[danfruehauf]

Still doesn't work, the latest update is that I get that:

May 20 11:11:00 localhost NetworkManager[1107]: <info>  [1558314660.5685] audit: op="connection-activate" uuid="d0a1a843-98ad-41dc-831b-7a8139771a8e" name="tfx-jump ssh" pid=2351 uid=1000 result="success"
May 20 11:11:00 localhost NetworkManager[1107]: <info>  [1558314660.5718] vpn-connection[0x563d7e8f8350,d0a1a843-98ad-41dc-831b-7a8139771a8e,"tfx-jump ssh",0]: Started the VPN service, PID 2876
May 20 11:11:00 localhost NetworkManager[1107]: <info>  [1558314660.5833] vpn-connection[0x563d7e8f8350,d0a1a843-98ad-41dc-831b-7a8139771a8e,"tfx-jump ssh",0]: Saw the service appear; activating connection
May 20 11:11:00 localhost audit[960]: USER_AVC pid=960 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=method_call interface=org.freedesktop.NetworkManager.VPN.Plugin member=NeedSecrets dest=:1.392 spid=1107 tpid=2876 scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:system_r:NetworkManager_ssh_t:s0 tclass=dbus permissive=0#012 exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'
May 20 11:11:00 localhost NetworkManager[1107]: <error> [1558314660.5859] vpn-connection[0x563d7e8f8350,d0a1a843-98ad-41dc-831b-7a8139771a8e,"tfx-jump ssh",0]: plugin NeedSecrets request #1 failed: An SELinux policy prevents this sender from sending this message to this recipient, 0 matched rules; type="method_call", sender=":1.16" (uid=0 pid=1107 comm="/usr/sbin/NetworkManager --no-daemon " label="system_u:system_r:NetworkManager_t:s0") interface="org.freedesktop.NetworkManager.VPN.Plugin" member="NeedSecrets" error name="(unset)" requested_reply="0" destination=":1.392" (uid=0 pid=2876 comm="/usr/libexec/nm-ssh-service --bus-name org.freedes" label="system_u:system_r:NetworkManager_ssh_t:s0")
May 20 11:11:00 localhost audit[960]: USER_AVC pid=960 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=method_call interface=org.freedesktop.NetworkManager.VPN.Plugin member=Disconnect dest=:1.392 spid=1107 tpid=2876 scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:system_r:NetworkManager_ssh_t:s0 tclass=dbus permissive=0#012 exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'