[BUG] Unable to connect to CRC remotely on F33 with systemd-resolved.

[dofinn]

Regarding: https://code-ready.github.io/crc/#connecting-to-remote-instance_gsg

Is it possible to set this up on F33 using systemd-resolved?

I have to managed to create a dummy interface

sudo ip link add type dummy crc
sudo ip link set dummy0 name crc
sudo ip link set crc up

and assign the following to it

sudo resolvectl domain crc '~crc.testing'
sudo resolvectl dns crc 192.168.0.70
resolvectl status crc                                                                                                                                    ✔  4m 15s  66  
Link 21 (crc)
Current Scopes: none                                                        
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
   DNS Servers: 192.168.0.70                                                
    DNS Domain: ~crc.testing   

However, I am still unable to resolve crc.testing to my server on my home network.

[dofinn]

Managed to get it somewhat working with

cat /etc/systemd/resolved.conf.d/crc.conf                                                                                                                                                          ✔  67  
[Resolve]
DNS=127.0.0.1
Domains=~apps-crc.testing ~crc.testing

+

grep 'listen-add\|crc' /etc/dnsmasq.conf                                                                                                                                                           ✔  67  
address=/apps-crc.testing/192.168.0.70
address=/api.crc.testing/192.168.0.70
address=/crc.testing/192.168.0.70
listen-address=127.0.0.1

So still requiring dnsmasq accept its running under its own systemd service instead of a network manager plugin because i dont want to tunnel my entire desktops dns via dnsmasq

[cfergeau]

My understanding is that the 'client' machine (the one not running crc) is running fedora 33, and the 'server' machine (the remote machine running crc) is running fedora 32. What you need on the client machine is to use 192.168.0.70 as the DNS server for ~testing queries on the interface connected to your local network (the one which can access 192.168.0.70). I'd try to configure this directly through nmcli or some NetworkManager UI rather than using a dispatcher file. It definitely should not be using a crc interface, unless this interface is connected to 192.168.0.70

[dofinn]

Hi @cfergeau . The above works fine for my use case. I am able to resolve my remote crc cluster in home nework regardless of if im on wifi or ethernet. I wanted an abstraction above the interfaces. Thanks.

[praveenkumar]

@dofinn Can we close this issue or mark it as documentation side?

[cfergeau]

Yes this is a documentation issue.

[kowen-rh]

How should we document this? What would we consider the "recommended" approach in general?

[cfergeau]

I would favour what I vaguely described in https://github.com/code-ready/crc/issues/1954#issuecomment-773869575 (I'd nede to configure this myself to be able to describe it more clearly). Not sure how workable this would be, especially as this has the limitation @dofinn pointed out https://github.com/code-ready/crc/issues/1954#issuecomment-774904308 Maybe something system-wide can be done in systemd-resolve (https://www.freedesktop.org/software/systemd/man/resolved.conf.html# ) or NetworkManager, but we have to be careful not to send all DNS requests to the crc VM imo.

[dofinn]

So i would be in favour of documenting https://github.com/code-ready/crc/issues/1954#issuecomment-773813647 for F33 only. This sends traffic only destined for the crc.testing domain to your remote machine that hosts CRC.

The complete solution is those configurations + sudo systemctl enable --now dnsmasq

[kowen-rh]

I assume that this isn't related specifically to Fedora 33, but more generally to using systemd-resolved -- is that correct? Could we expect to see the same problem in other distributions with a similar networking setup?

[cfergeau]

Yes, that's correct, all distributions using systemd-resolved would have issues with the remote setup currently described in the documentation

[kowen-rh]

Okay, good, we can have separate docs for each networking setup in that case. Thank you!

Would anyone mind helping me out with the actual contents of the configuration file for systemd-resolved in that case?

[dofinn]

$ cat /etc/systemd/resolved.conf.d/crc.conf                                                                                                                                                
[Resolve]
DNS=127.0.0.1
Domains=~apps-crc.testing ~crc.testing
$ cat /etc/systemd/resolved.conf                                                                                                                                                                
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See resolved.conf(5) for details

[Resolve]
#DNS=
#FallbackDNS=
#Domains=
#LLMNR=no
#MulticastDNS=no
#DNSSEC=no
#DNSOverTLS=no
#Cache=yes
#DNSStubListener=yes

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[kowen-rh]

Working on documenting this now.

Forgive my ignorance of the topic, but I'm uncertain about the discrete steps we need to document in order for this procedure to work. I could use some more specificity in each step simply because I don't fully understand how/why these configurations work as described here. :)

From what I can see here, it looks like we're still ultimately accessing the remote with dnsmasq, correct? Are these additional steps required in addition to the currently-documented procedure, or do we need another procedure altogether that has different steps?

[cfergeau]

I managed to configure something vaguely working by following the suggestions from this issue: On the client/local machine:

$ cat /etc/systemd/resolved.conf.d/crc.conf                                                                                                                                                
[Resolve]
DNS=127.0.0.1
Domains=~apps-crc.testing ~crc.testing

$ cat /etc/dnsmasq.conf 
user=dnsmasq
group=dnsmasq
interface=lo
bind-interfaces
listen-address=127.0.0.1
address=/apps-crc.testing/192.168.1.126
address=/api.crc.testing/192.168.1.126
address=/crc.testing/192.168.1.126

$ systemctl start dnsmasq

The remote machine is configured using haproxy as in the current documentation.

After this, dig foo.crc.testing returns 192.168.1.126, but name resolution for *.testing seems really slow in some cases (resolvectl query, curl -I, ...), and accessing the webconsole from firefox takes ages even ifit eventually works.

[themr0c]

Is this issue still relevant? cc: @jsliacan

[fgharo]

@dofinn What exact steps did you use to get the client to be able to access the server? I am having the same problem and was going to post a new issue but saw this issue was already open. I have a client machine that can't seem to access the crc/Openshift instance running on a server machine. I know its been awhile so if you happen to remember I would be grateful. I tried carrying out your suggestions in this thread in various ways to no success. I take that the dummy interface was one approach distinct from the service unit approach? Did you run each of those approaches in addition to the client setup talked about in the article (https://crc.dev/crc/#connecting-to-remote-instance_gsg)?

If I login to the server host via ssh, and then access either by cli or web console works fine. Accessing from a second client machine with Fedora 32 works fine too. So the problem seems to occur with clients of Fedora 33 or higher. I would rather get this working on the client that is running Fedora 36 where is where I am running into the problem.

@cfergeau I tried your steps as well with the /etc/systemd/resolved.conf.d/crc.conf file but as you had already noted, it is super slow. Accessing via the terminal with oc just hangs there until timeout, and when I try to access the web console via the browser it is super slow as well probably like 5 or more minutes for login to finally complete.

$ oc login -u developer https://api.crc.testing:6443
error: dial tcp: lookup api.crc.testing: i/o timeout - verify you have provided the correct host and port and that the server is currently running.

And this is the web page message I got from the client's browser after trying to get to address https://console-openshift-console.apps-crc.testing:

This site can’t be reached oauth-openshift.apps-crc.testing’s server IP address could not be found.

Here are further details about my client / server setup:

Server details:

$ cat /etc/os-release 
NAME=Fedora
VERSION="34 (Workstation Edition)"
...

$ crc version
WARN A new version (2.20.0) has been published on https://developers.redhat.com/content-gateway/file/pub/openshift-v4/clients/crc/2.20.0/crc-linux-amd64.tar.xz 
CRC version: 2.9.0+9591a8f
OpenShift version: 4.11.3
Podman version: 4.2.0

I setup the server according to this section https://crc.dev/crc/#setting-up-remote-server_gsg

Client details:

$ cat /etc/os-release 
NAME="Fedora Linux"
VERSION="36 (Workstation Edition)"
...
$ oc version
Client Version: 4.13.0
Kustomize Version: v4.5.7

I setup the client according to this section https://crc.dev/crc/#connecting-to-remote-instance_gsg

I can ssh from the client to the server just fine and I can ping that server host from the client just fine, so I don't think it has anything to do with the way the network configuration is setup on the server. Most likely it is the way the network configuration on the client is setup.

[fgharo]

@dofinn @cfergeau Nevermind! I came across this crc issue and in that thread copied jotak's answer which worked for me. I seem to be able to login via web console and terminal.