search

crc-org / crc

CRC is a tool to help you run containers. It manages a local OpenShift 4.x cluster, Microshift or a Podman VM optimized for testing and development purposes
https://crc.dev
Apache License 2.0
1.25k stars 236 forks source link

[BUG] Failed to query DNS from host: Invalid IP for api.crc.testing #1309

Closed pottacode closed 4 years ago

pottacode commented 4 years ago

General information

CRC version

crc version: 1.11.0+883ca49
OpenShift version: 4.4.5 (embedded in binary)

CRC status

crc status --log-level debug
DEBU Running '/Users/pottavia/.crc/bin/oc/oc get co -ojson --kubeconfig /Users/pottavia/.crc/machines/crc/kubeconfig --context admin --cluster crc' 
DEBU Command failed: exit status 1                
DEBU stdout:                                      
DEBU stderr: error: stat /Users/pottavia/.crc/machines/crc/kubeconfig: no such file or directory 
DEBU error: stat /Users/pottavia/.crc/machines/crc/kubeconfig: no such file or directory 
ERRO stat /Users/pottavia/.crc/machines/crc/id_rsa: no such file or directory

CRC config

N/A

Host Operating System

sw_vers
ProductName:    Mac OS X
ProductVersion: 10.15.5
BuildVersion:   19F101

Steps to reproduce

  1. crc setup
  2. crc start -p ./pull-secret --log-level debug

Expected

Actual

INFO Check DNS query from host ...
DEBU api.crc.testing resolved to [40.68.249.35]
WARN api.crc.testing resolved to [40.68.249.35] but 192.168.64.2 was expected ERRO Failed to query DNS from host: Invalid IP for api.crc.testing

Logs

Before gather the logs try following if that fix your issue

$ crc delete -f
$ crc cleanup
$ crc setup
$ crc start --log-level debug

Please consider posting the output of crc start --log-level debug on http://gist.github.com/ and post the link in the issue.

https://gist.github.com/pottacode/2f42bb5128072647b0f8216a06c1f2cb

gbraad commented 4 years ago

Please verify your DNS settings;

WARN api.crc.testing resolved to [40.68.249.35] but 192.168.64.2 was expected

a result is given that does not match. Can you verify if you are using a similar service as reported in: https://github.com/code-ready/crc/issues/1300#issuecomment-642383480. If so, please override your nameserver on start or disable this 'service'.

The address resolves to pages owned/operated by Windtre (3) in italy. This is likely a DNS interception/snooping to provide a service for mistyped or unresolvable addresses. Often used to collect additional advertisement money.

ron1 commented 4 years ago

Note that overriding the nameserver with the crc flag does not correct the problem for me. I had to explicitly change the dns server on my router.

pottacode commented 4 years ago

I replaced the DNS provided by my ISP by overriding with 1.1.1.1 and 1.0.0.1 in the router:

image

I am now facing a different error:

INFO Check DNS query from host ...
ERRO Failed to query DNS from host: lookup api.crc.testing on 192.168.178.1:53: no such host Making call to close driver server (crc) Calling .Close Successfully made call to close driver server Making call to close connection to plugin binary Making call to close driver server (crc) Calling .Close Successfully made call to close driver server Making call to close connection to plugin binary

Following is the content of resolv.conf file:

domain fritz.box
nameserver 192.168.178.1
gbraad commented 4 years ago

did you perform:

crc delete -f
crc cleanup
crc setup
crc start

to make sure this is a clean run. the resolving for the testing domain should be handled by the resolver, so not sure why it touches the router for this.

pottacode commented 4 years ago

I have executed the above list of suggested commands and after a new run I am still getting the following error:

INFO Check DNS query from host ...                
ERRO Failed to query DNS from host: lookup api.crc.testing on 192.168.178.1:53: no such host 
Making call to close driver server
(crc) Calling .Close
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(crc) Calling .Close
Successfully made call to close driver server
Making call to close connection to plugin binary

cat /etc/resolver/testing 

port 53
domain testing
nameserver 192.168.64.2
search_order 1%

scutil --dns

DNS configuration

resolver #1
  search domain[0] : fritz.box
  nameserver[0] : 192.168.178.1
  if_index : 6 (en0)
  flags    : Request A records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

resolver #2
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300000

resolver #3
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300200

resolver #4
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300400

resolver #5
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300600

resolver #6
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300800

resolver #7
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 301000

resolver #8
  domain   : testing
  nameserver[0] : 192.168.64.2
  port     : 53
  flags    : Request A records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)
  order    : 1

DNS configuration (for scoped queries)

resolver #1
  search domain[0] : fritz.box
  nameserver[0] : 192.168.178.1
  if_index : 6 (en0)
  flags    : Scoped, Request A records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)
gbraad commented 4 years ago

@praveenkumar @cfergeau Any thoughts?

Seems the resolver prioritizes resolver #1 also for the .testing domain?

praveenkumar commented 4 years ago

@pottacode This looks like the resolver #8 which have details about the testing domain is not resolve, can you just restart your network manually ( like if you are on wireless or wired just disconnect and connect?), Also with this release we did add a fallback path using goodhosts which should put the VM ip to /etc/hosts against some of the domains. Can you please share the /etc/hosts file content ?

pottacode commented 4 years ago

Here is the content of /etc/hosts 

##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting. Do not change this entry.
##
127.0.0.1 localhost
255.255.255.255 broadcasthost
::1 localhost

# Added by Docker Desktop
# To allow the same kube context to work on the host and the container:
127.0.0.1 kubernetes.docker.internal
# End of section
192.168.64.2 api.crc.testing oauth-openshift.apps-crc.testing console-openshift-console.apps-crc.testing default-route-openshift-image-registry.apps-crc.testing
praveenkumar commented 4 years ago

@pottacode The content in the /etc/hosts looks as expected I am not sure why then it query to external dns to resolve the api.crc.testing when it can find it to /etc/hosts. Can you try to ping api.crc.testing and see if that able to resolve to 192.168.64.2 correctly?

pottacode commented 4 years ago

@praveenkumar attached you see the log with the failure (just replicated) and the ping as per your request:

INFO Check DNS query from host ...                
ERRO Failed to query DNS from host: lookup api.crc.testing on 192.168.178.1:53: no such host 
Making call to close driver server
(crc) Calling .Close
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(crc) Calling .Close
Successfully made call to close driver server
Making call to close connection to plugin binary
pottavia@pottavia-MBP Downloads % ping api.crc.testing
PING api.crc.testing (192.168.64.2): 56 data bytes
64 bytes from 192.168.64.2: icmp_seq=0 ttl=64 time=0.154 ms
64 bytes from 192.168.64.2: icmp_seq=1 ttl=64 time=0.342 ms
64 bytes from 192.168.64.2: icmp_seq=2 ttl=64 time=0.207 ms
^C
--- api.crc.testing ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.154/0.234/0.342/0.079 ms
praveenkumar commented 4 years ago

@pottacode Can you please try following and let me know the output of this program?

$ cat testhost.go 
package main

import (
    "fmt"
    "io/ioutil"
    "net"
    "runtime"
)

func main() {
    fmt.Println("GOOS:", runtime.GOOS)
    p, err := ioutil.ReadFile("/etc/hosts")
    fmt.Println("# /etc/hosts:")
    fmt.Println(string(p))

    addrs, err := net.LookupIP("api.crc.testing")
    fmt.Println("net.LookupHost addrs:", addrs, "err:", err)

    net.DefaultResolver.PreferGo = true

    addrs, err = net.LookupIP("api.crc.testing")
    fmt.Println("net.LookupHost addrs:", addrs, "err:", err)
}

$ go run testhost.go

Also I attached cross compiled binary for same, do try to run it also and let me know the output. testhost.zip

pottacode commented 4 years ago

@praveenkumar here is the output generated by testhost:

go run testhost.go 
GOOS: darwin
# /etc/hosts:

net.LookupHost addrs: [192.168.64.2] err: <nil>
net.LookupHost addrs: [] err: lookup api.crc.testing on 192.168.178.1:53: no such host

As I noticed non content listed under #/etc/hosts: section, I verified I had the following permissions:

ls -la /etc/hosts
-rw-------  1 root  wheel  515 Jun 17 12:47 /etc/hosts

I modified in:

ls -la /etc/hosts        
-rw-r--r--  1 root  wheel  515 Jun 18 11:09 /etc/hosts

and now when running: crc start --log-level debug I got:

INFO Check DNS query from host ...                
DEBU api.crc.testing resolved to [192.168.64.2]

CRC started !!!

psrin7 commented 4 years ago

@pottacode I have the same issue. May I know what did you edit. Did you create your /etc/hosts file?

$ sudo go run readFile.go
GOOS: darwin
# /etc/hosts:
127.0.0.1 localhost
255.255.255.255 broadcasthost
::1 localhost
192.168.64.12 api.crc.testing oauth-openshift.apps-crc.testing console-openshift-console.apps-crc.testing default-route-openshift-image-registry.apps-crc.testing
net.LookupHost addrs: [192.168.64.12] err: <nil>
net.LookupHost addrs: [192.168.64.12] err: <nil>
$ nslookup api.crc.testing
Server:     192.168.1.1
Address:    192.168.1.1#53

** server can't find api.crc.testing: NXDOMAIN

$ scutil --dns
DNS configuration

resolver #1
  nameserver[0] : 192.168.1.1
  if_index : 10 (en0)
  flags    : Request A records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

resolver #2
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300000

resolver #3
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300200

resolver #4
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300400

resolver #5
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300600

resolver #6
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300800

resolver #7
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 301000

resolver #8
  domain   : testing
  nameserver[0] : 192.168.64.12
  port     : 53
  flags    : Request A records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)
  order    : 1

DNS configuration (for scoped queries)

resolver #1
  nameserver[0] : 192.168.1.1
  if_index : 10 (en0)
  flags    : Scoped, Request A records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

$ cat /etc/resolver/testing
port 53
domain testing
nameserver 192.168.64.12
codingFonnegra commented 4 years ago

@psrin7 I think @pottacode just changed the file permissions for /etc/hosts

sudo chmod 644 /etc/hosts

then it looks like this

ls -l /etc/hosts -rw-r--r-- 1 root wheel 930 Jun 25 11:47 /etc/hosts

I had the same problem and changing the pemissions fixed it.

cfergeau commented 4 years ago

/etc/hosts permissions are going to be fixed by https://github.com/code-ready/crc/pull/1350