search

firecracker-microvm / firecracker

Secure and fast microVMs for serverless computing.
http://firecracker-microvm.io
Apache License 2.0
26.09k stars 1.81k forks source link

Guest VM can't access internet #1585

Closed xiekeyang closed 4 years ago

xiekeyang commented 4 years ago

What I want:

I setup one micro vm on aws i3.metal, and ssh into the guest vm successfully.

Then I want to connect to internet from guest vm, but it is failed.

Problem:

It is failed when ping google failed

localhost:~# ping www.google.com -c 2
ping: bad address 'www.google.com'

information:

I refer similar issues like #696 and #988 on community, but guest internet is still unreachable. It is reachable by host.

The related env variables are that:

IFACE_ID="eth0"

# Setup TAP device that uses proxy ARP
MASK_LONG="255.255.255.252"

MASK_SHORT="/30"

FC_IP="169.254.0.1"

TAP_IP="169.254.0.2"

FC_MAC="02:FC:00:00:00:00"

network setting:

Set host network according to On The Host

And set guest network according to In The Guest

The network request is:

curl -i -X PUT -H 'Accept: application/json' -H 'Content-Type: application/json' \
--unix-socket /tmp/firecracker-sb0.sock \
-d '{
  "iface_id": "eth0",
  "guest_mac": "02:FC:00:00:00:00",
  "host_dev_name": "fc-tap0"
}' \
http://localhost/network-interfaces/eth0

check tap dev on host:

$ ip addr list

...

fc-tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether c2:6c:f0:0c:b0:df brd ff:ff:ff:ff:ff:ff
    inet 169.254.0.2/30 scope global fc-tap0
       valid_lft forever preferred_lft forever

check ip forward:

$ cat /proc/sys/net/ipv4/ip_forward
1

check network interface on guest:

# ifconfig
eth0      Link encap:Ethernet  HWaddr 02:FC:00:00:00:00  
          inet addr:169.254.0.1  Bcast:169.254.0.3  Mask:255.255.255.252
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:529 errors:0 dropped:0 overruns:0 frame:0
          TX packets:467 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:41332 (40.3 KiB)  TX bytes:40832 (39.8 KiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:24 errors:0 dropped:0 overruns:0 frame:0
          TX packets:24 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1776 (1.7 KiB)  TX bytes:1776 (1.7 KiB)

check ip route on guest:

# ip route list
default via 169.254.0.2 dev eth0 
169.254.0.0/30 dev eth0 proto kernel scope link src 169.254.0.1

I didn't set DNS server on guest (/etc/resolv.conf), which is blank. I've no idea on this, could anyone help to take a look? Thanks a lot!

serban300 commented 4 years ago

What happens if you try to ping the IP address behind www.google.com from the guest ?

ping 172.217.3.164

You cand find the IP address by doing dig www.google.com from the host.

xiekeyang commented 4 years ago

@serban300

ping ip seems fail as well:

localhost:~# ping 172.217.3.164 -c 2
PING 172.217.3.164 (172.217.3.164): 56 data bytes

--- 172.217.3.164 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss
xiekeyang commented 4 years ago

Addition:

guest can connect to host server:

localhost:~# curl http://172.31.59.129:8080 -d "hello world"
Welcome to echo-server!  Here's what I know.
......
hello world

172.31.59.129:8080 is an echo server on host.

serban300 commented 4 years ago

@xiekeyang are you sure you are routing the traffic out of the tap device through the right host network interface ?

Can you provide the output of the ifconfig command on the host ? And the commands that you used in order to route the traffic out of the tap device through the host network interface ?

xiekeyang commented 4 years ago

@serban300 it really because I made mistake on setting iptables (error on host network interface). I fix it to:

iptables -t nat -A POSTROUTING -o $HOST_IFACE -j MASQUERADE
iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i $TAP_DEV -o $HOST_IFACE -j ACCEPT

And now it works to ping internet ip address. Thanks so much.

But how to set the guest DNS server, as to reach internet domain (ping www.google.com)?

host ifconfig info is:

$ ifconfig
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:ea:fe:16:f9  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 172.31.59.129  netmask 255.255.240.0  broadcast 172.31.63.255
        inet6 fe80::426:beff:feb7:3d22  prefixlen 64  scopeid 0x20<link>
        ether 06:26:be:b7:3d:22  txqueuelen 1000  (Ethernet)
        RX packets 1056  bytes 100091 (100.0 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 783  bytes 130974 (130.9 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

fc-tap0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 169.254.0.2  netmask 255.255.255.252  broadcast 0.0.0.0
        ether 8e:2f:90:cb:d9:8a  txqueuelen 1000  (Ethernet)
        RX packets 183  bytes 21263 (21.2 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 242  bytes 22177 (22.1 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 132  bytes 10506 (10.5 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 132  bytes 10506 (10.5 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
xiekeyang commented 4 years ago

guest DNS server seems unwork to follow enp4s0, 172.31.59.1

serban300 commented 4 years ago

I think you will have to set up the DNS servers yourself under /etc/resolv.conf

xiekeyang commented 4 years ago

@serban300 It works now. Thanks so much for your help! 👍

zxianx commented 3 years ago

@xiekeyang i can ping an ip ， config /etc/resolv.conf still cant reach internet domain， meserver 8.8.8.8， How did you solve it

ustiugov commented 3 years ago

@xiekeyang you can set up a nameserver upon a VM creation, like we do in vHive