Open tsibley opened 3 years ago
tsibley
commented
3 years ago @Saviq Oh! Is this not expected to work? (re: your bug → enhancement label switch)
I assumed the .multipass domain was intended to work on the host too, but maybe that was never the intent? If not, then this would be a nice enhancement, and it was surprisingly easy (at least with systemd-resolved).
Saviq
commented
3 years ago @tsibley indeed, not, it's not implemented yet. Sure, it's trivial with systemd-resolved, but what about other platforms? What about Linux hosts that don't have systemd-resolved?
We generally dislike adding features that only work on a subset of platforms, and while this would be great to have, we have to consider that it may only be possible on Linux.
Our next release will have bridging, at which point that could be used to address <hostname>.<your home domain>, another option to consider is the .local domain, advertised via zeroconf.
tsibley
commented
3 years ago Nod. I didn't mean to suggest adding the feature just for Linux hosts. One of the attractions of Multipass to me is the multi-platform host support and consistency across platforms.
I'd be in support of solutions like <instance-name>.local or <instance-name>.multipass.local, as those were things I naturally tried a couple days ago during my initial "troubleshooting." The <instance-name>.<home-domain> option is interesting, but it's harder to write portable instructions/documentation for that since <home-domain> will vary.
Saviq
commented
3 years ago but it's harder to write portable instructions/documentation for that since
<home-domain>will vary.
Usually your home domain will be searched, so as long as your instance names were unique with any physical hosts, just <instance-name> would work, too.
None of the above are exclusive, either :)
tsibley
commented
3 years ago Ah, both true! :-)
Out of curiosity, I did some digging about if this is supportable on macOS. It appears that domain-specific DNS resolvers have long been possible on that platform by creating files under /etc/resolver/. For example, Multipass could put the following under /etc/resolver/multipass:
domain multipass
search multipass
nameserver 10.x.x.1A few pages with more information:
tsibley
commented
3 years ago On Linux hosts without systemd-resolved, there are a few other potential solutions as well, like those documented by Arch Linux.
tsibley
commented
3 years ago and because I couldn't resist, on Windows 10 (maybe earlier too?) it looks like there's a PowerShell command for setting domain-specific DNS servers: https://docs.microsoft.com/en-us/powershell/module/dnsclient/add-dnsclientnrptrule?view=win10-ps
Saviq
commented
3 years ago Cool, thanks a lot for the research @tsibley!
The problem is that on both macOS and Windows, it's them that provide the resolvers. And so far I have not found a way to tell them that .multipass is a domain they should resolve for.
tsibley
commented
3 years ago The problem is that on both macOS and Windows, it's them that provide the resolvers. And so far I have not found a way to tell them that
.multipassis a domain they should resolve for.
@Saviq Hmm. So you're saying that Multipass instances on macOS and Windows cannot resolve <instance-name>.multipass domains even from within the instances themselves, unlike instances on Linux?
Saviq
commented
3 years ago @Saviq Hmm. So you're saying that Multipass instances on macOS and Windows cannot resolve
<instance-name>.multipassdomains even from within the instances themselves, unlike instances on Linux?
Unfortunately yes, we don't control the DNS resolvers on Windows or macOS, and there's no way we can tell them to resolve .multipass.
tsibley
commented
3 years ago Ok. It seems then that <instance>.multipass could work inside and outside of instances on all platforms if Multipass could (1) run its own super minimal DNS resolver for VM instances and (2) use the platform-specific methods I briefly researched and linked to above in November to register (1) as the preferred resolver for the .multipass domain.
The minimal resolver could only handle very simple queries and still be useful for this. I imagine it could either be implemented in C++ as part of Multipass (especially if there's a decent library for stubbing most of this) or perhaps using existing software on some platforms, like dnsmasq on Linux.
nikhiljohn10
commented
3 years ago @Saviq Regarding this issue, I want to know more about the workaround for this issue in macOS without doing anything with the instance. Like, install and configure dnsmasq with a zsh script. Please share any such links or scripts that configure multipass to use dnsmasq in macOS 11.
Saviq
commented
3 years ago @nikhiljohn10 there are no such scripts, it's just not possible with how networking is currently set up, because bootpd is what gives out DHCP on the network the instances connect to.
nikhiljohn10
commented
3 years ago what about using a parallel DHCP and DNS server dedicated to multipass and DNS server forward request to macOS's DNS servers?
Saviq
commented
3 years ago what about using a parallel DHCP and DNS server dedicated to multipass and DNS server forward request to macOS's DNS servers?
We'd need to set up our own network, too, which is likely not worth it for just this.
We're working (#1857) on a qemu-based solution for macOS which will allow bridging in the long run, so you'll be able to use your router's configuration. And then there's the .local domain, which your instances can advertise on.
Saviq
commented
3 years ago @nikhiljohn10 just install avahi-daemon on the instances and you can then refer to them that way:
$ multipass launch --cloud-init - <<EOF
packages: [avahi-daemon]
EOF
Launched: decorous-barbet
$ multipass exec impish-bandicoot ping decorous-barbet.local
PING decorous-barbet.local (192.168.64.63) 56(84) bytes of data.
64 bytes from 192.168.64.63 (192.168.64.63): icmp_seq=1 ttl=64 time=0.398 ms
...@nikhiljohn10 just install
avahi-daemonon the instances and you can then refer to them that way:$ multipass launch --cloud-init - <<EOF packages: [avahi-daemon] EOF Launched: decorous-barbet $ multipass exec impish-bandicoot ping decorous-barbet.local PING decorous-barbet.local (192.168.64.63) 56(84) bytes of data. 64 bytes from 192.168.64.63 (192.168.64.63): icmp_seq=1 ttl=64 time=0.398 ms ...
This is only partially working. the ping command is working. but dig or nslookup are not working on other virtual machines.
When I try the commands in same vm, it works.
ubuntu@stepca:~$ ping stepca.local -c2
PING stepca.local (192.168.65.16) 56(84) bytes of data.
64 bytes from stepca (192.168.65.16): icmp_seq=1 ttl=64 time=0.016 ms
64 bytes from stepca (192.168.65.16): icmp_seq=2 ttl=64 time=0.043 ms
--- stepca.local ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1047ms
rtt min/avg/max/mdev = 0.016/0.029/0.043/0.013 ms
ubuntu@stepca:~$ nslookup stepca.local
Server: 127.0.0.53
Address: 127.0.0.53#53
Non-authoritative answer:
Name: stepca.local
Address: 192.168.65.16
Name: stepca.local
Address: fe80::c8ee:94ff:fea8:b020
ubuntu@stepca:~$ dig stepca.local
; <<>> DiG 9.16.1-Ubuntu <<>> stepca.local
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7708
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;stepca.local. IN A
;; ANSWER SECTION:
stepca.local. 0 IN A 192.168.65.16
;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Mon Aug 16 11:44:44 IST 2021
;; MSG SIZE rcvd: 57But when it do these commands in another vm,
ubuntu@subscriber:~$ ping stepca.local -c2
PING stepca.local (192.168.65.16) 56(84) bytes of data.
64 bytes from 192.168.65.16 (192.168.65.16): icmp_seq=1 ttl=64 time=0.346 ms
64 bytes from 192.168.65.16 (192.168.65.16): icmp_seq=2 ttl=64 time=0.471 ms
--- stepca.local ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1076ms
rtt min/avg/max/mdev = 0.346/0.408/0.471/0.062 ms
ubuntu@subscriber:~$ nslookup stepca.local
Server: 127.0.0.53
Address: 127.0.0.53#53
** server can't find stepca.local: SERVFAIL
ubuntu@subscriber:~$ dig stepca.local
; <<>> DiG 9.16.1-Ubuntu <<>> stepca.local
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 60808
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;stepca.local. IN A
;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Mon Aug 16 11:46:19 IST 2021
;; MSG SIZE rcvd: 41This is important for my work. I am creating a demo tutorial for step-ca using multipass. It works perfectly in Ubuntu. Windows looked like a lost cause. But I at least need to make it work in macOS.
@Saviq I could not understand how to control mdns or avahi. Please tell me if there is anything I can do to fix this?
Saviq
commented
3 years ago This is only partially working. the ping command is working. but dig or nslookup are not working on other virtual machines.
That's because the tools you mention talk to name servers directly, they don't go through NSS.
$ grep mdns /etc/nsswitch.conf
hosts: files mdns4_minimal [NOTFOUND=return] dnsAll that should not prevent services to resolve those domains through NSS, like ping does, and basically everything should. What's the actual problem you're experiencing with the demo?
nikhiljohn10
commented
3 years ago What's the actual problem you're experiencing with the demo?
ubuntu@subscriber:~$ step ca bootstrap \
> --ca-url https://stepca.local \
> --fingerprint 5995e7954803de536f6cbbd01b0e9f5dc7b61466aee1b71bf8fd926fa85c72c3
error downloading root certificate: client.Root;
client GET https://stepca.local/root/5995e7954803de536f6cbbd01b0e9f5dc7b61466aee1b71bf8fd926fa85c72c3
failed: Get "https://stepca.local/root/5995e7954803de536f6cbbd01b0e9f5dc7b61466aee1b71bf8fd926fa85c72c3":
dial tcp: lookup stepca.local on 127.0.0.53:53: server misbehavingThis error is only found with macOS and avahi-daemon installation inside each VM. Hence it uses 'local' as TLD. Since it works in Ubuntu with multipass domain name, I am assuming it have something to do with dns resolution using avahi.
nikhiljohn10
commented
3 years ago $ grep mdns /etc/nsswitch.conf hosts: files mdns4_minimal [NOTFOUND=return] dns
Could not find this file in macOS. (Currently using macOS 11.5.1)
nikzjon@macpro step-ca-tutorials % grep mdns /etc/nsswitch.conf
grep: /etc/nsswitch.conf: No such file or directoryInside ubuntu VM, it is as follows.
ubuntu@client:~$ cat /etc/nsswitch.conf
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.
passwd: files systemd
group: files systemd
shadow: files
gshadow: files
hosts: files dns
networks: files
protocols: db files
services: db files
ethers: db files
rpc: db files
netgroup: nisdial tcp: lookup stepca.local on 127.0.0.53:53: server misbehavingThis error is only found with macOS and avahi-daemon installation inside each VM. Hence it uses 'local' as TLD. Since it works in Ubuntu with multipass domain name, I am assuming it have something to do with dns resolution using avahi.
If you can ping stepca.local from subscriber, the above suggests step is doing something wrong when resolving.
Could not find this file in macOS. (Currently using macOS 11.5.1)
Yeah it's specific to Linux.
Inside ubuntu VM, it is as follows.
hosts: files dnsDo you have avahi-daemon, and the recommended libnss-mdns installed? That should add the mdns bit.
Can you come to IRC for a more interactive debugging session? https://web.libera.chat/?channels=#multipass
nikhiljohn10
commented
3 years ago If you can
ping stepca.localfromsubscriber, the above suggestsstepis doing something wrong when resolving.
ping is working in all VMs. Other VM's IP addresses are reflected. I have raised the same question to step-ca devs.
Do you have
avahi-daemon, and the recommendedlibnss-mdnsinstalled? That should add themdnsbit.
Yes, I have installed avahi-daemon as vm is created. It seems like it already installed libnss-mdns as well.
ubuntu@stepca:~$ sudo apt install libnss-mdns
Reading package lists... Done
Building dependency tree
Reading state information... Done
libnss-mdns is already the newest version (0.14.1-1ubuntu1).
libnss-mdns set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.Can you come to IRC for a more interactive debugging session? https://web.libera.chat/?channels=#multipass
I'll join soon.
maraino
commented
3 years ago step-ca is a program in Go, and depending on how it is compiled it will use /etc/resolv.conf and /etc/hosts if it's using the pure go resolver and it will use system calls if it's using CGO. The released builds are currently using the pure go resolver, and a custom resolver can be passed using --resolver 1.2.3.4:53
Saviq
commented
3 years ago Thanks @maraino for that detail!
@nikhiljohn10 here's how to enable mDNS inside systemd-resolved. It's a bit more involved than I'd like (partly due to netplan bug LP#1830507), but it seems to do the trick:
# mdns.yaml
write_files:
- path: /etc/systemd/resolved.conf
append: true
content: |
MulticastDNS=yes
- path: /etc/systemd/system/mdns@.service
content: |
[Service]
Type=oneshot
ExecStart=/usr/bin/systemd-resolve --set-mdns=yes --interface=%i
After=sys-subsystem-net-devices-%i.device
[Install]
WantedBy=sys-subsystem-net-devices-%i.device
runcmd:
- systemctl restart systemd-resolved.service
- systemctl start mdns@enp0s2.service
- systemctl enable mdns@enp0s2.service$ multipass launch --cloud-init mdns.yaml
Launched: qualified-cod
$ multipass launch --cloud-init mdns.yaml
Launched: handy-troll
$ multipass exec qualified-cod -- dig handy-troll.local | grep -A1 'ANSWER SECTION'
;; ANSWER SECTION:
handy-troll.local. 97 IN A 192.168.65.51@Saviq @maraino It worked!!! It works even without installing avahi!!! Awesome work. Thank you so much for all the help. Now it works in macOS as well.
Saviq
commented
3 years ago Yeah systemd-resolved implements the same functionality that avahi does, but in a different way. Glad it's going for you :)
CyrosX
commented
2 years ago @Saviq Some changes have to be made for Ubuntu 22.04: systemd-resolve --set-mdns=yes --interface=%I -> resolvectl mdns %i yes
write_files:
- path: /etc/systemd/resolved.conf
append: true
content: |
MulticastDNS=yes
- path: /etc/systemd/system/mdns@.service
content: |
[Service]
Type=oneshot
ExecStart=/usr/bin/resolvectl mdns %i yes
After=sys-subsystem-net-devices-%i.device
[Install]
WantedBy=sys-subsystem-net-devices-%i.device
runcmd:
- systemctl restart systemd-resolved.service
- systemctl start mdns@enp0s1.service
- systemctl enable mdns@enp0s1.service
joes
commented
2 years ago This seems to do the trick for dns resolution for instance.multipass on the host when using lxd as multipass local.driver on my Ubuntu 22.04 host:
# Set dns mode
sudo lxc network set mpbr0 dns.mode=managed
# Set dns domain
sudo lxc network set mpbr0 dns.domain=multipass
sudo lxc network show mpbr0
dns_address=$(sudo lxc network get mpbr0 ipv4.address | sed -En -e 's/([0-9.]+).*/\1/p')
dns_domain=$(sudo lxc network get mpbr0 dns.domain)
cat <<EOF | sudo tee /etc/systemd/system/multipass-dns-mpbr0.service
[Unit]
Description=LXD per-link DNS configuration for mpbr0 (multipass)
BindsTo=sys-subsystem-net-devices-mpbr0.device
After=sys-subsystem-net-devices-mpbr0.device
[Service]
Type=oneshot
ExecStart=/usr/bin/resolvectl dns mpbr0 "$dns_address"
ExecStart=/usr/bin/resolvectl domain mpbr0 "~${dns_domain}"
ExecStopPost=/usr/bin/resolvectl revert mpbr0
RemainAfterExit=yes
[Install]
WantedBy=sys-subsystem-net-devices-mpbr0.device
EOF
# Enable and restart service
sudo systemctl daemon-reload
sudo systemctl enable --now multipass-dns-mpbr0Adapted from: https://linuxcontainers.org/lxd/docs/master/howto/network_bridge_resolved/#network-bridge-resolved-configure
Describe the bug Instance names don't resolve on the host as
instance-name.multipass(but they do within instances).To Reproduce Launched an instance with
multipass launch example, verified it was running withmultipass list, and then ranhost example.multipassandping example.multipass.Expected behavior
example.multipassresolves to the instance IP and ping succeeds.Logs Logs starting from
multipass launchand covering until the failedping(as above).-- Logs begin at Wed 2019-09-18 17:00:27 PDT. -- Nov 05 11:01:51 whunk multipassd[506]: Checking for images to update… Nov 05 11:02:28 whunk multipassd[506]: QMP: {"timestamp": {"seconds": 1604602948, "microseconds": 522995}, "event": "RTC_CHANGE", "data": {"offset": 1}} Nov 05 11:13:38 whunk multipassd[506]: QMP: {"timestamp": {"seconds": 1604603618, "microseconds": 507805}, "event": "RTC_CHANGE", "data": {"offset": 1}} Nov 05 11:24:50 whunk multipassd[506]: QMP: {"timestamp": {"seconds": 1604604290, "microseconds": 507926}, "event": "RTC_CHANGE", "data": {"offset": 1}} Nov 05 11:25:31 whunk multipassd[506]: starting: qemu-img info /var/snap/multipass/common/cache/multipassd/vault/images/focal-20201102/ubuntu-20.04-server-cloudimg-amd64.img Nov 05 11:25:31 whunk multipassd[11969]: Applying AppArmor policy: multipass.qemu-img Nov 05 11:25:31 whunk multipassd[506]: starting: qemu-system-x86_64 --version Nov 05 11:25:31 whunk multipassd[506]: starting: qemu-img resize /var/snap/multipass/common/data/multipassd/vault/instances/example/ubuntu-20.04-server-cloudimg-amd64.img 5368709120 Nov 05 11:25:31 whunk multipassd[11978]: Applying AppArmor policy: multipass.qemu-img Nov 05 11:25:31 whunk multipassd[506]: starting: qemu-img snapshot -l /var/snap/multipass/common/data/multipassd/vault/instances/example/ubuntu-20.04-server-cloudimg-amd64.img Nov 05 11:25:32 whunk multipassd[506]: process working dir '/snap/multipass/2857/qemu' Nov 05 11:25:32 whunk multipassd[506]: process program 'qemu-system-x86_64' Nov 05 11:25:32 whunk multipassd[506]: process arguments '--enable-kvm, -device, virtio-scsi-pci,id=scsi0, -drive, file=/var/snap/multipass/common/data/multipassd/vault/instances/example/ubuntu-20.04-server-cloudimg-amd64.img,if=none,format=qcow2,discard=unmap,id=hda, -device, scsi-hd,drive=hda,bus=scsi0.0, -smp, 1, -m, 1024M, -device, virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:81:da:13, -netdev, tap,id=hostnet0,ifname=tap-34560714794,script=no,downscript=no, -qmp, stdio, -cpu, host, -chardev, null,id=char0, -serial, chardev:char0, -nographic, -cdrom, /var/snap/multipass/common/data/multipassd/vault/instances/example/cloud-init-config.iso' Nov 05 11:25:32 whunk multipassd[506]: starting: qemu-system-x86_64 -nographic -dump-vmstate /tmp/multipassd.oNb506 Nov 05 11:25:32 whunk multipassd[506]: qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5] Nov 05 11:25:32 whunk multipassd[506]: starting: qemu-system-x86_64 --enable-kvm -device virtio-scsi-pci,id=scsi0 -drive file=/var/snap/multipass/common/data/multipassd/vault/instances/example/ubuntu-20.04-server-cloudimg-amd64.img,if=none,format=qcow2,discard=unmap,id=hda -device scsi-hd,drive=hda,bus=scsi0.0 -smp 1 -m 1024M -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:81:da:13 -netdev tap,id=hostnet0,ifname=tap-34560714794,script=no,downscript=no -qmp stdio -cpu host -chardev null,id=char0 -serial chardev:char0 -nographic -cdrom /var/snap/multipass/common/data/multipassd/vault/instances/example/cloud-init-config.iso Nov 05 11:25:32 whunk multipassd[12127]: Applying AppArmor policy: multipass.example.qemu-system-x86_64 Nov 05 11:25:32 whunk multipassd[506]: process started Nov 05 11:25:32 whunk multipassd[506]: QMP: {"QMP": {"version": {"qemu": {"micro": 1, "minor": 11, "major": 2}, "package": "(Debian 1:2.11+dfsg-1ubuntu7.32)"}, "capabilities": []}} Nov 05 11:25:32 whunk multipassd[506]: QMP: {"return": {}} Nov 05 11:25:34 whunk multipassd[506]: QMP: {"timestamp": {"seconds": 1604604334, "microseconds": 326892}, "event": "RESET", "data": {"guest": true}} Nov 05 11:25:34 whunk multipassd[506]: VM restarting Nov 05 11:25:34 whunk multipassd[506]: QMP: {"timestamp": {"seconds": 1604604334, "microseconds": 333014}, "event": "RESET", "data": {"guest": true}} Nov 05 11:25:43 whunk multipassd[506]: QMP: {"timestamp": {"seconds": 1604604343, "microseconds": 672581}, "event": "NIC_RX_FILTER_CHANGED", "data": {"name": "net0", "path": "/machine/peripheral/net0/virtio-backend"}} Nov 05 11:25:46 whunk dnsmasq-dhcp[875]: DHCPDISCOVER(mpqemubr0) 52:54:00:81:da:13 Nov 05 11:25:46 whunk dnsmasq-dhcp[875]: DHCPOFFER(mpqemubr0) 10.84.39.36 52:54:00:81:da:13 Nov 05 11:25:46 whunk dnsmasq-dhcp[875]: DHCPDISCOVER(mpqemubr0) 52:54:00:81:da:13 Nov 05 11:25:46 whunk dnsmasq-dhcp[875]: DHCPOFFER(mpqemubr0) 10.84.39.36 52:54:00:81:da:13 Nov 05 11:25:46 whunk dnsmasq-dhcp[875]: DHCPREQUEST(mpqemubr0) 10.84.39.36 52:54:00:81:da:13 Nov 05 11:25:46 whunk dnsmasq-dhcp[875]: DHCPACK(mpqemubr0) 10.84.39.36 52:54:00:81:da:13 example Nov 05 11:25:47 whunk multipassd[506]: Trying SSH on 10.84.39.36:22 Nov 05 11:26:18 whunk multipassd[506]: QMP: {"timestamp": {"seconds": 1604604378, "microseconds": 634637}, "event": "RTC_CHANGE", "data": {"offset": 1}}Additional info
multipass info exampleAdditional context I was able to make this work by manually configuring DNS for the multipass interface with
systemd-resolve:When I then
ping example.multipass, it's successful, and I see some log lines from dnsmasq:The multipass network is 10.84.39.1/24 and my home network is 10.0.0.1/24. I wonder if the problem arises because my home network is also under the 10/8 space?