USB image - Network unreachable

[abitrolly]

netboot.xyz.usb USB image from https://boot.netboot.xyz/ipxe/netboot.xyz.usb fails to reach the network. Not from real hardware connected to LAN, not from QEMU on this laptop. The error message is the same.

$ md5sum netboot.xyz.usb
10f28984666d1e836f7b650615088708  netboot.xyz.usb

$ sudo dd if=netboot.xyz.usb of=/dev/sdc
2752+0 records in
2752+0 records out
1409024 bytes (1.4 MB, 1.3 MiB) copied, 2.20912 s, 638 kB/s

$ sudo qemu-system-x86_64 -hda /dev/sdc

[abitrolly]

While debugging it appears that nslookup resolves addresses as IPv6.

iPXE> nslookup address ipxe.org
iPXE> echo ${address}
2001:ba8:0:1d4::6950:5845

http://ipxe.org/cmd/nslookup

But my network doesn't support IPv6.

[abitrolly]

Route command shows two net0 addresses too.

iPXE> route
net0: 192.168 ...
net0: fe80:: ...

My router seems to be IPv6 compatible, and that's probably the reason I get IPv6 address and get resolve names, but it seems that upstream provider is not. How to make iPXE fallback to IPv4 is IPv6 fails?

[pali]

Same problem there. It is because iPXE itself has broken IPv6 support.

I compiled my own iPXE version with disabled IPv6 support. Then network in this iPXE started working correctly.

But because https://boot.netboot.xyz/menu.ipxe is doing "Attempting to chain to latest version..." it is not possible to use my own working version of iPXE with netboot.xyz. netboot.xyz just replace working iPXE with its own iPXE version with broken IPv6 support and therefore netboot.xyz is unusable.

[antonym]

Hmmm... yeah, I think I see the problem. I'm just using the dhcp command so it's trying all interfaces and trying all types of networking regardless to if the network upstream can support it.

I could try doing:

ifconf -c dhcp || ifconf -c ipv6 || goto netconfig

https://github.com/antonym/netboot.xyz/blob/master/ipxe/disks/netboot.xyz#L18

That may force ipv4 first, then fallback to ipv6 and then go to manual config. (http://ipxe.org/cmd/ifconf)

[antonym]

Just dumping notes here, but I could also do some validation to ensure we can hit boot.netboot.xyz:

ping --count 1 boot.netboot.xyz && goto ipv4_good || goto ipv4_bad

[abitrolly]

@antonym ifconf -c dhcp still configures ipv6. Seems like there is no way to disable it. I asked here http://lists.ipxe.org/pipermail/ipxe-devel/2018-November/006337.html still waiting for replies.

ping: command not found.

netboot.xyz 1.04

[NiKiZe]

ping could be disabled (by firewalls), i would suggest trying to hit a http/https endpoint to determine if it is good and then have fallback, maybe even check the error code.

@abitrolly I remember there being a few other discussions on the list about ipv6 recently, maybe that could help.

Just to have it in text ... error in image is http://ipxe.org/280a6011 Network unreachable

[hugalafutro]

Hi, I am hitting this exact same issue with netboot.xyz running inside a docker container, is there any way to make this work? I use OpenWRT router, I tried disabling the ipv6 interfaces, but it just killed my internet even though I don't use ipv6 nor does my isp provide it.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[abitrolly]

@hugalafutro if your ISP doesn't support ipv6, how could disabling it kill the internet? Can you post what you do?

[pali]

@abitrolly: I have not tested iPXE again but at the time when I wrote my last commit it was because iPXE prefered IPv6 connection and if there was none from ISP then it it obiously killed the internet. Note that on the local network can be running DHCPv6 server or RA packets with just non-routable ULA addresses, it is perfectly fine to have assigned ULA or LL address even when you do not have IPv6 internet connection.

[abitrolly]

@pali do you mean iPXE device could access the network, but could not access the internet?

[pali]

@abitrolly: basically yes.

And there was also another issue in iPXE. On statefull DHCPv6-only network (when ISP or other router provides real IPv6 internet) iPXE detected that it is IPv6 capable (probably via RA) and prefered to use IPv6. But iPXE was not able to assign IPv6 address from statefull DHCPv6 server (maybe it did not have implemented statefull DHCPv6 client?) and just throwed error that Network unreachable.

[pali]

So in both cases, completely disabling IPv6 in iPXE fixed those issues. iPXE in never tried to use IPv6 and so it use IPv4 which worked.

[razamatan]

I'm hitting this issue chainloading netboot.xyz from my working ipxe with no ipv6 support...

[lukyrys]

Same problem via PXE, tried many version, both DHCP or DHCP-undionly tested with same result.

when i hit to "Manual network configuration" and I'll just go through it without change any value then its working. (https still same error but http works). I think the eth0 linkup is late before the dhcp request is in progress. Some extra check if eth is up or sleep may help?

[romanchyla]

was facing the same issue (v2.0.33), on OpenWrt disabling IPv6 solved it:

'sysctl -w net.ipv6.conf.all.disable_ipv6=1',
        'echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6',
        'sysctl -w net.ipv6.conf.default.disable_ipv6=1'```

[woeisme]

Came across the same issue today, afaik ipv6 already disabled on my network (openwrt / lede)

[xeijin]

@woeisme the option in OpenWRT is buried pretty deep, but it worked for me after unticking the box:

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[razamatan]

still a problem... still needs some love.

[foxt]

Can confirm, on all 3 machines I've tried netboot on, all of them required me to go into manual configuration and press enter a few times to get it to work

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[hello-smile6]

STILL BROKEN, BOT! STOP CLOSING IT!

[Openanonwriter]

Having the same issue, we do not have IPV6 on the IPXE machine. Did a capture using wireshark, I can see that it is looking up AAAA records and failing as it should, but does not look for IPV4 after, just hard fails. Manually setting IPv4 it works, but dhcp does not.

[donhector]

I was also getting the network unreachable error message when trying to spin up a Libvirt VM that was configured to boot from its NIC interface (virtio). This VM uses the default BIOS firmware (not UEFI)

The NIC was configured to be attached to the host network (ie not attached to the libvirt's default network, which uses nat via its own virb0 device, but to a custom libvirt network I had created that uses bridge mode via the br0 host device.)

This allows the libvirt VM to receive an IP in the same range as the Libvirt host and pretty much any other devices in my home LAN (ie 192.168.1.0/24)

My home LAN router is OpenWRT, which provides DNS, DHCP (ie, is the one leasing those 192.168.1.0/24 IPS) and TFTP amongst other stuff.

In the OpenWRT UI I had configured TFPT to serve directly from the url http://boot.netboot.xyz/ipxe/netboot.xyz.kpxe (since my VM is not configured for UEFI)

On boot, the VM would receive an IP in the desired 192.168.1.0/24 range, and would also receive the desired TFPT url provided by OpenWRT, but then it would fail trying to pull the actual netboot.xyz.kpxe file from the web with the dreaded network unreachable error message.

Thanks to the pointers here around IPv6 being the culprit and following @xeijin suggestion, totally disabling DHCPv6 in OpenWRT for my LAN fixed the problem for me:

Hope that helps other people as well as a temporary workaround until the ipxe file itslef handles resorting to IPv4 when needed

NOTE: In my setup, using https in the url did not work, I had to settle using http.

NOTE2: Your mileage might vary, but it was no big deal for me to disable IPv6 inside my Lan.

[steve6375]

My ISP does not support ipv6. Any fix for netboot.xyz.iso ? Seems to be magically working now! Maybe my ISP changed something?

[SimonMcN]

okay. I'm stumped. If I do m at boot and choose all the options it presents it works. but it doesn't work automatically ?

[joshenders]

okay. I'm stumped. If I do m at boot and choose all the options it presents it works. but it doesn't work automatically ?

This was my experience as well.

I'm using:

# netboot.xyz bootloaders generated at Thu Feb 16 00:54:02 UTC 2023
# iPXE Commit: https://github.com/ipxe/ipxe/commit/cff857461be443339aa39d614635d9a4eae8f8b2
...
f9432f382b8c507d15f7ad17a4f432e91a89d0e680fb95fa80d29cedd96d959f *netboot.xyz.iso

HP Microserver Gen10+ with Intel i350-based 1GbE directly connected to the ONT of an IPv4-only fiber ISP.

[dedene]

okay. I'm stumped. If I do m at boot and choose all the options it presents it works. but it doesn't work automatically ?

This was my experience as well.

I'm using:

# netboot.xyz bootloaders generated at Thu Feb 16 00:54:02 UTC 2023

# iPXE Commit: https://github.com/ipxe/ipxe/commit/cff857461be443339aa39d614635d9a4eae8f8b2

...

f9432f382b8c507d15f7ad17a4f432e91a89d0e680fb95fa80d29cedd96d959f *netboot.xyz.iso

HP Microserver Gen10+ with Intel i350-based 1GbE directly connected to the ONT of an IPv4-only fiber ISP.

I'm observing the same with latest version, regardless of ipv6 configured on the network or not.

Was it working for you with that version?

[joshenders]

okay. I'm stumped. If I do m at boot and choose all the options it presents it works. but it doesn't work automatically ?

This was my experience as well.

I'm using:

# netboot.xyz bootloaders generated at Thu Feb 16 00:54:02 UTC 2023

# iPXE Commit: https://github.com/ipxe/ipxe/commit/cff857461be443339aa39d614635d9a4eae8f8b2

...

f9432f382b8c507d15f7ad17a4f432e91a89d0e680fb95fa80d29cedd96d959f *netboot.xyz.iso

HP Microserver Gen10+ with Intel i350-based 1GbE directly connected to the ONT of an IPv4-only fiber ISP.

I'm observing the same with latest version, regardless of ipv6 configured on the network or not.

Was it working for you with that version?

I ended up trying the .iso version written to a usb flash drive using balena etcher.

[gema-arta]

in

QEMU emulator version 8.0.0 (v8.0.0-12024-gd6b71850be-dirty)
Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers

qemu-system-x86_64 -M q35 -cpu qemu64 -m 4G -L . -nic user,model=virtio -cdrom netboot.xyz-multiarch.iso  -boot menu=on -usb

or use

qemu-system-x86_64 -M q35 -cpu qemu64 -m 4G -L . -nic user,model=virtio-net-pci-transitional -hda netboot.xyz-multiarch.img  -boot menu=on -usb

or basically

qemu-system-x86_64 -nic user,tftp=.,bootfile=http://boot.netboot.xyz/ipxe/netboot.xyz.lkrn -nographic

result

iPXE 1.21.1+ (gb00935) -- Open Source Network Boot Firmware -- https://ipxe.org
Features: DNS HTTP HTTPS iSCSI TFTP SRP VLAN AoE ELF MBOOT PXE bzImage COMBOOT M
enu PXEXT
netboot.xyz - v2.x

Configuring (net0 52:54:00:12:34:56)...... ok
https://boot.netboot.xyz/menu.ipxe... Network unreachable (https://ipxe.org/280a6011)
HTTPS appears to have failed... attempting HTTP
http://boot.netboot.xyz/menu.ipxe... Network unreachable (https://ipxe.org/280a6011)
HTTP has failed, localbooting...
No bootable device.
QEMU: Terminated

show ip6 get local link address who added automatically

This is qemu problem?

[razamatan]

it's not specific to qemu. i'm still having this issue on physical/standalone pc's.

[gema-arta]

@razamatan

it's not specific to qemu. i'm still having this issue on physical/standalone pc's.

This is globally ipxe ipv6/dns name resolving and dual v4/v6 stack trouble. But the ipv6=off qemu workaround works for me. qemu-system-x86_64 -nic user,ipv6=off,tftp=.,bootfile=http://boot.netboot.xyz/ipxe/netboot.xyz.lkrn -nographic

[antonym]

The issue is really upstream with iPXE, I could probably build images that were separate single stack to see if that would work better but ideally I wouldn't have to create more images.

If v6 is enabled on the network, it prefers the v6 DNS regardless of whether or not it'll work, and I haven't found a good way to shut off the stack and prefer v4 DNS if v6 is present. Trying to disable or work around it doesn't work in the script.

[dedene]

Is there any pointer you could give on how to build these single stack (i.e. ipv4 only) images? I’d be happy to play around with it and see if it solves the issue.

I agree ideally it should be solved in iPXE but in the meanwhile this could be a good workaround.

[antonym]

I made a few changes to the boot-loader in development to see if it can more gracefully handle v4/v6 hybrid scenarios so the end result is that you end up in the menu and I can hopefully avoid having to split the disks up into separate stacks. Can anyone experiencing the issue try these boot-loaders out and see if it helps any in the scenario that usually doesn't work for you? I'm not currently utilizing v6 in my environment so it's difficult to simulate.

https://s3.amazonaws.com/dev.boot.netboot.xyz/9a04a0741e078c770c54c99bce5fdfb960507e98/index.html

If it see's netX/dns6 set from DHCP, it'll attempt v6 connectivity first. If that fails, it'll clear out the netX/dns6 variable and attempt another reload. If netX/dns6 is not set, it should abandon trying to use the v6 stack and fall back to booting over v4.

Actual changes are:

https://github.com/netbootxyz/netboot.xyz/commit/5e9d77fefa22931acef808280ca48c16d040e3e5 https://github.com/netbootxyz/netboot.xyz/commit/9a04a0741e078c770c54c99bce5fdfb960507e98

[dedene]

@antonym Awesome, thx for having a look at this! I can confirm it is an improvement: with the new images it automatically boots into the Netboot.xyz menu without manually setting the network config.

However after selecting an image (for installation or live cd) I still get the same Network Unreachable error. I made a small screen recording to show: https://cln.sh/97kZdn2WNTqcM50ZCkMp You can see when booting in the menu it says "attempting IPv4..." but then after selecting a distro it fails again probably due to ipv6?

[antonym]

It looks like it's failing on https but passing on http. The menu can be pulled from either, but anything pulled from Github is going to be https so it may be worth digging into why it's unable to load https. Date/time on machine might be one issue.

[dedene]

@antonym Sorry for the late reply. Indeed it's failing on https. But when pressing 'm' and just hitting enter for manual network config, https seems to works just fine. Also, NTP is running on the host, so I'm quite sure the Date/Time is configured correctly for the VM.

Any ideas why the https could be failing for the automatic flow, but working using the failsafe menu and manual (ipv4) network configuration?

[ManuLinares]

Hasn't been any news on this. Still on my network I have all ipv6 dhcp6 disabled, but when I start my machine I always have to press "m" to enter failsafe mode, then accepts the defaults and it works. Automatically it doesn't.

Any fix for this? Any workaround?

[nrvale0]

I guess I'm not adding much here except to say I'm seeing the exact same behavior as @ManuLinares et al.

[nrvale0]

After following the above link to the iPXE mailing list and then learning that iPXE will default to IPv6 for assets if it gets an AAAA record during DNS query, I adjusted my gateway's DNS settings (Unbound) to block all AAAA queries and the PXE-booting node is now able to download its boot images without a "Network unreachable..." message.

I think iPXE still needs better logic (and perhaps some tunables) to address this issue but here is a work-around config for Unbound:

server:
  private-address: ::/0

[joshenders]

After following the above link to the iPXE mailing list and then learning that iPXE will default to IPv6 for assets if it gets an AAAA record during DNS query, I adjusted my gateway's DNS settings (Unbound) to block all AAAA queries and the PXE-booting node is now able to download its boot images without a "Network unreachable..." message.

I think iPXE still needs better logic (and perhaps some tunables) to address this issue but here is a work-around config for Unbound:

server:

  private-address: ::/0

Globally disabling AAAA responses/IPv6 on your network feels too heavy handed of a solution to me.

Here's an alternate solution to only filter AAAA responses for just netboot.xyz:

Copy the script below to /etc/unbound.

https://gist.github.com/FiloSottile/e2cffde2bae1ea0c14eada229543aebd

Modify the domains dict to just one entry of netboot.xyz.:

domains = [
     "netboot.xyz.",
#    "netflix.com.",
#    "nflxso.net.",
]

Update your /etc/unbound/unbound.conf to contain the following directives:

server:
    module-config: "python"

python:
    python-script: "/etc/unbound/filter-aaaa.py"

Ideally, this could be scoped even further to specific source addresses but this is probably better than filtering out half the internet over an iPXE bug 😃.

[nrvale0]

Neither my internal network nor my upstream ISP supports IPv6 so those records aren't of any use to me. And its not the first time I've had problems with IPv6 records in DNS replies. Yes, if someone's upstream ISP requires IPv6 that's going to be too Big Hammer.

Regardless, it's a data point to suggest that the iPXE "heuristic" is a bit...well, it could use some more attention. ;)

[joshenders]

Neither my internal network nor my upstream ISP supports IPv6 so those records aren't of any use to me. And its not the first time I've had problems with IPv6 records in DNS replies. Yes, if someone's upstream ISP requires IPv6 that's going to be too Big Hammer.

Regardless, it's a data point to suggest that the iPXE "heuristic" is a bit...well, it could use some more attention. ;)

Just exhausted seeing low effort "disable IPv6" solutions. Your internal network likely does support it, you've just not bothered to set it up or disabled it 🤪. It's 2024, all major hardware and operating systems made in the last 20 years support IPv6 and half the Internet is accessible over IPv6, it's time to move on.

[nrvale0]

Regardless, it's a data point to suggest that the iPXE "heuristic" is a bit...well, it could use some more attention. ;)

Just exhausted seeing low effort "disable IPv6" solutions. Your internal network likely does support it, you've just not bothered to set it up or disabled it 🤪. It's 2024, all major hardware and operating systems made in the last 20 years support IPv6 and half the Internet is accessible over IPv6, it's time to move on.

You should refrain from assuming my requirements and whether or not they are "low effort". I can assure you that the x-vendor support for IPv6 is barely better now than it was 20 years ago because I worked at a research lab that helped with a number of the implementations.

The fact that we are here discussing a work-around for this half-hearted iPXE "heuristic" says a lot about about the community's understanding and the ecosystem's support of the technology as well.

[joshenders]

You should refrain from assuming my requirements and whether or not they are "low effort".

I find it odd that you consider a one line change higher effort than the complicated multi-step Python solution I posted.

Users are going to discover your solution via search engine and needlessly block access to half the internet on their network as a work around; it's irresponsible.

The fact that we are here discussing a work-around for this half-hearted iPXE "heuristic" says a lot about about the community's understanding and the ecosystem's support of the technology as well.

You're not wrong. I hope this gets fixed in iPXE. Again, in 2024 it's inexcusable to not have a working IPv6 stack.

[foxt]

Your internal network likely does support it, you've just not bothered to set it up or disabled it 🤪. It's 2024, all major hardware and operating systems made in the last 20 years support IPv6 and half the Internet is accessible over IPv6, it's time to move on.

You're right. My internal network, and all the devices on it do support IPv6, and of course a lot of the internet supports IPv6. But, I don't have a ethernet cable long enough to run to Big Ben, that's even if The Elders would let me directly plug in, so there has to be someone in between my local network and The Internet, and at least in the UK, your chances are 50/50 if you'll get an IPv6, or get told there's still enough v4 left

[nrvale0]

FWIW, I have now been running for months with my "irresponsible" solution above with zero negative consequences. I am at least two hops from any network supporting IPv6 and upstream routers handle that transition without my intervention.

This as a data point to suggest viability of either shipping a binary without IPv6 support or, preferably, providing a working config option to disable it until the iPXE IPv6 support is properly sorted.