Client on Separate Subnet

[MeCJay12]

Hello! I have got the booting on the same LAN network down. I want to move this client to another subnet. That subnet has its own DHCP server (pfSense). What options do I need to add to my DHCP server to get the client to PXE boot from another subnet? I read through this ticket but I didn't get anywhere with it. Thanks in advance!

[jonwitts]

Can yo not just have the Pi Server in the same subnet as the Pis? That's what we do and it works a treat...

[maxnet]

Sorry, but I have no pfSense setup here to test with. As said in the other thread, the settings do look ok from a quick look.

You would have to sniff packets to see if the DHCP server works as advertised though.

[maxnet]

BTW do note that the Pi pretends to be an old fashioned (BIOS) computer when PXE booting. Perhaps your DHCP server does not respond to the Pi because you filled in an UEFI boot file name in the pfsense webinterface.

But hard to tell without knowing what goes over the line.

[MeCJay12]

I'm zeroing in on the issue. I noticed in wireshark the pi is connecting to the piserver and requesting a file. it is requesting /var/lib/piserver/tftproot/bootcode.bin which doesn't exist. The bootcode.bin file is in a directory named after the Pi's mac. Mac addresses don't get past the router so how is the Pi Server supposed to recognize the client across networks?

[maxnet]

Ah, good point. Forgot about that. That is indeed not going to work this way.

Do may work if you use piserver as DHCP server, and tell pfSense to use dhcp relaying to the piserver IP. dnsmasq does know the MAC addresses of clients it handed out leases to, even if they live in another subnet.

[maxnet]

BTW do you have a need to serve different operating system images to different Pi? If not just symlink /var/lib/piserver/tftproot to /var/lib/piserver/os/name-of-os/boot

Alternatively, it is also possible to put just bootcode.bin in tftproot, and create symlinks based on Pi serial number instead of MAC address for the other boot files. But that is something you would need to do manually, and is not supported in the GUI. (As auto-detecting serial numbers is more difficult than MAC)

[MeCJay12]

Ideally I want to be able to swap OSes so I can't symlink an OS. I'm going to try the DHCP relay idea next but it isn't ideal either. Maybe that is a feature request? IP based OS delegation. Use static DHCP leases for the IP?

[maxnet]

Don't think that is a common enough use-case to add to the GUI. You do can make IP-based symlinks manually though, as it does is an option supported by dnsmasq.

[MeCJay12]

Ooooo. Can you point me in the correct direction to configure this?

[maxnet]

Ooooo. Can you point me in the correct direction to configure this?

Piserver puts "tftp-unique-root=mac" in /etc/dnsmasq.d/piserver While it should be just "tftp-unique-root" if you want to use IP-based folders. You may need to make the file immutable (with chattr) after your changes, to prevent Piserver overwriting your changes.

Can then make symlinks manually in /var/lib/piserver/tftproot for the IPs. Can either point them to the boot folder of the OS that IP should boot. Or do it indirectly, make a symlink from the IP to the MAC symlink. That way you can still change the OS in the Piserver GUI.

[MeCJay12]

Awesome. I'll let you know how that goes.

I did play around with it some more and what I found was symlinking the tftproot folder to the OS /boot folder didn't work. I had to copy the the whole /boot folder into tftproot (so maybe I just goofed the symlink). Once I did that, it would boot but I couldn't log in.

[MeCJay12]

If I change tftp-unique-root=mac to tftp-unique-root=ip I cannot get it to boot. Watching the log I get cannot find /var/lib/piserver/tftpboot/bootcode.bin instead of the IP or even the mac being in there. Is there anything else piserver does to the tftp server I need to change?

[maxnet]

Watching the log I get cannot find /var/lib/piserver/tftpboot/bootcode.bin instead of the IP

If it cannot find a folder for the IP, it falls back to the root folder. So that is "normal" if there is something wrong with the folder/symlink.

[Lukanite]

I had a similar issue for my setup with firewalls, and I found from the Wireshark captures that after it receives bootcode.bin, the Pi would try to access files from its own folder with its serial number pre-pended (e.g. 94xxxxxx/start.elf).

That folder's last 6 chars happens to line up with the Pi's Ethernet MAC address, so I wrote a patch for DNSMasq to translate those folder names into the dash-separated format PiServer uses. As for bootcode.bin, I just put that in the base of PiServer's tftproot folder (next to all of the directory symlinks for each Pi). Here's the diff if you're using DNSMasq as well, but the folder renaming trick should be pretty generally applicable:

From b975f901b9b96d1367ea3c54c85dc26a36d72ec9 Mon Sep 17 00:00:00 2001
From: Alan Zhang <alan@lucar.io>
Date: Tue, 31 Jul 2018 15:22:01 -0400
Subject: [PATCH] Added Raspberry Pi PXE serial identification and file path
 rewriting code

---
 src/tftp.c | 40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/src/tftp.c b/src/tftp.c
index bccca69..509017d 100644
--- a/src/tftp.c
+++ b/src/tftp.c
@@ -439,6 +439,46 @@ void tftp_request(struct listener *listen, time_t now)
    }
       else if (filename[0] == '/')
    daemon->namebuff[0] = 0;
+
+     //Translate Raspberry Pi serials - Alan Zhang hack
+     const char * validchars = "0123456789abcdef";
+     if (filename[0] != '/') {
+         char * c = filename;
+         int i;
+         int ispiser = 1;
+         for (i = 0; i<8; i++) {
+             if (strchr(validchars, *c) == NULL) {
+                 ispiser = 0;
+                 break;
+             }
+             c++;
+         }
+         if (*c != '/') ispiser = 0;
+         if (ispiser) {
+             //Is Pi serial, build MAC
+             //my_syslog(LOG_INFO, _("IS PI SERIAL"));
+             char macstring[19] = { 'b', '8', '-', '2', '7', '-', 'e', 'b', '-' };
+             macstring[9] = filename[2];
+             macstring[10] = filename[3];
+             macstring[11] = '-';
+             macstring[12] = filename[4];
+             macstring[13] = filename[5];
+             macstring[14] = '-';
+             macstring[15] = filename[6];
+             macstring[16] = filename[7];
+             macstring[17] = '/';
+             macstring[18] = '\0';
+             //Skip serial in output filename
+             filename += 9;
+             //Add MAC to output filename
+             strncat(daemon->namebuff, macstring, (MAXDNAME - 1) - strlen(daemon->namebuff));
+         }
+         else {
+             //Not Pi serial, do nothing
+             my_syslog(LOG_INFO, _("NOT PI SERIAL"));
+         }
+     }
+
       strncat(daemon->namebuff, filename, (MAXDNAME-1) - strlen(daemon->namebuff));

       /* check permissions and open file */
-- 
2.17.1

[MeCJay12]

Do you have a Github page with more info?

[Lukanite]

Sorry I haven't had much time; I put some info on how this technique works on a page: https://github.com/Lukanite/PiServer-PXE-Serial-Translator/wiki

The DNSMasq patch basically does this but automatically.

[MeCJay12]

That worked to get it to boot across networks but login doesn't work with this trick. Any ideas?

[Lukanite]

Hmm, DNS resolution for Piserver might be failing. Try adding an entry with the Piserver IP to the hosts file of your boot image? (That'd be /etc/hosts inside your image)

127.0.0.1       localhost
::1             localhost ip6-localhost ip6-loopback
ff02::1         ip6-allnodes
ff02::2         ip6-allrouters

127.0.1.1       raspberrypi
xxx.xxx.xxx.xxx   piserver

[MeCJay12]

That worked. Getting closer. What do the clients do for DNS? My network is handing out DNS servers fine for other clients but the Pi client cannot resolve anything. Nslookup isn't included in Raspian Stretch Lite. I can ping my DNS servers and the Internet by IP.

[maxnet]

By default Piserver images have /etc/resolv.conf modified to let the clients use the Piserver server (dnsmasq) as DNS server. Make sure nothing in your network is blocking DNS packets from/to the Piserver host.

[MeCJay12]

Looks like it's working. Here are my steps for future users. I will add-on if I find anything else. I encourage features be added to enable cross-subnet network booting.

On Pi: -Boot into Raspian and use raspi-config to enable network booting -Get Pi serial number: cat /proc/cpuinfo | grep Serial

Pi Server: -Run Pi Server app and follow prompts (requires Pi PXE booting on same subnet for initial setup) -Make a symlink in /var/lib/piserver/tftproot/ to /var/lib/piserver//boot -Copy bootcode.bin from an OS folder to /var/lib/piserver/tftproot/

Client Shell (inside Pi Server App): -Add /etc/hosts file entry for 'piserver' to IP of the Pi Server -Add/change /etc/resolv.conf nameserver line to desired nameservers -Make the resolve file immutable with chatter: chattr +i /etc/resolv.conf

DHCP Server Settings: -Option 43 "Raspberry Pi Boot" -Option 60 "PXEClient" -Enable Network booting and add Pi Server IP as 'Next Server'

[Lukanite]

Glad to hear! Just wanted to add a note that the automatic serial-to-mac translation won't work for Raspberry Pi 4's when they get PXE boot, since the serials no longer follow the Ethernet MAC.

[GDatviashvili]

Glad to hear! Just wanted to add a note that the automatic serial-to-mac translation won't work for Raspberry Pi 4's when they get PXE boot, since the serials no longer follow the Ethernet MAC.

on Rpi3b+ it works ??

[maxnet]

Just wanted to add a note that the automatic serial-to-mac translation won't work for Raspberry Pi 4's when they get PXE boot, since the serials no longer follow the Ethernet MAC.

The Pi 4 boot firmware now has an option to ask for Ethernet MAC instead of serial itself.

See: https://github.com/raspberrypi/rpi-eeprom/blob/master/firmware/raspberry_pi4_network_boot_beta.md

Requires that you use the pieeprom-2019-10-08.bin boot firmware file, set BOOT_ORDER=0x21 and TFTP_PREFIX=2 and flash it to EEPROM.

[Lukanite]

@maxnet That's awesome! Looking forward to testing it. @Gio1998 Yes, this works on RPi 3B+.