embedded-linux: Struggling to mount/boot with nfs root filesystem

[poodle-dog]

I'd love a suggestion from the bootlin team about how I might be making a mistake with my nfs mount. I'm following along with the BeagleBone Black guide posted here - https://bootlin.com/doc/training/embedded-linux-bbb/embedded-linux-bbb-labs.pdf

Had absolutely no trouble getting Uboot or the kernel up and running. My trouble has been entirely with getting the rootfs to mount using nfs.

I've tried this using both Ethernet (which I set up using my own research) and Ethernet-over-USB as the guide recommends. The furthest I have gotten is as follows, which shows the failure I get when appending the debug kernel argument to Uboot's bootargs.

[    5.260682] IP-Config: Guessing netmask 255.255.255.0
[    5.265773] IP-Config: Complete:
[    5.269012]      device=eth0, hwaddr=c8:a0:30:aa:d7:67, ipaddr=192.168.1.206, mask=255.255.255.0, gw=255.255.255.255
[    5.279619]      host=192.168.1.206, domain=, nis-domain=(none)
[    5.285583]      bootserver=255.255.255.255, rootserver=192.168.1.136, rootpath=
[    5.308516] VFS: Mounted root (nfs filesystem) on device 0:16.
[    5.322989] devtmpfs: mounted
[    5.327279] Freeing unused kernel image (initmem) memory: 1024K
[    5.341193] Run /sbin/init as init process
[    5.345318]   with arguments:
[    5.348295]     /sbin/init
[    5.351038]   with environment:
[    5.354191]     HOME=/
[    5.356558]     TERM=linux
[    5.368425] Run /etc/init as init process
[    5.372605]   with arguments:
[    5.375586]     /etc/init
[    5.378214]   with environment:
[    5.381406]     HOME=/
[    5.383776]     TERM=linux
[    5.389236] Run /bin/init as init process
[    5.393397]   with arguments:
[    5.396379]     /bin/init
[    5.399007]   with environment:
[    5.402195]     HOME=/
[    5.404565]     TERM=linux
[    5.408415] Run /bin/sh as init process
[    5.412384]   with arguments:
[    5.415365]     /bin/sh
[    5.417819]   with environment:
[    5.420992]     HOME=/
[    5.423361]     TERM=linux
[    5.429003] Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.
[    5.443278] ---[ end Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance. ]---

Here are my bootargs from Ethernet over USB:

bootargs=console=ttyS0,115200n8 root=/dev/nfs ip=192.168.0.100:::::usb0 g_ether.dev_addr=f8:dc:7a:00:00:02 g_ether.host_addr=f8:dc:7a:00:00:01 nfsroot=192.168.0.1:/home/daddy/Tools/nfsroot,nfsvers=3,tcp rw earlyprintk nfsrootdebug

And bootargs from Ethernet:

bootargs=console=ttyS0,115200n8 root=/dev/nfs ip=192.168.1.206 nfsroot=192.168.1.111:/home/daddy/Tools/nfsroot,nfsvers=3,tcp rw earlyprintk nfsrootdebug

I have some confidence that the Ethernet arguments are correct, as I'm able to get the following message in /var/log/syslog when the device is configured to boot from Ethernet:

Jul 20 17:06:02 warbucks rpc.mountd[40276]: authenticated mount request from 192.168.1.206:849 for /home/daddy/Tools/nfsroot (/home/daddy/Tools/nfsroot)
Jul 20 17:08:18 warbucks rpc.mountd[40276]: authenticated mount request from 192.168.1.206:778 for /home/daddy/Tools/nfsroot (/home/daddy/Tools/nfsroot)
Jul 20 17:09:00 warbucks rpc.mountd[40276]: authenticated mount request from 192.168.1.206:691 for /home/daddy/Tools/nfsroot (/home/daddy/Tools/nfsroot)

I also have no problem getting either Uboot or the DTS file via TFTP, which gives me yet another sign that the networking connection is A-OK.

I also am pretty sure I have /etc/exports configured correctly,

# /etc/exports: the access control list for filesystems which may be exported
#       to NFS clients.  See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes       hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4        gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes  gss/krb5i(rw,sync,no_subtree_check)
#

/home/daddy/Tools/nfsroot 192.168.1.206(rw,no_root_squash,no_subtree_check)

I have run sudo exportfs -r to export these settings after editing them, to no effect. I have also changed the client from 192.168.1.206 to * to be super permissive, just to be sure that isn't the source of the problem either.

I've also modified the ownership and permissions of my local busybox binary through the following steps:

• sudo chown root:root bin/busybox
• sudo chmod u+s bin/busybox ...but neither of these have the desired effect.

I'm also pretty sure my directory structure is correct (filtered for brevity):

daddy@warbucks:~/Tools/nfsroot$ tree -P 'init|sh|busybox'
.
├── bin
│   ├── busybox
│   ├── init -> busybox
│   └── sh -> busybox
├── dev
├── etc
│   └── init.d
├── sbin
│   └── init -> ../bin/busybox
└── usr
    ├── bin
    └── sbin

8 directories, 4 files

I have also changed options in busybox's menuconfig to change these to hard-links instead of soft-links - this also had no effect.

What am I missing? What else can I check to be sure of the source of the problem? Any suggestions about file permissions, directory structures, logs I can check, or other dumb "gotchas" are most welcome.

[poodle-dog]

And, yes, to be 100% transparent, I named the build machine daddy, so I can claim that I have daddy issues when things don't work. (Very, very stupid joke, but it's the only thing enjoyable about this scenario!) :laughing:

[tpetazzoni]

Your NFS is working fine as reported by the VFS: Mounted root (nfs filesystem) on device 0:16. line. Could you check if your Busybox binary is statically linked?

[poodle-dog]

Great call @tpetazzoni - that turned out to be the problem. I could have sworn I had ticked the option to statically link, but I apparently did not. I recompiled with the statically linked binary, reran make install, and got to a shell first go.

Thanks a bunch for your help, and for all of your wonderful tutorials. They are head and shoulders above all other embedded linux resources I can find online. :heart: (Do you sell bootlin t-shirts? I'd buy one. :shirt: :penguin:)

[tpetazzoni]

Glad to hear the issue is resolved. And no we do not sell T-Shirts (maybe we should!). If you want to support us, the best way it to just spread the word about our engineering and training services around you!