System: Failed to start Prepare for overlay fs

[amboar]

Trying to boot a system I encounter:

[FAILED] Failed to start Prepare for overlay fs.
See 'systemctl status prepare-overlay.service' for details.
[DEPEND] Dependency failed for /home.
[DEPEND] Dependency failed for Local File Systems.
[DEPEND] Dependency failed for /etc.
[DEPEND] Dependency failed for Mount machine-id.
[  OK  ] Mounted /var/volatile.
[   30.390000] systemd-journald[644]: Received request to flush runtime journal from PID 1
[  OK  ] Reached target Timers.
         Starting Prepare for overlay fs...
[  OK  ] Closed Syslog Socket.
         Starting Network Service...
[  OK  ] Reached target Chassis0 power on after reset.
[  OK  ] Reached target Login Prompts.
[  OK  ] Reached target Host0 running after reset.
[  OK  ] Reached target Phosphor Object Mapper.
[  OK  ] Reached target Webserver (pre).
         Starting Load/Save Random Seed...
[  OK  ] Reached target Sockets.
[  OK  ] Started Flush Journal to Persistent Storage.
[FAILED] Failed to start Prepare for overlay fs.
See 'systemctl status prepare-overlay.service' for details.
[DEPEND] Dependency failed for /home.
[DEPEND] Dependency failed for Emergency Shell.
[DEPEND] Dependency failed for Emergency Mode.
[FAILED] Failed to start Load/Save Random Seed.
See 'systemctl status systemd-random-seed.service' for details.
         Starting Create Volatile Files and Directories...
[  OK  ] Started Network Service.
[  OK  ] Reached target Network.
[   31.330000] 8021q: adding VLAN 0 to HW filter on device eth0
[FAILED] Failed to start Create Volatile Files and Directories.
See 'systemctl status systemd-tmpfiles-setup.service' for details.
[FAILED] Failed to start Network Time Synchronization.
See 'systemctl status systemd-timesyncd.service' for details.
[  OK  ] Reached target System Time Synchronized.
[   34.900000] ftgmac100 1e660000.ethernet eth0: NCSI interface up

And everything hangs. Need to powercycle the machine at the PDU. Not sure what build is on the machine, but it's one containing kernel da78428ac543d6d9ac2493dfa77364eb300a47cb.

[mdmillerii]

@eddiejames

[eddiejames]

@amboar I would need to examine the system in the failed state. Did you do anything weird on this system before this happened?

[amboar]

@eddiejames nothing I'd consider weird. I netbooted a buildroot environment, then tried to reboot into the on-flash environment and it was hosed. I didn't touch the flash in the buildroot environment, was just playing with GPIOs.

[gkeishin]

Seen on latest Master

[FAILED] Failed to start Prepare for overlay fs.
See 'systemctl status prepare-overlay.service' for details.
[DEPEND] Dependency failed for /etc.
[DEPEND] Dependency failed for Local File Systems.
[DEPEND] Dependency failed for Mount machine-id.
[DEPEND] Dependency failed for /home.
[DEPEND] Dependency failed for Emergency Shell.
[DEPEND] Dependency failed for Emergency Mode.
[  OK  ] Closed Syslog Socket.

Now its really bricked the system on few reboots. Attaching the last current serial console log

• [x] Journald SERIAL_CONSOLE_HANG.txt

[eddiejames]

I'm not sure why the prepare-overlay service fails, but it should be possible to make sure it never fails by forcing the bash to return true... Got a change for @gkeishin to try https://gerrit.openbmc-project.xyz/#/c/7724/ as I haven't been able to recreate myself.

[amboar]

I'm not sure why the prepare-overlay service fails, but it should be possible to make sure it never fails by forcing the bash to return true...

What? Surely we should not be doing this. Do you have evidence that it's spuriously failing?

[gkeishin]

@amboar fix for this https://gerrit.openbmc-project.xyz/#/c/7724/

[rfrandse]

Abandoned ~https://gerrit.openbmc-project.xyz/7724 prepare-overlay service: Don't remove entire work directory~ ~Resolves #2514 Witherspoon: Failed to start Prepare for overlay fs~

[gkeishin]

Looked good. We ran a 100 BMC reboot test which usually hits this, now worked clean at least in this 100 run I ran overnight.

[swe12345]

Observing the same issue on wsbmc011 Driver Details

root@witherspoon:~# cat /etc/os-release
ID="openbmc-phosphor"
NAME="Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro)"
VERSION="v2.0-52"
VERSION_ID="v2.0-52-g158f8de"
PRETTY_NAME="Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro) v2.0-52"
BUILD_ID="v2.0"

[FAILED] Failed to start Prepare for overlay fs.
See 'systemctl status prepare-overlay.service' for details.
[DEPEND] Dependency failed for /etc.
[DEPEND] Dependency failed for Local File Systems.
[DEPEND] Dependency failed for Mount machine-id.
[DEPEND] Dependency failed for /home.
[DEPEND] Dependency failed for Emergency Shell.
[DEPEND] Dependency failed for Emergency Mode.
[FAILED] Failed to start Prepare for overlay fs.
See 'systemctl status prepare-overlay.service' for details.
         Starting Flush Journal to Persistent Storage...
[   20.890000] systemd-journald[958]: Received request to flush runtime journal from PID 1
[  OK  ] Started Flush Journal to Persistent Storage.
         Starting Create Volatile Files and Directories...
[FAILED] Failed to start Create Volatile Files and Directories.
See 'systemctl status systemd-tmpfiles-setup.service' for details.
         Starting Network Time Synchronization...
[  OK  ] Started Network Time Synchronization.
[  OK  ] Reached target System Time Synchronized.
[   23.130000] ftgmac100 1e660000.ethernet eth0: NCSI: configuring channel 0
[   23.140000] ftgmac100 1e660000.ethernet eth0: NCSI: Handler for packet type 0x8b returned -34
[   23.170000] ftgmac100 1e660000.ethernet eth0: NCSI interface up