Failing to sync on shutdown but works on demand

[davidjade]

I can't quite figure out what is happening here. It is failing to sync on shutdown but if I run a manual sync, it works and I don't see the errors below during shutdown.

It seems like the overlay tools are failing because they can't access /opt/zram

Here's the log of the failed shutdown.

Any ideas here? It used to work.

zram-config stop 2022-02-16-01:20:56
ztab remove log /zram1 /var/log /opt/zram/log.bind
Stopping services that interfere with zram device configuration.
removeZdir: Beginning removal of device /dev/zram1.
umount: /var/log (overlay1) unmounted
mergeOverlay: Beginning merge of /opt/zram/opt/zram/log.bind.
ls: cannot access '/opt/zram/zram1/upper': Host is down
/opt/zram:
/opt/zram:
ls: cannot access '/opt/zram/zram1': Host is down
total 12
drwxr-xr-x 4 root root 4096 Feb 16 00:43 .
drwxr-xr-x 4 root root 4096 Jan 31 19:08 ..
drwxr-xr-x 3 root root 4096 Jan 31 13:29 opt
d????????? ? ?    ?       ?            ? zram1

/opt/zram/opt/zram/log.bind:
total 28
drwxr-xr-x 3 root root   4096 Feb 16 01:16 .
drwxr-xr-x 3 root root   4096 Jan 31 13:29 ..
-rw-rw---- 1 root utmp      0 Feb 16 00:45 btmp
-rw-rw-r-- 1 root utmp 292292 Feb 16 01:13 lastlog
drwx------ 2 root root   4096 Feb 16 00:45 private
-rw-rw-r-- 1 root utmp   6528 Feb 16 01:13 wtmp
lrwxrwxrwx 1 root root     32 Feb 16 00:44 zram-config -> /usr/local/share/zram-config/log
Upper directory not specified.
Try './overlay --help' for more information.
ztab remove swap /zram0 zram-config0
removeZswap: Beginning removal of device /dev/zram0.
swapoff /dev/zram0
removeZswap: Device /dev/zram0 removed.
removed '/tmp/zram-device-list.rev'
removed '/tmp/zram-device-list'
Restarting services that interfere with zram device configuration.

Here's my config

# swap  alg             mem_limit       disk_size       swap_priority   page-cluster    swappiness
swap    lzo-rle         100M            300M            75              0               150

# dir   alg             mem_limit       disk_size       target_dir              bind_dir
#dir    lzo-rle         50M             150M            /home/pi                /opt/zram/pi.bind

# log   alg             mem_limit       disk_size       target_dir              bind_dir                oldlog_dir
log     lzo-rle         40M             120M            /var/log                /opt/zram/log.bind

[ecdye]

It would appear that your system is unmounting the /opt directory on shutdown before the merge has completed. What distro, and version are you running.

[ecdye]

I pushed a change (4a62be0) that will now (hopefully) require the /opt/zram directory to be mounted while there service is running (including while stopping). Please update and see if the issue is resolved.

[davidjade]

I am using a distro from here: dietpi.com v8.0.2

I attempted to update zram-config but the update process seems to have hung right after this message:

Stopping zram-config.service
zram-config stop 2022-02-16-09:57:16

I eventually ctrl+c and tried again and it then finished the update. Afterwards I checked that /etc/systemd/system/zram-config.service was correct.

Then I rebooted, changed some files in /var/log, and rebooted again, and looked at the log. The sync did not happen and I still see this:

zram-config stop 2022-02-16-10:01:50
ztab remove log /zram1 /var/log /opt/zram/log.bind
Stopping services that interfere with zram device configuration.
removeZdir: Beginning removal of device /dev/zram1.
umount: /var/log (overlay1) unmounted
mergeOverlay: Beginning merge of /opt/zram/opt/zram/log.bind.
ls: cannot access '/opt/zram/zram1/upper': Host is down
/opt/zram:
ls: cannot access '/opt/zram/zram1': Host is down

I do suspect that my distro is the element of change here as I think I was previously using v7.9 when I set this all up. Unfortunately I had to upgrade that for other reasons and didn't notice this broke zram-config in the process. Not sure what they changed that causes the conflict now. DietPi is a stripped-down distro to reduce memory usage. I have used a very basic set up with not too many options enabled or additional stuff installed.

[ecdye]

Odd, I think that your analysis about it being distro specific is probably correct. I'm not sure what would cause the ordering issues on shutdown that you are observing but seeing as how it had worked fine as of yet on regular Debian base installs, I don't know that there is much more that I can do to fix the issue, as I do not personally run DietPi.

Feel free to let me know if you find the solution or open a PR with a fix and I would be happy to merge.

[davidjade]

I don't know if this is relevant, but it seems RequiresMountsFor can be ignored is some circumstances:

https://unix.stackexchange.com/questions/281650/systemd-unit-requiresmountsfor-vs-conditionpathisdirectory

Is the zram mounted as 'noauto'? I don't know enough to figure this out (but I am trying).

[ecdye]

No zram is not mounted as noauto because the mounts are generated by zram-config at the time of start.

I think that the issue may be caused by DietPi's custom affinity and speed optimizations to systems to make it a super small and fast distro. These optimizations have caused a conflict and in preference of speed it looks like the system is not waiting for the zram-config.service to stop before moving on in the shutdown. This is purely speculation based on a little bit of reading, and as such may not be true.

To really solve the issue you may need to figure out what exactly the DietPi dev's have changed about the way SystemD operates compared to a default install of Debian.

[davidjade]

A bit of progress. Clean re-image of everything and it is working so I don't think it is DietPi specifically. I've got a few additional packages I was using to install to narrow down when/where the conflict is.

[davidjade]

It seems to be fixed and I am not sure why. luckily I kept a detailed log of every change I made to the system and that allowed me to rebuild the image from scratch, more or less. In my newly built image, zram-config seems to work and sync at shutdown reliably. I've tested it quite a few times after each package install or config change I made. So it must have been something on the old image that I cannot reproduce with a new system install - sorry about that. I'll keep checking and if I see it again I'll try and figure out what breaks it. So I think this bug can be closed.

[ecdye]

Great, I'm glad you were able to fix the issue. Let me know if you are ever able to find the cause and I'll investigate it some more.