read-only zram fs IN operations ??

[mstormi]

My openhab logging to files stopped yesterday for no apparent reason such as disk full. OH logging via log:tail in Karaf console keeps working and I believe the reason is in zram. I guess the trigger was just the logfile about to be rotated.

"Dateisystem ist nur lesbar" means "read-only filesystem"

[12:09:57] root@mysmarthouse:/var/log/openhab# ls -l insgesamt 241236 -rw-r--r-- 1 openhab openhabian 0 Nov 28 14:05 audit.log -rw-r--r-- 1 openhab openhabian 13160053 Jan 10 11:40 events.log -rw-r--r-- 1 openhab openhabian 16777234 Jan 6 13:17 events.log.1 -rw-r--r-- 1 openhab openhabian 16777278 Jan 6 22:50 events.log.2 -rw-r--r-- 1 openhab openhabian 16777237 Jan 7 13:52 events.log.3 -rw-r--r-- 1 openhab openhabian 16777255 Jan 8 01:06 events.log.4 -rw-r--r-- 1 openhab openhabian 16777303 Jan 8 12:25 events.log.5 -rw-r--r-- 1 openhab openhabian 16777334 Jan 8 21:15 events.log.6 -rw-r--r-- 1 openhab openhabian 16777322 Jan 9 13:07 events.log.7 -rw-r--r-- 1 openhab openhabian 16572273 Jan 10 11:40 openhab.log -rw-r--r-- 1 openhab openhabian 17157184 Dez 20 18:05 openhab.log.1 -rw-r--r-- 1 openhab openhabian 12080636 Dez 20 18:08 openhab.log.2 -rw-r--r-- 1 openhab openhabian 3400040 Dez 20 19:28 openhab.log.3 -rw-r--r-- 1 openhab openhabian 16777253 Jan 1 18:50 openhab.log.4 -rw-r--r-- 1 openhab openhabian 16777316 Jan 3 22:35 openhab.log.5 -rw-r--r-- 1 openhab openhabian 16777281 Jan 6 00:48 openhab.log.6 -rw-r--r-- 1 openhab openhabian 16777220 Jan 7 23:03 openhab.log.7 [12:09:58] root@mysmarthouse:/var/log/openhab# mv openhab.log.? events.log.? /volatile/temp/ mv: das Entfernen von 'openhab.log.1' ist nicht möglich: Das Dateisystem ist nur lesbar mv: das Entfernen von 'openhab.log.2' ist nicht möglich: Das Dateisystem ist nur lesbar mv: das Entfernen von 'openhab.log.3' ist nicht möglich: Das Dateisystem ist nur lesbar mv: das Entfernen von 'openhab.log.4' ist nicht möglich: Das Dateisystem ist nur lesbar mv: das Entfernen von 'openhab.log.5' ist nicht möglich: Das Dateisystem ist nur lesbar mv: das Entfernen von 'openhab.log.6' ist nicht möglich: Das Dateisystem ist nur lesbar mv: das Entfernen von 'openhab.log.7' ist nicht möglich: Das Dateisystem ist nur lesbar [12:10:28] root@mysmarthouse:/var/log/openhab# touch xyz touch: 'xyz' kann nicht berührt werden: Das Dateisystem ist nur lesbar [12:16:35] root@mysmarthouse:/var/log/openhab# cd .. [12:18:05] root@mysmarthouse:/var/log# touch xyz touch: 'xyz' kann nicht berührt werden: Das Dateisystem ist nur lesbar [12:18:08] root@mysmarthouse:/var/log# zramctl NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT /dev/zram2 zstd 600M 407,9M 77,6M 200M 4 /opt/zram/zram2 /dev/zram1 zstd 1000M 381,9M 6,7M 186M 4 /opt/zram/zram1 /dev/zram0 lzo-rle 600M 182,3M 42,8M 76,2M 4 [SWAP]

[13:43:53] root@mysmarthouse:/var/log# mount -o remount,rw /var/log [13:44:31] root@mysmarthouse:/var/log# mount|grep log /dev/mmcblk0p2 on /opt/zram/log.bind type ext4 (rw,noatime) overlay2 on /var/log type overlay (rw,relatime,lowerdir=/opt/zram/log.bind,upperdir=/opt/zram/zram2/upper,workdir=/opt/zram/zram2/workdir,redirect_dir=on) [13:44:34] root@mysmarthouse:/var/log# mount -o remount,rw overlay2 [13:45:00] root@mysmarthouse:/var/log# mount -o remount,rw /dev/mmcblk0p2 [13:45:12] root@mysmarthouse:/var/log# touch x touch: 'x' kann nicht berührt werden: Das Dateisystem ist nur lesbar

note it is not the lowerfs but the upperfs: [13:52:52] root@mysmarthouse:/var/log# touch /opt/zram/log.bind/x [13:52:52] root@mysmarthouse:/var/log# touch /opt/zram/zram2/upper/x touch: '/opt/zram/zram2/upper/x' kann nicht berührt werden: Das Dateisystem ist nur lesbar [13:55:14] root@mysmarthouse:/var/log# mount -o remount,rw /dev/zram2 /opt/zram/zram2/upper mount: /opt/zram/zram2/upper: mount point not mounted or bad option. [14:14:17] root@mysmarthouse:/var/log# mount -o remount,rw /dev/zram2 /opt/zram/zram2 mount: /opt/zram/zram2: cannot remount /dev/zram2 read-write, is write-protected. [14:14:21] root@mysmarthouse:/var/log#

[mstormi]

totally unrelated just fyi: https://github.com/systemd/zram-generator

[ecdye]

If it is read only that means that it is full. That is default behavior on linux for a full filesystem.

[mstormi]

There can be other reasons. It isn't nowhere full, see zramctl output and df shows 64% at most. I kept the state if you have more ideas to validate but full it isn't

[ecdye]

Have you recently restarted zram without rebooting, and are you on the latest version of zram?

[mstormi]

Not sure if I did but for sure not recently in time with the issue to show up. Installation is some weeks young but as there aren't versions I'm not sure if it's latest. Anything particular you want to check for ? BTW had to reboot the machine now the state is gone.

[ecdye]

TBH, outside of filling up the device I'm not sure what would cause that. Perhaps it was just a weird Linux bug.