Question about ram usage

[mczerski]

Hi, its not an issue, I just want to ask a question to know if this project will fulfill my needs. Currently i'm using overlayfs with tmpfs upper dir and read only lower dir on raspberry pi. The reason is to lower sd card wear. But after long uptime, the tmpfs is eating all the ram and i cannot find solution for that. Does this project adress this issue ? And if yes, how ?

[StuartIanNaylor]

Already does the same but with zram so the compression will make it last approx 3x longer. sudo zram-config enable-ephemeral on reboot will load the whole root into zram / overlayfs. If you change the alg to zlib or zstd if enabled then 4x approx maybe more.

Otherwise you can just create specific zdir for high write needs via /etc/ztab.

One thing is I am trying to understand tmpfs as what you are saying shouldn't happen as it should push out old idle pages to swap. So that only frequent writes stay in tmpfs but aged static mem hoggers gets pushed to a swap. That should work and actually you shouldn't run out of tmpfs but may have to play with swapiness to lower write activity. Its actually a really good method surpassing ram limits whilst using swapiness to dictate that yes there are writes but of extremely low level with static idle files.

With overlayfs being CoW the copy up to tmpfs should only be for writes which are often just file modifications or appends so actually small amounts go a long way. You can increase this via zram compression but I haven't enabled the writeback cache that tmpfs has by default. On raspbian for some reason the WB cache has not been enabled even if you have to manually manipulate /proc/sys to enable it. So it isn't in the kernel default so haven't bothered, but why its disabled is a mystery.

[mczerski]

The way I have configured my system is that the root is read only without swap. Only /var directory is mounted as overlayfs with tmpfs on top. I thing that using swap with sd card is dangerous because high activity on swap will cause large wear on sd card. The problem I had with this setup was that after long run there was a lot log files rotated by logrotate in /var/log and every rotation causes all log files to rename so all goes to upper layer :/ The other thing in my setup is regular syncs of the overlayed /var directory back to the sdcard and reboot (of caurse i need to remout the filesystem rw). But with logrotate problem it is not enough. For now i fixed it by changing logrotate policy of file names (include date instead of sequential number).

So I thing that the fundamental problem with overlayfs + tmpfs is that it requires the amount of ram that the oerlayed directory occupies on the hard drive to be fully reliable. If there was a Way to sync back data from upper to lower without the need for reboot it probably solve my problem.

[StuartIanNaylor]

zram-swap is compressed memory swap it doesn't access disk as unlike tmpfs it doesn't have a write-back cache auto-enabled. There is no such thing as compressed system memory but there is near memcpy speed compressed memory swap that tmpfs and system will use and garner minimums depending on alg used 300% for LZs and approx 400% for zlib / zstd where there overhead on cpu is vastly outweighed my the memory availability. You are crippling both system and tmpfs by not applying a single large zram swap with high swappinness of 80-100.

An overlayfs merge so far can only take place when offline by the only existing tool its so strange to have a kernel inclusion that has absolutely zero supporting tools. I would of used Aufs which has but in most distro's its not included and needs a recompile or extra linux headers. I am not really sure about the politics and rationale behind aufs but there is a concerted effort for it to be depreciated so actually overlayfs is a hobsons choice than actual choice. I think you can do an online merge before shutdown but you can not have any file changes or access on the files that have or are being merged. That is why there is a ro ephemeral mode that can be enabled or disabled or you can select individual directories via ztab as much of the system is static and really doesn't need to be part of the ro overlay.

Logrotate prob will but your tmpf mem allocation must be very meagre or your starting the initial ro setup with a lot of unnecessary logs? If you have a look at the zlog part of zram-conf it includes a log-rotate directive as the default will not move logs off a singular device, which I am glad you mentioned as either I forgot or somehow with git lost changes that just got added back from log2zram. As well as shipping to persistent only live logs are retained and all logs in olddir are old so its a convient way to identify and do whatever you wish with those. The zram-config.logrotate is just prefixed with 00_ so it runs first and those are global directives so will be applied to all that come after.

olddir /opt/zram/log.old
createolddir 755 root root
renamecopy

Really you should have loads of space but what you can do is ship logs to persistent either a usb or network mount using the olddir as I do with zlogs as it picks a persistant folder rather than zram.

The ephermal command is the script that will be installed to /usr/local/share/zram-config/ro-root.sh edit these lines for memsize of write area

echo "lz4" > /sys/block/zram0/comp_algorithm
echo "1000M" > /sys/block/zram0/disksize
echo "333M" > /sys/block/zram0/mem_limit

lz4 usually doesn't dip below 300% compression and the effective disksize is always a bit of guesswork and I err low than high. If the alg is changed to "deflate" prob will not dip below 400% but you could have highly compressed files that get minimal compression so zram hard control is done by actual mem_limit even if it does seem slightly back to front. In the above we are talking 1gb of write space and I am wondering how many logs you have on start?

I would also set a zram swap device with something similar on the 1gb pi and adjust by half on the 512mb versions. That should give you a lot of room for manoeuvre and you have a choice of either zram or tmpf but both should be accompanied by non disk based zram swap.