Putting /tmp/, /run/, and /var/tmp/ on zram?

[jonesmz]

I needed to install a package that wasn't available in a binhost, and it took over 12 hours to compile and run tests. I think a large amount of the problem was disk access.

Thoughts on mounting /tmp/, /run/, and /var/tmp/ on zram block devices?

[AlexMikhalev]

@jonesmz do you have example configuration described in blog or doc? I am keen to implement at least as the script.

[jonesmz]

I just installed the 'zram-init' ebuild, and then ran systemctl enable --now zram_swap zram_tmp zram_var_tmp

You're also supposed to edit /boot/cmdline.txt and /etc/modprob.d/zram.conf, but i'm not sure how necessary that is.

[AlexMikhalev]

Seems like zswap already part of the image: https://github.com/sakaki-/genpi64-overlay/tree/master/sys-apps/rpi3-zswap Did you notice any performance improvements after enabling zram init?

[jonesmz]

zswap is not the same as zram.

Yes, enabling zram_var_tmp speeds up small package compilations enormously.

[AlexMikhalev]

Looking at write up here zswap seems to be more appropriate for Rpi 3/4. Have you tried zram init vs putting /var/tmp/ on tmpfs?

[jonesmz]

Why is compressed in-memory "swap" more appropriate for rpi3/4 for purposes of reducing disk access to /var/tmp/ while emerging packages?

I used zram via zram-init for /var/tmp/. I did not bother with tmpfs because the rpi3 and rpi4 are memory constrained.

[perkinslr]

ZSwap can only be safely used in fairly limited circumstances. It does not write compressed pages to the swap device. This means that, once the zswap cache is full, it must uncompress the LRU page, which can up the memory pressure and cause a nasty decompression cascade as it tries to dump a significant portion of the zswap to the underlying device.

The reason this is an insurmountable problem with zswap is that the zswap device must be able to be removed on-the-fly, which means it cannot intercede on the de-caching of pages from the real swap devices. Basically, it's a dead-end for most use cases.

ZRAM acts as an independent swap device, it won't evict pages, it'll just stop accepting new ones. It can refuse to accept incompressible pages, forcing them to get written to a lower priority swap device, but that doesn't cause the decompression cascade as those pages were never compressed.

Actually, zram acts as an independent block device, so it can be mounted on the filesystem, not just used for swap.

Additionally, zram can manage its own backing device (the option for that is disabled in your kernel config). Unlike zswap, this backing device does not appear as a swap target, so compressed pages can safely be written directly to it. There are a couple of issues with using a backing device. For one thing, it must be a block device, not a file. This can be handled via the loopback system, but comes with an I/O penalty. For the other, major, issue, it puts the zram device into async mode. This means writes to it silently (to the program, they show in dmesg) fail if the device fills up. Also, the user must explicitly trigger writeback to the backing device, which can take a while, and does not cause the device to block on I/O, so if you dump enough data fast enough, you can cause the device to choke.

Anyway, zram without a backing device works amazingly well, without adding writes to your flash storage. For compiling on gentoo, it usually manages about a 3:1 ratio. For this use case, zram-init works fine, you just set it up in /etc/conf.d/zram-init, add /etc/init.d/zram-init to the default runlevel, and you're good to go.

For zram with a backing device, I found zram-init to be a poor choice. For one thing, you need to create the loopback device if you are using a file, which you could do via an /etc/init.d/loop0 script or similar, but that's not great. For another, you need some way to start and stop jobs to give the slower backing storage time to catch up when you are dumping lots of data to zram. And finally, zram swap can still choke the whole system performance if the program using all the memory isn't isolated to its own memory cgroup.

All that said, I have a late-alpha, early-beta script I use to manage zram, with a backing device, and automated cgroup. I wrote it in a single pass in about half an hour, it really should be rewritten to be cleaner, but I was under a time constraint and haven't had time to clean it up. (Specifically, it should really be rewritten to use context managers, as it stands, it can leave a mess if it crashes during the setup phase). It's available here: https://gist.github.com/perkinslr/79622b10dae66050eae22f5e62dcd6e4

[sakaki-]

30 Oct 2020: sadly, due legal obligations arising from a recent change in my 'real world' job, I must announce I am standing down as maintainer of this project with immediate effect. For the meantime, I will leave the repo up (for historical interest, and since the images may be of use still in certain applications); however, there will be no further updates to the underlying binhost etc., nor will I be accepting / actioning further pull requests or bug reports from this point. Email requests for support will also have to be politely declined, so, please treat this as an effective EOL notice.

For further details, please see my post here.

Many thanks for your interest in this project!

With sincere apologies, sakaki ><