Add swap space in installed system

[probonopd]

Add swap in installed system

I noticed that the system can lock up in low-RAM situations, which can happen when editing video with Shotcut while running a web browser on a machine with 4 GB RAM. Hence we should add some swap space by default.

One way would be to use a swap partition/dataset. This would probably be the cleanest way.

Another way would be to add a swap file along those lines. This has the advantage that it can be done retroactively without having to repartition the disk. (But thinking about ZFS, one can probably also retroactively add a dedicated dataset for swap without having to repartition...)

# See https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/adding-swap-space.html

# TODO: Check whether we have 8192 MB free and only continue if so

# TODO: Check whether swap0 is already mentioned in /etc/fstab and only continue if it is not

# Create swap file
dd if=/dev/zero of=/usr/swap0 bs=1m count=8192
chmod 0600 /usr/swap0

# Inform the system about the swap file
cat >> /etc/fstab <<\EOF
md99    none    swap    sw,file=/usr/swap0,late 0   0
EOF

# Add swap space immediately
swapon -aL

What is the best practice when it comes to swap on ZFS?

Windows automatically uses dynamically growing swap files (I think). What does the Mac do?

[probonopd]

Indeed the "crashes" I had experienced before had to do with the lack of swap:

/var/log/messages:Jan 21 18:35:23 Users-RevoOne-RL85 kernel: pid 4043 (java), jid 0, uid 1001, was killed: out of swap space
/var/log/messages:Jan 21 19:57:27 Users-RevoOne-RL85 kernel: pid 5642 (shotcut), jid 0, uid 1001, was killed: out of swap space
/var/log/messages:Jan 21 20:22:53 Users-RevoOne-RL85 kernel: pid 5662 (shotcut), jid 0, uid 1001, was killed: out of swap space

Somehow we must bring messges like these to the user's attention, probably through notifications.

[ctsrc]

Some things to note, as said by one of the comments at https://news.ycombinator.com/item?id=25789809

Swap is not meant to enable you to escape the memory demands of your workload. If your workload requires more physical memory than you have, then swap will probably not help you.

However just because your software allocates data in memory doesn't mean it needs to be there. Rarely used memory could be swapped out to cache more frequently used files on disk instead, increasing the overall throughput of the system. This could be true whether or not you have an excess of physical memory for your workload.

Personally I run both my desktop and my server without swap.

Running out of memory means that either your hardware has too little RAM and you should buy more, or that your software is not tuned to the hardware that it is running on.

Using swap as a band-aid for this problem is not a good solution IMO.

Excessive swapping wears out your SSDs and it makes work that should be fast on a system that has enough RAM for it or with right tuning of the sotware, turns into a slow and frustrating experience.

I recommend against enabling swapping by default. It's better IMO to have software that consume too much memory actually be killed, so that users will know that they either need to buy more RAM or to modify or replace the software they are using or otherwise adapting the tasks that they are trying to do by using other means of getting them done.

[probonopd]

Thanks for the background information @ctsrc. I agree that just adding more RAM would be the best option, but it is not always feasible, especially with the kind of machines we are targeting (e.g., Macs that are no longer supported by the vendor).

It seems to me that enabling swap results in a much more stable system where processes don't just get killed. Even though the swap space is almost entirely unused.

So for me, adding swap greatly improved things...

More opinions wanted. What is your perspective @vermaden?

[probonopd]

https://klarasystems.com/articles/exploring-swap-on-freebsd/

Should I enable swap in 2021?

We believe it is worthwhile for like-minded FreeBSD users to explore enabling swapping again. NVMe drives are commonplace and have access latencies in the tens of microseconds, several orders of magnitude smaller than what was standard just a decade ago. FreeBSD’s performance under memory pressure has continued to improve and the kernel is strenuously stress-tested on a continuous basis. Applications especially sensitive to memory access latency can be wired using mlock(2), while the system as a whole may benefit from the improved memory efficiency that swapping can provide.

[vermaden]

More opinions wanted. What is your perspective @vermaden?

In the old times when 512 MB or 1 GB RAM was 'quite large' I believed in the oldschool 2*RAM rule so for 1 GB RAM I would use 2 GB of swap space.

Today my laptop has 16 GB of RAM and I can upgrade it to 32 GB RAM when needed.

On that 16 GB RAM system I use 1 GB of swap space (on raw GPT partition - not on ZFS) and its used 0% for most of the time. Sometimes it 'hits' 1% but nothing more.

I would not waste 32 GB of SSD space for that 2*RAM rule when I have 16 GB RAM and 1 GB swap space is almost always at 0% but 512 MB to 1 GB of 'just in case' swap space should be more then enough IMHO.

Hope that helps.

[probonopd]

Thanks @vermaden for commenting.

On that 16 GB RAM system I use 1 GB of swap space (on raw GPT partition - not on ZFS) and its used 0% for most of the time. Sometimes it 'hits' 1% but nothing more.

Please elaborate the logic behind this. During my tests I had the suspicion that swap space helps the system even though very little of it - or nothing at all - actually gets used. Is this suspicion correct? Or why else would you want to up your space from 16 to 17?

[vermaden]

Its 'just in case' there.

I do not have anything against swap if some not now needed data can be put on swap and make RAM free.

[grahamperrin]

For privacy and security: prefer encrypted swap.

… oldschool 2*RAM rule …

Thanks, I was previously unaware of that old-school rule. x2 is what I settled upon, through experimentation, a couple of years ago, after (more than once) finding FreeBSD's default to be far too small.

At some point, probably when I began using a more powerful notebook, I seem to have settled upon 16 GB real + 16 GB swap i.e. x1. If I recall correctly, some types of software build were tricky if there was less swap available.

A few minutes ago, with around 7 GB of my 16 GB swap partition in use (internal hard disk drive, OpenZFS), whilst restoring two virtual machines from a mobile hard disk drive on USB (3 GB memory allowed for Windows, 4 GB for helloSystem) and building x11/screengrab with two of four CPUs:

– and whilst those things progress, I'm typing this in Firefox with scores of extensions enabled.

More useful than (simply) observing the amount of swap in use, output from:

gstat -p

As expected, the combined memory requirements of those two virtual machines was, on top of everything else, too much, application performance suffered, so I closed the Windows VM. Note the 'red alerts' in the gstat window:

YMMV.

[probonopd]

/me wonders what this monstrous thing is

It looks like a Hamburger menu that wants revenge for my post about hamburger menus ;-)

[grahamperrin]

Not revenge :-) but coincidentally it's no longer necessary – https://matrix.to/#/!EKNFbsWSwXpDOGLRex:matrix.org/$QvRs_gkC-1D67x1nQnfR22V-p0w6pUTDJ_QSqfOdpo8?via=matrix.org&via=fosdem.org&via=t2bot.io

[grahamperrin]

Hamburgers aside,

… x2 is what I settled upon, through experimentation, a couple of years ago, after (more than once) finding FreeBSD's default to be far too small.

Thinking back … at the time I might have had only 8 GB real memory in the notebook, with a preference for 16 GB swap. At some point I doubled the amount of real memory, and either:

• found no need to increase the swap; or
• more likely, I couldn't be bothered to reformat the drive.

In any case: 16 GB swap seems good for me. I've had almost all real + all swap used on numerous occasions. Exhaustion (typically leading to an automated kill of Firefox) is extremely rare. YMMV.

[crees]

Agreed, you won't get automated kills unless you do run out of memory. This is the advantage of having swap-- you get the clue of the system slowing down hugely. I think it's madness to have windows evaporating in front of your eyes, when no other OS behaves like this by default.

No operating system disables swap out of sympathy for the SSDs afaik- it's just what happens with them, and they are far cheaper than they used to be, so this is less of an issue. The performance is also much better.

My suggestion would probably be a swap file, for the flexibility mentioned, and also because the user can easily see the size of the file if they were curious about it. It also has the advantage of being much easier to resize.

Use truncate rather than dd to create the swapfile.

In terms of size... twice the memory up to a maximum of 16 GB as a straw man. It's perfectly possible to allocate extra files if necessary.

[grahamperrin]

Thanks.

Incidentally, (although I don't see any fit for this with the current design of helloSystem): sysutils/swapexd

[probonopd]

Agreed, you won't get automated kills unless you do run out of memory. This is the advantage of having swap-- you get the clue of the system slowing down hugely. I think it's madness to have windows evaporating in front of your eyes, when no other OS behaves like this by default.

Full ack.

It would be nice if we could give the user a notification in case RAM and/or even swap is running close to full. (On a 4GB system, having 10-20 tabs open in the Falkon browser can lead to this situation for me - is this normal? I wonder whether Falkon is smart enough to purge the RAM for the background tabs...)

[grahamperrin]

https://github.com/helloSystem/ISO/issues/109#issue-791447052

… lock up in low-RAM situations, …

If ever you want a kernel panic in an out of memory situation, set vm.panic_on_oom to 1 or greater.

[probonopd]

Actually I'd like the largest process to get killed as it is now, but then give some information on screen about what just happened ("The application foo was closed because there is not enough memory"). I think this is how other systems do it?

[grahamperrin]

From https://lists.freebsd.org/pipermail/freebsd-questions/2014-October/261962.html

… /var/log/messages should be listing exactly what was killed and when. It's probably on the console too.

Example messages: Out of memory | The FreeBSD Forums

[probonopd]

So possbily we should be running a daemon that watches that file all the time and triggers a dialog box whenever a OOM event happened? I wonder whether this can be done without needing just another background process

[probonopd]

Received this piece of advice:

Regarding helloSystem issue https://github.com/helloSystem/ISO/issues/109 where the question is asked whether a swap file or a swap partition should be used, see https://lists.freebsd.org/pipermail/freebsd-hackers/2021-March/057046.html

As result, swapfile swapping is more prone to the trivial and unavoidable deadlocks where the pagedaemon thread, which produces free memory, needs more free memory to make a progress. Swap write on the raw partition over simple partitioning scheme directly over HBA are usually safe, while e.g. zfs over geli over umass is the worst construction.

Based on that information I would avoid using a swap file or a swap ZFS volume.

[grahamperrin]

In context: http://freebsd.1045724.x6.nabble.com/The-out-of-swap-killer-makes-poor-choices-td6449386.html

Do we need to distinguish between a swap file and a swap partition?

No problems here with a GELI-encrypted swap partition.

(I'll test with swap disabled … when I last did so, the experience was poor …)

[probonopd]

The problem is with partitioning. Users hate partitioning. Users are afraid of it. Because users can lose data.

So if we really need a swap partition, the installer should set one up right from the start.

[grahamperrin]

the installer should set one up

👍

[probonopd]

As the consensus of experts seems to be that a partition is preferable to files,

the installer should set one up

indeed.

• Would probably be easy by changing to something like ZFSBOOT_SWAP_SIZE="8G" in furybsd-install
• Might want to do this under certain conditions only (e.g., target disk is large enough and physical RAM is small, target disk is not a removable USB device,...)
• Should also adjust the required disk space that the graphical installer interface shows accordingly

[probonopd]

On a related note, I am tempted to do enable RAM compression, something like this:

#!/bin/sh

# Exit if RAM compression is already on
sysctl kern.geom.conftxt | grep md7 >/dev/null && exit 0

# Create a ramdisk that is compressed
mdconfig -a -t malloc -o compress -o reserve -s 512m -u 7

# Use the compressed ramdisk as swap space
swapon /dev/md7

# Show information
swapinfo -m

Instead of 512m we should dynamically find out how much free RAM there is, and use that (or a portion of it).

[grahamperrin]

When might swap become a feature? It's a significant drawback.

https://github.com/helloSystem/ISO/issues/109#issue-791447052

… ZFS, one can probably also retroactively add a dedicated dataset for swap …

Probably not a good idea; see for example https://github.com/openzfs/zfs/issues/7734

[probonopd]

Does GhostBSD use RAM compression of some sort?

   Mem: 69M Active, 1028K Inact, 20M Laundry, 615M Wired, 69M Free
   ARC: 45M Total, 7383K MFU, 22M MRU, 256K Anon, 428K Header, 15M Other
        4448K Compressed, 27M Uncompressed, 6.21:1 Ratio

[ctsrc]

Also pretty interesting how macOS does it. According to https://forums.macrumors.com/threads/mac-not-writing-swap-files.2219502/

Usually, on macos swap files are dynamic and will be created and deleted as needed

and

The dynamic_pager daemon manages swap files and is run on demand

[ctsrc]

Also http://newosxbook.com/MOXiI.pdf (17MB PDF), via http://newosxbook.com/1stEdIsFree.html

Chapter 12 "Commit to memory: mach virtual memory" has a subsection titled "Paging Policy Management", which includes a little bit of text about dynamic_pager of macOS.

However, that edition of the book came out in 2013 and I don't know how much of the book is still current. Other books which may be more up to date (idk, haven't read em) are available for purchase from the main page of the same site where the author gives the 1st edition away, linked in this comment.

[probonopd]

Hi @ctsrc, interesting reading. Added that book to https://github.com/helloSystem/hello/wiki#must-read.

Now, do you happen to know how to achieve a similar logic on FreeBSD?

[grahamperrin]

Gentle bump. The complete absence of swap is astonishing:

[probonopd]

So if we really need a swap partition, the installer should set one up right from the start.

Do you know how to do do it?

Could it possibly be as easy as changing

https://github.com/helloSystem/ISO/blob/93cb8e31dcbc978307c0e3c419b333e01c84a85f/overlays/uzip/furybsd-live-settings/files/usr/local/bin/furybsd-install#L88

to a reasonable value? What would be a "reasonable value"?

Maybe if the target disk is larger than 32 GB and is not a removable device, then set

export ZFSBOOT_SWAP_SIZE="8g"

to prevent swap from being created on USB sticks and microSD cards.

[grahamperrin]

Does GhostBSD use RAM compression of some sort?

   Mem: 69M Active, 1028K Inact, 20M Laundry, 615M Wired, 69M Free
   ARC: 45M Total, 7383K MFU, 22M MRU, 256K Anon, 428K Header, 15M Other
        4448K Compressed, 27M Uncompressed, 6.21:1 Ratio

From top(1) DESCRIPTION OF MEMORY:

   ZFS ARC Stats
     These stats are only displayed when the ARC is in use.

[grahamperrin]

… What would be a "reasonable value"?

https://github.com/helloSystem/ISO/issues/109#issuecomment-765698107 above from @vermaden. Plus, in FreeBSD Forums:

• Solved - How Much Swap (2021-04-13)
• Swap battle stories (2021-04-30)
• Solved - Swap - Is it still really necessary? (2022-05-13).

Reading the latter for the first time, today, I see a case where 10 G swap was insufficient for large PDF files in TeXstudio on a computer with 16 G RAM.

https://old.reddit.com/r/freebsd/comments/tnvkoz/-/i25hubu/?context=2 there's a more extreme example. And so on.

---

I habitually prefer 16 G swap when installing FreeBSD.

On my everyday notebook (16 G RAM + 16 G swap), I'm very frequently glad that I didn't restrict myself to less.

Note:

Maybe if the target disk is larger than 32 GB and …

– I would not give 16 GB of a ~33 GB device to swap.

[grahamperrin]

… It would be nice if we could give the user a notification in case RAM and/or even swap is running close to full. …

Maybe an option to notify, when swap capacity (e.g. as shown by swapinfo (pstat(8)) exceeds a certain percentage, however:

• I should not enable the option by default.

Alarms might create more questions than answers. Too few people are aware of relevant sysctls, and so on.

% uptime ; swapinfo -h
 8:12a.m.  up  7:57, 7 users, load averages: 3.06, 2.70, 2.49
Device              Size     Used    Avail Capacity
/dev/ada0p2.eli      16G     9.3G     6.7G    58%
% 

[probonopd]

With export ZFSBOOT_SWAP_SIZE=8g, just hangs on

==> /tmp/Installer.err <==
+ export 'ZFSBOOT_BOOTFS_NAME=default'
+ export 'ZFSBOOT_DATASET_NESTING=1'
+ export 'BOOT_BLOCKS_FROM_DISTSET=1'
+ export 'ZFSBOOT_VDEV_TYPE=stripe'
+ export 'ZFSBOOT_SWAP_SIZE=8g'
+ export 'ZFSBOOT_CONFIRM_LAYOUT=1'
+ export 'nonInteractive=YES'
+ export 'ZFSBOOT_BOOT_TYPE=BIOS+UEFI'
+ env
+ bsdinstall zfsboot

[probonopd]

Argh. There is something strange going on in bsdinstall zfsboot.

When we set ZFSBOOT_SWAP_SIZE=0, it works

When we set ZFSBOOT_SWAP_SIZE=16g, it just stalls and does not continue. Possibly because our virtual vmdk disk is not large enough, or because formatting so much hard disk takes a long time?

Possibly we should disable swap on VMs by default...

When we set ZFSBOOT_SWAP_SIZE=16000, we are back to

kern.geom.part.mbr.enforce_chs: 0 -> 0
kern.geom.part.mbr.enforce_chs: 0 -> 0
kern.geom.part.mbr.enforce_chs: 0 -> 0
kern.geom.part.mbr.enforce_chs: 0 -> 0
kern.geom.part.mbr.enforce_chs: 0 -> 0

Need to retest with real hardware.

[probonopd]

Actually, the problems do not occur on real hardware with a large enough disk. So for now, let's enable using a 16 GB swap partition only if the target disk is at least 80 GB in size (yes, I know, it's somewhat arbitrary... but better than no default logic at all).

Maybe the

kern.geom.part.mbr.enforce_chs: 0 -> 0
kern.geom.part.mbr.enforce_chs: 0 -> 0
kern.geom.part.mbr.enforce_chs: 0 -> 0

message is totally misleading and just means "not enough space on disk"?

[probonopd]

Swap is working in helloSystem 0.8.1.