Support KSM

[ktsaou]

I don't know if this the right place for this request.

Given the limited memory of a raspberry Pi, I suggest to provide compiled kernels with CONFIG_KSM=y

KSM is a memory de-duplication feature, for applications willing to use it. More about it here.

netdata uses this to lower its memory requirements. Other applications may use it too.

[popcornmix]

Have you tried this? An option like this has the possibility to harm performance:

KSM's scans may use a lot of processing power: some installations will disable KSM for that reason.

so we won't enable this without strong evidence that the benefits outweigh the costs.

[ktsaou]

When KSM is enabled at the kernel is just available for the user to enable it.

So, if you build a kernel with CONFIG_KSM=y you will just get a few files in /sys/kernel/mm/ksm. Nothing else happens. There is no performance penalty (apart I guess from the memory this code occupies into the kernel).

The files that CONFIG_KSM=y offers include:

• /sys/kernel/mm/ksm/run by default 0. You have to set this to 1 for the kernel to spawn ksmd.
• /sys/kernel/mm/ksm/sleep_millisecs, by default 20. The frequency ksmd should evaluate memory for deduplication.
• /sys/kernel/mm/ksm/pages_to_scan, by default 100. The amount of pages ksmd will evaluate on each run.

So, by default ksmd is just disabled. It will not harm performance and the user/admin can control the CPU resources he/she is willing ksmd to use.

For netdata for example, I use sleep_millisecs=1000, pages_to_scan=100. With these settings ksmd does not even appear in the running process list.

[popcornmix]

So what is the effect for you when enabled? Are you seeing increased free memory with this enabled?

[ktsaou]

It depends on the application.

The applications opt-in / register memory for deduplication.

For netdata, you can have 50% to 60% memory savings.

Check this:

[ktsaou]

And these the CPU resources it uses:

As you can see, 1% CPU every a few seconds.

[ktsaou]

The effect on netdata is so high, because netdata registers its in-memory round robin database. This database stores the values collected from the system. As you understand there are many metrics that are constant, or just zero.

Deduplication happens at the page level (4KB), and netdata uses by default 1 hour of data filled with 32bit numbers, so each metric is about 3,5 pages. If a metric is constant, 3 of these pages will be merged to 1, saving 2. If more metrics have the same constant value (zeros for example), then even more pages will be merged.

In the chart I posted above, you can see that sharing = 116MB merged to shared = 1,65MB, while unshared = 87MB were unique and could not be merged with anything else.

[P33M]

Do these applications ever encounter hard memory constraints in realistic scenarios (i.e. OOM-kills) without KSM enabled?

[ktsaou]

Do these applications ever encounter hard memory constraints in realistic scenarios (i.e. OOM-kills) without KSM enabled?

netdata can be sized according to your needs. The user / admin can configure the size of the in-memory round robin database. By default it will use about 15-20 MB. Even at this size however, KSM will offer 25-30% savings.

I am sure you will find netdata worths the trouble. Check it at http://netdata.firehol.org. It is good!

[Ruffio]

@ktsaou has your issue been resolved? If so, please close this issue. Thanks.

[ktsaou]

This is not resolved. KSM is still missing in rpi kernels.

[abpostelnicu]

I agree on this one KSM should be available in the default raspbian kernel. It would be very helpfl for monitoring application such as netdata

[ktsaou]

Do these applications ever encounter hard memory constraints in realistic scenarios (i.e. OOM-kills) without KSM enabled?

and keep in mind netdata adjusts its OOM score to 1000, so it is the first to be killed in case the system starves for memory.

Enabling KSM for RPi will really help a lot of people out there.

netdata is booming, we have 45k installations in the last 7 months (http://my-netdata.io). netdata is included on home automation systems (https://home-assistant.io/blog/2016/12/17/text-to-speech-aquostv-flic-zamg/?platform=hootsuite) and all kinds of embedded applications, due to its efficiency.

So, please give us KSM on RPi.

Thanks!

[sfx2000]

No impact to enable the item - it's default is not to run, so the user has to explicitly turn it on.

Saves a lot of time not having to build a custom kernel..

[sfx2000]

I'll update here - should be fine on Pi2/Pi3 for the arm7 kernel - I don't have a Pi1/PiZero/CM so can't say much about ARM6, but again, this is low risk - as one does need to enable it first, and even then, only code that is KSM aware will use it. Most folks will never know it's there, but for those that are looking to develop around it, or teach others, it's a good thing to have.

More info here - https://www.kernel.org/doc/Documentation/vm/ksm.txt

Those who are exploring things like containers and VM's can benefit here - and even Android has this option to enable/disable KSM.

[ktsaou]

Just to add: it is useful on RPi 1 too.

[sfx2000]

Won't say anything about ARM6 -- localizing this to the A7/A53 (e.g. Pi2/Pi3) makes sense..

edit...

Anyways - I'm ok with the potential hit...

Nice explanation about memory management here -

http://events.linuxfoundation.org/sites/events/files/slides/Efficient_Memory_Management_on_Mobile_Devices_0.pdf

[jbenz]

Any update on this consideration?

[JamesH65]

Ping @popcornmix @pelwell

[sfx2000]

Seems like this has request has been pushed back/deprecated...

Again - enabling this on the kernel default config itself, just makes it available... one still has to explicitly enable this feature on run-time - most commonly is via rc.local, if not done, it's not active...

For for multiple process tasks - esp. with python, there's a benefit to doing this...

for testing - once the kernel module is enable, one has to activate it...

/etc/rc.local - this is normally the place.

echo 1 > /sys/kernel/mm/ksm/run echo 1000 > /sys/kernel/mm/ksm/sleep_millisecs

Combined with Zram, KSM can help out quite a bit...

[Ferroin]

@sfx2000 KSM requires an opt-in both from the sysadmin (/sys/kernel/mm/ksm/run), and from the application itself (madvise(MADV_MERGABLE)). Just running multiple copies of the same thing isn't going to get you any benefit unless that application makes the required madvise call. AFAIK, the only two widespread applications that have that call are QEMU and netdata. IOW, unless someone is running one of those (and multiple copies of QEMU in most cases if they're using that), or has some custom app that makes the needed madvise calls, they won't see any benefit.

[sfx2000]

Understood - let me manage my code... ksm works for me, netdata does benefit, vm's might benefit, but like @Ferroin mentions, not a big deal for the mainstream.

It's basically a double opt-in, so again, low risk...

[sfx2000]

so where do we land here?

I'm not seeing a good reason why "not" to do this change, esp. since it's in upstream and working well... most folks don't know it's there, unless they are looking for it.

turn the bit on....

yes or no, would like to sort this ticket, and reasons as to why not.

[squirrelboot]

Can you make this demonstrable to users like me?

I understand that I need to compile the kernel with the config option "CONFIG_KSM=y" set. But I don't see where or how we are given instructions on how to do this. I followed a guide here to install netdata.

Reviewing the successful installation, it pointed out that this system would benefit from the KSM installation. I however, do not know how to proceed with that.

While we wait for it to be pre-compiled in the kernels, it would be fantastic for us now to learn how to do this.

Thank you in advance!

[Ferroin]

@squirrelboot You can find info on building a custom kernel for the Pi here. Once you've generated the base config (with, for example, make bcm2709_defconfig), you can then use make menuconfig to tweak the configuration. You can then find the KSM option in the 'Kernel Features' sub-menu.

[squirrelboot]

Thanks!!

[Fieel]

So is building your own kernel is the only solution? I was also following the netdata guide https://docs.netdata.cloud/database/#ksm and I am really confused.

Also, I'm running a really old (~2012) raspberry pi 2B with a armhf processor, such as the one on the zero, 500 MB of ram are a real constraint in netdata and using KSM is virtually the only solution.

[sfx2000]

I think this is going to fall into the "not doing pile"...

There are alternative platforms (other ARM based SBC's) and other OS platforms that support the Raspberry Pi - if one really wants to run Netdata, there are choices. And of course, other than the memory savings, Netdata does run just fine on Pi2/Pi3, just takes a bit more memory.

While it would be nice to see this item pulled in - the project kernel maintainers do need to juggle the various wants (and needs) to keep a stable platform that supports the overall project goals.

[Fieel]

Understandable!

[mijutu]

KSM is not just for netdata or some special cases like virtual machines. With ksm_preload any user who runs two or more copies of any application might benefit from it.

I'd like to see KSM enabled on the default raspbian kernel.

[agneevX]

Almost 2 years now, and it's as stagnant as it was from the start.

[nethershaw]

KSM has been in the kernel since 2.6.32. In case anyone is counting, that was 11 years ago. Let upstream worry about its stability and just build in the 88 kilobytes of heavily commented code; the risk is zero. You can safely give everyone the choice of switching it on and forget about it.

If anyone wants to actually assume the risk of using KSM (the worst case scenario likely being that exactly nothing will happen), they still have to apt install ksmtuned -- which is in all of Raspberry Pi OS buster repositories already -- or flip the bits in sysfs by hand. This of course assumes the user knows it exists, which in turn safely indicates that the user knows what it is.

Consider also that Raspberry Pi OS already builds in and enables KVM, which is a decision within the realm of applicability to this discussion, and one that has much larger implications for the kernel. It does not make sense to build in one and not the other, especially for so trivial a cost.

Edit: the case for applications which benefit from KSM has already been expertly made in earlier comments, netdata chief among them. I just want to throw my hat in, too. As a sysadmin, I use it widely at work, and I use it on all of my machines at home including three different generations of Raspberry Pis. If you want to encourage users to learn with their devices, you can hardly do better than good, simple instrumentation -- but if "mainstream" users (ignoring the problematic use of that word) have to go through extra hoops to use it without potentially prohibitive memory overhead, they probably won't.

[6543]

why not just compile it in by default and disable it by default ... so nothing will change but it's easy to enable