I am low on RAM!

[mercertom]

If your applications want more memory, they just take back a chunk that the disk cache borrowed. Disk cache can always be given back to applications immediately! You are not low on ram!

This isn't true. Something is eating the RAM, and won't give it back. The use case is Chromium or Firefox with lots of tabs, then quit them. There will be additional RAM taken up than before the browser was run.

A few numbers to show what I mean: On bootup: <2GB used, usually ~1.7GB Open a bunch of tabs in FF/Chromium, eventually more than 8GB are used Quit FF/Chromium, and 3.5GB are used.

What is the 3.5-1.7= 1.8GB? Let's call it X.

Is X a memory cache? A memory leak? Why do both Firefox and Chromium reproduce the exact seem results?

Opening tabs eventually completely fills my 8GB RAM, system freezes, then eventually tabs die/freeze/crash, freeing just enough memory to continue.

The correct function, that I wish Ubuntu had, would be giving whatever the hell X is back to applications when they want RAM to run. However, when I launch an app, and most of the RAM has been eaten up by X, whether it was created by either of the 2 most mainstream browsers, no RAM is reclaimed from X. It remains until I reboot.

Use of "free" and other tools has not helped me diagnose WTF X is.

Any ideas?

[koalaman]

Can you post free output before and after? Make sure chrome/firefox exits completely and isn't running their preloaded single instance processes in the background.

[mercertom]

About a minute after quitting chromium.

              total        used        free      shared  buff/cache   available
Mem:        8035708     4037332     2202604     1014656     1795772     2713256
Swap:      12441592           0    12441592

A little while later.

              total        used        free      shared  buff/cache   available
Mem:        8035708     2497680     4168392      552812     1369636     4715280
Swap:      12441592           0    12441592

A day later, moments after a fill up to 7.9GB and all the tabs crash.

              total        used        free      shared  buff/cache   available
Mem:        8035708     5855132      739016     1012864     1441560      923084
Swap:      12441592      975712    11465880

After a similar fill-up and crash.

              total        used        free      shared  buff/cache   available
Mem:        8035708     5777812      638932      714256     1618964     1273920
Swap:      12441592           0    12441592

Some time later, opened chromium, ran it for a few hours (not to max RAM), closed it:

              total        used        free      shared  buff/cache   available
Mem:        8035708      737080     5869896      438592     1428732     6604848
Swap:      12441592      524436    11917156

Right after boot:

              total        used        free      shared  buff/cache   available
Mem:        8035708      689232     6449124      322912      897352     6769772
Swap:      12441592           0    12441592

After Chromium quits:

              total        used        free      shared  buff/cache   available
Mem:        8035708      967988     5677848      531960     1389872     6284744
Swap:      12441592         336    12441256

[koalaman]

Sorry, by "before" and "after", I meant immediately before running Chromium, and then immediately after quitting, with no other activity in between.

Also, please confirm that you're checking ps aux output for stray processes and potentially terminating them, not just quitting Chromium.

[mercertom]

The states offer more detail than just those 2 states, as well as showing multiple repetitions of booting fresh, running chromium, and quitting.

Not sure what you mean by ps aux. It's messy and long list, no way to say definitively there isn't a stray process. I am waiting after quitting chromium a couple minutes, so it should be completely quit.

Twice since upgrading to 17.04, I've had chromium freeze my computer and never recover, I'm assuming when it fills the 8GB of physical RAM, and just a few 100MB of the 12GB of swap. Last time it happened was yesterday.

I'm going to reboot, and do a before and after Chromium "free" right now. Will comment with free outputs in a few minutes.

[mercertom]

Right after boot:

              total        used        free      shared  buff/cache   available
Mem:        8035708      730640     6325556      452396      979512     6603724
Swap:      12441592           0    12441592

Run Chromium & quit

              total        used        free      shared  buff/cache   available
Mem:        8035708      838704     5395272      762124     1801732     6160332
Swap:      12441592           0    12441592

ps aux | grep chrom returned

thomas    6553  0.0  0.0  14240   924 pts/0    S+   16:34   0:00 grep --color=au

[koalaman]

So there's a 900MB reduction in free memory, where 300MB is shm/tmpfs, 100MB is increased application usage, and 500MB is disk cache (for a total of 400MB reduced available memory)

The 300MB difference in shared is probably the easiest to look into. You can list POSIX shm with ipcs and tmpfs usage shows up in df. Maybe you have /tmp on tmpfs so that temp files take up memory instead of disk space?

[mercertom]

------ Shared Memory Segments -------- key shmid owner perms bytes nattch status
0x00000000 12812288 thomas 600 524288 2 dest
0x00000000 327681 thomas 600 524288 2 dest
0x00000000 360450 thomas 600 16777216 2
0x00000000 557059 thomas 600 524288 2 dest
0x00000000 9961477 thomas 600 524288 2 dest
0x00000000 9994246 thomas 600 16777216 2 dest
0x00000000 5963783 thomas 600 393216 2 dest
0x00000000 14352392 thomas 600 1048576 2 dest
0x00000000 9011209 thomas 600 524288 2 dest
0x00000000 3670026 thomas 600 524288 2 dest
0x00000000 7831563 thomas 600 1048576 2 dest
0x00000000 12845068 thomas 600 4194304 2 dest
0x00000000 12877837 thomas 600 8388608 2 dest

No idea how to make sense of this.

/tmp is in root. I don't know how to check if /tmp is on tmpfs.

$ df -h /dev/shm Filesystem Size Used Avail Use% Mounted on tmpfs 3.9G 117M 3.8G 3% /dev/shm

With my chrome open and 6.1G of RAM used. Doesn't seem to be unusually large, but I'll try to run this command when I get close to full RAM.

Do you recommend any other diagnostic ideas?

[mercertom]

Also, how do I check whether disk cache is releasing? One important claim made on linuxatemyram.com is that whenever an app wants RAM, it will steal it back from disk cache. My suspicion is that this is not true. I would expect based on the article that linux will free the disk cache before I get to full RAM. If you look at my free outputs where I was closer to 8G (or over), has the disk cache released from RAM to let chrome/ff bloat replace it?

[koalaman]

I don't know why you're seeing crashes when you still have lots of available swap. I don't see any reason to assume that it's memory related at all. Applications crash when they run completely out of memory, but they usually just get slow and unresponsive when they start swapping.

It's hard to reason about a large application running over several days with unknown allocation patterns and measurements taken days apart, which allocates an unknown amount of data before crashing. It's way easier in a simple experiment like http://www.linuxatemyram.com/play.html

In your close-to-8G crash, there's just 400MB of non-shared disk cache in use. Normally I would have expected a modern distro in active use to have several GB cached at that point, but then again, Chromium could have been slowly growing over time and displacing the cache.

Linux chooses between dropping disk cache and swapping out old pages based on the swappiness setting. 0 will always drop disk cache, 100 will swap or drop cache with equal pressure. If you have a high swappiness, you will see some swapping before all disk cache is gone.

If yours is high, that's one possible explanation for why there is small amount of disk cache left. Another is that Chromium wrote a 400MB core dump when it crashed, and that this is what's currently in the crash. Or it could have been dirty pages that weren't available to be dropped at the time.

[mercertom]

My swappiness is and always has been either 0 or 1. I hate to write to SSD when RAM is still available. Makes no sense.

[mercertom]

When it freezes, I think it's close to 8GB of physical, but it's never ever close to filling swap. Would an application go unresponsive, meaning mouse completely goes away, when it starts swapping? Most times, I don't notice any slowdowns when the first few 100MB start swapping from FF or Chromium. I have an SSD, and I would not expect applications to get unresponsive even if I didn't have an SSD, and they started writing to swap. Maybe slowing a tiny bit, but like in FF or Chrome, would you expect to see extremely slow to switch tabs or to execute a command or mouse crawling extremely slowly with major multi-second delay? That's the kind of thing I'm seeing, which seems entirely a problem with OS / memory management, not Chromium or FF, which are just going from X MB to X+1 MB of memory.

[mercertom]

Also, nautilus cache will get a little big, like 1GB if I use a bunch of windows, but as soon as I close all nautilus windows, it deletes the cache (at least in 17.04 it seems)

[koalaman]

I would definitely expect switching tabs in FF or Chrome to take 5, 10 or 30 seconds if the application is swapping heavily to SSD.

I wouldn't expect a small amount of swap to affect it because it would belong to some other, less recently used process.

With some minor parts of the browser swapped out, I would expect occasional, second-long stutter and lag. If significant portions of it gets swapped out, I would expect it to be unusable for regular browsing.

[mercertom]

Oh, wow, that's really interesting. Why is that? And once it swaps out the tab's 50-150MB worth, would you expect it to be responsive again? I'm pretty shocked by this. I would have always thought mouse remains responsive, and maybe a 1-second or so lag to grab in a tab's worth of memory from swap.

[koalaman]

Mostly experience, but it also makes sense. PCI-e flash has a latency at least a hundred times greater than DRAM (not counting all the other overhead of paging and context switching), so I'd expect a slowdown on that order of magnitude.

I wouldn't really expect it to be perfectly responsive again, because tabs presumably access a lot of the same internal browser data, and they all keep running in the background to some degree and therefore churning the working set. Every time some data is loaded in, other data is written out.

I would only expect the behavior you describe if I recently freed a bunch of memory, so that a tab could load in a few hundred MB without anything else important having to be swapped out.

[mercertom]

Thank you for explaining all that.

I'll try to run some of these diagnostic commands if/when I spot these problems in the future.

[Technohacker]

It could be possible that Linux keeps the program binaries/shared libraries in memory to facilitate faster relaunch. Try closing the browser and reopening. The second start should be faster since everything's in RAM

[mercertom]

i have heard of preload. how would I test that? with a ssd and modern os, apps open near instantly. how do I disable preload? I need to measure a tiny difference in time accurately.

On Oct 10, 2017 09:26, "K3v1n Kur14k053" notifications@github.com wrote:

It could be possible that Linux keeps the program binaries/shared libraries in memory to facilitate faster relaunch. Try closing the browser and reopening. The second start should be faster since everything's in RAM

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/koalaman/linuxatemyram.com/issues/1#issuecomment-335471579, or mute the thread https://github.com/notifications/unsubscribe-auth/AJ347MfFJfBBw1fF5ILsgMO5_ei4ryaSks5sq3BygaJpZM4M6hLF .

[Technohacker]

@mercertom there's the time command. prefix time to any command and it will measure the amount of time it takes to run (when using for browsers, make sure you use a background flag, so it exits once the browser has loaded)

As for disabling it, I assume like the disk cache, it can't be disabled.

[mercertom]

also any way to just measure that directly in a RAM usage metric?

[mercertom]

also, what's the background flag?

I do 'time firefox', but googled "background flag" and see nothing pertinent.

[Technohacker]

@mercertom you can check htop, REServed and SHaRed memory. and by background flag, i meant any kind of program argument that can make the browser process quit after it shows the GUI.

For firefox, ignore that. just run time firefox

[mercertom]

So what should I do for background flag on firefox?

I'm doing 'time firefox'. how do i set a flag for it to quit after GUI?

[Technohacker]

@mercertom i edited my comment. you can ignore that as firefox does it automatically. just run time firefox

[mercertom]

i don't think so. it stays open until i close it manually with 'time firefox'.

[Technohacker]

@mercertom Hmm, it seems to work as expected for me with Firefox Developer edition

[cityvigil]

ref : https://unix.stackexchange.com/questions/33381/getting-information-about-a-process-memory-usage-from-proc-pid-smaps

Calculate physical memory usage for all processes grep Pss /proc/[1-9]*/smaps | awk '{total+=$2}; END {printf "%d kB\n", total }'

Calculate the memory occupied by processes in non-process folders

awk '/Pss:/{ sum += $2 } END { print sum }' /proc/$$/smaps

Use the above two commands to compare memory usage between a fresh reboot, running Chromium, and quitting Chromium.