poornimag
commented
7 years ago Fuse/gfapi client usually allocates ~25-30MB of space(as calculated from the inside memory accounting counters). The memory breakdown is as follows:
- 12.2MB is iobufs
- 9MB is allocated by io-stats.
- 0.8MB by inode_table etc.
But, 25-30MB is not what a user see, when we do simple top -p of any gfapi/FUSE process. We see the following:
| No. of glfs_init | VIRT | RSS |
|---|---|---|
| 0 | 35MB | 1.1MB |
| 1 | 540MB | 10.5MB |
| 2 | 811MB | 15.7MB |
| 3 | 1.2GB | 22.9MB |
For analysing the huge VIRT memory consumed by FUSE/gfapi, will just understand memory consumed by some sample multithreaded program [1]. This program just creates 4 threads and allocates 1K bytes of memory. The top -p output shows that the VIRT size of this process, when all the 4 threads are running, is:
| VIRT | RSS | SHR |
|---|---|---|
| 294MB | 0.4MB | 0.3MB |
294MB is rather a really high memory consumption for a program that just creates 4 threads and allocates 1K each. The pmap output breaks down this VIRT memory consumption, below is the output. What we can see is that the [anon] accounts for the most of the memory, of which the 65404K(64MB) and 8192K(8MB) [anon] allocations constitute the most. After playing around with the thread count and some googling, turns out that, each thread creates allocates 72MB at the minimum:
- 4K ----- [ anon ]
- 144K rw--- [ anon ]
- 8192K rw--- [ anon ]-> Thread stack
- 65404K ----- [ anon ]-> Each thread mem pool (by malloc?) for solving thread contention? But, all this 72*4(288MB) allocated for 4 threads doesn't actually map to any physical memory, its just the address space reservations. With the 64bit systems the address space is so huge that the OS can afford to reserve the address space. If run on older RHEL versions with 32 bit systems, the VIRT memory wouldn't be this huge.
Coming back to the gfapi memory consumption, attached is a sample program that just performs glfs_init()/mount once [2], and here is the top consumption of the same:
| VIRT | RSS | SHR |
|---|---|---|
| 523MB | 10.3MB | 3.75MB |
The pmap output for the same is below. As discussed earlier, the memory breakdown of the gfapi process is as follows:
- Memory consumed by mapping all the shared objects: 90MB
- Memory consumed by 6 threads for stack and mempool: 295MB
- Memory mapped for locale archive: 102MB
- Rest of it: 36MB This is almost same as the memory allocation got by internal mem-accounting counters. (TODO: Need to account for another 10MB (36MB- 26MB alloced by process))
Conclusion: So we need not really bother much about the high VIRT consumption, as it doesn't necessarily mean allocation allocations made in the process. But 25-30MB is still a high number, when we want to scale for 10000s of clients (meaning ~300GB of RAM), thus the goal with thin client is to reduce the memory consumption from 25-30MB to 3-5MB (which will need 50GB for 10000 clients).
gluster-ant
ShyamsundarR
mykaul
amarts
stale[bot]
As of today, bringing up any gfapi client pre allocs ~ 25MB (at the minimum). This is too huge when gfapi is loaded in samba processes/qemu/block etc. We need to reduce the consumption to be < 2-3MB.