Different output between psutil and "free" command

[GoogleCodeExporter]

What steps will reproduce the problem?
1. import psutil
2. mem = psutil.phymem_usage()
3. print mem.total, mem.used, mem.free

What is the expected output?
This is from "free" command:
total      used        free
2569812    2458704     111108

What do you see instead?
This is from psutil:
total  used   free
642453 615393 27060

What version of psutil are you using? What Python version?
psutil 0.4.0 and Python 2.7.1

On what operating system? Is it 32bit or 64bit version?
Linux Mint Katya 2.6.38-8-generic, 32bit version

Please provide any additional information below.
I want to check my physical memory usage using psutil. What I got from psutil 
is different from "free" command. Is my simple script using psutil wrong? Thank 
you in advance.

Original issue reported on code.google.com by studia...@gmail.com on 2 Dec 2011 at 1:43

[GoogleCodeExporter]

psutil results are expressed in bytes while "free" express them in... KB maybe? 
I'm not sure.
To compare results you should use MB instead ("free -m").
Example:

import psutil, os
mem = psutil.phymem_usage()
print "psutil: total=%s used=%s free=%s\n" % (mem.total / 1048576,
                                              mem.used / 1048576,
                                              mem.free / 1048576)
os.system('free -m')

On my linux box I get this output:

psutil: total=3961 used=2757 free=1203

             total       used       free     shared    buffers     cached
Mem:          3961       2757       1203          0        630        645
-/+ buffers/cache:       1481       2479
Swap:         1999         67       1932

Therefore, I'd say they match.
I'm noticing the doc doesn't mention that psutil.phymem_usage() is supposed to 
match free utility.
I'm going to fix that.

Original comment by g.rodola on 2 Dec 2011 at 2:20

[GoogleCodeExporter]

Hi, thanks for your quick reply.

You said in Documentation that, by default, psutil.phymem_usage() return value 
is in bytes. free -b will return total, used, and free memory in bytes too. 
However, I still  don't get the match result.

--- 1st trial --- 

mem = psutil.phymem_usage()
print 'total:', mem.total, 'used:', mem.used, 'free:', mem.free
os.system('free -b')

This is output on my Linux Mint Katya:

total: 642453 used: 476598 free: 165855

             total       used       free     shared    buffers     cached
Mem:    2631487488 1952526336  678961152          0   55832576 1497534464
-/+ buffers/cache:  399159296 2232328192
Swap:   2012213248          0 2012213248

It will give an almost match result when I multiply each value from psutil by 
4096:

--- 2nd trial ---

print 'total:', mem.total*4096, 'used:', mem.used*4096, 'free:', mem.free*4096
os.system('free -b')

Output:

 total: 2631487488 used: 1948819456 free: 682668032

             total       used       free     shared    buffers     cached
Mem:    2631487488 1948946432  682541056          0   56283136 1492238336
-/+ buffers/cache:  400424960 2231062528
Swap:   2012213248          0 2012213248

I continue my trial to "free -k" that returns value in kilo Bytes. It will 
almost match if I multiply value from psutil by 4.

--- 3rd trial ---

print 'total:', mem.total*4, 'used:', mem.used*4, 'free:', mem.free*4
os.system('free -k')

total: 2569812 used: 1903144 free: 666668

             total       used       free     shared    buffers     cached
Mem:       2569812    1903516     666296          0      55840    1451104
-/+ buffers/cache:     396572    2173240
Swap:      1965052          0    1965052

Then, i try "free -m" that returns value in mega Bytes. It will give an almost 
match result if I divide value from psutil by 256

--- 4th trial ---

print 'total:', mem.total/256, 'used:', mem.used/256, 'free:', mem.free/256
os.system('free -m')

total: 2509 used: 1887 free: 621

             total       used       free     shared    buffers     cached
Mem:          2509       1887        621          0         55       1435
-/+ buffers/cache:        397       2112
Swap:         1918          0       1918

Any idea about value 4096, 4, and 256? I have stuck here for the last 5 hours. 
Thank you :)

Original comment by studia...@gmail.com on 2 Dec 2011 at 3:39

[GoogleCodeExporter]

Can you post the contents of /proc/meminfo from your system? This seems really 
odd, I'd like to go direct to the source and see what it looks like before we 
assume its a psutil issue. 

Original comment by jlo...@gmail.com on 2 Dec 2011 at 3:46

[GoogleCodeExporter]

Okay, here is the contents of my /proc/meminfo:

MemTotal:        2569812 kB
MemFree:          970800 kB
Buffers:           58436 kB
Cached:          1083416 kB
SwapCached:            0 kB
Active:           715356 kB
Inactive:         806328 kB
Active(anon):     373392 kB
Inactive(anon):   156612 kB
Active(file):     341964 kB
Inactive(file):   649716 kB
Unevictable:          16 kB
Mlocked:              16 kB
HighTotal:       1703752 kB
HighFree:         483768 kB
LowTotal:         866060 kB
LowFree:          487032 kB
SwapTotal:       1965052 kB
SwapFree:        1965052 kB
Dirty:                 0 kB
Writeback:             8 kB
AnonPages:        379600 kB
Mapped:           110212 kB
Shmem:            150172 kB
Slab:              37620 kB
SReclaimable:      24928 kB
SUnreclaim:        12692 kB
KernelStack:        2792 kB
PageTables:         8156 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     3249956 kB
Committed_AS:    2416732 kB
VmallocTotal:     122880 kB
VmallocUsed:       28612 kB
VmallocChunk:      53448 kB
HardwareCorrupted:     0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       4096 kB
DirectMap4k:       24568 kB
DirectMap4M:      884736 kB

Thank you ^_^

Original comment by studia...@gmail.com on 2 Dec 2011 at 3:50

[GoogleCodeExporter]

What's the exact problem we're talking about? The lack of a perfect match when 
using bytes / kilo bytes?

Original comment by g.rodola on 2 Dec 2011 at 4:05

[GoogleCodeExporter]

Yes, we're talking about matching psutil.phymem_usage() and free output. Based 
on described Documentation, psutil should match with free output, isn't it? I 
want to use psutil in my server so I double check it with existing tool :)

Thanks. 

Original comment by studia...@gmail.com on 2 Dec 2011 at 4:35

[GoogleCodeExporter]

My mistake, I didn't realize we had moved to C code on the Linux platform for 
physical memory information. In that case it's most likely a casting issue. 
I'll have to look up the sysinfo docs to confirm but I think that's the 
problem. 

Original comment by jlo...@gmail.com on 2 Dec 2011 at 4:41

[GoogleCodeExporter]

http://stackoverflow.com/questions/4229415/c-sysinfo-returning-bad-values-i686

http://linux.about.com/library/cmd/blcmdl2_sysinfo.htm

The values returned by sysinfo are multiples of the "mem_unit" value. Our code 
is currently not calculating the values correctly since we're returning 
totalram, freeram, bufferram directly from sysinfo. 

Original comment by jlo...@gmail.com on 2 Dec 2011 at 4:44

[GoogleCodeExporter]

Original comment by jlo...@gmail.com on 2 Dec 2011 at 4:45

• Changed state: Started
• Added labels: OpSys-Linux

[GoogleCodeExporter]

Okay, it seems that I have same problem with psutil.virtmem_usage(). Should I 
start a new issue or continue this issue to report psutil.virtmem_usage() 
problem?

Thanks.

Original comment by studia...@gmail.com on 2 Dec 2011 at 4:45

[GoogleCodeExporter]

psutil and free results already match as shown in my first message using mega 
bytes as unit of measurement.
You're not getting a perfect match when using bytes / kilo bytes because:

- the unit of measurement is more precise
- more importantly: when you call "free" the system allocates/uses an extra 
quantity of memory *for the free command itself*

Therefore there's no way to have "free" and "used" values matching perfectly - 
the only value which keeps being exactly the same is the total memory, which 
isn't subject to any change deriving from the creation of new processes (in 
this case: the python interpreter and the free command).

Original comment by g.rodola on 2 Dec 2011 at 4:56

[GoogleCodeExporter]

virtmem is a different function and it just reads from the /proc/meminfo file 
so it should match these two lines: 

SwapTotal:       1965052 kB
SwapFree:        1965052 kB

if the virtmem totals don't match /proc/meminfo SwapTotal and SwapFree in 
bytes, then I would open a new issue to track that separately. 

Original comment by jlo...@gmail.com on 2 Dec 2011 at 4:56

[GoogleCodeExporter]

Ok, I've committed changes in r1231 for get_physmem() in the C code to convert 
the values to unsigned long long byte values for memory. Note that I changed 
the Py_BuildValue to use KKK instead of lll for the 3 values, as it's possible 
on a 64 bit system that memory values in bytes will be > long int size. Tested 
on my machine but ideally we can get it tested on 64 and 32 bit systems with 
various platforms.

Original comment by jlo...@gmail.com on 2 Dec 2011 at 5:04

• Changed state: FixedInSVN

[GoogleCodeExporter]

> psutil and free results already match as shown in my first message using 
> mega bytes as unit of measurement.

That may be true on your system, but it won't be the case on all systems due to 
the way sysinf() is implemented currently. We needed to make a code change in 
psutil to make sure the values get calculated correctly for all Linux 
platforms. 

> Therefore there's no way to have "free" and "used" values matching perfectly
> the only value which keeps being exactly the same is the total memory

This is correct, as Giampaolo I wouldn't expect psutil or free to match exactly 
for the used/free values, but they should at least be in the same ballpark. 

Original comment by jlo...@gmail.com on 2 Dec 2011 at 5:10

[GoogleCodeExporter]

> That may be true on your system, but it won't be the case on all systems 
> due to the way sysinf() is implemented currently

Clear. Good catch.
Can you also update HISTORY?

Original comment by g.rodola on 2 Dec 2011 at 5:15

[GoogleCodeExporter]

HISTORY file updated

Original comment by jlo...@gmail.com on 2 Dec 2011 at 5:17

[GoogleCodeExporter]

> if the virtmem totals don't match /proc/meminfo SwapTotal and SwapFree in 
bytes,
> then I would open a new issue to track that separately. 

I have to double check psutil.virtmem_usage() first to describe a detail 
problem.

> This is correct, as Giampaolo I wouldn't expect psutil or free to match 
exactly
> for the used/free values, but they should at least be in the same ballpark.

I can understand about perfect match. It's reasonable to understand. 

> That may be true on your system, but it won't be the case on all systems due 
to
> the way sysinf() is implemented currently. We needed to make a code change in
> psutil to make sure the values get calculated correctly for all Linux 
platforms. 

Then, should I commit to SVN to try this little correction to sysinf?

Thanks.

Original comment by studia...@gmail.com on 2 Dec 2011 at 6:02

[GoogleCodeExporter]

Yes, if you want to try the latest SVN revision it should hopefully correct the 
problem for you :) 

Original comment by jlo...@gmail.com on 2 Dec 2011 at 6:11

[GoogleCodeExporter]

Okey, thank you for you both ^_^

Original comment by studia...@gmail.com on 2 Dec 2011 at 6:19

[GoogleCodeExporter]

Just FYI, I have tried latest SVN revision and it fixed my problem with 
psutil.phymem_usage() output.

Thank you :)

Original comment by studia...@gmail.com on 7 Dec 2011 at 1:29

[GoogleCodeExporter]

Good. This is a high priority issue IMO. Wonder why it worths a new release.

Original comment by g.rodola on 7 Dec 2011 at 1:31

• Added labels: Priority-High
• Removed labels: Priority-Medium

[GoogleCodeExporter]

Original comment by g.rodola on 14 Dec 2011 at 5:56

• Added labels: Milestone-0.4.1

[GoogleCodeExporter]

Original comment by g.rodola on 14 Dec 2011 at 11:18

• Changed state: Fixed

[GoogleCodeExporter]

This is now fixed in 0.4.1 version.

Original comment by g.rodola on 14 Dec 2011 at 11:51

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Updated csets after the SVN -> Mercurial migration:
r1231 == revision 4328ec6cf2fe

Original comment by g.rodola on 2 Mar 2013 at 12:05