search

shanecode / psutil

Automatically exported from code.google.com/p/psutil
Other
0 stars 0 forks source link

psutil.disk_io_counters metrics too high on osx #343

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?
1.just call psutil.disk_io_counters() method

arcanus:webui sebastianwebber$ python
Python 2.7.3 (default, Dec  4 2012, 22:51:24) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> psutil.disk_io_counters()
iostat(read_count=755705L, write_count=1579596L, read_bytes=102353456128L, 
write_bytes=201152184320L, read_time=14720944724L, write_time=27322048809L)
>>> psutil.__version__
'0.6.1'

comparing this data with iostat, the write|read_bytes seems wrong:
arcanus:webui sebastianwebber$ iostat -I -c 10
          disk0           disk2       cpu     load average
    KB/t xfrs   MB     KB/t xfrs   MB  us sy id   1m   5m   15m
  134.93 2408988 317422.32    15.57 19171 291.56   5  5 90  1.60 2.00 1.96
  179.38  26  4.55    73.88  68  4.91   7  9 84  1.60 2.00 1.96
  156.90  31  4.75    67.73  75  4.96   5  8 88  1.60 2.00 1.96
  140.23  35  4.79    46.62 116  5.28   5  6 88  1.87 2.04 1.98
  132.97  37  4.80    34.35 170  5.70   6  7 87  1.87 2.04 1.98
  148.10  78 11.28    45.49 290 12.88  13 13 74  1.87 2.04 1.98
  201.27  88 17.30    56.47 306 16.88   7 14 80  1.87 2.04 1.98
  312.03 135 41.14   107.78 351 36.95   7 13 80  1.87 2.04 1.98
  336.32 165 54.19   107.50 352 36.95   7 10 82  1.88 2.04 1.98
  187.55 301 55.13   126.55 416 51.41   6 12 82  1.88 2.04 1.98

arcanus:webui sebastianwebber$ python
Python 2.7.3 (default, Dec  4 2012, 22:51:24) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import time, psutil
>>> con=0
>>> while con < 10:
...     psutil.disk_io_counters()
...     time.sleep(1)
...     con += 1
... 
iostat(read_count=780701L, write_count=1631905L, read_bytes=102513880576L, 
write_bytes=230456704000L, read_time=12475371900L, write_time=29870109263L)
iostat(read_count=780701L, write_count=1632025L, read_bytes=102513880576L, 
write_bytes=230457490432L, read_time=12475371900L, write_time=29870252188L)
iostat(read_count=780701L, write_count=1632025L, read_bytes=102513880576L, 
write_bytes=230457490432L, read_time=12475371900L, write_time=29870252188L)
iostat(read_count=780701L, write_count=1632025L, read_bytes=102513880576L, 
write_bytes=230457490432L, read_time=12475371900L, write_time=29870252188L)
iostat(read_count=780701L, write_count=1632027L, read_bytes=102513880576L, 
write_bytes=230457531392L, read_time=12475371900L, write_time=29870253983L)
iostat(read_count=780701L, write_count=1632027L, read_bytes=102513880576L, 
write_bytes=230457531392L, read_time=12475371900L, write_time=29870253983L)
iostat(read_count=780701L, write_count=1632027L, read_bytes=102513880576L, 
write_bytes=230457531392L, read_time=12475371900L, write_time=29870253983L)
iostat(read_count=780701L, write_count=1632027L, read_bytes=102513880576L, 
write_bytes=230457531392L, read_time=12475371900L, write_time=29870253983L)
iostat(read_count=780701L, write_count=1632027L, read_bytes=102513880576L, 
write_bytes=230457531392L, read_time=12475371900L, write_time=29870253983L)
iostat(read_count=780709L, write_count=1632035L, read_bytes=102514605568L, 
write_bytes=230458878976L, read_time=12475693016L, write_time=29870274777L)

I'm running Mac OS X Montain Lion.

Original issue reported on code.google.com by sebastia...@gmail.com on 5 Dec 2012 at 8:02

GoogleCodeExporter commented 9 years ago 
Keep in mind that psutil values are expressed in bytes while it seems iostat is 
using... megabytes? I'm not sure.

Original comment by g.rodola on 5 Dec 2012 at 8:06

GoogleCodeExporter commented 9 years ago 
Sorry but read_bytes can't be 95gb... or can?

>>> print 'kb: ' + str(102514605568L / 1024)
kb: 100111919
>>> print 'kb: ' + str(102514605568L / 1024 / 1024)
kb: 97765
>>> print 'mb: ' + str(102514605568L / 1024 / 1024)
mb: 97765
>>> print 'gb: ' + str(102514605568L / 1024 / 1024 / 1024)
gb: 95

Original comment by sebastia...@gmail.com on 5 Dec 2012 at 8:16

GoogleCodeExporter commented 9 years ago 
It might be possible as they are *total cumulative* bytes since last boot.
I wrote this simple script which writes and read a 1MB file every second.

====================================================================
import psutil, time

def read():
    f = open('/tmp/xxx', 'r')
    f.read(99999999)
    f.close()

def write():
    f = open('/tmp/xxx', 'w')
    f.write('x' * 1024 * 1024)  # 1MB
    f.flush()
    f.close()

while 1:
    write()
    read()
    io = psutil.disk_io_counters()
    print "read=%sMB, written=%sMB" % (io.read_bytes / 1024 / 1024,
                                       io.write_bytes / 1024 / 1024)
    time.sleep(1)
====================================================================

Here's the output:

bash-3.2$ python foo2.py 
read=499MB, written=767MB
read=499MB, written=768MB
read=499MB, written=769MB
read=499MB, written=770MB
read=499MB, written=771MB

It shows that the increment of the written bytes is very accurate, so I would 
rule out a precision issue.
The read counter apparently doesn't change though, so there might be another 
problem worth investigating.
After re-running the script after a couple of minutes the read bytes counter 
increased though, so it might be that the kernel updates that counter only 
after a certain period of time and we can't do nothing about that (just an 
hypothesis).

Original comment by g.rodola on 5 Dec 2012 at 9:17

GoogleCodeExporter commented 9 years ago 
What i really need is how i get the input/output bytes when the method is 
called. The data since boot is useful, but i need just information that is 
happening now. This is possible? if is, how i can do this?

Anyway, thanks for your help.

Original comment by sebastia...@gmail.com on 10 Dec 2012 at 10:14

GoogleCodeExporter commented 9 years ago 
IO metrics since last boot is everything you need.
What you have to do is subtract input/output metrics every second, and you'll 
have a real time representation of disk IO usage:

while 1:
    io1 = psutil.disk_io_counters()
    time.sleep(1)
    io2 = psutil.disk_io_counters()
    # bytes which were read/written in the last (1) second
    r = io2.read_bytes - io1.read_bytes
    w = io2.write_bytes - io1.write_bytes
    print "read=%sMB/sec, written=%sMB/sec" % (r / 1024 / 1024,
                                               w / 1024 / 1024)

The same concept is adopted by iotop.py and nettop.py example scripts:
http://code.google.com/p/psutil/source/browse/trunk/examples/iotop.py
http://code.google.com/p/psutil/source/browse/trunk/examples/nettop.py

Anyways, as for what concerns this issue I'm going to assume everything is fine.
Closing it out.

Original comment by g.rodola on 10 Dec 2012 at 2:26