search

nikademus79 / psutil

Automatically exported from code.google.com/p/psutil
Other
0 stars 0 forks source link

Real time disk IO counters #206

Closed GoogleCodeExporter closed 8 years ago

GoogleCodeExporter commented 8 years ago 
Similarly to network_io_counters (see issue 150) it would be good for psutil to 
provide a disk_io_counters() function returning the number of bytes read and 
written from/to disk.
With this we'd have the possibility to write tools such as iotop [1].

>>> psutil.disk_io_counters()
iostat(read_bytes=2334342, write_bytes=12353453)

As an extra, if the OS provides the necessary hooks, it might be good to 
provide also the number of read()/write() calls, as per Process.get_io_counters 
method (see issue 64).

I haven't looked into any implementation yet so I'm still not sure what the 
implications are and whether this can be done on all platforms.

[1] http://guichaz.free.fr/iotop/

Original issue reported on code.google.com by g.rodola on 14 Sep 2011 at 10:18

GoogleCodeExporter commented 8 years ago 
My request would be to have disk_io_counters() behave just like 
network_io_counters() with a total keyword argument and per-device/partition 
read/write stats.  I'll get cracking on the Mac OS X implementation.

Original comment by jcscoob...@gmail.com on 14 Sep 2011 at 10:31

GoogleCodeExporter commented 8 years ago 
From what I see, iostat doesn't distinguish transfer information into 
read/write information.  I'll keep looking but I'm not seeing how this might 
work.  If you know of a better utility to get read/write information for a 
disk, I'll be glad to look at it as a reference.  I also will keep looking into 
what is available to iostat just in case it has the information it needs but 
just doesn't deliver it.

Original comment by jcscoob...@gmail.com on 20 Sep 2011 at 7:27

GoogleCodeExporter commented 8 years ago 
It just so happens you can get read/write, I just needed to dig deeper into the 
code.  I'm good to go now.  I'll have the following:

>>> psutil.disk_io_counters()
iostat(name=disk0, reads=1234, writes=1234, read_bytes=1234, write_bytes=1234, 
read_time=1234, write_time=1234)

With all of this, we could reproduce the output of iostat.  What do you think?

Original comment by jcscoob...@gmail.com on 20 Sep 2011 at 7:44

GoogleCodeExporter commented 8 years ago 
Good to know you figured this out.
Thinking back about higher level API I think it would be better off to stick 
with something like this (same for psutil.network_io_counters() - issue 150; I 
think I'm going to modify that in the same manner):

>>> psutil.disk_io_counters()
iostat(...)
>>> psutil.disk_io_counters(per_disk=True)
{'/dev/sda0': iostat(...),
 '/dev/sda1': iostat(...),
 '/dev/sda2': iostat(...)}
>>>

I think this is better for two reasons:

- total disk io is usually what you want
- a dict is better in case you want to know io stats of a particular partition 
given its mount point (in contrast to iterating over the whole list).

I'm sorry about this, I hope you don't mind.
As for the lower level C API I think it is good to return a list of tuples as 
such:

[('name', ...), ('name', ...), ('name', ...), ...]

...and then provide the total in python, similarly to what we're currently 
doing here:
http://code.google.com/p/psutil/source/browse/trunk/psutil/__init__.py#694

Question: what's read_time/write_time?

Original comment by g.rodola on 20 Sep 2011 at 10:11

GoogleCodeExporter commented 8 years ago 
Do you mind if I commit disk_io_counters prior to your refactoring?  It's not a 
big deal but it would be a lot less work for me.  As for the read/write time, 
it's the total amount of time (in nanoseconds) reading/writing to disk.  It's 
used to create the xfrs/s and MB/s for iostat.  I could see this being very 
useful for tools using psutil for averages/metrics...like I'm doing.  ;)

Original comment by jcscoob...@gmail.com on 21 Sep 2011 at 12:40

GoogleCodeExporter commented 8 years ago 
Since I'm asking you to wait, I could have a patch submitted by end of day.  I 
have things working locally and would need to just write a suitable unit test, 
like I did for netstat comparison for network I/O.

Original comment by jcscoob...@gmail.com on 21 Sep 2011 at 12:41

GoogleCodeExporter commented 8 years ago 
Here is my current output for OS X:

>>> import psutil
>>> psutil.disk_io_counters()
[iostat(name='disk0', reads=342757L, writes=820590L, read_bytes=5084622336L, 
write_bytes=21940955136L, read_time=1540016653215L, write_time=3884665830052L)]
>>> psutil.disk_io_counters(total=True)
iostat(reads=342758L, writes=820598L, read_bytes=5084626432L, 
write_bytes=21941061632L, read_time=1540029707276L, write_time=3884671096030L)
>>> psutil.disk_io_counters()
[iostat(name='disk2', reads=66L, writes=13L, read_bytes=594944L, 
write_bytes=189952L, read_time=113436291L, write_time=224393426L), 
iostat(name='disk0', reads=342774L, writes=820897L, read_bytes=5084923904L, 
write_bytes=21945732096L, r
ead_time=1540136725056L, write_time=3885044760892L)]
>>> psutil.disk_io_counters(total=True)
iostat(reads=342840L, writes=820910L, read_bytes=5085518848L, 
write_bytes=21945922048L, read_time=1540250161347L, write_time=3885269154318L)

I run it twice, once with only one disk (primary HD) being found and the second 
time after plugging in a USB thumbdrive.  I'll get you a patch when I get a 
unit test up and running.

Original comment by jcscoob...@gmail.com on 21 Sep 2011 at 3:10

GoogleCodeExporter commented 8 years ago 
Attached is my preliminary patch.  I followed the psutil.network_io_usage() 
documentation/enablement/testing/approach for this so it should be pretty 
similar.  I did have to start using the CoreFoundation and IOKit frameworks for 
Mac OS X but those are OS supplied so it shouldn't be a big deal.  I also 
skipped on writing a more thorough test since my last one wasn't committed.  We 
can work together to come up with ideal tests if you'd like but for now, it's 
par for the course I'd say.  Let me know what I need to do to get this 
committed.

Original comment by jcscoob...@gmail.com on 21 Sep 2011 at 3:15

Attachments:

GoogleCodeExporter commented 8 years ago 
On OSX 10.5.8 it compiles fine but this is what I get on module import:

users-virtualbox-3:psutil user$ python
Python 2.5.1 (r251:54863, Jun 17 2009, 20:37:34) 
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
/Library/Python/2.5/site-packages/psutil-0.3.0-py2.5-macosx-10.5-i386.egg/_psuti
l_osx.py:3: UserWarning: Module _psutil_osx was already imported from 
/Library/Python/2.5/site-packages/psutil-0.3.0-py2.5-macosx-10.5-i386.egg/_psuti
l_osx.pyc, but /Users/user/psutil is being added to sys.path
  import sys, pkg_resources, imp
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "psutil/__init__.py", line 73, in <module>
    import psutil._psosx as _psplatform
  File "psutil/_psosx.py", line 9, in <module>
    import _psutil_osx
  File "build/bdist.macosx-10.5-i386/egg/_psutil_osx.py", line 7, in <module>
  File "build/bdist.macosx-10.5-i386/egg/_psutil_osx.py", line 6, in __bootstrap__
ImportError: 
dlopen(/Users/user/.python-eggs/psutil-0.3.0-py2.5-macosx-10.5-i386.egg-tmp/_psu
til_osx.so, 2): Symbol not found: ___CFConstantStringClassReference
  Referenced from: /Users/user/.python-eggs/psutil-0.3.0-py2.5-macosx-10.5-i386.egg-tmp/_psutil_osx.so
  Expected in: dynamic lookup

Original comment by g.rodola on 21 Sep 2011 at 8:48

GoogleCodeExporter commented 8 years ago 
Giampaolo, double check and make sure you're not running python with "import 
psutil" while you're in the source dir for psutil. I did that the other day and 
got the same error, because it tries to import psutil from the cwd.

Original comment by jlo...@gmail.com on 21 Sep 2011 at 12:26

GoogleCodeExporter commented 8 years ago 
I removed the build directory, then re-installed psutil with "python setup.py 
install", then moved into /tmp directory but I still get the same error:

users-virtualbox-3:tmp user$ pwd
/tmp
users-virtualbox-3:tmp user$ python
Python 2.5.1 (r251:54863, Jun 17 2009, 20:37:34) 
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "build/bdist.macosx-10.5-i386/egg/psutil/__init__.py", line 73, in <module>
  File "build/bdist.macosx-10.5-i386/egg/psutil/_psosx.py", line 9, in <module>
  File "build/bdist.macosx-10.5-i386/egg/_psutil_osx.py", line 7, in <module>
  File "build/bdist.macosx-10.5-i386/egg/_psutil_osx.py", line 6, in __bootstrap__
ImportError: 
dlopen(/Users/user/.python-eggs/psutil-0.3.0-py2.5-macosx-10.5-i386.egg-tmp/_psu
til_osx.so, 2): Symbol not found: ___CFConstantStringClassReference
  Referenced from: /Users/user/.python-eggs/psutil-0.3.0-py2.5-macosx-10.5-i386.egg-tmp/_psutil_osx.so
  Expected in: dynamic lookup
>>> 

Google brought me to this: 
https://trac.macports.org/ticket/14758
...but I'm not sure what to do.

Original comment by g.rodola on 21 Sep 2011 at 12:41

GoogleCodeExporter commented 8 years ago 
That gets rid of the UserWarning at least, and for the symbol error for 
___CFConstantStringClassReference it looks like you need to add the flag 
"-framework CoreFoundation" though I don't know offhand what the equivalent 
configuration would be with distutils

Original comment by jlo...@gmail.com on 21 Sep 2011 at 12:59

GoogleCodeExporter commented 8 years ago 
Answer here: 
http://stackoverflow.com/questions/1676384/how-to-pass-flag-to-gcc-in-python-set
up-py-script

you need to add 

    extra_link_args=['-framework', 'CoreFoundation']

to the Extension definition it looks like.

Original comment by jlo...@gmail.com on 21 Sep 2011 at 1:02

GoogleCodeExporter commented 8 years ago 
Giampaolo, I committed changes in r1124 to add this flag to setup.py - let me 
know if that works on your 10.5 system.

Original comment by jlo...@gmail.com on 21 Sep 2011 at 1:05

GoogleCodeExporter commented 8 years ago 
Now error changed into:

>>> import psutil
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "build/bdist.macosx-10.5-i386/egg/psutil/__init__.py", line 73, in <module>
  File "build/bdist.macosx-10.5-i386/egg/psutil/_psosx.py", line 9, in <module>
  File "build/bdist.macosx-10.5-i386/egg/_psutil_osx.py", line 7, in <module>
  File "build/bdist.macosx-10.5-i386/egg/_psutil_osx.py", line 6, in __bootstrap__
ImportError: 
dlopen(/Users/user/.python-eggs/psutil-0.3.0-py2.5-macosx-10.5-i386.egg-tmp/_psu
til_osx.so, 2): Symbol not found: _kIOMasterPortDefault
  Referenced from: /Users/user/.python-eggs/psutil-0.3.0-py2.5-macosx-10.5-i386.egg-tmp/_psutil_osx.so
  Expected in: dynamic lookup

Original comment by g.rodola on 21 Sep 2011 at 1:30

GoogleCodeExporter commented 8 years ago 
Ok, try r1125 which adds -framework IOKit also

Original comment by jlo...@gmail.com on 21 Sep 2011 at 1:37

GoogleCodeExporter commented 8 years ago 
Ok, it is fixed now, thanks.
I'm going to commit OSX patch soon.

Original comment by g.rodola on 21 Sep 2011 at 1:39

GoogleCodeExporter commented 8 years ago 
Checked in in r1126.
Later today I'm gonna change the API to use a dict and add the Linux 
implementation.
Just one thing: what do read_time and write_time represent? Seconds spent in 
reading/writing from/to disk?

Original comment by g.rodola on 21 Sep 2011 at 1:51

GoogleCodeExporter commented 8 years ago 
Per earlier comment "it's the total amount of time (in nanoseconds) 
reading/writing to disk"

Original comment by jlo...@gmail.com on 21 Sep 2011 at 1:58

GoogleCodeExporter commented 8 years ago 
You know, I was wondering how it was compiling in the first place.  I had to 
use "-framework CoreFoundation -framework IOKit" when I compiled via gcc but 
when I ran "python setup.py [build|install]" it wasn't complaining and it was 
working.  Weird.  I must have had some left over build or site-package 
installed.  My bad.  (What's funny was I had those extra_compile_args in a 
previous patch and removed them.  *sigh*)

Original comment by jcscoob...@gmail.com on 21 Sep 2011 at 4:23

GoogleCodeExporter commented 8 years ago 
s/extra_compile_args/extra_link_args/

Original comment by jcscoob...@gmail.com on 21 Sep 2011 at 4:24

GoogleCodeExporter commented 8 years ago 
If it makes you feel any better, it compiled fine here on 10.6 without the 
flags added, so they must be being linked automatically or something on later 
versions of OS X.

Original comment by jlo...@gmail.com on 21 Sep 2011 at 4:26

GoogleCodeExporter commented 8 years ago 
Crap, I feel so stupid now.  I am still baffled how it worked locally for me in 
the first place.  Anyhow, the answer you found was the solution, using 
extra_link_args in setup.py's Extension for OS X.  Again, I apologize.

Original comment by jcscoob...@gmail.com on 21 Sep 2011 at 4:27

GoogleCodeExporter commented 8 years ago 
Yes, that does make me feel much better. ;)

Original comment by jcscoob...@gmail.com on 21 Sep 2011 at 4:32

GoogleCodeExporter commented 8 years ago 
[deleted comment]
GoogleCodeExporter commented 8 years ago 
API changed as described in comment #4 ( 
http://code.google.com/p/psutil/issues/detail?id=206#c4 ) checked in as r1128.
I also added some integrity tests. We should add more to compare results 
against iostat cmdline utility.

Original comment by g.rodola on 21 Sep 2011 at 6:58

GoogleCodeExporter commented 8 years ago 
I can submit a patch for comparing to iostat output.  I need to write something 
for work that will do iostat usage/parsing until the next psutil release 
anyways.  I'll submit the patch shortly.

Original comment by jcscoob...@gmail.com on 21 Sep 2011 at 7:01

GoogleCodeExporter commented 8 years ago 
For those of you not following Issue 150, I've taken the suggested refactoring 
for the call/return of disk_io_counters and implemented the same approach for 
network_io_counters.  Should help for consistency.

Original comment by jcscoob...@gmail.com on 21 Sep 2011 at 8:55

GoogleCodeExporter commented 8 years ago 
I have created Issue 210 for tracking the enhanced tests required to prove 
psutil.disk_io_counters() is working properly.

Original comment by jcscoob...@gmail.com on 21 Sep 2011 at 10:37

GoogleCodeExporter commented 8 years ago 
Linux implementation added in r1130.

Original comment by g.rodola on 22 Sep 2011 at 11:48

GoogleCodeExporter commented 8 years ago 
Windows implementation added in r1149.

Original comment by jcscoob...@gmail.com on 13 Oct 2011 at 4:40

GoogleCodeExporter commented 8 years ago 
I started looking into BSD implementation.
After reading iostat source code [1] I started using 
devstat_compute_statistics() [2] until I figured out that other that read/write 
bytes it doesn't provide the other 4 information which are useful to us: 
read/write calls and read/write times.

It seems devicestat (9) [3] provides exactly everything we need but I can't 
manage to use it (I'm supposed to use devstat_add_entry() but it's not even 
defined in sys/devicestat.h!).

Anyway, in case it could be useful, I attach this patch based on iostat's 
approach which provides read/write bytes only.

[1] 
http://gitorious.org/freebsd/freebsd/blobs/3bbca39ec7441d70323c6727efa2a865fe3e5
442/usr.sbin/iostat/iostat.c#line661
[2] 
http://www.gsp.com/cgi-bin/man.cgi?section=3&topic=devstat_compute_statistics
[3] http://www.gsp.com/cgi-bin/man.cgi?section=9&topic=devstat

Original comment by g.rodola on 13 Oct 2011 at 8:13

Attachments:

GoogleCodeExporter commented 8 years ago 
Ok, I figured out how to do it.
FreeBSD implementation checked in as r1176.
This is now implemented on all platforms.

Original comment by g.rodola on 21 Oct 2011 at 10:04

GoogleCodeExporter commented 8 years ago

Original comment by g.rodola on 21 Oct 2011 at 11:44

GoogleCodeExporter commented 8 years ago

Original comment by g.rodola on 21 Oct 2011 at 11:45

GoogleCodeExporter commented 8 years ago

Original comment by g.rodola on 29 Oct 2011 at 3:44

GoogleCodeExporter commented 8 years ago 
[deleted comment]
GoogleCodeExporter commented 8 years ago 
Updated csets after the SVN -> Mercurial migration:
r251 == revision 24d321225795
r1124 == revision a1d51a7482e4
r1125 == revision b1c1bf0143b1
r1126 == revision 39750cf12924
r1128 == revision 59b453c08bc9
r1130 == revision 6ca667aedc68
r1149 == revision f14fbfcf2b6d
r1176 == revision edd9514cc641

Original comment by g.rodola on 2 Mar 2013 at 12:03