search

end18 / psutil

Automatically exported from code.google.com/p/psutil
Other
0 stars 0 forks source link

Real time network IO counters #150

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?
1. if can show the Process's IO, i means: psutil support a function
like 
def get_disk_io(pid=None):
    return (write_speed, read_speed)

def get_net_io(pid=None):
    return (recv_speed, send_speed)

2.
3.

What is the expected output? What do you see instead?

What version of psutil are you using? What Python version?

On what operating system? Is it 32bit or 64bit version?

Please provide any additional information below.

Original issue reported on code.google.com by bychya...@gmail.com on 18 Feb 2011 at 6:04

GoogleCodeExporter commented 9 years ago 
There's already something for process IO counters (see issue 64).
By using that, get_disk_io() might be implemented, although I'm not sure 
whether something so high level fits well in a python lib.

As for get_net_io() I'm not sure what we might do. I doubt that kind of 
information is supported on many platforms.

Original comment by g.rodola on 18 Feb 2011 at 10:03

GoogleCodeExporter commented 9 years ago 
Personal reminder.
On linux network IO stats can be retrieved via /proc/net/dev and 
/proc/diskstats for the disk IO.

Original comment by g.rodola on 11 Jun 2011 at 6:28

GoogleCodeExporter commented 9 years ago 
On FreeBSD checkout man devstat and iostat cmdline utility.

Original comment by g.rodola on 11 Jun 2011 at 8:38

GoogleCodeExporter commented 9 years ago 
If this is something you want to implement, I have the network I/O implemented 
for Darwin.  Just need  a reason to create a patch.

Original comment by jcscoob...@gmail.com on 23 Jun 2011 at 6:17

GoogleCodeExporter commented 9 years ago 
Please do.
This is something I'd really love to see added.

Original comment by g.rodola on 24 Jun 2011 at 9:22

GoogleCodeExporter commented 9 years ago 
Changing this issue so that we track network IO counters only; disk IO counters 
will be tracked into a separate issue.
What I have in mind is something like this:

>>> psutil.network_io_counters()
iostat(bytes_sent=..., bytes_recv=...)

...or, if the the platform has support for it, also include the number of 
packets sent/received:

>>> psutil.network_io_counters()
iostat(bytes_sent=..., bytes_recv=..., pkts_sent=..., pkts_recv=...)

Original comment by g.rodola on 27 Jun 2011 at 8:43

GoogleCodeExporter commented 9 years ago 
This is not possible on Windows without using ETW kernel event tracing, and you 
don't want to do that.

Original comment by wj32...@gmail.com on 21 Jul 2011 at 11:29

GoogleCodeExporter commented 9 years ago 
I don't see any movement on this.  Is this still on the table?  I could 
implement OS X pretty quickly but I don't see any stubs to base my work off of. 
 I don't mind using the above "psutil.network_io_counters()" and it would 
return the following:

(packets_in, packets_out, bytes_in, bytes_out)

Just let me know.  OS X implementation is working locally so support is just a 
patch away once we can get a few suggestions made known.

Original comment by jcscoob...@gmail.com on 7 Sep 2011 at 4:12

GoogleCodeExporter commented 9 years ago 
As I said earlier in the thread, please do provide a patch.
The API I had in mind still stands:

>>> psutil.network_io_counters()
iostat(bytes_sent=..., bytes_recv=..., pkts_sent=..., pkts_recv=...)

Unfortunately I haven't had much time lately, and this issue requires a lot of 
effort because the implementation is completely different on every platform.
Having a patch for OSX would surely speed up the process, altough I'm not sure 
when I'll have the time to look into other platform implementations and start 
writing some actual code.

Original comment by g.rodola on 7 Sep 2011 at 7:42

GoogleCodeExporter commented 9 years ago 
I've submitted patches in the past but usually against an already in place API. 
 It just felt right to make a last ditch effort for some clarity before doing 
things my way only to find out you had something else in mind.  Expect a patch 
sometime today.

Original comment by jcscoob...@gmail.com on 7 Sep 2011 at 4:35

GoogleCodeExporter commented 9 years ago 
Yeah, sure.
The most important thing is the C implementation which extracts the information 
from the OS. The upper python layer and API aren't really important at this 
stage.

Original comment by g.rodola on 7 Sep 2011 at 4:49

GoogleCodeExporter commented 9 years ago 
Attached is a patch that adds network_io_counters for Mac OS X, complete with 
tests.  Since I'm adding a new API I might not had done things they way you 
wanted but I did my best to merge your conventions with the information in this 
patch.  Right now, this works just like mentioned above:

>>> import psutil
>>> psutil.network_io_counters()
iostat(bytes_sent=179001106, bytes_rcvd=1633284276, packets_sent=1564655, 
packets_rcvd=1916980)

Let me know if there is anything I can do to clean this patch up so it can be 
submitted.  Also, I really think we could add another feature to do 
per-interface stats and interface-specific stats, kind of how it's done for 
partition/disk usage.  I was gonna implement that now but I figured I'd wait 
until we had consensus on getting the totals in first.

Original comment by jcscoob...@gmail.com on 8 Sep 2011 at 11:18

Attachments:

GoogleCodeExporter commented 9 years ago 
I realize now that the test I wrote actually parses the output of "netstat -bi" 
to validate the psutil output.  I started thinking we could use that logic to 
implement this for supporting OSes.  Just an idea...

Original comment by jcscoob...@gmail.com on 9 Sep 2011 at 4:24

GoogleCodeExporter commented 9 years ago 
I haven't tried it yet but at a first look your patch looks fine to me.
Do you know if this might work on FreeBSD as well?
I'd say let's leave per-interface alone for now: we're not sure whether we can 
do that on all platforms.

Original comment by g.rodola on 9 Sep 2011 at 8:44

GoogleCodeExporter commented 9 years ago 
I had the same question after doing it OS X specific.  Based on the header 
files I used for reference, it's possible but I don't have a FreeBSD box to 
test on.  I guess I could install one on a VM.

Original comment by jcscoob...@gmail.com on 9 Sep 2011 at 8:58

GoogleCodeExporter commented 9 years ago 
I'll try to see whether your patch works on FreeBSD so that you won't have to 
install it.

As for per-interface, I'm looking at Linux implementation now and noticed that 
data can naturally be gathered in a per-interface fashion.
Considering that FreeBSD implemention should be similar to OSX's maybe it makes 
sense to rethink our API right now and return a list of namedtuples for every 
interface including the interface name. Example:

>>> psutil.network_io_counters()
[iostat(interface='eth0', bytes_sent=1, bytes_rcvd=2, packets_sent=3, 
packets_rcvd=4),
 iostat(interface='eth1', bytes_sent=1, bytes_rcvd=2, packets_sent=3, packets_rcvd=4),
 iostat(interface='lo', bytes_sent=1, bytes_rcvd=2, packets_sent=3, packets_rcvd=4),
 ...]

Original comment by g.rodola on 9 Sep 2011 at 9:09

GoogleCodeExporter commented 9 years ago 
I was thinking the same thing.  It would be nice to be able to get a list when 
necessary but not make it harder to get a total.  I don't see any other place 
where we do this type of thing.  The closest is psutil.cpu_times() where you 
have a function argument that dictates returning a list of results vs. 
returning a total.  I guess it's up to you.  Just let me know what you'd like 
me to do.  If you can tell me the Python interface(s) for the function(s), I'll 
update my patch accordingly.  Let me know the FreeBSD support too and I'll 
refactor if necessary.  I guess you can just include me where you want. ;)

Original comment by jcscoob...@gmail.com on 9 Sep 2011 at 9:16

GoogleCodeExporter commented 9 years ago 
I think in this case it makes more sense to return a list of nametuples as the 
default.
Perhaps we can add a "total=False" argument and return single a namedtuple 
instead, but this should be done in python, at the highest level.

> If you can tell me the Python interface(s) for the function(s), 
> I'll update my patch accordingly. 

I'd be for sticking with this API:

>>> psutil.network_io_counters()
[iostat(interface='eth0', bytes_sent=1, bytes_rcvd=2, packets_sent=3, 
packets_rcvd=4),
 iostat(interface='eth1', bytes_sent=1, bytes_rcvd=2, packets_sent=3, packets_rcvd=4),
 iostat(interface='lo', bytes_sent=1, bytes_rcvd=2, packets_sent=3, packets_rcvd=4),
 ...]

...therefore the C function should return a list of (name, bsent, brcv, psent, 
prcvd) tuples.
It is fine for the C extension to return plain tuples. You can then convert 
them to namedtuples in python (psutil/_psosx.py).

Original comment by g.rodola on 9 Sep 2011 at 9:34

GoogleCodeExporter commented 9 years ago 
Linux implementation committed as r1117.

Original comment by g.rodola on 9 Sep 2011 at 9:43

GoogleCodeExporter commented 9 years ago 
total parameter added in r1118.

Original comment by g.rodola on 9 Sep 2011 at 9:58

GoogleCodeExporter commented 9 years ago 
I'll update my patch for OS X based on the updated code in Subversion.

Original comment by jcscoob...@gmail.com on 9 Sep 2011 at 10:17

GoogleCodeExporter commented 9 years ago 
I've updated my patch.  Let me know if you need me to change anything.  Also, 
for testing I don't see any real test for this new stuff.  I didn't port the 
test I wrote in the original patch because I wasn't sure why it wasn't used in 
your tests.  I can port it if need be but applying my attached patch shouldn't 
be held up while I do that, assuming you're gonna apply it of course.

Original comment by jcscoob...@gmail.com on 9 Sep 2011 at 11:07

Attachments:

GoogleCodeExporter commented 9 years ago 
OSX patch committed as r1119.
I also gave credits to you (you are Jeremy Whitlock, right?).
Let me know if that's fine with you.
I tried to adapt your original OSX test but it keeps failing.

Original comment by g.rodola on 10 Sep 2011 at 3:57

GoogleCodeExporter commented 9 years ago 
I tried to copy & paste your patch on FreeBSD and this is what I get:

psutil/_psutil_bsd.c: In function 'get_network_io_counters':
psutil/_psutil_bsd.c:761: error: 'NET_RT_IFLIST2' undeclared (first use in this 
function)
psutil/_psutil_bsd.c:761: error: (Each undeclared identifier is reported only 
once
psutil/_psutil_bsd.c:761: error: for each function it appears in.)
psutil/_psutil_bsd.c:783: error: dereferencing pointer to incomplete type
psutil/_psutil_bsd.c:785: error: dereferencing pointer to incomplete type
psutil/_psutil_bsd.c:785: error: 'RTM_IFINFO2' undeclared (first use in this 
function)
psutil/_psutil_bsd.c:787: error: invalid use of undefined type 'struct 
if_msghdr2'
psutil/_psutil_bsd.c:795: error: dereferencing pointer to incomplete type
psutil/_psutil_bsd.c:796: error: dereferencing pointer to incomplete type
psutil/_psutil_bsd.c:797: error: dereferencing pointer to incomplete type
psutil/_psutil_bsd.c:798: error: dereferencing pointer to incomplete type
error: command 'cc' failed with exit status 1

I fixed 'NET_RT_IFLIST2' and 'RTM_IFINFO2' errors by including sys/socket.h and 
net/route.h but other errors remain.

Original comment by g.rodola on 10 Sep 2011 at 4:13

GoogleCodeExporter commented 9 years ago 
Update about OSX: it seems bytes/packets received are always 0.
The script in attachment shows the network usage in real time.
On linux it seems to work fine (while I download a big file with the browser it 
more or less reflects the number of bytes shown by the browser download 
manager) but on OSX it clearly doesn't work as download bytes are always 0.

Original comment by g.rodola on 10 Sep 2011 at 4:54

Attachments:

GoogleCodeExporter commented 9 years ago 
I don't see the same issue with bytes/packets being 0 except for certain 
devices, which also report the same metrics via "netstat -bi".  Here is an 
example of what I have, my code prior to your applying my patch:

>>> psutil.network_io_counters()
[iostat(interface='lo0', bytes_sent=84498890, bytes_rcvd=84491065, 
packets_sent=430981, packets_rcvd=430991), iostat(interface='gif0', 
bytes_sent=0, bytes_rcvd=0, packets_sent=0, packets_rcvd=0), 
iostat(interface='stf0', bytes_sent=0, bytes_rcvd=0, packets_sent=0, 
packets_rcvd=0), iostat(interface='en0', bytes_sent=342, bytes_rcvd=0, 
packets_sent=0, packets_rcvd=0), iostat(interface='en1', bytes_sent=220966030, 
bytes_rcvd=-1835371131, packets_sent=2140419, packets_rcvd=2745809), 
iostat(interface='fw0', bytes_sent=346, bytes_rcvd=0, packets_sent=0, 
packets_rcvd=0), iostat(interface='vboxnet0', bytes_sent=0, bytes_rcvd=0, 
packets_sent=0, packets_rcvd=0)]

I'm not understanding the problem.  As for my name in the patch, that's fine.  
I usually use "Jeremy Whitlock <jwhitlock@apache.org>" for projects I 
contribute to but I wasn't specific so no big deal.  As I was about to submit 
this comment, I see your script and I am able to see non-0 bytes sent/rcvd.  I 
do see a formatting issue where the k/B/etc. aren't always being displayed or 
are being displayed "stacked"  (0.5kk).  So, is there a problem with the 
network_io_counters() on OS X or with your implementation of the script?  I can 
look further into it regardless.

Original comment by jcscoob...@gmail.com on 10 Sep 2011 at 8:21

GoogleCodeExporter commented 9 years ago 
FreeBSD implementation checked in as r1120.
As for the script problem on OSX I'll take a deeper look to figure out what's 
wrong. Maybe it's my fault.

Original comment by g.rodola on 13 Sep 2011 at 5:25

GoogleCodeExporter commented 9 years ago 
Ok, problem was numbers types which are expressed by using a u_int64_t type.
This is now fixed in r1121.

Original comment by g.rodola on 13 Sep 2011 at 6:16

GoogleCodeExporter commented 9 years ago 
It seems we can get this information also on Windows without going crazy (at 
least, this is my impression) by using GetIfEntry():
http://stackoverflow.com/questions/5213629/how-to-enumerate-all-available-networ
k-interfaces
http://msdn.microsoft.com/en-us/library/aa365939(v=vs.85).aspx

Original comment by g.rodola on 14 Sep 2011 at 7:54

GoogleCodeExporter commented 9 years ago 
Would you like to refactor the call/result of network_io_counters to be like 
what you suggested/implemented in disk_io_counters?  
(http://code.google.com/p/psutil/issues/detail?id=206#c4)  I can do the work if 
you've not started already.

Original comment by jcscoob...@gmail.com on 21 Sep 2011 at 7:59

GoogleCodeExporter commented 9 years ago 
I have this implemented across all currently-implemented OSes.  Here is the 
output of both disk_io_counters and network_io_counters for comparison.  Some 
minor cleanup and I'll submit a patch.

>>> import psutil
>>> psutil.disk_io_counters()
iostat(read_count=349825L, write_count=897726L, read_bytes=5204112896L, 
write_bytes=22961667072L, read_time=1631324184229L, write_time=4326592882937L)
>>> psutil.disk_io_counters(perdisk=True)
{'disk0': iostat(read_count=349826L, write_count=897726L, 
read_bytes=5204116992L, write_bytes=22961667072L, read_time=1631336311003L, 
write_time=4326592882937L)}
>>> psutil.network_io_counters()
iostat(bytes_sent=342302872L, bytes_rcvd=5570448096L, packets_sent=3600502L, 
packets_rcvd=5141945L)
>>> psutil.network_io_counters(pernic=True)
{'gif0': iostat(bytes_sent=0L, bytes_rcvd=0L, packets_sent=0L, 
packets_rcvd=0L), 'en0': iostat(bytes_sent=342L, bytes_rcvd=0L, 
packets_sent=0L, packets_rcvd=0L), 'en1': iostat(bytes_sent=317063256L, 
bytes_rcvd=5545205708L, packets_sent
=3522199L, packets_rcvd=5063634L), 'lo0': iostat(bytes_sent=25240482L, 
bytes_rcvd=25243127L, packets_sent=78312L, packets_rcvd=78317L), 'stf0': 
iostat(bytes_sent=0L, bytes_rcvd=0L, packets_sent=0L, packets_rcvd=0L), 
'vboxnet0': iostat(
bytes_sent=0L, bytes_rcvd=0L, packets_sent=0L, packets_rcvd=0L), 'fw0': 
iostat(bytes_sent=346L, bytes_rcvd=0L, packets_sent=0L, packets_rcvd=0L)}

Original comment by jcscoob...@gmail.com on 21 Sep 2011 at 8:45

GoogleCodeExporter commented 9 years ago 
Attached is a patch that will refactor network_io_counters to be more like 
disk_io_counters in how it's called, total being the default, and the return 
type of dict instead of list.  During this I also found that 
network_io_counters on OS X was leaking memory and I've fixed this as well.

Original comment by jcscoob...@gmail.com on 21 Sep 2011 at 8:53

Attachments:

GoogleCodeExporter commented 9 years ago 
I have created Issue 210 for tracking the enhanced tests required to prove 
psutil.network_io_counters() is working properly.

Original comment by jcscoob...@gmail.com on 21 Sep 2011 at 10:37

GoogleCodeExporter commented 9 years ago 
Your patch looks good to me. You can go on and commit AFAICT.

Original comment by g.rodola on 22 Sep 2011 at 11:21

GoogleCodeExporter commented 9 years ago 
As of r1132, this has been committed to source control.

Original comment by jcscoob...@gmail.com on 22 Sep 2011 at 3:04

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
Windows implementation added in r1153.

Original comment by jcscoob...@gmail.com on 13 Oct 2011 at 8:01

GoogleCodeExporter commented 9 years ago

Original comment by g.rodola on 21 Oct 2011 at 11:34

GoogleCodeExporter commented 9 years ago

Original comment by g.rodola on 21 Oct 2011 at 11:44

GoogleCodeExporter commented 9 years ago

Original comment by g.rodola on 21 Oct 2011 at 11:45

GoogleCodeExporter commented 9 years ago

Original comment by g.rodola on 29 Oct 2011 at 3:44

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
Updated csets after the SVN -> Mercurial migration:
r1117 == revision 99bc14825f65
r1118 == revision 3a92e8ef7a71
r1119 == revision b404050fd676
r1120 == revision 08e02cf133ba
r1121 == revision ac004112a205
r1132 == revision 516510dc23a3
r1153 == revision df719375414c

Original comment by g.rodola on 2 Mar 2013 at 11:59

GoogleCodeExporter commented 9 years ago 
Are there any plans to extend this support to the Process class as 
"get_net_io", in the same fashion that exists for "get_io_counters"? Would be 
really awesome.

Original comment by justinis...@gmail.com on 16 Jun 2013 at 3:49

GoogleCodeExporter commented 9 years ago 
You mean per-process net io counters? I don't think there are platforms 
exposing that kind of info except maybe Windows, if my memory serves me right.

Original comment by g.rodola on 16 Jun 2013 at 9:33

GoogleCodeExporter commented 9 years ago 
Ya, sorry about that. After I made that request, I researched it a bit and came 
to find the same ruling. Makes perfect sense now why that feature doesn't exist 
:-)

Original comment by justinis...@gmail.com on 16 Jun 2013 at 9:49