disk_io_counters failed to find some disks on GNU/linux

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Just have some lvm partition on a virtual server (xen in my case) with no 
digit in its name, like this /proc/partitions : 

major minor  #blocks  name

 202        0   19922944 xvda
 202       16    1048576 xvdb

What is the expected output?
Any correct result

What do you see instead?
RuntimeError: couldn't find any physical disk

What version of psutil are you using? What Python version?
psutil 0.6.1 on python 2.6.6

On what operating system? Is it 32bit or 64bit version?
GNU/linux debian squeeze, 64 bits, in a xen domU

Please provide any additional information below.

The file /proc/diskstats correctly show some informations : 

 202       0 xvda 27846519 29828 1002976690 121876532 13449001 43255546 453670328 1152982588 0 79363348 1275022004
 202      16 xvdb 55557 1134 453464 412056 38511 33466 575840 1699516 0 272092 2111504

It's not a real physical disk, it's a lvm volume, but the virtual server show 
it as a real disk.

In _pslinux.py , line 322, I can see that to be added as a partition, the name 
of the partition must end with a digit, but that's not always the case. This is 
the bug.

Original issue reported on code.google.com by mani...@gmail.com on 16 Nov 2012 at 5:18

[GoogleCodeExporter]

Sorry for replying so late.
Can you please paste the output of the following commands?

$ cat /proc/diskstats
$ cat /proc/partitions
$ sudo fdisk -l

I tried to debug this issue and it seems that the isdigit() check is necessary.
Here's why:

giampaolo@ubuntu:~/svn/psutil$ cat /proc/diskstats
...
   8       0 sda 88010 382993 5037912 1761940 167386 162715 5640296 4226620 0 786756 5990656
   8       1 sda1 166 0 1328 180 0 0 0 0 0 180 180
   8       2 sda2 162 0 1296 640 0 0 0 0 0 640 640
   8       3 sda3 16040 34410 1845834 1431572 21578 27070 444832 1489176 0 151096 2922972
   8       4 sda4 2 0 12 64 0 0 0 0 0 64 64
   8       5 sda5 161 31 1536 1516 0 0 0 0 0 1516 1516
   8       6 sda6 54591 328533 2673104 105052 2467 57933 483200 483340 0 97180 588364
   8       7 sda7 16721 20019 513466 222712 134044 77712 4712264 2054888 0 402808 2277988

'sda' seems to represent the entire disk 'as a whole' while the sda1, sda2, 
etc. looks like sda's partitions.
sudo fdisk -l seems to confirm my thesis.
Include 'sda' in the list of returned metrics would be incorrect.

Original comment by g.rodola on 12 Dec 2012 at 2:47

• Added labels: OpSys-Linux

[GoogleCodeExporter]

Original comment by g.rodola on 12 Dec 2012 at 3:35

• Changed state: Started

[GoogleCodeExporter]

No problem, time is hard to find sometimes :)

Here are the output : 

$ cat /proc/diskstats
 202       0 xvda 35225468 35266 1280851930 158475256 17086558 51539118 549080896 1457575900 0 101625076 1616234844
 202      16 xvdb 66347 1262 540808 499404 45155 37351 660080 1953716 0 321644 2453044

$ cat /proc/partitions
 202        0   19922944 xvda
 202       16    1048576 xvdb

$ sudo fdisk -l

Disk /dev/xvda: 20.4 GB, 20401094656 bytes
255 heads, 63 sectors/track, 2480 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/xvda doesn't contain a valid partition table

Disk /dev/xvdb: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/xvdb doesn't contain a valid partition table

This case is somehow specific (but not only my case), because it is in a 
virtual server, so this disk are note physical, it's a LVM volume on the dom0, 
but the virtual server doesn't make a difference, and produce statistics.

In this case, the isdigit is not necessary, but if you remove it, the case you 
write here will not be correctly checked.

Can I suggest some test on the major number of the partition ? It seems that it 
is specific of the hdd type, and the 202 value in my case seems to show that it 
is some virtual partition. But I don't know if someones uses xvda1 xvda2 in 
their virtual server and what it will output on /proc/diskstats :/

Regards,

Original comment by mani...@gmail.com on 12 Dec 2012 at 10:37

[GoogleCodeExporter]

According to "fdisk -l" you have no partitions whatsoever.
If that's the case are you even able to write on disk?
If not then I'd say that psutil's behavior is legitimate otherwise I can't see 
a reliable way to deal with this case.

Can you please try this and paste the output?

>>> import _psutil_linux
>>> _psutil_linux.get_disk_partitions()

Original comment by g.rodola on 12 Dec 2012 at 12:28

[GoogleCodeExporter]

As you see, I am able to write on disk : 

$ mount
/dev/xvda on / type ext3 (rw,errors=remount-ro)
[snip]

has it's my / , it is better that I'm able to write on It. 

There is no partition because it is not a physical disk, it is a LVM volume.

Here is the output :

>>> import _psutil_linux
>>> _psutil_linux.get_disk_partitions()
[('/dev/xvda', '/', 'ext3', 'rw,errors=remount-ro'), ('tmpfs', '/lib/init/rw', 
'tmpfs', 'rw,nosuid,mode=0755'), ('proc', '/proc', 'proc', 
'rw,noexec,nosuid,nodev'), ('sysfs', '/sys', 'sysfs', 
'rw,noexec,nosuid,nodev'), ('udev', '/dev', 'tmpfs', 'rw,mode=0755'), ('tmpfs', 
'/dev/shm', 'tmpfs', 'rw,nosuid,nodev'), ('devpts', '/dev/pts', 'devpts', 
'rw,noexec,nosuid,gid=5,mode=620')]

If you need it, I can create a small virtual server and give you access then 
you can see it and test if needed.

Regards,

Original comment by mani...@gmail.com on 12 Dec 2012 at 2:23

[GoogleCodeExporter]

Could you paste the output of iostat command?

Original comment by g.rodola on 13 Dec 2012 at 3:26

[GoogleCodeExporter]

Here it is :

Linux 2.6.32-5-xen-amd64 (myvirtualmachine)     13/12/2012  _x86_64_    (4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,14    0,00    0,05    0,23    0,01   99,57

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
xvda              5,43       133,05        56,86 1293070250  552655816
xvdb              0,01         0,06         0,07     541144     667048

Original comment by mani...@gmail.com on 13 Dec 2012 at 5:18

[GoogleCodeExporter]

Hi,

We get this issue also, an easy way to reproduce is to start up a CentOS 6.3 
instance on the Amazon EC2 Marketplace via 
https://aws.amazon.com/marketplace/pp/B00A6L6F9I

This launches with one disk, /dev/xvde and no separate partitions and thus no 
disks are found.

The following code does seem to work correctly but only limited testing has 
been done so far:
In psutil/_pslinux.py:

#This checks through the existing partitions (which are loaded in reverse order 
so that any numbered partitions are found first to see if a numbered partition 
for a disk has been found in which case the disk itself is ignored, it is only 
called when the last character of a partition name is not a digit

def check_partition_match(partition,partitions):
        partition_length = len(partition)
        for p in partitions:
                match_string = p[:partition_length]
                if partition == match_string:
                        return True;

#First part of disk_io_counters updated, changes are that the partitions are 
run through in reverse and if the last character is not a digit then the above 
function is called.

def disk_io_counters():
    """Return disk I/O statistics for every disk installed on the
    system as a dict of raw tuples.
    """
    # man iostat states that sectors are equivalent with blocks and
    # have a size of 512 bytes since 2.4 kernels. This value is
    # needed to calculate the amount of disk I/O in bytes.
    SECTOR_SIZE = 512

    # determine partitions we want to look for
    partitions = []
    f = open("/proc/partitions", "r")
    try:
        lines = f.readlines()[2:]
    finally:
        f.close()
    for line in reversed(lines):
        _, _, _, name = line.split()
        if name[-1].isdigit():
            partitions.append(name)
        else:
           if check_partition_match(name,partitions) != True:
                partitions.append(name)

I'm sure this could probably be written more efficiently but it seems to work 
correctly!

Regards

James

Original comment by ja...@cirrushq.com on 21 Feb 2013 at 9:21

[GoogleCodeExporter]

Could you guys please try the patch in attachment and confirm everything works 
as expected?

Original comment by g.rodola on 23 Feb 2013 at 3:25

Attachments:

• disk-io-linux.patch

[GoogleCodeExporter]

Updated patch including comments and unit tests.

Original comment by g.rodola on 23 Feb 2013 at 3:48

Attachments:

• disk-io-linux.patch

[GoogleCodeExporter]

Hello,

Tried with this patch, everything works correctly for me.

Regards,

Original comment by mani...@gmail.com on 24 Feb 2013 at 4:10

[GoogleCodeExporter]

Fixed in r1565.

Original comment by g.rodola on 24 Feb 2013 at 6:56

• Changed state: FixedInSVN
• Added labels: Milestone-0.6.2

[GoogleCodeExporter]

Thanks, to confirm, it also works correctly for me.

Original comment by ja...@cirrushq.com on 25 Feb 2013 at 9:52

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Updated csets after the SVN -> Mercurial migration:
r1565 == revision 97061fc808e6

Original comment by g.rodola on 2 Mar 2013 at 12:13

[GoogleCodeExporter]

Original comment by g.rodola on 8 Apr 2013 at 1:21

• Changed state: FixedInHG

[GoogleCodeExporter]

Original comment by g.rodola on 11 Apr 2013 at 9:17

• Added labels: Milestone-0.7.0
• Removed labels: Milestone-0.6.2

[GoogleCodeExporter]

Original comment by g.rodola on 12 Apr 2013 at 6:21

• Changed state: Fixed