thomas-krenn / TKperf

Performance test tool for SSDs, HDDs and RAID devices
https://www.thomas-krenn.com/en/wiki/TKperf
GNU General Public License v3.0
50 stars 23 forks source link

TKPerf hangs with no output if mdadm thinks disks might be part of an existing array #18

Open mwberry opened 7 years ago

mwberry commented 7 years ago

I recently started a tkperf run against a number of disks. I wanted to validate the performance of the disks in a RAID 0 array. I started the test, noticed that it invoked mdadm and then I disconnected from my session to let it run. The next day I connected and noticed that mdadm was still running. The log file did not have anything in it indicating why it was taking so long or whether it was blocked. I straced mdadm and noticed that it was blocked on reading file descriptor 0, so I suspected that it was prompting the user to confirm something.

So I killed TKPerf and sure enough, if I ran the same mdadm command the first thing it did was prompt me that some of the disks have what looked like raid headers. After I killed TKPerf, it did log information into the log file, but due to what appears to be buffering that output didn't appear while it was hung.

TKPerf should probably do its own check and then zero-out possible RAID, LVM, and LUKS headers before calling any external tools. Alternatively, it could give mdadm additional arguments to auto-agree or force creation.

$ cat raid.cfg
{
    "devices": [
        "/dev/sdc",
        "/dev/sdd",
        "/dev/sde",
        "/dev/sdf",
        "/dev/sdg",
        "/dev/sdh",
        "/dev/sdi",
        "/dev/sdj"
    ],
    "raidlevel": 0,
    "type": "sw_mdadm"
}
$ sudo tkperf raid XXXX_raid_0_8 /dev/md0 -c raid.cfg 2>&1 | tee tkperf.log
# wait 24 hours
^C
$ cat XXXX_raid_0_8.log
20170522 00:36 root     INFO     # Initialiazing test lat
20170522 00:36 root     INFO     # Checking for device /dev/md0
20170522 00:36 root     INFO     # Creating raid device /dev/md0
20170522 00:36 root     INFO     # Command line: /sbin/mdadm --create /dev/md0 --quiet --metadata=default --level=0 --raid-devices=8 /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj
20170522 20:06 root     ERROR    mdadm encountered an error: mdadm: /dev/sdc appears to be part of a raid array:
    level=raid0 devices=0 ctime=Thu Jan  1 00:00:00 1970
mdadm: partition table exists on /dev/sdc but will be lost or
       meaningless after creating array
mdadm: /dev/sdd appears to be part of a raid array:
    level=raid0 devices=0 ctime=Thu Jan  1 00:00:00 1970
mdadm: partition table exists on /dev/sdd but will be lost or
       meaningless after creating array
mdadm: /dev/sde appears to be part of a raid array:
    level=raid0 devices=0 ctime=Thu Jan  1 00:00:00 1970
mdadm: partition table exists on /dev/sde but will be lost or
       meaningless after creating array
mdadm: /dev/sdf appears to be part of a raid array:
    level=raid0 devices=0 ctime=Thu Jan  1 00:00:00 1970
mdadm: partition table exists on /dev/sdf but will be lost or
       meaningless after creating array
mdadm: /dev/sdg appears to be part of a raid array:
    level=raid0 devices=0 ctime=Thu Jan  1 00:00:00 1970
mdadm: partition table exists on /dev/sdg but will be lost or
       meaningless after creating array
mdadm: /dev/sdh appears to be part of a raid array:
    level=raid0 devices=0 ctime=Thu Jan  1 00:00:00 1970
mdadm: partition table exists on /dev/sdh but will be lost or
       meaningless after creating array
mdadm: /dev/sdi appears to be part of a raid array:
    level=raid0 devices=0 ctime=Thu Jan  1 00:00:00 1970
mdadm: partition table exists on /dev/sdi but will be lost or
       meaningless after creating array
mdadm: /dev/sdj appears to be part of a raid array:
    level=raid0 devices=0 ctime=Thu Jan  1 00:00:00 1970
mdadm: partition table exists on /dev/sdj but will be lost or
       meaningless after creating array
Continue creating array? 
20170522 20:06 root     ERROR    # Could not fetch initial information for /dev/md0
mwberry commented 7 years ago
$ tkperf --version
TKperf Version: 2.1
$ git log | head -n 1
commit 8967b6dbf2b701b83cdef229f1ec8fa088718748
gschoenberger commented 6 years ago

I've taken a look at the man page, maybe the "-R" option helps?

-R, --run Insist that mdadm run the array, even if some of the components appear to be active in another array or filesystem. Normally mdadm will ask for confirmation before including such components in an array. This option causes that question to be suppressed.