behlendorf
commented
12 years ago Thanks for the additional debugging. It appears the following is happening (why I'm not sure)
- The rsync process is blocked waiting for free space in the next transaction group.
- The current transaction group is in the process of being sync'ed out and has issued multiple writes.
- One of those writes required that a new space map be loaded from disk and it's waiting for it to be read.
- For some reason that space map can't be read from disk yet isn't returning an error code. Everything is blocked waiting for this I/O to complete.
It would be interested to see both zpool events and zpool status to see if there are any issues with the drives.
stephane-chazelas
ryao
durin42
dswartz
akorn
v0.6.0.53 on debian 3.0.0-16 amd64
First time occurrence of this kind of problem.
Today, I've got a bunch of processes hung in "D" state (though the kernel doesn't report the tasks being hung).
Most I/Os to zfs (raidz1, compress, dedup) hang (system calls don't return) except for a few files (possible cached).
There is disk activity. dstat report heavy read access to all 6 drives in the zpool, though no end-user application seems to be doing any I/O. No write I/O to the disks.
No write I/O spotted.
The above sorted by cumulative read I/O over a period of time. I've been stracing 27852 all along, and no system call ever returned (it's been killed (SIGTERM) long before that but hasn't terminated yet)
There's no special zfs operation like removal of volumes ongoing.
a sysrq-w shows
for all the z_wr_iss threads.
and for blocked userspace application: