Closed prometheanfire closed 11 years ago
This looks new to me. The core would be helpful but I don't have the symbols for your build of the zfs tools. Can you fire up gdb and post the backtrace from the core.
gdb /sbin/zfs <core>
bt
here you go
now with more debug symbols
This indicates that the ZFS_IOC_SEND ioctl which was passed to the kernel contained an invalid argument. It would be useful to see what those arguments were, perhaps there was an invalid snapshot name. Can you either:
dump_ioctl()
with a few printf's to output the zc.zc_name, zc.zc_cookie, zc.zc_obj, zc.zc_sendobj, and zc.zc_fromobj value before the call to zio_ioctl
. Or,trace-cmd
. You should be able to run the following which will show where the EINVAL is being returned in the kernel.trace-cmd record -p function_graph -g zfs_ioc_send zfs send ...
Here's the patch http://bpaste.net/show/89072/
did this help at all?
Static analysis from issue #1392 suggests a possible cause:
[C:\repo\zfs\lib\libzfs\libzfs_sendrecv.c:2824]: (error) Dereferencing 'cp' after it is deallocated / released
@behlendorf As easy as it should be to fix that, I do not have time right now, so it is all yours.
@prometheanfire Let me get you a patch to try to provide the needed debugging (maybe today). Also if you get a chance the trace-cmd
log would be very helpful.
@ryao I don't think #1392 is likely the root cause since the reason for the core is due to an explicit abort()
not a dereference of an invalid address.
@behlendorf I don't have access to the trace-cmd but do you have that patch for me to test?
I narrowed it down to a specific dataset that I cannot send.
zfs list NAME USED AVAIL REFER MOUNTPOINT khorne-m4-zp00 107G 125G 136K none khorne-m4-zp00/GENTOO 10.5G 125G 136K none khorne-m4-zp00/GENTOO/distfiles 5.97G 125G 5.97G /usr/portage/distfiles khorne-m4-zp00/GENTOO/packages 3.61G 125G 3.61G /usr/portage/packages khorne-m4-zp00/GENTOO/portage 983M 125G 923M /usr/portage khorne-m4-zp00/HOME 91.0G 125G 144K /home khorne-m4-zp00/HOME/mthode 91.0G 125G 90.9G /home/mthode khorne-m4-zp00/HOME/root 30.9M 125G 30.8M /root khorne-m4-zp00/ROOT 5.76G 125G 136K none khorne-m4-zp00/ROOT/rootfs 5.76G 125G 48.8M / khorne-m4-zp00/ROOT/rootfs/OPT 372M 125G 372M /opt khorne-m4-zp00/ROOT/rootfs/USR 5.09G 125G 5.06G /usr khorne-m4-zp00/ROOT/rootfs/VAR 269M 125G 267M legacy
zfs list -t snap NAME USED AVAIL REFER MOUNTPOINT khorne-m4-zp00@20130506-backup 0 - 136K - khorne-m4-zp00/GENTOO@20130506-backup 0 - 136K - khorne-m4-zp00/GENTOO/distfiles@20130506-backup 0 - 5.97G - khorne-m4-zp00/GENTOO/packages@20130506-backup 0 - 3.61G - khorne-m4-zp00/GENTOO/portage@20130506-backup 15.8M - 923M - khorne-m4-zp00/HOME@20130506-backup 0 - 144K - khorne-m4-zp00/HOME/mthode@20130506-backup 35.7M - 90.9G - khorne-m4-zp00/HOME/root@20130506-backup 0 - 30.8M - khorne-m4-zp00/ROOT@20130506-backup 0 - 136K - khorne-m4-zp00/ROOT/rootfs@20130506-backup 184K - 48.8M - khorne-m4-zp00/ROOT/rootfs/OPT@20130506-backup 0 - 372M - khorne-m4-zp00/ROOT/rootfs/USR@20130506-backup 13.1M - 5.06G - khorne-m4-zp00/ROOT/rootfs/VAR@20130506-backup 520K - 267M -
works zfs send -R khorne-m4-zp00/GENTOO@${date}-backup | pigz -c > /mnt/backups/laptop/${date}-GENTOO-zfssend.gz zfs send -R khorne-m4-zp00/HOME@${date}-backup | pigz -c > /mnt/backups/laptop/${date}-HOME-zfssend.gz zfs send -R khorne-m4-zp00/ROOT/rootfs@${date}-backup | pigz -c > /mnt/backups/laptop/${date}-rootfs-zfssend.gz
doesn't work zfs send -R khorne-m4-zp00/ROOT@${date}-backup | pigz -c > /mnt/backups/laptop/${date}-ROOT-zfssend.gz internal error: Invalid argument
zfs send khorne-m4-zp00/ROOT@20130506-backup > asd internal error: Invalid argument Aborted
zdb khorne-m4-zp00/ROOT@20130506-backup Dataset khorne-m4-zp00/ROOT@20130506-backup [ZPL], ID 271, cr_txg 2143498, 136K, 6 objects
Object lvl iblk dblk dsize lsize %full type
0 7 16K 16K 56.0K 16K 18.75 DMU dnode
-1 1 16K 512 8K 512 100.00 ZFS user/group used
-2 1 16K 512 8K 512 100.00 ZFS user/group used
1 1 16K 512 8K 512 100.00 ZFS master node
2 1 16K 512 8K 512 100.00 SA master node
3 1 16K 512 8K 512 100.00 ZFS delete queue
4 1 16K 512 8K 512 100.00 ZFS directory
5 1 16K 1.50K 8K 1.50K 100.00 SA attr registration
6 1 16K 16K 16K 32K 100.00 SA attr layouts
zdb khorne-m4-zp00/ROOT
Dataset khorne-m4-zp00/ROOT [ZPL], ID 38, cr_txg 11, 136K, 6 objects
Object lvl iblk dblk dsize lsize %full type
0 7 16K 16K 56.0K 16K 18.75 DMU dnode
-1 1 16K 512 8K 512 100.00 ZFS user/group used
-2 1 16K 512 8K 512 100.00 ZFS user/group used
1 1 16K 512 8K 512 100.00 ZFS master node
2 1 16K 512 8K 512 100.00 SA master node
3 1 16K 512 8K 512 100.00 ZFS delete queue
4 1 16K 512 8K 512 100.00 ZFS directory
5 1 16K 1.50K 8K 1.50K 100.00 SA attr registration
6 1 16K 16K 16K 32K 100.00 SA attr layouts
I noticed a couple of odd things as well about the dataset that is giving the errors.
NAME TYPE CREATION USED AVAIL REFER RATIO MOUNTED ORIGIN QUOTA RESERV VOLSIZE VOLBLOCK RECSIZE MOUNTPOINT SHARENFS CHECKSUM COMPRESS ATIME DEVICES EXEC SETUID RDONLY ZONED SNAPDIR ACLINHERIT CANMOUNT XATTR COPIES VERSION UTF8ONLY NORMALIZATION CASE VSCAN NBMAND SHARESMB REFQUOTA REFRESERV PRIMARYCACHE SECONDARYCACHE USEDSNAP USEDDS USEDCHILD USEDREFRESERV DEFER_DESTROY USERREFS LOGBIAS DEDUP MLSLABEL SYNC REFRATIO WRITTEN CLONES SNAPDEV
khorne-m4-zp00/GENTOO filesystem Fri Aug 31 15:55 2012 12.5G 72.1G 136K 1.00x no - none none - - 128K none off on off on on on on off off hidden restricted on on 1 5 on formD sensitive off off off none none all all 0 136K 12.5G 0 - - latency off none standard 1.00x 0 - hidden
khorne-m4-zp00/ROOT filesystem Fri Aug 31 15:55 2012 8.91G 72.1G 136K 1.93x no - none none - - 128K none off on on on on on on off off hidden restricted on on 1 - - - - off off off none none all all 0 136K 8.91G 0 - - latency off none standard 1.00x 0 - hidden
these pop out at me
version = - utf8only = - normalization = - case = -
The output of trace-cmd revealed that failing call path in the kernel is the following which returns EINVAL.
zfs_ioc_send()->dmu_send()->zfs_get_zplprop()
Unfortunately the error code from zfs_get_zplprop() is lost we don't know exactly why it was unable to fetch the version. However, based on the previous comment I suspect that somehow the property is outright missing. How that could happen isn't at all clear, but this chunk of kernel code can't cope with it and correctly errors out.
You can apply the following patch to get the exact error code.
diff --git a/module/zfs/dmu_send.c b/module/zfs/dmu_send.c
index b256803..79ce9da 100644
--- a/module/zfs/dmu_send.c
+++ b/module/zfs/dmu_send.c
@@ -464,7 +464,10 @@ dmu_send(objset_t *tosnap, objset_t *fromsnap, boolean_t fr
#ifdef _KERNEL
if (dmu_objset_type(tosnap) == DMU_OST_ZFS) {
uint64_t version;
- if (zfs_get_zplprop(tosnap, ZFS_PROP_VERSION, &version) != 0) {
+ err = zfs_get_zplprop(tosnap, ZFS_PROP_VERSION, &version);
+ if (err) {
+ printk("zfs_get_zplprop(%p, %d, %d) = %d\n",
+ tosnap, ZFS_PROP_VERSION, version, err);
kmem_free(drr, sizeof (dmu_replay_record_t));
return (EINVAL);
}
If you're more ambitious simply you can apply this path to comment out the version check and set version=ZPL_VERSION_SA
for the purposes of a test. It may be able to successfully send the dataset.
diff --git a/module/zfs/dmu_send.c b/module/zfs/dmu_send.c
index b256803..3157b37 100644
--- a/module/zfs/dmu_send.c
+++ b/module/zfs/dmu_send.c
@@ -463,10 +463,13 @@ dmu_send(objset_t *tosnap, objset_t *fromsnap, boolean_t f
#ifdef _KERNEL
if (dmu_objset_type(tosnap) == DMU_OST_ZFS) {
- uint64_t version;
+ uint64_t version = ZPL_VERSION_SA;
+
+#if 0
if (zfs_get_zplprop(tosnap, ZFS_PROP_VERSION, &version) != 0) {
kmem_free(drr, sizeof (dmu_replay_record_t));
return (EINVAL);
+#endif
}
if (version == ZPL_VERSION_SA) {
DMU_SET_FEATUREFLAGS(
I was more ambitious
./tmp/trace-cmd/trace-cmd record -p function_graph -F zfs send khorne-m4-zp00/ROOT@20130807 > /mnt/20130807-backup-zfssend.gz plugin function_graph warning: cannot send 'khorne-m4-zp00/ROOT@20130807': Input/output error ^Ctrace-cmd: Interrupted system call trace-cmd: Interrupted system call recorder error in splice inputrecorder error in splice input
trace-cmd: Interrupted system call recorder error in splice input offset=200000 offset=3c0000 offset=3d9000 offset=4eb000
http://dev.gentoo.org/~prometheanfire/crash-logs/zfs-send-trace-2.out.xz
closing this since it seems like it's just corruption on my end
pool: khorne-m4-zp00 state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://zfsonlinux.org/msg/ZFS-8000-8A scan: scrub in progress since Thu Aug 8 14:40:30 2013 152G scanned out of 156G at 166M/s, 0h0m to go 0 repaired, 96.95% done config:
NAME STATE READ WRITE CKSUM
khorne-m4-zp00 ONLINE 0 0 266
root ONLINE 0 0 1.04K
errors: Permanent errors have been detected in the following files:
khorne-m4-zp00:<0x0>
khorne-m4-zp00/ROOT:<0x1>
khorne-m4-zp00/ROOT:<0x3>
<0xcd>:<0x1>
<0xea>:<0x0>
I ran into something that looks quite similar to this today. I've been using the ZFS Stable Ubuntu PPA (Raring) and just today updated my ZFS debs from 0.6.1 to 0.6.2. After the update, my recursive zfs sends no longer worked:
sudo zfs send -R -v -I griffin-ssd@20130906_020002 griffin-ssd@20130906_221801 | sudo zfs receive griffin-sata/REP/griffin-ssd
internal error: Invalid argument
I downgraded to 0.6.1 and the send worked properly. Unfortunately I'm in the middle of a couple projects and am not totally comfortable running additional tests/patches right now. However, I would be happy to do so in a few weeks if you would like additional testing.
@ziuchkovski There was an accidentally incompatibility introduced between the 0.6.2 user space utilities and the 0.6.1 kernel modules related to zfs send
. We saw the same thing after the 0.6.2 tag was made. Make sure your're using the 0.6.2 modules if you install the 0.6.2 user space utilities. You'll need to manual unload and load the new kernel modules. Rolling the user space utilities back to 0.6.1 will avoid the issue as well which you've already discovered.
@behlendorf Thanks, that was the problem. Userspace/module incompatibility didn't occur to me. Reloading the modules after the 0.6.2 upgrade fixed the issue.
@ziuchkovski Glad to hear it. Unfortunately, you should expect similar breakage when updating to 0.6.3 it looks like it's going to be unavoidable. Sorry.
@behlendorf Good to know, thanks. Now I know what to look for.
zfs send -Rvvvvvvvv khorne-m4-zp00@20130403-backup | pigz -c > /mnt/backups/${date}-backup-zfssend.gz send from @ to khorne-m4-zp00@20130403-backup estimated size is 15K send from @ to khorne-m4-zp00/ROOT@20130403-backup estimated size is 15K send from @ to khorne-m4-zp00/ROOT/rootfs@20130403-backup estimated size is 70.9M send from @ to khorne-m4-zp00/ROOT/rootfs/USR@20130403-backup estimated size is 5.30G send from @ to khorne-m4-zp00/ROOT/rootfs/OPT@20130403-backup estimated size is 595M send from @ to khorne-m4-zp00/ROOT/rootfs/VAR@20130403-backup estimated size is 1.04G send from @ to khorne-m4-zp00/GENTOO@20130403-backup estimated size is 14.5K send from @ to khorne-m4-zp00/GENTOO/portage@20130403-backup estimated size is 340M send from @ to khorne-m4-zp00/GENTOO/distfiles@20130403-backup estimated size is 4.67G send from @ to khorne-m4-zp00/GENTOO/packages@20130403-backup estimated size is 2.69G send from @ to khorne-m4-zp00/HOME@20130403-backup estimated size is 15.5K send from @ to khorne-m4-zp00/HOME/root@20130403-backup estimated size is 30.3M send from @ to khorne-m4-zp00/HOME/mthode@20130403-backup estimated size is 95.1G total estimated size is 110G TIME SENT SNAPSHOT internal error: Invalid argument
here, have a core dump http://dev.gentoo.org/~prometheanfire/crash-logs/zfs-core-0.6.1-20130403.dump