freebsd-pefs / pefs

PEFS - stacked cryptographic file system for FreeBSD
http://pefs.io/
76 stars 14 forks source link

kernel panics on i386/amd64 9.2 when writing to mounted filesystem on zfs #12

Open beyondcreed opened 10 years ago

beyondcreed commented 10 years ago

UPDATE: same happens on amd64 system when using pefs from pkgng, below is i386 example

uname -a FreeBSD node3 9.2-RELEASE FreeBSD 9.2-RELEASE #0 r255898: Fri Sep 27 03:52:52 UTC 2013 root@bake.isc.freebsd.org:/usr/obj/usr/src/sys/GENERIC i386

pkg info pefs*

pefs-kmod-2013.10.13

Dec 22 14:15:18 node3 kernel: Fatal trap 12: page fault while in kernel mode Dec 22 14:15:18 node3 kernel: cpuid = 0; apic id = 00 Dec 22 14:15:18 node3 kernel: fault virtual address = 0x4 Dec 22 14:15:18 node3 kernel: fault code = supervisor read, page not present Dec 22 14:15:18 node3 kernel: instruction pointer = 0x20:0xc052a0e7 Dec 22 14:15:18 node3 kernel: stack pointer = 0x28:0xeffe2a08 Dec 22 14:15:18 node3 kernel: frame pointer = 0x28:0xeffe2a18 Dec 22 14:15:18 node3 kernel: code segment = base 0x0, limit 0xfffff, type 0x1b Dec 22 14:15:18 node3 kernel: = DPL 0, pres 1, def32 1, gran 1 Dec 22 14:15:18 node3 kernel: processor eflags = interrupt enabled, resume, IOPL = 0 Dec 22 14:15:18 node3 kernel: current process = 1533 (vi) Dec 22 14:15:18 node3 kernel: trap number = 12 Dec 22 14:15:18 node3 kernel: panic: page fault Dec 22 14:15:18 node3 kernel: cpuid = 0

commands executed in following order ( note the use of camellia xts instead of aes xts)

UPDATE: same thing happens when default algorithm is in use ( AES128 XTS mode )

commands taken from https://wiki.freebsd.org/PEFS

mkdir ~/private ~/private.enc pefs addchain -a camellia128 -f -Z ~/private.enc pefs mount ~/private.enc ~/private pefs addkey -c ~/private cd ~/private vi hello trying to save file kernel panics, system reboots

root on zfs setup as per: https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/9.0-RELEASE

UPDATE:

pefs works on amd64 when using example from man pefs

       % mkdir ~/Private
       % pefs addchain -fZ ~/Private
       Enter parent key passphrase:
       Reenter parent key passphrase:
       % pefs mount ~/Private ~/Private
       % pefs addkey -c ~/Private
       Enter passphrase:
       ...
       % pefs unmount ~/Private

the only difference here is same directory is used in mount, could wiki have an error?

beyondcreed commented 10 years ago

closing this, it seems possibly wiki has error

glk commented 10 years ago

Can't reproduce on CURRENT, didn't install 9.2 vm to verify it yet.

What do you mean by "wiki error"?

Error log doesn't contain backtrace, backtrace would be really helpful.

Could you also check if problem persists if building pefs from ports. I don't know which kernel packages are build against. Kernel ABI is supposed to remain stable but could have been accidentally broken.

beyondcreed commented 10 years ago

by "wiki error" i meant this

https://wiki.freebsd.org/PEFS

pefs mount ~/private.enc ~/private

here private.enc and private are being used, in pefs man page example it is like this

pefs mount ~/Private ~/Private

If I use what wiki suggests, kernel panics, however the example in man page works fine

sorry didn't have backtrace, I can supply if this is still legitimate problem, however it is working fine for me now that I am using example from man page

yes the same problem persists even from ports, I tried building on amd64, but again this is only when I do "pefs mount ~/private.enc ~/private"

thanks.

On Tuesday 24 December 2013 20:01:50 Gleb Kurtsou wrote:

Can't reproduce on CURRENT, didn't install 9.2 vm to verify it yet.

What do you mean by "wiki error"?

Error log doesn't contain backtrace, backtrace would be really helpful.

Could you also check if problem persists if building pefs from ports. I don't know which kernel packages are build against. Kernel ABI is supposed to remain stable but could have been accidentally broken.


Reply to this email directly or view it on GitHub: https://github.com/glk/pefs/issues/12#issuecomment-31191090

Alex Pavlovic - CEO Taskforce-1 Industries Ltd. http://www.taskforce-1.com +1.888.630.5733 Ext. 500 Google+: alex.pavlovic@taskforce-1.com Skype: alex.pavlovic.tf1

glk commented 10 years ago

I was able to reproduce it with 9.2 packages.

I believe the problem to be KABI breakage between 9.0 and 9.2 when VOP_ADD_WRITECOUNT was introduced. Although it's not clear why "pefs mount /a /a" doesn't panic instantaneously..

The following VOP methods are initialized correctly:

  vop_add_writecount = 0xc7ab7360 <pefs_aes_encrypt>, 
  vop_spare4 = 0xc7ab7340 <pefs_aes_decrypt>, 
  vop_spare5 = 0xc7ab7310 <pefs_aes_keysetup>

Backtrace:

root@fbsd-92-i386:/var/crash # kgdb /boot/kernel/kernel.symbols vmcore.0
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...

Unread portion of the kernel message buffer:

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x4
fault code      = supervisor read, page not present
instruction pointer = 0x20:0xc052a0e7
stack pointer           = 0x28:0xe7fd9a08
frame pointer           = 0x28:0xe7fd9a18
code segment        = base 0x0, limit 0xfffff, type 0x1b
            = DPL 0, pres 1, def32 1, gran 1
processor eflags    = interrupt enabled, resume, IOPL = 0
current process     = 816 (vi)
trap number     = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
#0 0xc0b1810f at kdb_backtrace+0x4f
#1 0xc0adf38f at panic+0x16f
#2 0xc0f4c563 at trap_fatal+0x323
#3 0xc0f4c668 at trap_pfault+0xf8
#4 0xc0f4da29 at trap+0x599
#5 0xc0f365cc at calltrap+0x6
#6 0xc0f6ef6a at VOP_ADD_WRITECOUNT_APV+0xaa
#7 0xc0b8e72c at vn_open_cred+0x57c
#8 0xc0b8e87b at vn_open+0x3b
#9 0xc0b89ccc at kern_openat+0x1ec
#10 0xc0b8a0e5 at kern_open+0x35
#11 0xc0b8a120 at sys_open+0x30
#12 0xc0f4ce53 at syscall+0x443
#13 0xc0f36661 at Xint0x80_syscall+0x21
Uptime: 3m3s
Physical memory: 1003 MB
Dumping 204 MB: 189 173 157 141 125 109 93 77 61 45 29 13

Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/modules/pefs.ko...done.
Loaded symbols for /boot/modules/pefs.ko
Reading symbols from /boot/kernel/crypto.ko...Reading symbols from /boot/kernel/crypto.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/crypto.ko
#0  doadump (textdump=1) at pcpu.h:249
249 pcpu.h: No such file or directory.
    in pcpu.h
(kgdb) bt
#0  doadump (textdump=1) at pcpu.h:249
#1  0xc0adf0d5 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:449
#2  0xc0adf3d2 in panic (fmt=<value optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:637
#3  0xc0f4c563 in trap_fatal (frame=0xe7fd99c8, eva=4)
    at /usr/src/sys/i386/i386/trap.c:1044
#4  0xc0f4c668 in trap_pfault (frame=0xe7fd99c8, usermode=0, eva=4)
    at /usr/src/sys/i386/i386/trap.c:896
#5  0xc0f4da29 in trap (frame=0xe7fd99c8) at /usr/src/sys/i386/i386/trap.c:555
#6  0xc0f365cc in calltrap () at /usr/src/sys/i386/i386/exception.s:170
#7  0xc052a0e7 in rijndael_encrypt (ctx=0x0, src=0x0, 
    dst=0xcd080700 "P\a\bН \a\bНа8jЗ4ь>Е")
    at /usr/src/sys/crypto/rijndael/rijndael-api.c:58
#8  0xc0f6ef6a in VOP_ADD_WRITECOUNT_APV (vop=0xc7abd580, a=0xe7fd9ae8)
    at vnode_if.c:4236
#9  0xc0b8e72c in vn_open_cred (ndp=0xe7fd9b5c, flagp=0xe7fd9c24, cmode=420, 
    vn_open_flags=0, cred=0xc6865800, fp=0xc59e94d0) at vnode_if.h:1791
#10 0xc0b8e87b in vn_open (ndp=0xe7fd9b5c, flagp=0xe7fd9c24, cmode=420, 
    fp=0xc59e94d0) at /usr/src/sys/kern/vfs_vnops.c:113
#11 0xc0b89ccc in kern_openat (td=0xc5bf02f0, fd=-100, 
    path=0x284cf090 <Address 0x284cf090 out of bounds>, pathseg=UIO_USERSPACE, 
    flags=1538, mode=438) at /usr/src/sys/kern/vfs_syscalls.c:1132
#12 0xc0b8a0e5 in kern_open (td=0xc5bf02f0, 
---Type <return> to continue, or q <return> to quit---q
path=0x284cf090 <Address 0x284cf090 Quit
) at /usr/src/sys/kern/vfs_syscalls.c:1083
#13 0xc0b8a120 in sys_open (td=0xc5bf02f0, uap=0xe7fd9ccc)
    at /usr/src/sys/kern/vfs_syscalls.c:1059
#14 0xc0f4ce53 in syscall (frame=0xe7fd9d08) at subr_syscall.c:135
#15 0xc0f36661 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:270
#16 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) fr 8
#8  0xc0f6ef6a in VOP_ADD_WRITECOUNT_APV (vop=0xc7abd580, a=0xe7fd9ae8)
    at vnode_if.c:4236
4236    vnode_if.c: No such file or directory.
    in vnode_if.c
(kgdb) p *vop
$1 = {vop_default = 0xc1261120, vop_bypass = 0, 
  vop_islocked = 0xc0b6a170 <vop_stdislocked>, 
  vop_lookup = 0xc0b67170 <vfs_cache_lookup>, 
  vop_cachedlookup = 0xc7ab6390 <pefs_lookup>, 
  vop_create = 0xc7ab55c0 <pefs_create>, vop_whiteout = 0, 
  vop_mknod = 0xc7ab4ed0 <pefs_mknod>, vop_open = 0xc7ab4250 <pefs_open>, 
  vop_close = 0xc7ab1390 <pefs_close>, vop_access = 0xc7ab3fb0 <pefs_access>, 
  vop_accessx = 0xc7ab3f40 <pefs_accessx>, 
  vop_getattr = 0xc7ab2610 <pefs_getattr>, 
  vop_setattr = 0xc7ab4020 <pefs_setattr>, vop_markatime = 0, 
  vop_read = 0xc7ab24f0 <pefs_read>, vop_write = 0xc7ab34e0 <pefs_write>, 
  vop_ioctl = 0xc7ab4380 <pefs_ioctl>, vop_poll = 0, vop_kqfilter = 0, 
  vop_revoke = 0, vop_fsync = 0xc0b6a460 <vop_stdfsync>, 
  vop_remove = 0xc7ab5450 <pefs_remove>, vop_link = 0xc7ab5350 <pefs_link>, 
  vop_rename = 0xc7ab5a20 <pefs_rename>, vop_mkdir = 0xc7ab58b0 <pefs_mkdir>, 
  vop_rmdir = 0xc7ab5730 <pefs_rmdir>, 
  vop_symlink = 0xc7ab5040 <pefs_symlink>, 
  vop_readdir = 0xc7ab3720 <pefs_readdir>, 
  vop_readlink = 0xc7ab1cd0 <pefs_readlink>, 
  vop_inactive = 0xc7ab1560 <pefs_inactive>, 
  vop_reclaim = 0xc7ab3050 <pefs_reclaim>, vop_lock1 = 0xc7ab2bb0 <pefs_lock>, 
  vop_unlock = 0xc7ab0f20 <pefs_unlock>, 
  vop_bmap = 0xc0b69a90 <vop_eopnotsupp>, 
  vop_strategy = 0xc0b6a030 <vop_panic>, 
---Type <return> to continue, or q <return> to quit---
  vop_getwritemount = 0xc7ab13f0 <pefs_getwritemount>, 
  vop_print = 0xc7ab1520 <pefs_print>, 
  vop_pathconf = 0xc7ab1290 <pefs_pathconf>, vop_advlock = 0, 
  vop_advlockasync = 0, vop_advlockpurge = 0, vop_reallocblks = 0, 
  vop_getpages = 0xc0b6a000 <vop_stdgetpages>, 
  vop_putpages = 0xc0b69fa0 <vop_stdputpages>, 
  vop_getacl = 0xc7ab13c0 <pefs_getacl>, vop_setacl = 0, vop_aclcheck = 0, 
  vop_closeextattr = 0, vop_getextattr = 0, vop_listextattr = 0, 
  vop_openextattr = 0, vop_deleteextattr = 0, vop_setextattr = 0, 
  vop_setlabel = 0, vop_vptofh = 0xc7ab1350 <pefs_vptofh>, 
  vop_vptocnp = 0xc0b6b130 <vop_stdvptocnp>, vop_allocate = 0, vop_advise = 0, 
  vop_unp_bind = 0, vop_unp_connect = 0, vop_unp_detach = 0, vop_is_text = 0, 
  vop_set_text = 0, vop_unset_text = 0, vop_get_writecount = 0, 
  vop_add_writecount = 0xc7ab7360 <pefs_aes_encrypt>, 
  vop_spare4 = 0xc7ab7340 <pefs_aes_decrypt>, 
  vop_spare5 = 0xc7ab7310 <pefs_aes_keysetup>}
(kgdb) q
beyondcreed commented 10 years ago

Just another update, on i386 mounting via same directory and trying to write as per below panics as well

   % mkdir ~/Private
   % pefs addchain -fZ ~/Private
   Enter parent key passphrase:
   Reenter parent key passphrase:
   % pefs mount ~/Private ~/Private
   % pefs addkey -c ~/Private
   Enter passphrase:
   % echo 'bye' > ~/Private/myfile
beyondcreed commented 10 years ago

Just builtworld with STABLE 10 on i386 as per below, mounting and writing works fine using man page example. Built pefs from ports afterwards.

       % mkdir ~/Private
       % pefs addchain -fZ ~/Private
       Enter parent key passphrase:
       Reenter parent key passphrase:
       % pefs mount ~/Private ~/Private
       % pefs addkey -c ~/Private
       % echo 'hello' > ~/Private/world

FreeBSD node3.grid1 10.0-PRERELEASE FreeBSD 10.0-PRERELEASE #1 r259961: Fri Dec 27 21:55:13 PST 2013 root@node3.grid1:/usr/obj/usr/src/sys/NODE3-GRID1 i386

glk commented 10 years ago

I've send follow up to the same issue discussed in Nov on stable@. No response yet.

http://marc.info/?l=freebsd-stable&m=138795016121536&w=2

droe commented 10 years ago

FWIW, I can confirm this on two 9.2-RELEASE-p3 systems, amd64, GENERIC kernel installed by freebsd-update, on UFS, with pefs installed through pkgng (pefs-kmod-2013.10.13), crashes kernel on first write:

% mkdir ~/Private
% pefs mount ~/Private ~/Private
% pefs addkey -c ~/Private
% echo test > ~/Private/test # <-- crashes the kernel

kgdb backtrace (via panicmail):

#0  doadump (textdump=<value optimized out>) at pcpu.h:234             
#1  0xffffffff8090d486 in kern_reboot (howto=260)                             
    at /usr/src/sys/kern/kern_shutdown.c:449                                  
#2  0xffffffff8090d987 in panic (fmt=0x1 <Address 0x1 out of bounds>)         
    at /usr/src/sys/kern/kern_shutdown.c:637                                  
#3  0xffffffff80b79eca in vm_fault_hold (map=0xfffffe0002000000,              
    vaddr=18446743523957809152, fault_type=1 '\001', fault_flags=0,           
    m_hold=0x0) at /usr/src/sys/vm/vm_fault.c:285                             
#4  0xffffffff80b7a663 in vm_fault (map=0xfffffe0002000000,                   
    vaddr=18446743523957809152, fault_type=<value optimized out>,             
    fault_flags=0) at /usr/src/sys/vm/vm_fault.c:229                          
#5  0xffffffff80cf264f in trap_pfault (frame=0xffffff80003e1700, usermode=0)  
    at /usr/src/sys/amd64/amd64/trap.c:772                                  
#6  0xffffffff80cf2a24 in trap (frame=0xffffff80003e1700)                     
    at /usr/src/sys/amd64/amd64/trap.c:463                                    
#7  0xffffffff80cdbd53 in calltrap ()                                         
    at /usr/src/sys/amd64/amd64/exception.S:232                               
#8  0xffffffff80343628 in rijndaelEncrypt (rk=0xffffff80003e1fe8,             
    Nr=<value optimized out>, pt=<value optimized out>,                       
    ct=0x4 <Address 0x4 out of bounds>)                                       
    at /usr/src/sys/crypto/rijndael/rijndael-alg-fst.c:966                    
#9  0xffffffff80d968d8 in VOP_ADD_WRITECOUNT_APV (vop=0xffffffff81851ea0,     
    a=0xffffff80003e1920) at vnode_if.c:4236                                  
#10 0xffffffff809bad48 in vn_open_cred (ndp=0xffffff80003e19d0,               
    flagp=0xffffff80003e19cc, cmode=420, vn_open_flags=<value optimized out>, 
    cred=0xfffffe00028b9e00, fp=0xfffffe0002abc7d0) at vnode_if.h:1791        
#11 0xffffffff809b676a in kern_openat (td=0xfffffe00028c9490, fd=-100,        
    path=0x80141f9c0 <Address 0x80141f9c0 out of bounds>,                     
    pathseg=UIO_USERSPACE, flags=1538, mode=<value optimized out>)            
    at /usr/src/sys/kern/vfs_syscalls.c:1132                                  
#12 0xffffffff80cf18ba in amd64_syscall (td=0xfffffe00028c9490, traced=0)     
    at subr_syscall.c:135                                                     
#13 0xffffffff80cdc037 in Xfast_syscall ()                                    
    at /usr/src/sys/amd64/amd64/exception.S:391                               
#14 0x0000000800de0d3c in ?? ()