Stat of directory causes segfault, kernel BUG at fs/namei.c:1199

[rschlaikjer]

System information

Type	Version/Name
Distribution Name	Debian
Distribution Version	Jessie/Testing
Linux Kernel	4.10.12
Architecture	amd64
ZFS Version	0.6.5.9-5
SPL Version	0.6.5.9-1

Describe the problem you're observing

I have a build output directory that has become unstatable - attempting to remove the directory or inspect it with most programs causes the following userspace error:

ross@stirrup:/h/r/j/c/s/out$ strace stat Production/gen/chrome/app/policy/android/values-v21 2>&1 | tail -n 10
read(3, "", 4096)                       = 0
close(3)                                = 0
open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US.utf8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.utf8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
lstat("Production/gen/chrome/app/policy/android/values-v21",  <unfinished ...>) = ?
+++ killed by SIGSEGV +++

This causes a stacktrace to be emitted to syslog:

Jun 16 14:15:03 stirrup kernel: [688028.854162] ------------[ cut here ]------------
Jun 16 14:15:03 stirrup kernel: [688028.855562] kernel BUG at fs/namei.c:1199!
Jun 16 14:15:03 stirrup kernel: [688028.857379] invalid opcode: 0000 [#41] SMP
Jun 16 14:15:03 stirrup kernel: [688028.859192] Modules linked in: nls_utf8(E) nls_cp437(E) vfat(E) fat(E) uas(E) usb_storage(E) bnep(E) ipt_MASQUERADE(E) nf_nat_masquerade_ipv4(E) xfrm_user(E) xfrm_algo(E) iptable_nat(E) nf_conntrack_ipv4(E) nf_defrag_ipv4(E) nf_nat_ipv4(E) xt_addrtype(E) iptable_filter(E) xt_conntrack(E) nf_nat(E) nf_conntrack(E) libcrc32c(E) crc32c_generic(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) fuse(E) hid_generic(E) intel_rapl(E) usbhid(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) kvm_intel(E) kvm(E) irqbypass(E) crct10dif_pclmul(E) crc32_pclmul(E) ghash_clmulni_intel(E) pcbc(E) zfs(POE) snd_hda_codec_realtek(E) nouveau(E) snd_hda_codec_generic(E) zunicode(POE) ttm(E) zavl(POE) zcommon(POE) snd_hda_intel(E) drm_kms_helper(E) znvpair(POE) eeepc_wmi(E) aesni_intel(E) snd_hda_codec(E) drm(E) asus_wmi(E)
Jun 16 14:15:03 stirrup kernel: [688028.865126]  aes_x86_64(E) spl(OE) sparse_keymap(E) mxm_wmi(E) evdev(E) i2c_algo_bit(E) crypto_simd(E) snd_hda_core(E) glue_helper(E) snd_hwdep(E) cryptd(E) snd_pcm(E) snd_timer(E) iTCO_wdt(E) snd(E) iTCO_vendor_support(E) soundcore(E) sg(E) mei_me(E) shpchp(E) mei(E) serio_raw(E) pcspkr(E) hci_uart(E) btbcm(E) btqca(E) btintel(E) bluetooth(E) battery(E) rfkill(E) wmi(E) video(E) intel_lpss_acpi(E) intel_lpss(E) mfd_core(E) acpi_als(E) tpm_tis(E) kfifo_buf(E) tpm_tis_core(E) tpm(E) industrialio(E) button(E) acpi_pad(E) sunrpc(E) coretemp(E) adt7475(E) hwmon_vid(E) ip_tables(E) x_tables(E) autofs4(E) ext4(E) crc16(E) jbd2(E) mbcache(E) sd_mod(E) crc32c_intel(E) psmouse(E) ahci(E) libahci(E) i2c_i801(E) libata(E) r8169(E) xhci_pci(E) mii(E) xhci_hcd(E) scsi_mod(E) usbcore(E) fan(E) thermal(E) i2c_hid(E)
Jun 16 14:15:03 stirrup kernel: [688028.874815]  hid(E) fjes(E)
Jun 16 14:15:03 stirrup kernel: [688028.874818] CPU: 0 PID: 16538 Comm: stat Tainted: P      D W  OE   4.10.12-stirrup #1
Jun 16 14:15:03 stirrup kernel: [688028.874818] Hardware name: System manufacturer System Product Name/Z170-P, BIOS 0601 11/16/2015
Jun 16 14:15:03 stirrup kernel: [688028.874819] task: ffff9a09d972a080 task.stack: ffffa53d2f800000
Jun 16 14:15:03 stirrup kernel: [688028.874821] RIP: 0010:follow_managed+0x2e0/0x310
Jun 16 14:15:03 stirrup kernel: [688028.874821] RSP: 0018:ffffa53d2f803c30 EFLAGS: 00010246
Jun 16 14:15:03 stirrup kernel: [688028.874822] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000010
Jun 16 14:15:03 stirrup kernel: [688028.874822] RDX: ffff9a1030a12520 RSI: ffffa53d2f803d50 RDI: ffffa53d2f803cd8
Jun 16 14:15:03 stirrup kernel: [688028.874823] RBP: ffffa53d2f803cd8 R08: ffff9a0f40000000 R09: ffff9a05204ab045
Jun 16 14:15:03 stirrup kernel: [688028.874823] R10: ffff9a0f40000038 R11: 00000000ce645327 R12: 0000000003255777
Jun 16 14:15:03 stirrup kernel: [688028.874823] R13: ffff9a0f40000000 R14: ffffa53d2f803d50 R15: 0000000000000000
Jun 16 14:15:03 stirrup kernel: [688028.874824] FS:  00007feb25c9d3c0(0000) GS:ffff9a107e400000(0000) knlGS:0000000000000000
Jun 16 14:15:03 stirrup kernel: [688028.874824] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 16 14:15:03 stirrup kernel: [688028.874825] CR2: 0000556c271ba108 CR3: 0000000e0bb5f000 CR4: 00000000003406f0
Jun 16 14:15:03 stirrup kernel: [688028.874825] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jun 16 14:15:03 stirrup kernel: [688028.874825] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Jun 16 14:15:03 stirrup kernel: [688028.874826] Call Trace:
Jun 16 14:15:03 stirrup kernel: [688028.874827]  ? lookup_fast+0x1da/0x300
Jun 16 14:15:03 stirrup kernel: [688028.874828]  ? walk_component+0x48/0x450
Jun 16 14:15:03 stirrup kernel: [688028.874829]  ? path_lookupat+0x52/0x110
Jun 16 14:15:03 stirrup kernel: [688028.874829]  ? filename_lookup+0xb1/0x180
Jun 16 14:15:03 stirrup kernel: [688028.874831]  ? schedule+0x32/0x80
Jun 16 14:15:03 stirrup kernel: [688028.874833]  ? kmem_cache_alloc+0xf6/0x200
Jun 16 14:15:03 stirrup kernel: [688028.874834]  ? getname_flags+0x6f/0x1e0
Jun 16 14:15:03 stirrup kernel: [688028.874835]  ? vfs_fstatat+0x59/0xb0
Jun 16 14:15:03 stirrup kernel: [688028.874836]  ? SYSC_newlstat+0x2d/0x60
Jun 16 14:15:03 stirrup kernel: [688028.874837]  ? syscall_trace_enter+0x8c/0x2f0
Jun 16 14:15:03 stirrup kernel: [688028.874838]  ? do_sys_open+0x193/0x210
Jun 16 14:15:03 stirrup kernel: [688028.874839]  ? do_syscall_64+0x5c/0x170
Jun 16 14:15:03 stirrup kernel: [688028.874840]  ? entry_SYSCALL64_slow_path+0x25/0x25
Jun 16 14:15:03 stirrup kernel: [688028.874840] Code: e8 36 58 01 00 48 8b 14 24 e9 d3 fe ff ff 48 8b 7d 00 e8 b4 73 01 00 48 8b 14 24 e9 76 ff ff ff 48 83 f8 eb 74 12 44 89 e3 eb 9e <0f> 0b 4c 8b 6d 08 31 db e9 38 fd ff ff 41 f6 46 38 10 74 e7 45 
Jun 16 14:15:03 stirrup kernel: [688028.874849] RIP: follow_managed+0x2e0/0x310 RSP: ffffa53d2f803c30
Jun 16 14:15:03 stirrup kernel: [688028.874878] ---[ end trace e984190633c92eaa ]---

zdb output for the directory:

ross@stirrup:/h/ross$ sudo zdb -dddd tank/chromium/out 76606
Dataset tank/chromium/out [ZPL], ID 156, cr_txg 1395, 3.89G, 59458 objects, rootbp DVA[0]=<0:2353b50600:200> DVA[1]=<0:c01389800:200> [L0 DMU objset] fletcher4 lz4 LE contiguous unique double size=800L/200P birth=391202L/391202P fill=59458 cksum=13b51e2c97:6b776dcec2a:1341f7cc8a49e:268342585fcfff

    Object  lvl   iblk   dblk  dsize  lsize   %full  type
     76606    1    16K    512      0    512  100.00  ZFS directory
                                        168   bonus  System attributes
    dnode flags: USED_BYTES USERUSED_ACCOUNTED 
    dnode maxblkid: 0
    path    /Production/gen/chrome/app/policy/android/values-v21
    uid     1000
    gid     1000
    atime   Thu Jun  1 14:06:30 2017
    mtime   Thu Jun  1 14:07:19 2017
    ctime   Thu Jun  1 14:07:19 2017
    crtime  Thu Jun  1 14:06:30 2017
    gen 137660
    mode    40755
    size    3
    parent  76605
    links   2
    pflags  40800000144
    microzap: 512 bytes, 1 entries

        restriction_values.xml = 78223 (type: Regular File)

The BUG at fs/namei.c:1199 is this assert here: http://elixir.free-electrons.com/linux/v4.10.12/source/fs/namei.c#L1199

Unfortunately I do not know of a way to reproduce this issue.

[rschlaikjer]

Properties of the filesystem:

ross@stirrup:/h/ross$ sudo zfs get all tank/chromium/out
NAME               PROPERTY              VALUE                              SOURCE
tank/chromium/out  type                  filesystem                         -
tank/chromium/out  creation              Wed May 24 12:28 2017              -
tank/chromium/out  used                  3.89G                              -
tank/chromium/out  available             80.1G                              -
tank/chromium/out  referenced            3.89G                              -
tank/chromium/out  compressratio         1.00x                              -
tank/chromium/out  mounted               yes                                -
tank/chromium/out  quota                 none                               default
tank/chromium/out  reservation           none                               default
tank/chromium/out  recordsize            128K                               default
tank/chromium/out  mountpoint            /home/ross/chromium/src/out/  local
tank/chromium/out  sharenfs              off                                default
tank/chromium/out  checksum              on                                 default
tank/chromium/out  compression           off                                default
tank/chromium/out  atime                 off                                inherited from tank/chromium
tank/chromium/out  devices               on                                 default
tank/chromium/out  exec                  on                                 default
tank/chromium/out  setuid                on                                 default
tank/chromium/out  readonly              off                                default
tank/chromium/out  zoned                 off                                default
tank/chromium/out  snapdir               hidden                             default
tank/chromium/out  aclinherit            restricted                         default
tank/chromium/out  canmount              on                                 default
tank/chromium/out  xattr                 on                                 default
tank/chromium/out  copies                1                                  default
tank/chromium/out  version               5                                  -
tank/chromium/out  utf8only              off                                -
tank/chromium/out  normalization         none                               -
tank/chromium/out  casesensitivity       sensitive                          -
tank/chromium/out  vscan                 off                                default
tank/chromium/out  nbmand                off                                default
tank/chromium/out  sharesmb              off                                default
tank/chromium/out  refquota              none                               default
tank/chromium/out  refreservation        none                               default
tank/chromium/out  primarycache          all                                default
tank/chromium/out  secondarycache        all                                default
tank/chromium/out  usedbysnapshots       0                                  -
tank/chromium/out  usedbydataset         3.89G                              -
tank/chromium/out  usedbychildren        0                                  -
tank/chromium/out  usedbyrefreservation  0                                  -
tank/chromium/out  logbias               latency                            default
tank/chromium/out  dedup                 off                                default
tank/chromium/out  mlslabel              none                               default
tank/chromium/out  sync                  standard                           default
tank/chromium/out  refcompressratio      1.00x                              -
tank/chromium/out  written               3.89G                              -
tank/chromium/out  logicalused           3.87G                              -
tank/chromium/out  logicalreferenced     3.87G                              -
tank/chromium/out  filesystem_limit      none                               default
tank/chromium/out  snapshot_limit        none                               default
tank/chromium/out  filesystem_count      none                               default
tank/chromium/out  snapshot_count        none                               default
tank/chromium/out  snapdev               hidden                             default
tank/chromium/out  acltype               off                                default
tank/chromium/out  context               none                               default
tank/chromium/out  fscontext             none                               default
tank/chromium/out  defcontext            none                               default
tank/chromium/out  rootcontext           none                               default
tank/chromium/out  relatime              off                                default
tank/chromium/out  redundant_metadata    all                                default
tank/chromium/out  overlay               off                                default

[stale[bot]]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.