ipmp crashes illumos

[vtoc]

After an update from 20180412T025853Z to 20180816T002409Z i got a reproducable crash on one of our nodes druring startup of one VM:

panic[cpu20]/thread=fffffe85bd11d480:
recursive mutex_enter, lp=fffffe861de6f400 owner=fffffe85bd11d480 thread=fffffe85bd11d480

fffffe00bc9812f0 unix:mutex_panic+58 ()
fffffe00bc981360 unix:mutex_vector_enter+2b7 ()
fffffe00bc981390 ip:ill_capability_done+4b ()
fffffe00bc981430 ip:ip_rput_dlpi_writer+86e ()
fffffe00bc981480 ip:ipsq_exit+81 ()
fffffe00bc9814c0 ip:ill_capability_wait+52 ()
fffffe00bc981500 ip:ill_dl_up+fd ()
fffffe00bc981590 ip:ipif_up+ed ()
fffffe00bc981630 ip:ip_sioctl_flags_tail+288 ()
fffffe00bc9816e0 ip:ip_sioctl_flags+42d ()
fffffe00bc981780 ip:ip_process_ioctl+102 ()
fffffe00bc9817d0 ip:ip_wput_nondata+1ba ()
fffffe00bc9817f0 ip:ip_helper_wput+23 ()
fffffe00bc981870 unix:putnext+225 ()
fffffe00bc981950 genunix:strdoioctl+2fd ()
fffffe00bc981b10 genunix:strioctl+348 ()
fffffe00bc981b70 genunix:ldi_ioctl+6c ()
fffffe00bc981bf0 ip:udp_ioctl+64 ()
fffffe00bc981c80 sockfs:so_ioctl+1b9 ()
fffffe00bc981cc0 sockfs:socket_ioctl+27 ()
fffffe00bc981d00 sockfs:socket_vop_ioctl+29 ()
fffffe00bc981d90 genunix:fop_ioctl+55 ()
fffffe00bc981eb0 genunix:ioctl+9b ()
fffffe00bc981f10 unix:brand_sys_sysenter+1d3 ()

This is caused by upping an IPMP interface, i could reproduce it a fresh VirtualBox instance using a new base-64-lts VM:

{
  "brand": "joyent",
  "image_uuid": "390639d4-f146-11e7-9280-37ae5c6d53d4",
  "zpool": "zones",
  "autoboot": true,
  "hostname": "ipmp.test.v",
  "delegate_dataset": true,
  "dns_domain": "test.v",
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "192.168.2.77",
      "netmask": "255.255.255.0",
      "gateway": "192.168.2.1",
      "allow_ip_spoofing": true,
      "primary": true
    },
    {
      "nic_tag": "admin",
      "ip": "192.168.2.78",
      "netmask": "255.255.255.0",
      "gateway": "192.168.2.1",
      "allow_ip_spoofing": true

    }
  ],
  "resolvers": [
    "192.168.2.1"
  ],
  "max_physical_memory": 512,
  "max_swap": 1024
}

All you need to do to crash the kernel is login to the VM and run:

ifconfig net0 deprecated -failover group ipmp0
ifconfig net1 deprecated -failover group ipmp0
ifconfig ipmp0 192.168.2.79 netmask 255.255.255.0 up

[melloc]

@vtoc This seems like it's probably fallout from OS-4683. I'll try to recreate this locally.

[danmcd]

A DEBUG kernel will not fail in a different way with this reproduction. Coredump will be forthcoming.

[danmcd]

Coredump is available upon request.

[danmcd]

How does this now-back-in-illumos-joyent OS-7069 alter anything about this?

[vtoc]

unchanged

Hostname: test
Release: 5.11
Kernel architecture: i86pc
Application architecture: amd64
Kernel version: SunOS 5.11 i86pc joyent_20181121T182506Z
Platform: i86pc

panic[cpu0]/thread=fffffe02b70cf420: 
recursive mutex_enter, lp=fffffe02c6256ac0 owner=fffffe02b70cf420 thread=fffffe02b70cf420

fffffe00056082f0 unix:mutex_panic+58 ()
fffffe0005608360 unix:mutex_vector_enter+2b7 ()
fffffe0005608390 ip:ill_capability_done+4b ()
fffffe0005608430 ip:ip_rput_dlpi_writer+86e ()
fffffe0005608480 ip:ipsq_exit+81 ()
fffffe00056084c0 ip:ill_capability_wait+5d ()
fffffe0005608500 ip:ill_dl_up+fd ()
fffffe0005608590 ip:ipif_up+ed ()
fffffe0005608630 ip:ip_sioctl_flags_tail+288 ()
fffffe00056086e0 ip:ip_sioctl_flags+42d ()
fffffe0005608780 ip:ip_process_ioctl+102 ()
fffffe00056087d0 ip:ip_wput_nondata+1ba ()
fffffe00056087f0 ip:ip_helper_wput+23 ()
fffffe0005608870 unix:putnext+225 ()
fffffe0005608950 genunix:strdoioctl+2fd ()
fffffe0005608b10 genunix:strioctl+348 ()
fffffe0005608b70 genunix:ldi_ioctl+6c ()
fffffe0005608bf0 ip:udp_ioctl+64 ()   
fffffe0005608c80 sockfs:so_ioctl+1b9 ()
fffffe0005608cc0 sockfs:socket_ioctl+27 ()
fffffe0005608d00 sockfs:socket_vop_ioctl+29 ()
fffffe0005608d90 genunix:fop_ioctl+55 ()
fffffe0005608eb0 genunix:ioctl+9b ()
fffffe0005608f10 unix:brand_sys_sysenter+1d3 ()