search

multipath-tcp / mptcp

⚠️⚠️⚠️ Deprecated 🚫 Out-of-tree Linux Kernel implementation of MultiPath TCP. 👉 Use https://github.com/multipath-tcp/mptcp_net-next repo instead ⚠️⚠️⚠️
https://github.com/multipath-tcp/mptcp_net-next
Other
889 stars 335 forks source link

Use after free warning in lib/refcount #363

Open cifvts opened 4 years ago

cifvts commented 4 years ago

Hello guys, I've found this error in one of my server. Nothing else seems happening around it but let me know if I can do anything to provide more information.

[16751.877150] ------------[ cut here ]------------
[16751.877151] refcount_t: underflow; use-after-free.
[16751.877173] WARNING: CPU: 1 PID: 17 at lib/refcount.c:187 refcount_sub_and_test_checked+0x3e/0x50
[16751.877174] Modules linked in: cls_fw cls_u32 xt_tcpudp sch_htb xt_policy drbg ansi_cprng authenc echainiv xfrm6_mode_tunnel xfrm4_mode_tunnel ipip fou ip_tunnel ip6_udp_tunnel udp_tunnel tun xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 af_key xfrm_algo dummy ip6table_filter ip6_tables nfit libnvdimm crct10dif_pclmul iptable_filter crc32_pclmul xt_statistic xt_mark ghash_clmulni_intel xt_connmark pcbc xt_conntrack xt_TPROXY nf_tproxy_ipv6 nf_tproxy_ipv4 xt_addrtype iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat aesni_intel nf_conntrack aes_x86_64 crypto_simd nf_defrag_ipv6 nf_defrag_ipv4 cryptd glue_helper libcrc32c ppdev snd_pcsp evdev snd_pcm intel_rapl_perf snd_timer serio_raw snd soundcore parport_pc parport button mptcp_fullmesh ip_tables x_tables ext4 crc32c_generic
[16751.877209]  crc16 mbcache jbd2 fscrypto dm_mod crc32c_intel nvme ena nvme_core i2c_piix4
[16751.877217] CPU: 1 PID: 17 Comm: ksoftirqd/1 Not tainted 4.19.67-mat-0.95 #8
[16751.877217] Hardware name: Amazon EC2 m5.xlarge/, BIOS 1.0 10/16/2017
[16751.877219] RIP: 0010:refcount_sub_and_test_checked+0x3e/0x50
[16751.877220] Code: 75 0c f0 0f b1 16 75 27 85 d2 0f 94 c0 c3 80 3d 3c 7a d0 00 00 75 15 48 c7 c7 58 b9 87 b8 c6 05 2c 7a d0 00 01 e8 22 ac c8 ff <0f> 0b 31 c0 c3 83 f8 ff 75 bf eb f6 66 0f 1f 44 00 00 48 89 fe bf
[16751.877222] RSP: 0018:ffffb56681933d80 EFLAGS: 00010282
[16751.877223] RAX: 0000000000000000 RBX: ffff9457540a9f80 RCX: 0000000000000006
[16751.877223] RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff9457d2a966b0
[16751.877224] RBP: ffff94574fe82bc0 R08: 0000000000000001 R09: 00000000000001da
[16751.877225] R10: ffff94574fe82bc0 R11: 0000000000000000 R12: ffff94579c1483f8
[16751.877226] R13: ffffffffb807c490 R14: dead000000000200 R15: 0000000000000003
[16751.877227] FS:  0000000000000000(0000) GS:ffff9457d2a80000(0000) knlGS:0000000000000000
[16751.877228] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[16751.877228] CR2: 00007f8de3ca8ff8 CR3: 000000011b60a006 CR4: 00000000007606e0
[16751.877231] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[16751.877232] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[16751.877233] PKRU: 55555554
[16751.877233] Call Trace:
[16751.877237]  mptcp_mpcb_put+0x12/0x50
[16751.877240]  mptcp_sock_destruct+0x71/0xf0
[16751.877243]  __sk_destruct+0x24/0x1c0
[16751.877247]  call_timer_fn+0x2b/0x130
[16751.877249]  run_timer_softirq+0x1d3/0x420
[16751.877251]  ? __switch_to_asm+0x41/0x70
[16751.877252]  ? __switch_to+0x8c/0x450
[16751.877253]  ? __switch_to_asm+0x41/0x70
[16751.877254]  ? __switch_to_asm+0x35/0x70
[16751.877257]  __do_softirq+0x10d/0x2c3
[16751.877261]  run_ksoftirqd+0x26/0x40
[16751.877264]  smpboot_thread_fn+0x10e/0x160
[16751.877266]  kthread+0xf8/0x130
[16751.877268]  ? sort_range+0x20/0x20
[16751.877269]  ? kthread_create_worker_on_cpu+0x70/0x70
[16751.877270]  ret_from_fork+0x35/0x40
[16751.877272] ---[ end trace 2dc0947227a2bc2a ]---
cpaasch commented 4 years ago

There is one place in mptcp_sub_close where we inc the refcnt and then schedule the work-queue. That could fail though, so ideally we should decrement again.

However, that does not explain the issue with the underflow.

@cifvts - Are there any other warnings,... above this particular warning here?

cifvts commented 4 years ago

I looked all my server and all of them has the same warning happening after a while. Nothing else seems to happens around that time but I've attached the logs so you can look better. logs.tar.gz

cpaasch commented 4 years ago

Thanks for the logs! It's yet unclear to me how this happens. I'm trying to trigger it with syzkaller. Let's see...

user747 commented 3 years ago

sorry to nercobump but adding this in case it helps

using trunk branch

uname -a
Linux test 5.4.69 #0 SMP Fri Oct 9 01:10:05 2020 aarch64 GNU/Linux
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97442.717021] ------------[ cut here ]------------
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97442.721650] refcount_t: increment on 0; use-after-free.
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97442.726907] WARNING: CPU: 0 PID: 0 at lib/refcount.c:156 refcount_inc_checked+0x40/0x48
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97442.734902] Modules linked in: ath9k ath9k_htc ath9k_common rtl8192cu rtl8192c_common rtl_usb rt2800usb rt2800lib qcserial pppoe ppp_async option ipw cdc_mbim brcmfmac ath9k_hw ath usb_wwan usb_serial_simple ti_usb_3410_5052 sr9700 smsc95xx smsc75xx sierra_net sierra rtlwifi rtl8xxxu rtl8187 rt2x00usb rt2x00lib rndis_host qmi_wwan pppox ppp_generic pl2303 oti6858 mt7601u mos7720 mmc_spi mct_u232 mcs7830 mac80211 lzo keyspan kalmia ipt_REJECT huawei_cdc_ncm garmin_gps ftdi_sio ebtable_nat ebtable_filter ebtable_broute dm9601 cypress_m8 cp210x ch341 cfg80211 cdc_subset cdc_ncm cdc_ether cdc_eem belkin_sa ax88179_178a asix ark3116 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_socket xt_recent xt_quota xt_policy xt_pkttype xt_owner xt_ndpi xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_iface xt_hl xt_helper xt_hashlimit xt_esp xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_condition xt_comment xt_addrtype xt_TRACE xt_TPROXY xt_TCPMSS xt_REDIRECT
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97442.734968]  xt_MASQUERADE xt_LOG xt_IPMARK xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY xt_ACCOUNT visor via_velocity via_rhine usbserial usbnet usbhid tulip ts_fsm ts_bm solos_pci slhc sky2 skge sis900 sis190 sch_cake rtl8150 r8712u(C) r8169 r8152 r6040 pegasus pcnet32 of_mmc_spi nf_tproxy_ipv6 nf_tproxy_ipv4 nf_socket_ipv6 nf_socket_ipv4 nf_reject_ipv4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_rtcache nf_conntrack_pptp nf_conntrack_netlink nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_broadcast ts_kmp nf_conntrack_amanda nf_conncount ne2k_pci macvlan lzo_decompress lzo_compress kaweth iptable_raw iptable_nat iptable_mangle iptable_filter ipt_ah ipt_ECN ipheth ip6table_raw ip_tables hso hid_generic forcedeth ezusb ethoc et131x ebtables ebt_vlan ebt_stp ebt_redirect ebt_pkttype ebt_mark_m ebt_mark
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97442.821794]  ebt_limit ebt_among ebt_802_3 e1000e e100 crc7 crc_itu_t crc_ccitt compat_xtables compat cdc_wdm cdc_acm brcmutil bnx2 atl2 atl1e atl1c atl1 asn1_decoder arptable_filter arpt_mangle arp_tables 8390 8250_pci 8139too 8139cp tcp_nanqinlang sch_teql sch_sfq sch_red sch_prio sch_pie sch_multiq sch_gred sch_fq sch_dsmark sch_codel em_text em_nbyte em_meta em_cmp act_simple act_police act_pedit act_ipt act_gact act_csum sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred snd_bcm2835(C) hid evdev i2c_gpio i2c_algo_bit i2c_dev spi_ks8995 ledtrig_heartbeat xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6table_nat nf_nat nf_conntrack nf_defrag_ipv6
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97442.908969]  nf_defrag_ipv4 ip6t_NPT nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 swconfig 3c59x ip6_gre ip_gre gre vmxnet3 e1000 ifb nat46 sit sctp libcrc32c ipcomp6 xfrm6_tunnel esp6 ah6 xfrm4_tunnel ipcomp esp4 ah4 ip6_tunnel netlink_diag tunnel6 tunnel4 ip_tunnel hfcpci hfcmulti veth tun snd_rawmidi snd_seq_device snd_pcm_oss snd_pcm_dmaengine snd_pcm snd_timer snd_mixer_oss snd_hwdep snd_compress snd soundcore mISDN_dsp l1oip mISDN_core xfrm_user xfrm_ipcomp af_key xfrm_algo autofs4 br2684 atm nls_utf8 zram zsmalloc natsemi eeprom_93cx6 sha1_generic md5 ghash_generic gf128mul gcm echainiv des_generic libdes deflate zlib_inflate zlib_deflate authenc crypto_acompress vfat fat nls_iso8859_1 nls_cp437 ahci libahci libata fsl_mph_dr_of ehci_platform ehci_fsl ehci_hcd gpio_button_hotplug tg3 b44 ssb ptp realtek pps_core
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.074406] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C        5.4.69 #0
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.081706] Hardware name: Raspberry Pi 4 Model B Rev 1.1 (DT)
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.087531] pstate: 60400005 (nZCv daif +PAN -UAO)
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.092314] pc : refcount_inc_checked+0x40/0x48
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.096836] lr : refcount_inc_checked+0x40/0x48
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.101356] sp : ffffffc010003df0
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.104660] x29: ffffffc010003df0 x28: ffffffc0109c50c0
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.109964] x27: ffffffc0109ad598 x26: ffffff807fb997b0
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.115267] x25: 0000000000000002 x24: dead000000000100
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.120570] x23: dead000000000122 x22: ffffff80775cb688
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.125873] x21: ffffffc010ab2000 x20: ffffff8073aab600
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.131175] x19: ffffff80775cb600 x18: 0000000000000000
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.136478] x17: 0000000000000000 x16: 0000000000000001
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.141781] x15: 0000000000000000 x14: 0720072007200720
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.147084] x13: 0720072007200720 x12: 0720072007200720
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.152386] x11: 0720072007200720 x10: 0720072007200720
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.157689] x9 : 07200720072e0765 x8 : 076507720766072d
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.162992] x7 : 0772076507740766 x6 : 0000000000000001
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.168294] x5 : ffffffc0103908b0 x4 : 0000000000000001
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.173597] x3 : ffffffc0109c8ce4 x2 : 0000000000000004
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.178900] x1 : 0000000000000004 x0 : 000000000000002b
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.184203] Call trace:
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.186641]  refcount_inc_checked+0x40/0x48
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.190819]  tcp_tsq_handler+0x108/0x160
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.194733]  tcp_tasklet_func+0xc8/0x100
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.198648]  tasklet_action_common.isra.19+0xac/0x150
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.203691]  tasklet_action+0x24/0x30
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.207344]  __do_softirq+0x11c/0x250
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.210997]  irq_exit+0x9c/0xb8
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.214129]  __handle_domain_irq+0x64/0xb8
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.218215]  gic_handle_irq+0x5c/0xb8
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.221868]  el1_irq+0xf0/0x1c0
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.225001]  arch_cpu_idle+0x10/0x18
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.228568]  do_idle+0x1e4/0x258
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.231787]  cpu_startup_entry+0x24/0x78
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.235701]  rest_init+0xb0/0xbc
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.238921]  arch_call_rest_init+0xc/0x14
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.242921]  start_kernel+0x3c4/0x3dc
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.246573] ---[ end trace b8b637b795f1dacf ]---