amarao
commented
9 years ago Update: I think it happens when I tried to add one more sFlow entry in the table:
ovs-vsctl -- --id=@br_int_sflow create sflow agent=eth0 target=\"192.168.11.42:6343\" header=64 sampling=1000 polling=10 -- set bridge br-int sflow=@br_int_sflow
blp
One of our nodes starts to trace aperiodically. We're gathing logs via remote syslog, here trace (it is a bit unordered because of UDP in syslog).
Configuration: openvswitch-switch 2.3.1 Linux: 3.13.0-52-generic
We using: neutron with vlans, sFlow (inside OVS) on 'external' bridge.
Trace:
[12803.874856] ------------[ cut here ]------------ [12803.876861] [] ? key_extract+0x6f8/0xa20 [openvswitch]
[12803.877797] [] ? kthread_create_on_node+0x1c0/0x1c0
[12803.877589] [] vhost_worker+0x101/0x1f0 [vhost]
[12803.875980] ffff8800720fa100 ffff881fff403ac0 ffff8800720fa100 ffff881fb6e58200
[12803.877383] [] netif_rx_ni+0x28/0x30
[12803.877504] [] handle_tx+0x1b9/0x4d0 [vhost_net]
[12803.875673] RBP: ffff881fff403ac8 R08: 00000000e0400000 R09: ffff881fff403bd0
[12803.874889] invalid opcode: 0000 [#1] SMP
[12803.876661] [] ? flow_lookup.isra.4+0x4a/0xc0 [openvswitch]
[12803.876973] [] ovs_vport_receive+0x2a/0x30 [openvswitch]
[12803.874870] kernel BUG at /build/buildd/linux-3.13.0/net/core/skbuff.c:1043!
[12803.874916] Modules linked in: ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_raw iptable_raw 8021q garp mrp vhost_net vhost macvtap macvlan dm_multipath scsi_dh nbd kvm_intel ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp bridge stp llc xfs ip6table_filter ip6_tables iptable_filter ip_tables x_tables openvswitch(OX) gre vxlan ip_tunnel libcrc32c ipmi_devintf gpio_ich dcdbas x86_pkg_temp_thermal intel_powerclamp coretemp kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd lpc_ich ipmi_si mei_me mei wmi shpchp acpi_power_meter mac_hid squashfs aufs igb i2c_algo_bit ixgbe e1000e tg3 dca ptp ahci pps_core libahci megaraid_sas mdio [last unloaded: kvm_intel]
[12803.877546] [] handle_tx_kick+0x15/0x20 [vhost_net]
[12803.876251]
[12803.877676] [] kthread+0xd2/0xf0
[12803.877714] [] ? kthread_create_on_node+0x1c0/0x1c0
[12803.877226] [] do_softirq+0xec/0x2c0
[12803.875907] CR2: 00007eff8c072400 CR3: 0000001f86331000 CR4: 00000000001427f0
[12803.876727] [] ovs_execute_actions+0x67/0x100 [openvswitch]
[12803.876551] [] ? netdev_send+0x4b/0xc0 [openvswitch]
[12803.877314] [] do_softirq+0x55/0x60
[12803.877268] [] do_softirq_own_stack+0x1c/0x30
[12803.875548] RSP: 0018:ffff881fff403a90 EFLAGS: 00010202
[12803.877017] [] netdev_frame_hook+0xc1/0x120 [openvswitch]
[12803.878203] RIP [] pskb_expand_head+0x20e/0x250
[12803.876219] Call Trace:
[12803.875802] FS: 0000000000000000(0000) GS:ffff881fff400000(0000) knlGS:0000000000000000
[12803.876060] 00000000000005b4 ffff881f5c0e6cfc 00000000ffffffff ffff881fff403b10
[12803.877061] [] netif_receive_skb_core+0x262/0x840
[12803.875715] R10: 0000000000000010 R11: ffff881f31d29d40 R12: 0000000000000000
[12803.875868] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12803.877632] [] ? vhost_dev_reset_owner+0x30/0x30 [vhost]
[12803.876904] [] ovs_dp_process_received_packet+0x4b/0x80 [openvswitch]
[12803.877307]
[12803.876594] [] do_execute_actions+0x3ac/0xbf0 [openvswitch]
[12803.876331] [] skb_checksum_help+0x123/0x190
[12803.875451] Hardware name: Dell Inc. PowerEdge R530/03XKDV, BIOS 1.1.10 03/10/2015
[12803.875587] RAX: 0000000000000002 RBX: ffff8800720fa100 RCX: 0000000000000020
[12803.877185] [] net_rx_action+0x152/0x250
[12803.877757] [] ret_from_fork+0x58/0x90
[12803.876374] [] queue_userspace_packet+0x3ae/0x400 [openvswitch]
[12803.875950] Stack:
[12803.876793] [] ovs_dp_process_packet_with_key+0x7f/0xf0 [openvswitch]
[12803.877145] [] process_backlog+0xae/0x1a0
[12803.877838] Code: ff ff ff 48 89 d7 48 89 55 c8 e8 ae 4f b4 ff 84 c0 48 8b 55 c8 75 a1 eb 9b 0f 1f 40 00 0f 0b 41 81 cf 00 20 00 00 e9 43 fe ff ff <0f> 0b 44 89 fe 4c 89 ef e8 e5 f4 ff ff 85 c0 74 12 48 89 df e8
[12803.875430] CPU: 0 PID: 17654 Comm: vhost-17644 Tainted: G OX 3.13.0-53-generic #89-Ubuntu
[12803.875495] RIP: 0010:[] [] pskb_expand_head+0x20e/0x250
[12803.875630] RDX: 00000000000006aa RSI: 0000000000000000 RDI: ffff8800720fa100
[12803.876441] [] ovs_dp_upcall+0x65/0x70 [openvswitch]
[12803.877464] [] tun_sendmsg+0x5a/0x80
[12803.878247] RSP
[12803.877103] [] netif_receive_skb+0x18/0x60
[12803.877425] [] tun_get_user+0x403/0x840
[12803.876258] [] pskb_pull_tail+0x50/0x360
[12803.876484] [] do_execute_actions+0x376/0xbf0 [openvswitch]
[12803.878792] ---[ end trace 4e8d5a14aaae59db ]---
[12803.875758] R13: ffff8800720fa100 R14: ffff881f5c0e6cfc R15: 00000000ffffffff
[12803.875473] task: ffff88301f389800 ti: ffff88300fcce000 task.ti: ffff88300fcce000
[12803.876140] ffffffff81618ef0 ffffffffa01a41c0 000005b4cd428000 ffff8800720fa100