search

coreos / bugs

Issue tracker for CoreOS Container Linux
https://coreos.com/os/eol/
147 stars 30 forks source link

net/core/dev.c skb_warn_bad_offload #158

Closed morgabra closed 9 years ago

morgabra commented 10 years ago

https://bugzilla.kernel.org/show_bug.cgi?id=82471

This affects alpha/beta kernel versions and significantly hurts network performance/reliability.

Disabling scatter/gather offloading appears to stop the pain: sudo ethtool -K bond0.101 sg off

Is this a useful thing to report here?

# similar trace is spammed
core@coreos-beta ~ $ dmesg | tail
[  462.446706] ------------[ cut here ]------------
[  462.447389] WARNING: CPU: 0 PID: 0 at /build/amd64-usr/var/tmp/portage/sys-kernel/coreos-kernel-3.16.2-r3/work/coreos-kernel-3.16.2/net/core/dev.c:2246 skb_warn_bad_offload+0xcf/0xdc()
[  462.449889] : caps=(0x0000000004197ba9, 0x00000000001b583b) len=2962 data_len=1376 gso_size=1448 gso_type=1 ip_summed=3
[  462.451491] Modules linked in: xt_addrtype xt_conntrack ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack bridge iptable_filter ip_tables bonding 8021q garp stp mrp llc nls_utf8 isofs ext4 crc16 mbcache jbd2 btrfs xor raid6_pq sd_mod crct10dif_generic crc_t10dif crct10dif_common coretemp x86_pkg_temp_thermal kvm_intel kvm crc32c_intel isci ahci libsas libahci aesni_intel aes_x86_64 glue_helper lrw gf128mul iTCO_wdt ablk_helper sb_edac libata scsi_transport_sas iTCO_vendor_support cryptd microcode ixgbe i2c_i801 edac_core hwmon lpc_ich scsi_mod ptp i2c_core mfd_core firmware_class pps_core mdio evdev button autofs4
[  462.460866] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W     3.16.2+ #2
[  462.461919] Hardware name: Quanta OCP-Q301 /OCP-Q301, BIOS F03B3A14.Signed 07/14/2014
[  462.463092]  0000000000000009 ffff88087f0034f8 ffffffff814d377b ffff88087f00fa68
[  462.464219]  ffff88087f003548 ffff88087f003538 ffffffff8106357c ffff88087f003518
[  462.466157]  ffff88084e9cece8 ffff880078f2b000 0000000000000001 0000000000000003
[  462.467284] Call Trace:
[  462.467652]  <IRQ>  [<ffffffff814d377b>] dump_stack+0x46/0x58
[  462.469552]  [<ffffffff8106357c>] warn_slowpath_common+0x8c/0xc0
[  462.470450]  [<ffffffff81063666>] warn_slowpath_fmt+0x46/0x50
[  462.471291]  [<ffffffff814d535d>] skb_warn_bad_offload+0xcf/0xdc
[  462.472243]  [<ffffffff813d9095>] skb_checksum_help+0x195/0x1a0
[  462.473124]  [<ffffffff813dc9e9>] dev_hard_start_xmit+0x489/0x550
[  462.474043]  [<ffffffff813dce08>] __dev_queue_xmit+0x358/0x4e0
[  462.476018]  [<ffffffff813dcfb0>] dev_queue_xmit+0x10/0x20
[  462.476838]  [<ffffffffa01a5cdd>] register_vlan_dev+0x145d/0x1640 [8021q]
[  462.477838]  [<ffffffff813dc8e5>] dev_hard_start_xmit+0x385/0x550
[  462.478741]  [<ffffffff813dce08>] __dev_queue_xmit+0x358/0x4e0
[  462.479704]  [<ffffffff813dcfb0>] dev_queue_xmit+0x10/0x20
[  462.480517]  [<ffffffff813e52a6>] neigh_resolve_output+0x126/0x220
[  462.481432]  [<ffffffff8141b9c0>] ? ip_finish_output2+0x360/0x360
[  462.482333]  [<ffffffff8141bd81>] ip_finish_output+0x3c1/0x890
[  462.483182]  [<ffffffff8141cb58>] ip_output+0x58/0x90
[  462.483935]  [<ffffffff8141c319>] ip_local_out_sk+0x39/0x50
[  462.484853]  [<ffffffff8141c483>] ip_queue_xmit+0x153/0x370
[  462.485676]  [<ffffffff8143390f>] tcp_transmit_skb+0x49f/0x900
[  462.486547]  [<ffffffff8143488d>] tcp_write_xmit+0x13d/0xc00
[  462.487376]  [<ffffffff81438063>] ? tcp_v4_md5_lookup+0x13/0x20
[  462.488250]  [<ffffffff814353c2>] __tcp_push_pending_frames+0x32/0xd0
[  462.489204]  [<ffffffff81430797>] tcp_rcv_established+0x357/0x730
[  462.490361]  [<ffffffff8143ab55>] tcp_v4_do_rcv+0x1d5/0x4e0
[  462.491196]  [<ffffffff8143c5cd>] tcp_v4_rcv+0x5bd/0x6f0
[  462.491981]  [<ffffffff81416a80>] ? ip_rcv_finish+0x370/0x370
[  462.492831]  [<ffffffff8140d784>] ? nf_hook_slow+0x74/0x130
[  462.493671]  [<ffffffff81416a80>] ? ip_rcv_finish+0x370/0x370
[  462.494560]  [<ffffffff81416b28>] ip_local_deliver_finish+0xa8/0x220
[  462.496098]  [<ffffffff81416e38>] ip_local_deliver+0x48/0x80
[  462.496942]  [<ffffffff81416798>] ip_rcv_finish+0x88/0x370
[  462.497752]  [<ffffffff81417118>] ip_rcv+0x2a8/0x3e0
[  462.498488]  [<ffffffff813daaa2>] __netif_receive_skb_core+0x5b2/0x750
[  462.499522]  [<ffffffff813dac61>] __netif_receive_skb+0x21/0x70
[  462.500401]  [<ffffffff813dae53>] netif_receive_skb_internal+0x33/0xa0
[  462.501362]  [<ffffffff813db8e8>] napi_gro_receive+0xc8/0x100
[  462.502209]  [<ffffffffa00f60ef>] ixgbe_alloc_rx_buffers+0x87f/0xd10 [ixgbe]
[  462.503250]  [<ffffffffa00f7209>] ixgbe_poll+0x4b9/0x860 [ixgbe]
[  462.504147]  [<ffffffff813db4d9>] net_rx_action+0x149/0x250
[  462.505958]  [<ffffffff81068277>] __do_softirq+0xe7/0x280
[  462.506768]  [<ffffffff81068695>] irq_exit+0xb5/0xc0
[  462.507498]  [<ffffffff814dd357>] do_IRQ+0x67/0x110
[  462.508216]  [<ffffffff814db3ed>] common_interrupt+0x6d/0x6d
[  462.509056]  <EOI>  [<ffffffff813a2c91>] ? cpuidle_enter_state+0x61/0xe0
[  462.510128]  [<ffffffff813a2c87>] ? cpuidle_enter_state+0x57/0xe0
[  462.511033]  [<ffffffff813a2dd7>] cpuidle_enter+0x17/0x20
[  462.511827]  [<ffffffff810aa17d>] cpu_startup_entry+0x2fd/0x450
[  462.512716]  [<ffffffff814c5e77>] rest_init+0x77/0x80
[  462.513467]  [<ffffffff81af60a2>] start_kernel+0x442/0x44f
[  462.514275]  [<ffffffff81af5a5f>] ? set_init_arg+0x58/0x58
[  462.516063]  [<ffffffff81af5117>] ? early_idt_handlers+0x117/0x120
[  462.517062]  [<ffffffff81af55f0>] x86_64_start_reservations+0x2a/0x2c
[  462.518683]  [<ffffffff81af5733>] x86_64_start_kernel+0x141/0x150
[  462.520274] ---[ end trace f6ebb5d9b6053ffd ]---
core@coreos-beta ~ $ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000
    link/ether <redacted> brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000
    link/ether <redacted> brd ff:ff:ff:ff:ff:ff
4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether <redacted> brd ff:ff:ff:ff:ff:ff
    inet6 <redacted>/64 scope link
       valid_lft forever preferred_lft forever
5: bond0.101@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether <redacted> brd ff:ff:ff:ff:ff:ff
    inet <redacted>/24 brd <redacted> scope global bond0.101
       valid_lft forever preferred_lft forever
    inet6 <redacted>/64 scope link
       valid_lft forever preferred_lft forever
6: bond0.401@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether <redacted> brd ff:ff:ff:ff:ff:ff
    inet <redacted>/20 brd <redacted> scope global bond0.401
       valid_lft forever preferred_lft forever
    inet6 <redacted>/64 scope link
       valid_lft forever preferred_lft forever
7: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
    link/ether <redacted> brd ff:ff:ff:ff:ff:ff
    inet <redacted>/16 scope global docker0
       valid_lft forever preferred_lft forever
# eth0/eth1 identical
core@coreos-beta ~ $ ethtool -k eth0
Features for eth0:
rx-checksumming: on
tx-checksumming: on
    tx-checksum-ipv4: on
    tx-checksum-ip-generic: off [fixed]
    tx-checksum-ipv6: on
    tx-checksum-fcoe-crc: off [fixed]
    tx-checksum-sctp: on
scatter-gather: on
    tx-scatter-gather: on
    tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
    tx-tcp-segmentation: on
    tx-tcp-ecn-segmentation: off [fixed]
    tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off
busy-poll: on [fixed]
core@coreos-beta ~ $ ethtool -k bond0
Features for bond0:
rx-checksumming: off [fixed]
tx-checksumming: on
    tx-checksum-ipv4: off [fixed]
    tx-checksum-ip-generic: on
    tx-checksum-ipv6: off [fixed]
    tx-checksum-fcoe-crc: off [fixed]
    tx-checksum-sctp: off [fixed]
scatter-gather: on
    tx-scatter-gather: on
    tx-scatter-gather-fraglist: off [requested on]
tcp-segmentation-offload: on
    tx-tcp-segmentation: on
    tx-tcp-ecn-segmentation: on
    tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: on [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: on
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]
core@coreos-beta ~ $ ethtool -k bond0.101
Features for bond0.101:
rx-checksumming: off [fixed]
tx-checksumming: on
    tx-checksum-ipv4: on
    tx-checksum-ip-generic: off
    tx-checksum-ipv6: on
    tx-checksum-fcoe-crc: off
    tx-checksum-sctp: off
scatter-gather: on
    tx-scatter-gather: on
    tx-scatter-gather-fraglist: off
tcp-segmentation-offload: on
    tx-tcp-segmentation: on
    tx-tcp-ecn-segmentation: off
    tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]
marineam commented 10 years ago

FYI @gregkh

gregkh commented 10 years ago

I'll watch this, but networking bugs should be reported to the netdev@vger.kernel.org mailing list, not on bugzilla, if they want to get fixed....

philips commented 10 years ago

@morgabra has someone filed this on netdev?

marineam commented 10 years ago

Someone forwarded the bugzilla message to netdev a long time ago but as far as I can tell that was all and it certainly didn't get a response: http://thread.gmane.org/gmane.linux.network/326672

marineam commented 10 years ago

A possibly related report: http://sourceforge.net/p/e1000/bugs/434/

The user in that case resolved it by a random shuffle of hardware but re-raising the issue with Intel folks is likely the best next step.

crawford commented 9 years ago

Please reopen if this is still an issue with the 4.2+ kernels.

ericentin commented 8 years ago

Hi, hope this is the right place to mention this but I'm getting this error on the latest coreos alpha channel build within a virtualbox guest on a Windows host. sudo ethtool -K eth1 sg offfixed it for me. Glad to add any more info, just let me know.

Mar 17 16:57:28 core01 kernel: ------------[ cut here ]------------
Mar 17 16:57:28 core01 kernel: WARNING: CPU: 0 PID: 328 at ../source/net/core/dev.c:2422 skb_warn_ba                              d_offload+0xd3/0x120()
Mar 17 16:57:28 core01 kernel: virtio_net: caps=(0x0000080400174a29, 0x0000000000000000) len=1654 da                              ta_len=1456 gso_size=1480 gso_type=2 ip_summed=0
Mar 17 16:57:28 core01 kernel: Modules linked in: xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4                               iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter nf_nat nf_connt                              rack br_netfilter bridge stp llc overlay nfsv3 nfs_acl nfs lockd grace sunrpc fscache nls_ascii nls_                              cp437 vfat fat ext4 crc16 mbcache jbd2 sd_mod crc32c_intel jitterentropy_rng hmac drbg virtio_net mo                              usedev ata_piix libata aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd scsi_mod p                              smouse evdev i2c_piix4 i2c_core virtio_pci virtio_ring virtio acpi_cpufreq tpm_tis tpm button sch_fq                              _codel ip_tables autofs4
Mar 17 16:57:28 core01 kernel: CPU: 0 PID: 328 Comm: kworker/0:1H Tainted: G        W       4.4.4-co                              reos #2
Mar 17 16:57:28 core01 kernel: Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12                              /01/2006
Mar 17 16:57:28 core01 kernel: Workqueue: rpciod rpc_wake_up_status [sunrpc]
Mar 17 16:57:28 core01 kernel:  0000000000000286 000000007e9d210b ffff880016cc37f0 ffffffff952d6973
Mar 17 16:57:28 core01 kernel:  ffff880016cc3838 ffffffff95819051 ffff880016cc3828 ffffffff950746b6
Mar 17 16:57:28 core01 kernel:  ffff88001706bb00 ffff880017355000 0000000000000002 ffff880017355000
Mar 17 16:57:28 core01 kernel: Call Trace:
Mar 17 16:57:28 core01 kernel:  [<ffffffff952d6973>] dump_stack+0x63/0x90
Mar 17 16:57:28 core01 kernel:  [<ffffffff950746b6>] warn_slowpath_common+0x86/0xc0
Mar 17 16:57:28 core01 kernel:  [<ffffffff9507474c>] warn_slowpath_fmt+0x5c/0x80
Mar 17 16:57:28 core01 kernel:  [<ffffffff952dc766>] ? ___ratelimit+0x86/0xe0
Mar 17 16:57:28 core01 kernel:  [<ffffffff95455203>] skb_warn_bad_offload+0xd3/0x120
Mar 17 16:57:28 core01 kernel:  [<ffffffff954596ce>] __skb_gso_segment+0x7e/0xd0
Mar 17 16:57:28 core01 kernel:  [<ffffffff95459a6f>] validate_xmit_skb.isra.99.part.100+0x12f/0x2b0
Mar 17 16:57:28 core01 kernel:  [<ffffffff9545a02b>] validate_xmit_skb_list+0x3b/0x60
Mar 17 16:57:28 core01 kernel:  [<ffffffff9547d851>] sch_direct_xmit+0x171/0x210
Mar 17 16:57:28 core01 kernel:  [<ffffffff9545a383>] __dev_queue_xmit+0x253/0x590
Mar 17 16:57:28 core01 kernel:  [<ffffffff9545a6d0>] dev_queue_xmit+0x10/0x20
Mar 17 16:57:28 core01 kernel:  [<ffffffff9549ae06>] ip_finish_output2+0x276/0x310
Mar 17 16:57:28 core01 kernel:  [<ffffffff9549c609>] ip_finish_output+0x139/0x1f0
Mar 17 16:57:28 core01 kernel:  [<ffffffff9548da43>] ? nf_hook_slow+0x73/0xd0
Mar 17 16:57:28 core01 kernel:  [<ffffffff9549d03e>] ip_output+0x6e/0xe0
Mar 17 16:57:28 core01 kernel:  [<ffffffff9549c702>] ? __ip_local_out+0x42/0x100
Mar 17 16:57:28 core01 kernel:  [<ffffffff9549c4d0>] ? ip_fragment.constprop.50+0x80/0x80
Mar 17 16:57:28 core01 kernel:  [<ffffffff9549c7f5>] ip_local_out+0x35/0x40
Mar 17 16:57:28 core01 kernel:  [<ffffffff9549d9f9>] ip_send_skb+0x19/0x40
Mar 17 16:57:28 core01 kernel:  [<ffffffff954c470d>] udp_send_skb+0x16d/0x280
Mar 17 16:57:28 core01 kernel:  [<ffffffff954c485e>] udp_push_pending_frames+0x3e/0x60
Mar 17 16:57:28 core01 kernel:  [<ffffffff954c5e41>] udp_sendpage+0x121/0x1a0
Mar 17 16:57:28 core01 kernel:  [<ffffffff9543b648>] ? sock_sendmsg+0x38/0x50
Mar 17 16:57:28 core01 kernel:  [<ffffffff9543b77b>] ? kernel_sendmsg+0x2b/0x30
Mar 17 16:57:28 core01 kernel:  [<ffffffffc05de458>] ? csum_partial_copy_to_xdr+0x12d8/0x4630 [sunrp                              c]
Mar 17 16:57:28 core01 kernel:  [<ffffffff954d2ad4>] inet_sendpage+0x74/0xd0
Mar 17 16:57:28 core01 kernel:  [<ffffffff954d2a60>] ? inet_sendmsg+0xa0/0xa0
Mar 17 16:57:28 core01 kernel:  [<ffffffffc05de5f5>] csum_partial_copy_to_xdr+0x1475/0x4630 [sunrpc]
Mar 17 16:57:28 core01 kernel:  [<ffffffffc05de76f>] csum_partial_copy_to_xdr+0x15ef/0x4630 [sunrpc]
Mar 17 16:57:28 core01 kernel:  [<ffffffffc05dc5a6>] xprt_transmit+0x66/0x350 [sunrpc]
Mar 17 16:57:28 core01 kernel:  [<ffffffffc05d86d9>] rpc_peeraddr+0x1369/0x14d0 [sunrpc]
Mar 17 16:57:28 core01 kernel:  [<ffffffffc05d8520>] ? rpc_peeraddr+0x11b0/0x14d0 [sunrpc]
Mar 17 16:57:28 core01 kernel:  [<ffffffffc05d8520>] ? rpc_peeraddr+0x11b0/0x14d0 [sunrpc]
Mar 17 16:57:28 core01 kernel:  [<ffffffffc05e33b5>] rpc_wake_up_status+0x2a5/0x6b0 [sunrpc]
Mar 17 16:57:28 core01 kernel:  [<ffffffffc05e37b5>] rpc_wake_up_status+0x6a5/0x6b0 [sunrpc]
Mar 17 16:57:28 core01 kernel:  [<ffffffff9508caf5>] process_one_work+0x155/0x410
Mar 17 16:57:28 core01 kernel:  [<ffffffff9508d59e>] worker_thread+0x4e/0x470
Mar 17 16:57:28 core01 kernel:  [<ffffffff9508d550>] ? rescuer_thread+0x320/0x320
Mar 17 16:57:28 core01 kernel:  [<ffffffff9508d550>] ? rescuer_thread+0x320/0x320
Mar 17 16:57:28 core01 kernel:  [<ffffffff95092928>] kthread+0xd8/0xf0
Mar 17 16:57:28 core01 kernel:  [<ffffffff95092850>] ? kthread_park+0x60/0x60
Mar 17 16:57:28 core01 kernel:  [<ffffffff95558c0f>] ret_from_fork+0x3f/0x70
Mar 17 16:57:28 core01 kernel:  [<ffffffff95092850>] ? kthread_park+0x60/0x60
Mar 17 16:57:28 core01 kernel: ---[ end trace 106ba202ee0e666d ]---
satishdotpatel commented 6 years ago

I am having same issue i have off LRO/GRO on NIC but still causing this error

[ 8162.302876] ------------[ cut here ]------------ [ 8162.302902] WARNING: CPU: 2 PID: 0 at net/core/dev.c:2573 skb_warn_bad_offload+0xcd/0xda [ 8162.302906] : caps=(0x00003e2007db78e9, 0x0000000000000000) len=5844 data_len=4344 gso_size=1448 gso_type=1 ip_summed=3 [ 8162.302908] Modules linked in: btrfs raid6_pq xor loop iptable_raw veth bonding ip6table_mangle xt_CHECKSUM sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vhost_net tun vhost ipmi_ssif macvtap macvlan iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_vs iTCO_wdt iptable_nat nf_conntrack_ipv4 iTCO_vendor_support nf_defrag_ipv4 nf_nat_ipv4 ipmi_si iptable_mangle iptable_filter ipt_REJECT nf_reject_ipv4 ipmi_devintf ipt_MASQUERADE pcspkr sg lpc_ich nf_nat_masquerade_ipv4 hpwdt hpilo ipmi_msghandler video nf_nat acpi_power_meter wmi ioatdma shpchp nf_conntrack pcc_cpufreq ip6table_filter ip6_tables ebtables dm_snapshot dm_bufio dm_multipath br_netfilter bridge 8021q garp mrp stp [ 8162.302967] llc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ixgbe drm crct10dif_pclmul crct10dif_common mdio crc32c_intel hpsa serio_raw ptp pps_core i2c_core scsi_transport_sas dca dm_mirror dm_region_hash dm_log dm_mod [ 8162.302994] CPU: 2 PID: 0 Comm: swapper/2 Kdump: loaded Tainted: G W ------------ 3.10.0-862.3.3.el7.x86_64 #1 [ 8162.302999] Hardware name: HP ProLiant BL460c Gen8, BIOS I31 06/01/2015 [ 8162.303001] Call Trace: [ 8162.303005] [] dump_stack+0x19/0x1b [ 8162.303014] [] warn+0xd8/0x100 [ 8162.303017] [] warn_slowpath_fmt+0x5f/0x80 [ 8162.303025] [] ? ___ratelimit+0x93/0x100 [ 8162.303030] [] skb_warn_bad_offload+0xcd/0xda [ 8162.303038] [] skb_checksum_help+0x195/0x1b0 [ 8162.303043] [] checksum_tg+0x29/0x30 [xt_CHECKSUM] [ 8162.303049] [] ipt_do_table+0x2eb/0x710 [ip_tables] [ 8162.303056] [] ? dev_hard_start_xmit+0x246/0x3b0 [ 8162.303061] [] ? local_bh_enable+0x17/0x20 [ 8162.303065] [] ? ipt_do_table+0x33e/0x710 [ip_tables] [ 8162.303071] [] iptable_mangle_hook+0x43/0x130 [iptable_mangle] [ 8162.303078] [] nf_iterate+0x98/0xe0 [ 8162.303081] [] nf_hook_slow+0xa8/0x110 [ 8162.303087] [] ip_output+0xce/0xe0 [ 8162.303091] [] ? ip_append_data.isra.50+0xa50/0xa50 [ 8162.303097] [] ip_forward_finish+0x6c/0x90 [ 8162.303100] [] ip_forward+0x37c/0x490 [ 8162.303106] [] ? netfilter_net_init+0x42/0x70 [ 8162.303109] [] ? ip_frag_mem+0x20/0x20 [ 8162.303114] [] ip_rcv_finish+0x90/0x370 [ 8162.303120] [] ip_rcv+0x2b9/0x410 [ 8162.303125] [] ? inet_del_offload+0x40/0x40 [ 8162.303131] [] netif_receive_skb_core+0x729/0xa20 [ 8162.303135] [] ? internal_add_timer+0x113/0x130 [ 8162.303141] [] netif_receive_skb+0x18/0x60 [ 8162.303145] [] netif_receive_skb_internal+0x40/0xc0 [ 8162.303153] [] ? ip_route_input_slow+0x5a7/0xc80 [ 8162.303158] [] netif_receive_skb+0x1c/0x70 [ 8162.303169] [] br_netif_receive_skb+0x28/0x60 [bridge] [ 8162.303176] [] br_pass_frame_up+0xc0/0x150 [bridge] [ 8162.303185] [] ? br_fdb_update+0xca/0x220 [bridge] [ 8162.303192] [] br_handle_frame_finish+0x221/0x580 [bridge] [ 8162.303199] [] br_nf_pre_routing_finish+0x180/0x450 [br_netfilter] [ 8162.303208] [] ? br_pass_frame_up+0x150/0x150 [bridge] [ 8162.303211] [] ? nf_iterate+0x98/0xe0 [ 8162.303218] [] br_nf_pre_routing+0x2e8/0x420 [br_netfilter] [ 8162.303223] [] ? br_nf_forward_ip+0x450/0x450 [br_netfilter] [ 8162.303228] [] nf_iterate+0x98/0xe0 [ 8162.303235] [] nf_hook_slow+0xa8/0x110 [ 8162.303242] [] br_handle_frame+0x222/0x320 [bridge] [ 8162.303247] [] ? tcp_gro_receive+0x207/0x320 [ 8162.303253] [] ? br_pass_frame_up+0x150/0x150 [bridge] [ 8162.303258] [] __netif_receive_skb_core+0x1fa/0xa20 [ 8162.303264] [] ? getnstimeofday64+0x3f/0xd0 [ 8162.303268] [] netif_receive_skb+0x18/0x60 [ 8162.303276] [] netif_receive_skb_internal+0x40/0xc0 [ 8162.303280] [] napi_gro_complete+0x7f/0x100 [ 8162.303285] [] napi_gro_flush+0x6d/0x90 [ 8162.303288] [] napi_complete_done+0x30/0x110 [ 8162.303302] [] ixgbe_poll+0x548/0x770 [ixgbe] [ 8162.303308] [] net_rx_action+0x26f/0x390 [ 8162.303315] [] do_softirq+0xf5/0x280 [ 8162.303318] [] call_softirq+0x1c/0x30 [ 8162.303324] [] do_softirq+0x65/0xa0 [ 8162.303327] [] irq_exit+0x105/0x110 [ 8162.303330] [] do_IRQ+0x56/0xf0 [ 8162.303334] [] common_interrupt+0x17c/0x17c [ 8162.303336] [] ? cpuidle_enter_state+0x57/0xd0 [ 8162.303343] [] cpuidle_idle_call+0xde/0x230 [ 8162.303346] [] arch_cpu_idle+0xe/0x40 [ 8162.303350] [] cpu_startup_entry+0x14a/0x1e0 [ 8162.303358] [] start_secondary+0x1f2/0x270 [ 8162.303366] [] start_cpu+0x5/0x14 [ 8162.303378] ---[ end trace 80e59d47a5b261e9 ]---

gregkh commented 6 years ago

You are using a very old and obsolete kernel version (3.10), please go get support from the vendor that is forcing you to do that, you are paying them money for this :)

satishdotpatel commented 6 years ago

its CentOS 7.5 :(

Does this message means its impacting network performance or its just reporting stuff and nothing to do with performance?

gregkh commented 6 years ago

If it is Centos, go ask on their developer forums, there is nothing that anyone here can do about this sorry.

You might want to use a more modern kernel, that one is very old and obsolete :(

satishdotpatel commented 6 years ago

I think you are right, i should just install kernel 4.x because there is no downside to do that i believe.

satishdotpatel commented 6 years ago

Finally i have upgraded my kernel to 4.17.12-1.el7.elrepo.x86_64 and i am still see same stack error which i posted earlier, I believe 4.17.x is the latest kernel so now where and what is the wrong here?

bgilbert commented 6 years ago

Locking this thread as off-topic for Container Linux. If you're seeing kernel issues with a current version of Container Linux, please report a new issue.