Open itamarst opened 2 years ago
leviport
commented
2 years ago Linux 6.0.2 will be available shortly and may fix this issue. I've been running 6.0.2 on a thelio-b3, and I haven't seen the same behavior, but I will continue to let it run to see if this bug occurs.
itamarst
commented
2 years ago Thanks, I will test once I am able to upgrade.
itamarst
commented
1 year ago I can go days without this happening, and then it happens again. I am now on kernel 6.0.2-76060002-generic #202210150739~1666289067~22.04~fe0ce53~dev-Ubuntu, but it just happened again:
[16616.450068] igc 0000:02:00.0 enp2s0: PCIe link lost, device now detached
[16616.450098] ------------[ cut here ]------------
[16616.450101] igc: Failed to read reg 0xc030!
[16616.450215] WARNING: CPU: 15 PID: 37722 at drivers/net/ethernet/intel/igc/igc_main.c:6197 igc_rd32+0xaf/0xc0 [igc]
[16616.450249] Modules linked in: rfcomm xt_nat nf_conntrack_netlink xfrm_user xfrm_algo xt_addrtype br_netfilter xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink bridge stp llc cmac algif_hash algif_skcipher af_alg bnep overlay snd_hda_codec_hdmi snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils snd_soc_hdac_hda snd_hda_ext_core snd_hda_codec_realtek snd_soc_acpi_intel_match snd_soc_acpi snd_hda_codec_generic soundwire_bus ledtrig_audio snd_soc_core snd_compress ac97_bus snd_pcm_dmaengine snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec snd_usb_audio snd_hda_core snd_usbmidi_lib intel_rapl_msr snd_hwdep intel_rapl_common intel_tcc_cooling x86_pkg_temp_thermal iwlmvm intel_powerclamp snd_pcm coretemp snd_seq_midi binfmt_misc
[16616.450372] mac80211 snd_seq_midi_event snd_rawmidi kvm_intel uvcvideo btusb snd_seq videobuf2_vmalloc btrtl videobuf2_memops btbcm videobuf2_v4l2 nls_iso8859_1 libarc4 snd_seq_device pmt_telemetry kvm btintel mei_hdcp mei_pxp pmt_class iwlwifi snd_timer spi_nor rndis_wlan videobuf2_common btmtk rapl bluetooth intel_cstate gigabyte_wmi videodev wmi_bmof mei_me ecdh_generic input_leds mtd snd ee1004 mc soundcore cfg80211 ecc mei intel_vsec intel_hid acpi_tad acpi_pad sparse_keymap mac_hid sch_fq_codel ipmi_devintf ipmi_msghandler msr parport_pc ppdev lp parport ramoops reed_solomon pstore_blk pstore_zone efi_pstore ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear system76_acpi(OE) hid_generic rndis_host usbhid cdc_ether hid usbnet mii system76_io(OE) i915 drm_buddy i2c_algo_bit ttm drm_display_helper cec rc_core drm_kms_helper syscopyarea
[16616.450533] sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel aesni_intel nvme crypto_simd cryptd drm nvme_core igc ahci i2c_i801 intel_lpss_pci spi_intel_pci xhci_pci intel_lpss i2c_smbus spi_intel libahci idma64 xhci_pci_renesas nvme_common wmi video pinctrl_alderlake
[16616.450586] CPU: 15 PID: 37722 Comm: kworker/15:0 Tainted: G OE 6.0.2-76060002-generic #202210150739~1666289067~22.04~fe0ce53~dev-Ubuntu
[16616.450596] Hardware name: System76 Thelio/Thelio, BIOS F5b Z5 10/11/2022
[16616.450600] Workqueue: events igc_watchdog_task [igc]
[16616.450619] RIP: 0010:igc_rd32+0xaf/0xc0 [igc]
[16616.450634] Code: c7 c6 18 85 6d c0 e8 87 29 ff dc 48 8b bb 28 ff ff ff e8 74 83 8b dc 84 c0 74 ae 44 89 ee 48 c7 c7 40 85 6d c0 e8 d2 05 f5 dc <0f> 0b e9 74 ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00
[16616.450639] RSP: 0018:ffffaf0c2b09bdb8 EFLAGS: 00010246
[16616.450644] RAX: 0000000000000000 RBX: ffff8bfda0fc6c58 RCX: 0000000000000000
[16616.450648] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[16616.450650] RBP: ffffaf0c2b09bdd8 R08: 0000000000000000 R09: 0000000000000000
[16616.450653] R10: 0000000000000000 R11: 0000000000000000 R12: 00000000ffffffff
[16616.450655] R13: 000000000000c030 R14: ffff8bfda0fc6000 R15: ffff8bfd81c9dd80
[16616.450659] FS: 0000000000000000(0000) GS:ffff8c0cff9c0000(0000) knlGS:0000000000000000
[16616.450663] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[16616.450667] CR2: 000000c0005ef010 CR3: 0000000329a10002 CR4: 0000000000770ee0
[16616.450671] PKRU: 55555554
[16616.450673] Call Trace:
[16616.450676] <TASK>
[16616.450683] igc_update_stats+0xa5/0x740 [igc]
[16616.450698] igc_watchdog_task+0xaa/0x310 [igc]
[16616.450712] process_one_work+0x21c/0x400
[16616.450721] worker_thread+0x50/0x3f0
[16616.450727] ? process_one_work+0x400/0x400
[16616.450732] kthread+0xeb/0x120
[16616.450741] ? kthread_complete_and_exit+0x20/0x20
[16616.450749] ret_from_fork+0x1f/0x30
[16616.450761] </TASK>
[16616.450763] ---[ end trace 0000000000000000 ]---Is it possible this is a hardware issue?
leviport
commented
1 year ago It's possible. If you suspect that to be the case, then I'd recommend opening a support ticket: https://support.system76.com/
However 6.0.2 has also been updated. It might be a good idea to give 6.0.6 a try before we make that conclusion.
itamarst
commented
1 year ago Rebooted, happened again, so possibly it's actually happening more frequently with 6.0.2. Will upgrade when 6.0.6 comes out and see if that helps.
(Apriori my bet is software bug, just because this is quite new computer, but who knows.)
itamarst
commented
1 year ago https://github.com/pop-os/linux/tree/master/drivers/net/ethernet/intel/igc shows last change was in July, so not optimistic that an upgrade will hep.
itamarst
commented
1 year ago https://www.reddit.com/r/buildapc/comments/xypn1m/network_card_intel_ethernet_controller_i225v_igc/ is a bunch of other people having the same issue, so I'm going to say it's a software bug. Someone there is reaching out to Intel, so I will share my crash with them.
itamarst
commented
1 year ago Still happening with 6.2.6:
[ 183.747456] ------------[ cut here ]------------
[ 183.747459] igc: Failed to read reg 0xc030!
[ 183.747587] WARNING: CPU: 10 PID: 8982 at drivers/net/ethernet/intel/igc/igc_main.c:6412 igc_rd32+0xa4/0xc0 [igc]
[ 183.747631] Modules linked in: rfcomm nf_conntrack_netlink xfrm_user xfrm_algo xt_addrtype br_netfilter vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) ccm cmac algif_hash algif_skcipher af_alg xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink bridge stp llc bnep overlay snd_hda_codec_hdmi snd_hda_codec_realtek snd_sof_pci_intel_tgl snd_hda_codec_generic snd_sof_intel_hda_common ledtrig_audio soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus snd_soc_core snd_compress ac97_bus snd_pcm_dmaengine snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec snd_usb_audio snd_hda_core snd_usbmidi_lib intel_rapl_msr snd_hwdep intel_rapl_common snd_pcm x86_pkg_temp_thermal snd_seq_midi snd_seq_midi_event
[ 183.747754] snd_rawmidi btusb intel_powerclamp btrtl btbcm coretemp snd_seq iwlmvm btintel uvcvideo videobuf2_vmalloc kvm_intel videobuf2_memops videobuf2_v4l2 btmtk binfmt_misc mei_hdcp mei_pxp snd_seq_device mac80211 kvm cmdlinepart bluetooth nls_iso8859_1 snd_timer videodev spi_nor pmt_telemetry rapl pmt_class libarc4 rndis_wlan snd intel_cstate iwlwifi videobuf2_common ecdh_generic gigabyte_wmi wmi_bmof input_leds mc ecc mtd ee1004 soundcore mei_me cfg80211 mei intel_vsec intel_hid sparse_keymap acpi_pad acpi_tad mac_hid sch_fq_codel msr parport_pc ppdev lp parport ramoops pstore_blk reed_solomon pstore_zone efi_pstore ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear system76_acpi(OE) hid_generic rndis_host cdc_ether usbhid usbnet hid mii system76_io(OE) i915 drm_buddy i2c_algo_bit ttm drm_display_helper cec rc_core drm_kms_helper syscopyarea sysfillrect
[ 183.747921] sysimgblt nvme crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 aesni_intel ahci i2c_i801 spi_intel_pci crypto_simd cryptd igc drm spi_intel i2c_smbus nvme_core libahci intel_lpss_pci intel_lpss xhci_pci nvme_common idma64 xhci_pci_renesas video wmi pinctrl_alderlake
[ 183.747974] CPU: 10 PID: 8982 Comm: kworker/10:3 Tainted: G OE 6.2.6-76060206-generic #202303130630~1683056921~22.04~02b04a1~dev-Ubuntu
[ 183.747982] Hardware name: System76 Thelio/Thelio, BIOS F5b Z5 10/11/2022
[ 183.747987] Workqueue: events igc_watchdog_task [igc]
[ 183.748024] RIP: 0010:igc_rd32+0xa4/0xc0 [igc]
[ 183.748056] Code: c7 c6 d8 15 79 c0 e8 2b 92 23 f3 48 8b bb 28 ff ff ff e8 af 04 d3 f2 84 c0 74 b4 44 89 e6 48 c7 c7 00 16 79 c0 e8 3c d5 56 f2 <0f> 0b eb a1 b8 ff ff ff ff 31 d2 89 d6 89 d7 c3 cc cc cc cc 0f 1f
[ 183.748062] RSP: 0018:ffffaf4f4a5d3db8 EFLAGS: 00010246
[ 183.748068] RAX: 0000000000000000 RBX: ffff967a2117ac60 RCX: 0000000000000000
[ 183.748073] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 183.748077] RBP: ffffaf4f4a5d3dd0 R08: 0000000000000000 R09: 0000000000000000
[ 183.748081] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000c030
[ 183.748084] R13: ffff967a2117a000 R14: 0000000000000000 R15: ffff967a2091dd80
[ 183.748089] FS: 0000000000000000(0000) GS:ffff96897fa80000(0000) knlGS:0000000000000000
[ 183.748094] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 183.748098] CR2: 00007f88ddb89000 CR3: 00000007a5e10006 CR4: 0000000000770ee0
[ 183.748104] PKRU: 55555554
[ 183.748107] Call Trace:
[ 183.748111] <TASK>
[ 183.748119] igc_update_stats+0xa5/0x750 [igc]
[ 183.748153] igc_watchdog_task+0xf7/0x570 [igc]
[ 183.748186] process_one_work+0x222/0x430
[ 183.748197] worker_thread+0x50/0x3e0
[ 183.748204] ? __pfx_worker_thread+0x10/0x10
[ 183.748211] kthread+0xe6/0x110
[ 183.748220] ? __pfx_kthread+0x10/0x10
[ 183.748229] ret_from_fork+0x29/0x50
[ 183.748238] </TASK>
[ 183.748240] ---[ end trace 0000000000000000 ]---Linux version:
$ uname -a
Linux l-overture 6.2.6-76060206-generic #202303130630~1683056921~22.04~02b04a1~dev-Ubuntu SMP PREEMPT_DY x86_64 x86_64 x86_64 GNU/LinuxI'm on Ubuntu 22.04 with System76 apt source added, as installed by System76 when my machine shipped.
skelly37
commented
1 year ago Still an issue. There has been some changes to igc — perhaps integrating the newest igc to the pop kernel would fix the issue? It started happening to me ~2 months ago, it's really annoying.
itamarst
commented
10 months ago Currently have 20 days uptime without the problem happening, so perhaps it's fixed? https://github.com/torvalds/linux/commit/d4a7ce642100765119a872d4aba1bf63e3a22c8a seems plausible. I'm on https://github.com/torvalds/linux/commit/d4a7ce642100765119a872d4aba1bf63e3a22c8a.
Update: Looks like copy/paste failure, I was trying to copy/paste uname output. I was on 6.5.x or something.
itamarst
commented
9 months ago Rebooted into 6.6.6 and it immediately crashed:
[ 74.324060] igc 0000:02:00.0 enp2s0: PCIe link lost, device now detached
[ 74.324067] ------------[ cut here ]------------
[ 74.324068] igc: Failed to read reg 0xc030!
[ 74.324116] WARNING: CPU: 10 PID: 1655 at drivers/net/ethernet/intel/igc/igc_main.c:6641 igc_rd32+0xa4/0xc0 [igc]
[ 74.324128] Modules linked in: rfcomm ccm nf_conntrack_netlink xfrm_user xfrm_algo xt_addrtype br_netfilter vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) cmac algif_hash algif_skcipher af_alg xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink bridge stp llc overlay bnep snd_hda_codec_hdmi snd_hda_codec_realtek snd_sof_pci_intel_tgl snd_hda_codec_generic snd_sof_intel_hda_common ledtrig_audio soundwire_intel snd_sof_intel_hda_mlink soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_generic_allocation soundwire_bus snd_soc_core snd_compress ac97_bus snd_pcm_dmaengine intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common x86_pkg_temp_thermal snd_hda_intel snd_intel_dspcfg intel_powerclamp snd_intel_sdw_acpi snd_hda_codec iwlmvm snd_usb_audio coretemp uvcvideo
[ 74.324158] snd_hda_core btusb snd_usbmidi_lib videobuf2_vmalloc uvc btrtl videobuf2_memops snd_hwdep kvm_intel snd_ump btintel btbcm binfmt_misc mac80211 videobuf2_v4l2 snd_seq_midi kvm btmtk pmt_telemetry mei_hdcp mei_pxp pmt_class libarc4 snd_seq_midi_event irqbypass nls_iso8859_1 snd_rawmidi videodev snd_seq bluetooth rapl cmdlinepart snd_pcm iwlwifi snd_seq_device videobuf2_common rndis_wlan spi_nor ecdh_generic mc input_leds snd_timer intel_cstate ecc gigabyte_wmi wmi_bmof mtd ee1004 cfg80211 snd mei_me soundcore mei intel_vsec intel_hid acpi_pad acpi_tad mac_hid sparse_keymap sch_fq_codel msr parport_pc ppdev lp parport efi_pstore ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear system76_acpi(OE) hid_generic usbhid rndis_host cdc_ether hid usbnet mii system76_io(OE) i915 drm_buddy i2c_algo_bit ttm crct10dif_pclmul crc32_pclmul polyval_clmulni drm_display_helper polyval_generic
[ 74.324198] ghash_clmulni_intel cec sha256_ssse3 rc_core sha1_ssse3 drm_kms_helper aesni_intel intel_lpss_pci nvme crypto_simd cryptd i2c_i801 intel_lpss spi_intel_pci igc drm xhci_pci nvme_core ahci i2c_smbus spi_intel libahci xhci_pci_renesas idma64 nvme_common video wmi pinctrl_alderlake
[ 74.324211] CPU: 10 PID: 1655 Comm: kworker/10:2 Tainted: G OE 6.6.6-76060606-generic #202312111032~1702306143~22.04~d28ffec~dev-Ubuntu
[ 74.324213] Hardware name: System76 Thelio/Thelio, BIOS F5b Z5 10/11/2022
[ 74.324214] Workqueue: events igc_watchdog_task [igc]
[ 74.324223] RIP: 0010:igc_rd32+0xa4/0xc0 [igc]
[ 74.324230] Code: c7 c6 60 d7 47 c0 e8 eb 31 5d d8 48 8b bb 28 ff ff ff e8 1f 67 0b d8 84 c0 74 b4 44 89 e6 48 c7 c7 88 d7 47 c0 e8 3c 0c 8a d7 <0f> 0b eb a1 b8 ff ff ff ff 31 d2 31 f6 31 ff c3 cc cc cc cc 0f 1f
[ 74.324232] RSP: 0018:ffffc90003287d88 EFLAGS: 00010246
[ 74.324233] RAX: 0000000000000000 RBX: ffff88811ece2cb8 RCX: 0000000000000000
[ 74.324234] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 74.324235] RBP: ffffc90003287da0 R08: 0000000000000000 R09: 0000000000000000
[ 74.324236] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000c030
[ 74.324236] R13: ffff88811ece2000 R14: 0000000000000000 R15: ffff88811fa14d40
[ 74.324237] FS: 0000000000000000(0000) GS:ffff88907fb00000(0000) knlGS:0000000000000000
[ 74.324238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 74.324239] CR2: 0000783bc08b1000 CR3: 0000000337a3c000 CR4: 0000000000f50ee0
[ 74.324240] PKRU: 55555554
[ 74.324241] Call Trace:
[ 74.324242] <TASK>
[ 74.324244] ? show_regs+0x6d/0x80
[ 74.324247] ? __warn+0x89/0x160
[ 74.324249] ? igc_rd32+0xa4/0xc0 [igc]
[ 74.324257] ? report_bug+0x17e/0x1b0
[ 74.324260] ? handle_bug+0x46/0x90
[ 74.324262] ? exc_invalid_op+0x18/0x80
[ 74.324263] ? asm_exc_invalid_op+0x1b/0x20
[ 74.324266] ? igc_rd32+0xa4/0xc0 [igc]
[ 74.324272] ? igc_rd32+0xa4/0xc0 [igc]
[ 74.324279] igc_update_stats+0xab/0x770 [igc]
[ 74.324287] igc_watchdog_task+0xa1/0x540 [igc]
[ 74.324294] ? __schedule+0x2d3/0x750
[ 74.324296] process_one_work+0x16c/0x350
[ 74.324298] worker_thread+0x306/0x440
[ 74.324300] ? _raw_spin_lock_irqsave+0xe/0x20
[ 74.324302] ? __pfx_worker_thread+0x10/0x10
[ 74.324304] kthread+0xef/0x120
[ 74.324306] ? __pfx_kthread+0x10/0x10
[ 74.324308] ret_from_fork+0x44/0x70
[ 74.324309] ? __pfx_kthread+0x10/0x10
[ 74.324311] ret_from_fork_asm+0x1b/0x30
[ 74.324313] </TASK>
[ 74.324314] ---[ end trace 0000000000000000 ]---
After running fine for a while, a few minutes our hours after a restart Ethernet stops working. A reboot fixes it. The network is fine (other machines are working, WiFi is working).
This is an Alder Lake Thelio, with Ubuntu 22.04.
The problem started happening today. I upgraded both BIOS and Linux today so maybe(?) it's the BIOS, unclear, but there's definitely an issue happening in the kernel, see below.
From dmesg: