search

xmm7360 / xmm7360-pci

PCI driver for Fibocom L850-GL modem based on Intel XMM7360 modem
420 stars 82 forks source link

Error loading module: Could not bring up command ring #99

Open ge0rg opened 3 years ago

ge0rg commented 3 years ago

The modem has been working fine for a while, then I haven't used it for some weeks, and now that I needed to use it again, it refuses. The laptop has 44d of uptime now, so maybe it got hung in some way.

I tried reloading the kernel module via make load and it loaded fine, but I got this in dmesg:

[Thu May 13 19:34:38 2021] xmm7360 0000:05:00.0: modem is ready
[Thu May 13 19:34:40 2021] xmm7360 0000:05:00.0: Could not bring up command ring
[Thu May 13 19:34:40 2021] xmm7360: probe of 0000:05:00.0 failed with error -110

I assumed that the device got hung up or went into some unreasonable state, so I tried to echo 1 > /sys/devices/pci0000:00/0000:00:02.5/0000:05:00.0/reset and to reload the module afterwards.

However, when loading the module, this kernel bug happened:

[Thu May 13 19:39:57 2021] xmm7360 0000:05:00.0: can't change power state from D3hot to D0 (config space inaccessible)
[Thu May 13 19:39:57 2021] xmm7360 0000:05:00.0: pci_alloc_irq_vectors
[Thu May 13 19:39:57 2021] ------------[ cut here ]------------
[Thu May 13 19:39:57 2021] WARNING: CPU: 12 PID: 32550 at kernel/workqueue.c:3037 __flush_work.isra.0+0x67/0x80
[Thu May 13 19:39:57 2021] Modules linked in: xmm7360(OE+) ip6_tunnel tunnel6 veth sr_mod vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) nls_iso8859_1 nls_cp437 omfs reiserfs isofs nls_utf8 udf crc_itu_t mmc_block rpcsec_gss_krb5 nfsv4 dns_resolver nfs nfs_ssc fscache nfnetlink_queue nfnetlink_log rtl2832_sdr e4000 rtl2832 i2c_mux dvb_usb_rtl28xxu dvb_usb_v2 dvb_core rc_core btrfs blake2b_generic ufs qnx4 hfsplus hfs cdrom minix vfat msdos fat jfs xfs loop sctp tun hid_plantronics ses enclosure sd_mod scsi_transport_sas sg uas usb_storage scsi_mod snd_usb_audio snd_usbmidi_lib snd_rawmidi cdc_acm hid_jabra hid_generic usbhid hid uinput ctr ccm xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter bridge stp llc overlay vmw_vsock_vmci_transport vsock vmw_vmci cmac rfcomm fuse snd_seq_dummy snd_hrtimer snd_seq snd_seq_device uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 bnep
[Thu May 13 19:39:57 2021]  videobuf2_common btusb btrtl btbcm btintel bluetooth jitterentropy_rng drbg ansi_cprng ecdh_generic ecc cpufreq_powersave cpufreq_userspace cpufreq_conservative tpm_crb iwlmvm snd_hda_codec_realtek mac80211 binfmt_misc snd_hda_codec_generic snd_hda_codec_hdmi libarc4 snd_hda_intel snd_intel_dspcfg soundwire_intel soundwire_generic_allocation edac_mce_amd snd_soc_core iwlwifi sdhci_pci kvm_amd cqhci sdhci snd_compress kvm soundwire_cadence cpufreq_ondemand snd_hda_codec xhci_pci irqbypass crc32_pclmul wmi_bmof cfg80211 snd_hda_core ccp mmc_core ghash_clmulni_intel xhci_hcd snd_hwdep soundwire_bus ehci_pci efi_pstore rapl ehci_hcd thinkpad_acpi snd_pcm joydev sp5100_tco usbcore k10temp snd_timer nvram ledtrig_audio watchdog ipmi_devintf snd_rn_pci_acp3x snd_pci_acp3x tpm_tis i2c_piix4 ucsi_acpi snd tpm_tis_core typec_ucsi ipmi_msghandler usb_common tpm soundcore typec rfkill rng_core wmi battery ac i2c_scmi acpi_cpufreq button nfsd auth_rpcgss nfs_acl videodev lockd grace mc
[Thu May 13 19:39:57 2021]  parport_pc ppdev sunrpc lp parport efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 algif_skcipher af_alg dm_crypt dm_mod raid10 raid456 libcrc32c crc32c_generic async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq raid1 raid0 multipath linear md_mod i915 amdgpu gpu_sched crc32c_intel ttm i2c_algo_bit drm_kms_helper cec nvme aesni_intel drm psmouse glue_helper libaes crypto_simd evdev cryptd nvme_core serio_raw t10_pi crc_t10dif crct10dif_generic r8168(OE) crct10dif_pclmul crct10dif_common video [last unloaded: xmm7360]
[Thu May 13 19:39:57 2021] CPU: 12 PID: 32550 Comm: insmod Tainted: G        W  OE     5.10.0-5-amd64 #1 Debian 5.10.24-1
[Thu May 13 19:39:57 2021] Hardware name: LENOVO 20U70004GE/20U70004GE, BIOS R19ET32W (1.16 ) 01/26/2021
[Thu May 13 19:39:57 2021] RIP: 0010:__flush_work.isra.0+0x67/0x80
[Thu May 13 19:39:57 2021] Code: 50 65 48 2b 14 25 28 00 00 00 75 24 48 83 c4 58 c3 48 8d 7c 24 28 88 44 24 07 e8 94 30 81 00 0f b6 44 24 07 eb d6 0f 0b eb d2 <0f> 0b 31 c0 eb cc e8 2e 8b 80 00 66 66 2e 0f 1f 84 00 00 00 00 00
[Thu May 13 19:39:57 2021] RSP: 0018:ffffbd3282003b58 EFLAGS: 00010246
[Thu May 13 19:39:57 2021] RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000000
[Thu May 13 19:39:57 2021] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff9f3a96780040
[Thu May 13 19:39:57 2021] RBP: 0000000000000000 R08: 0000000000000000 R09: ffffbd3282003968
[Thu May 13 19:39:57 2021] R10: ffffbd3282003960 R11: ffffffff9b2bca58 R12: ffff9f3a96780040
[Thu May 13 19:39:57 2021] R13: ffff9f4604d79780 R14: 00000000000002c0 R15: 0000000000000000
[Thu May 13 19:39:57 2021] FS:  00007faae0cb2500(0000) GS:ffff9f460f900000(0000) knlGS:0000000000000000
[Thu May 13 19:39:57 2021] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Thu May 13 19:39:57 2021] CR2: 0000561e57d3faa0 CR3: 000000035b122000 CR4: 0000000000350ee0
[Thu May 13 19:39:57 2021] Call Trace:
[Thu May 13 19:39:57 2021]  ? dev_printk_emit+0x4e/0x65
[Thu May 13 19:39:57 2021]  __cancel_work_timer+0x102/0x190
[Thu May 13 19:39:57 2021]  ? _dev_err+0x6c/0x83
[Thu May 13 19:39:57 2021]  xmm7360_dev_deinit+0x1f/0x120 [xmm7360]
[Thu May 13 19:39:57 2021]  xmm7360_probe+0x18d/0x1e0 [xmm7360]
[Thu May 13 19:39:57 2021]  local_pci_probe+0x42/0x80
[Thu May 13 19:39:57 2021]  ? _cond_resched+0x16/0x40
[Thu May 13 19:39:57 2021]  pci_device_probe+0xfd/0x1b0
[Thu May 13 19:39:57 2021]  really_probe+0xf2/0x440
[Thu May 13 19:39:57 2021]  driver_probe_device+0xe1/0x150
[Thu May 13 19:39:57 2021]  device_driver_attach+0xa1/0xb0
[Thu May 13 19:39:57 2021]  __driver_attach+0x8a/0x150
[Thu May 13 19:39:57 2021]  ? device_driver_attach+0xb0/0xb0
[Thu May 13 19:39:57 2021]  ? device_driver_attach+0xb0/0xb0
[Thu May 13 19:39:57 2021]  bus_for_each_dev+0x78/0xc0
[Thu May 13 19:39:57 2021]  bus_add_driver+0x12b/0x1e0
[Thu May 13 19:39:57 2021]  driver_register+0x8b/0xe0
[Thu May 13 19:39:57 2021]  ? xmm7360_net_xmit+0x190/0x190 [xmm7360]
[Thu May 13 19:39:57 2021]  do_one_initcall+0x44/0x1d0
[Thu May 13 19:39:57 2021]  ? do_init_module+0x23/0x260
[Thu May 13 19:39:57 2021]  ? kmem_cache_alloc_trace+0xf5/0x200
[Thu May 13 19:39:57 2021]  do_init_module+0x5c/0x260
[Thu May 13 19:39:57 2021]  __do_sys_finit_module+0xb1/0x110
[Thu May 13 19:39:57 2021]  do_syscall_64+0x33/0x80
[Thu May 13 19:39:57 2021]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[Thu May 13 19:39:57 2021] RIP: 0033:0x7faae0dd0f09
[Thu May 13 19:39:57 2021] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 57 3f 0c 00 f7 d8 64 89 01 48
[Thu May 13 19:39:57 2021] RSP: 002b:00007ffc7e60b948 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[Thu May 13 19:39:57 2021] RAX: ffffffffffffffda RBX: 0000561e59007790 RCX: 00007faae0dd0f09
[Thu May 13 19:39:57 2021] RDX: 0000000000000000 RSI: 0000561e57d4a3f0 RDI: 0000000000000003
[Thu May 13 19:39:57 2021] RBP: 0000561e57d4a3f0 R08: 0000000000000000 R09: 00007faae0e99240
[Thu May 13 19:39:57 2021] R10: 0000000000000003 R11: 0000000000000246 R12: 0000000000000000
[Thu May 13 19:39:57 2021] R13: 0000561e59007760 R14: 0000000000000000 R15: 0000000000000000
[Thu May 13 19:39:57 2021] ---[ end trace baf4161fc79ff1e4 ]---
[Thu May 13 19:39:57 2021] BUG: unable to handle page fault for address: 0000000000022b40
[Thu May 13 19:39:57 2021] #PF: supervisor write access in kernel mode
[Thu May 13 19:39:57 2021] #PF: error_code(0x0002) - not-present page
[Thu May 13 19:39:57 2021] PGD 0 P4D 0 
[Thu May 13 19:39:57 2021] Oops: 0002 [#1] SMP NOPTI
[Thu May 13 19:39:57 2021] CPU: 12 PID: 32550 Comm: insmod Tainted: G        W  OE     5.10.0-5-amd64 #1 Debian 5.10.24-1
[Thu May 13 19:39:57 2021] Hardware name: LENOVO 20U70004GE/20U70004GE, BIOS R19ET32W (1.16 ) 01/26/2021
[Thu May 13 19:39:57 2021] RIP: 0010:xmm7360_dev_deinit+0x10/0x120 [xmm7360]
[Thu May 13 19:39:57 2021] Code: 8b 87 88 00 00 00 48 8b b8 60 02 00 00 e9 48 ff ff ff 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 41 54 55 48 89 fd 48 83 c7 40 53 <c7> 87 00 2b 02 00 ed ff ff ff e8 31 1e 86 d7 48 83 bd 38 2b 02 00
[Thu May 13 19:39:57 2021] RSP: 0018:ffffbd3282003c08 EFLAGS: 00010202
[Thu May 13 19:39:57 2021] RAX: ffffbd32805fb000 RBX: ffffbd3282003c98 RCX: 0000000000000000
[Thu May 13 19:39:57 2021] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000040
[Thu May 13 19:39:57 2021] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff9f3a9679e628
[Thu May 13 19:39:57 2021] R10: ffffbd3282003960 R11: ffffffff9b2bca58 R12: 0000000000000000
[Thu May 13 19:39:57 2021] R13: ffff9f3741387000 R14: 00000000000002c0 R15: 0000000000000000
[Thu May 13 19:39:57 2021] FS:  00007faae0cb2500(0000) GS:ffff9f460f900000(0000) knlGS:0000000000000000
[Thu May 13 19:39:57 2021] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Thu May 13 19:39:57 2021] CR2: 0000000000022b40 CR3: 000000035b122000 CR4: 0000000000350ee0
[Thu May 13 19:39:57 2021] Call Trace:
[Thu May 13 19:39:57 2021]  xmm7360_remove+0x1a/0x60 [xmm7360]
[Thu May 13 19:39:57 2021]  xmm7360_probe+0x195/0x1e0 [xmm7360]
[Thu May 13 19:39:57 2021]  local_pci_probe+0x42/0x80
[Thu May 13 19:39:57 2021]  ? _cond_resched+0x16/0x40
[Thu May 13 19:39:57 2021]  pci_device_probe+0xfd/0x1b0
[Thu May 13 19:39:57 2021]  really_probe+0xf2/0x440
[Thu May 13 19:39:57 2021]  driver_probe_device+0xe1/0x150
[Thu May 13 19:39:57 2021]  device_driver_attach+0xa1/0xb0
[Thu May 13 19:39:57 2021]  __driver_attach+0x8a/0x150
[Thu May 13 19:39:57 2021]  ? device_driver_attach+0xb0/0xb0
[Thu May 13 19:39:57 2021]  ? device_driver_attach+0xb0/0xb0
[Thu May 13 19:39:57 2021]  bus_for_each_dev+0x78/0xc0
[Thu May 13 19:39:57 2021]  bus_add_driver+0x12b/0x1e0
[Thu May 13 19:39:57 2021]  driver_register+0x8b/0xe0
[Thu May 13 19:39:57 2021]  ? xmm7360_net_xmit+0x190/0x190 [xmm7360]
[Thu May 13 19:39:57 2021]  do_one_initcall+0x44/0x1d0
[Thu May 13 19:39:57 2021]  ? do_init_module+0x23/0x260
[Thu May 13 19:39:57 2021]  ? kmem_cache_alloc_trace+0xf5/0x200
[Thu May 13 19:39:57 2021]  do_init_module+0x5c/0x260
[Thu May 13 19:39:57 2021]  __do_sys_finit_module+0xb1/0x110
[Thu May 13 19:39:57 2021]  do_syscall_64+0x33/0x80
[Thu May 13 19:39:57 2021]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[Thu May 13 19:39:57 2021] RIP: 0033:0x7faae0dd0f09
[Thu May 13 19:39:57 2021] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 57 3f 0c 00 f7 d8 64 89 01 48
[Thu May 13 19:39:57 2021] RSP: 002b:00007ffc7e60b948 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[Thu May 13 19:39:57 2021] RAX: ffffffffffffffda RBX: 0000561e59007790 RCX: 00007faae0dd0f09
[Thu May 13 19:39:57 2021] RDX: 0000000000000000 RSI: 0000561e57d4a3f0 RDI: 0000000000000003
[Thu May 13 19:39:57 2021] RBP: 0000561e57d4a3f0 R08: 0000000000000000 R09: 00007faae0e99240
[Thu May 13 19:39:57 2021] R10: 0000000000000003 R11: 0000000000000246 R12: 0000000000000000
[Thu May 13 19:39:57 2021] R13: 0000561e59007760 R14: 0000000000000000 R15: 0000000000000000
[Thu May 13 19:39:57 2021] Modules linked in: xmm7360(OE+) ip6_tunnel tunnel6 veth sr_mod vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) nls_iso8859_1 nls_cp437 omfs reiserfs isofs nls_utf8 udf crc_itu_t mmc_block rpcsec_gss_krb5 nfsv4 dns_resolver nfs nfs_ssc fscache nfnetlink_queue nfnetlink_log rtl2832_sdr e4000 rtl2832 i2c_mux dvb_usb_rtl28xxu dvb_usb_v2 dvb_core rc_core btrfs blake2b_generic ufs qnx4 hfsplus hfs cdrom minix vfat msdos fat jfs xfs loop sctp tun hid_plantronics ses enclosure sd_mod scsi_transport_sas sg uas usb_storage scsi_mod snd_usb_audio snd_usbmidi_lib snd_rawmidi cdc_acm hid_jabra hid_generic usbhid hid uinput ctr ccm xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter bridge stp llc overlay vmw_vsock_vmci_transport vsock vmw_vmci cmac rfcomm fuse snd_seq_dummy snd_hrtimer snd_seq snd_seq_device uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 bnep
[Thu May 13 19:39:57 2021]  videobuf2_common btusb btrtl btbcm btintel bluetooth jitterentropy_rng drbg ansi_cprng ecdh_generic ecc cpufreq_powersave cpufreq_userspace cpufreq_conservative tpm_crb iwlmvm snd_hda_codec_realtek mac80211 binfmt_misc snd_hda_codec_generic snd_hda_codec_hdmi libarc4 snd_hda_intel snd_intel_dspcfg soundwire_intel soundwire_generic_allocation edac_mce_amd snd_soc_core iwlwifi sdhci_pci kvm_amd cqhci sdhci snd_compress kvm soundwire_cadence cpufreq_ondemand snd_hda_codec xhci_pci irqbypass crc32_pclmul wmi_bmof cfg80211 snd_hda_core ccp mmc_core ghash_clmulni_intel xhci_hcd snd_hwdep soundwire_bus ehci_pci efi_pstore rapl ehci_hcd thinkpad_acpi snd_pcm joydev sp5100_tco usbcore k10temp snd_timer nvram ledtrig_audio watchdog ipmi_devintf snd_rn_pci_acp3x snd_pci_acp3x tpm_tis i2c_piix4 ucsi_acpi snd tpm_tis_core typec_ucsi ipmi_msghandler usb_common tpm soundcore typec rfkill rng_core wmi battery ac i2c_scmi acpi_cpufreq button nfsd auth_rpcgss nfs_acl videodev lockd grace mc
[Thu May 13 19:39:57 2021]  parport_pc ppdev sunrpc lp parport efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 algif_skcipher af_alg dm_crypt dm_mod raid10 raid456 libcrc32c crc32c_generic async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq raid1 raid0 multipath linear md_mod i915 amdgpu gpu_sched crc32c_intel ttm i2c_algo_bit drm_kms_helper cec nvme aesni_intel drm psmouse glue_helper libaes crypto_simd evdev cryptd nvme_core serio_raw t10_pi crc_t10dif crct10dif_generic r8168(OE) crct10dif_pclmul crct10dif_common video [last unloaded: xmm7360]
[Thu May 13 19:39:57 2021] CR2: 0000000000022b40
[Thu May 13 19:39:57 2021] ---[ end trace baf4161fc79ff1e5 ]---
[Thu May 13 19:39:57 2021] RIP: 0010:xmm7360_dev_deinit+0x10/0x120 [xmm7360]
[Thu May 13 19:39:57 2021] Code: 8b 87 88 00 00 00 48 8b b8 60 02 00 00 e9 48 ff ff ff 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 41 54 55 48 89 fd 48 83 c7 40 53 <c7> 87 00 2b 02 00 ed ff ff ff e8 31 1e 86 d7 48 83 bd 38 2b 02 00
[Thu May 13 19:39:57 2021] RSP: 0018:ffffbd3282003c08 EFLAGS: 00010202
[Thu May 13 19:39:57 2021] RAX: ffffbd32805fb000 RBX: ffffbd3282003c98 RCX: 0000000000000000
[Thu May 13 19:39:57 2021] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000040
[Thu May 13 19:39:57 2021] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff9f3a9679e628
[Thu May 13 19:39:57 2021] R10: ffffbd3282003960 R11: ffffffff9b2bca58 R12: 0000000000000000
[Thu May 13 19:39:57 2021] R13: ffff9f3741387000 R14: 00000000000002c0 R15: 0000000000000000
[Thu May 13 19:39:57 2021] FS:  00007faae0cb2500(0000) GS:ffff9f460f900000(0000) knlGS:0000000000000000
[Thu May 13 19:39:57 2021] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Thu May 13 19:39:57 2021] CR2: 0000000000022b40 CR3: 000000035b122000 CR4: 0000000000350ee0
tgxn commented 3 years ago

Hey mate,

Can you try the following:

sudo make unload
make
sudo make load

I'd like to see if this still causes any kernel bugs. Otherwise, can you try restarting the laptop too? :)

I get the following in dmesg: [102692.336797] xmm7360 0000:04:00.0: modem is ready

I've noticed some similar issues where the connection cannot be reestablished, it's possible that sometimes the modem needs to be booted into windows to resolve any hangs/issues that we don't currently handle.

I've not seen any of these kernel messages, but could this indicate a possible hardware problem? can't change power state from D3hot to D0 (config space inaccessible)

tgxn commented 3 years ago

Error originates from here: https://github.com/xmm7360/xmm7360-pci/blob/9d9a2000cc70fc941e771e35f579becd9afe7445/xmm7360.c#L1399

It'd be nice to know what the output of xmm7360_cmd_ring_init is in your case, since that's what open the cmd ring for the modem.

tgxn commented 3 years ago

I'm now having this same error on my laptop too. :/

ge0rg commented 3 years ago

WORKAROUND that worked for me:

  1. make unload
  2. systemctl suspend
  3. resume
  4. make load
plajjan commented 3 years ago

I'm also seeing this:

[321112.836877] xmm7360 0000:02:00.0: modem is ready
[321114.871539] xmm7360 0000:02:00.0: Could not bring up command ring
[321114.872050] xmm7360: probe of 0000:02:00.0 failed with error -110

workaround does indeed work.

jhagberg commented 2 years ago

Seeing the same error "Could not bring up command ring" The workaround seems to get the modem back on track again