openwrt / packages

Community maintained packages for OpenWrt. Documentation for submitting pull requests is in CONTRIBUTING.md
GNU General Public License v2.0
4.03k stars 3.5k forks source link

modemmanager: improper detection of modem at boot #24831

Open arinc9 opened 3 months ago

arinc9 commented 3 months ago

Maintainer: @feckert, @aleksander0m Environment: aarch64, Banana Pi BPI-R4, OpenWrt SNAPSHOT

Description:

This issue only happens for the modem on the M.2 socket. Modems on the mPCIe sockets are detected properly. This issue is not there on Buildroot which utilises ModemManager 1.22.0 unmodified.

At boot, the modem won't be detected properly. The cdc_mbim driver is missing from the drivers section.

root@OpenWrt:/# mmcli -m 0
  --------------------------------
  General  |                 path: /org/freedesktop/ModemManager1/Modem/0
           |            device id: dcdcb513b4e963cc181b9b841f1986448b775bf7
  --------------------------------
  Hardware |         manufacturer: Quectel
           |                model: EP06
           |    firmware revision: EP06ELAR03A08M4G
           |            supported: gsm-umts, lte
           |              current: gsm-umts, lte
           |         equipment id: 
  --------------------------------
  System   |               device: /sys/devices/platform/soc/11200000.usb/usb1/1-1/1-1.1
           |              physdev: /sys/devices/platform/soc/11200000.usb/usb1/1-1/1-1.1
           |              drivers: option1
           |               plugin: quectel
           |         primary port: ttyUSB2
           |                ports: ttyUSB0 (ignored), ttyUSB1 (gps), ttyUSB2 (at), ttyUSB3 (at)
  --------------------------------
[...]

Same modem on Buildroot is detected properly.

# mmcli -m 0
  --------------------------------
  General  |                 path: /org/freedesktop/ModemManager1/Modem/0
           |            device id: 641e3f3b807779e6c119591c1f81ec5f1eb4cec8
  --------------------------------
  Hardware |         manufacturer: Quectel
           |                model: EP06-E
           |    firmware revision: EP06ELAR03A08M4G
           |         h/w revision: EP06-E
           |            supported: gsm-umts, lte
           |              current: gsm-umts, lte
           |         equipment id: 
  --------------------------------
  System   |               device: /sys/devices/platform/soc/11200000.usb/usb1/1-1/1-1.1
           |              physdev: /sys/devices/platform/soc/11200000.usb/usb1/1-1/1-1.1
           |              drivers: option, cdc_mbim
           |               plugin: quectel
           |         primary port: cdc-wdm0
           |                ports: cdc-wdm0 (mbim), ttyUSB0 (ignored), ttyUSB1 (gps), 
           |                       ttyUSB2 (at), ttyUSB3 (at), wwu1u1i4 (net)
  --------------------------------
[...]

The modem is detected properly on OpenWrt after manually resetting the modem.

root@OpenWrt:/# echo -ne 'AT+CFUN=1,1\r' > /dev/ttyUSB3

[ 1572.048735] option1 ttyUSB0: GSM modem (1-port) converter now disconnected from ttyUSB0
[ 1572.056888] option 1-1.1:1.0: device disconnected
[ 1572.061883] option1 ttyUSB1: GSM modem (1-port) converter now disconnected from ttyUSB1
[ 1572.069974] option 1-1.1:1.1: device disconnected
[ 1572.075153] option1 ttyUSB2: GSM modem (1-port) converter now disconnected from ttyUSB2
[ 1572.083284] option 1-1.1:1.2: device disconnected
[ 1572.088232] option1 ttyUSB3: GSM modem (1-port) converter now disconnected from ttyUSB3
[ 1572.096359] option 1-1.1:1.3: device disconnected
[ 1572.101197] cdc_mbim 1-1.1:1.4 wwan0: unregister 'cdc_mbim' usb-11200000.usb-1.1, CDC MBIM
[ 1576.999690] usb 1-1.1: new high-speed USB device number 7 using xhci-mtk
[ 1577.182439] option 1-1.1:1.0: GSM modem (1-port) converter detected
[ 1577.188805] usb 1-1.1: GSM modem (1-port) converter now attached to ttyUSB0
[ 1577.196039] option 1-1.1:1.1: GSM modem (1-port) converter detected
[ 1577.202515] usb 1-1.1: GSM modem (1-port) converter now attached to ttyUSB1
[ 1577.209773] option 1-1.1:1.2: GSM modem (1-port) converter detected
[ 1577.216224] usb 1-1.1: GSM modem (1-port) converter now attached to ttyUSB2
[ 1577.223485] option 1-1.1:1.3: GSM modem (1-port) converter detected
[ 1577.229925] usb 1-1.1: GSM modem (1-port) converter now attached to ttyUSB3
[ 1577.260206] cdc_mbim 1-1.1:1.4: cdc-wdm0: USB WDM device
[ 1577.265912] cdc_mbim 1-1.1:1.4 wwan0: register 'cdc_mbim' at usb-11200000.usb-1.1, CDC MBIM, d2:1b:6c:de:a5:e1

root@OpenWrt:/# mmcli -m 3
  -----------------------------------
  General  |                    path: /org/freedesktop/ModemManager1/Modem/3
           |               device id: 641e3f3b807779e6c119591c1f81ec5f1eb4cec8
  -----------------------------------
  Hardware |            manufacturer: Quectel
           |                   model: EP06-E
           |       firmware revision: EP06ELAR03A08M4G
           |          carrier config: ROW_Generic_3GPP
           | carrier config revision: 06010821
           |            h/w revision: EP06-E
           |               supported: gsm-umts, lte
           |                 current: gsm-umts, lte
           |            equipment id: 
  -----------------------------------
  System   |                  device: /sys/devices/platform/soc/11200000.usb/usb1/1-1/1-1.1
           |                 physdev: /sys/devices/platform/soc/11200000.usb/usb1/1-1/1-1.1
           |                 drivers: option1, cdc_mbim
           |                  plugin: quectel
           |            primary port: cdc-wdm0
           |                   ports: cdc-wdm0 (mbim), ttyUSB0 (ignored), ttyUSB1 (gps), 
           |                          ttyUSB2 (at), ttyUSB3 (at), wwan0 (net)
  -----------------------------------
[...]

All modems seem to be brought up by the kernel properly on OpenWrt.

[    5.319800] usb 1-1.1: new high-speed USB device number 3 using xhci-mtk
[    5.504782] option 1-1.1:1.0: GSM modem (1-port) converter detected
[    5.511125] usb 1-1.1: GSM modem (1-port) converter now attached to ttyUSB0
[    5.518193] option 1-1.1:1.1: GSM modem (1-port) converter detected
[    5.524514] usb 1-1.1: GSM modem (1-port) converter now attached to ttyUSB1
[    5.531587] option 1-1.1:1.2: GSM modem (1-port) converter detected
[    5.537899] usb 1-1.1: GSM modem (1-port) converter now attached to ttyUSB2
[    5.544968] option 1-1.1:1.3: GSM modem (1-port) converter detected
[    5.551283] usb 1-1.1: GSM modem (1-port) converter now attached to ttyUSB3
[    5.580444] cdc_mbim 1-1.1:1.4: cdc-wdm0: USB WDM device
[    5.586063] cdc_mbim 1-1.1:1.4 wwan0: register 'cdc_mbim' at usb-11200000.usb-1.1, CDC MBIM, d2:1b:6c:de:a5:e1

[    7.979821] usb 1-1.3: new high-speed USB device number 5 using xhci-mtk
[    8.182779] option 1-1.3:1.0: GSM modem (1-port) converter detected
[    8.195689] usb 1-1.3: GSM modem (1-port) converter now attached to ttyUSB4
[    8.202946] option 1-1.3:1.1: GSM modem (1-port) converter detected
[    8.209363] usb 1-1.3: GSM modem (1-port) converter now attached to ttyUSB5
[    8.216626] option 1-1.3:1.2: GSM modem (1-port) converter detected
[    8.223116] usb 1-1.3: GSM modem (1-port) converter now attached to ttyUSB6
[    8.230355] option 1-1.3:1.3: GSM modem (1-port) converter detected
[    8.236790] usb 1-1.3: GSM modem (1-port) converter now attached to ttyUSB7
[    8.260387] cdc_mbim 1-1.3:1.4: cdc-wdm1: USB WDM device
[    8.266104] cdc_mbim 1-1.3:1.4 wwan1: register 'cdc_mbim' at usb-11200000.usb-1.3, CDC MBIM, d2:1b:6c:de:a5:e1

[    8.619860] usb 1-1.2: new high-speed USB device number 6 using xhci-mtk
[    8.803905] option 1-1.2:1.0: GSM modem (1-port) converter detected
[    8.810375] usb 1-1.2: GSM modem (1-port) converter now attached to ttyUSB8
[    8.817625] option 1-1.2:1.1: GSM modem (1-port) converter detected
[    8.824117] usb 1-1.2: GSM modem (1-port) converter now attached to ttyUSB9
[    8.831353] option 1-1.2:1.2: GSM modem (1-port) converter detected
[    8.837806] usb 1-1.2: GSM modem (1-port) converter now attached to ttyUSB10
[    8.845125] option 1-1.2:1.3: GSM modem (1-port) converter detected
[    8.851585] usb 1-1.2: GSM modem (1-port) converter now attached to ttyUSB11
[    8.870625] cdc_mbim 1-1.2:1.4: cdc-wdm2: USB WDM device
[    8.876348] cdc_mbim 1-1.2:1.4 wwan2: register 'cdc_mbim' at usb-11200000.usb-1.2, CDC MBIM, d2:1b:6c:de:a5:e1
brada4 commented 3 months ago

How one of three possible working configurations is qualified as improper? In general you do not need modemmanager, native luci extensions suffice.

arinc9 commented 3 months ago

How one of three possible working configurations is qualified as improper?

Sorry, what does that mean? Where do you see that "one of three possible working configurations" do I claim is improper? I have clearly stated that the modem is not detected properly at boot.

In general you do not need modemmanager, native luci extensions suffice.

I disagree with your suggestion. I'd much rather prefer to have this issue fixed instead of using something else to evade the issue.

feckert commented 1 month ago

If this only happens at boot, it is usually because the modem is not yet ready to respond on the cdc_wdm interface. I suspect that it is not responding to the cdc_wdm interface during the probe phase of the ModemManager. The ModemManager then assumes that cdc_wdm (qmi/mbim) is not supported. As a workaround you could delay the start of ModemManager. But the cleanest solution would be that the ModemManager extends the probe phase and thus gives the modem more time to respond correctly during probing.

Will the modem be detected if you restart the ModemManager? This is definitely a timing thing.

I'm going to close this because I don't think we can do anything about it. If you disagree, feel free to reopen the ticket.

arinc9 commented 1 month ago

Will the modem be detected if you restart the ModemManager?

After running service modemmanager restart and waiting for a minute to get past the error: couldn't find the ModemManager process in the bus response, I see the same output I've mentioned in the issue report.

I'm going to close this because I don't think we can do anything about it. If you disagree, feel free to reopen the ticket.

As I've mentioned:

This issue is not there on Buildroot which utilises ModemManager 1.22.0 unmodified.

I think the problem stems from the modifications made to ModemManager for OpenWrt. Therefore, I believe that this issue report should remain open until the issue it reports is fixed. I see no option in this page to re-open the issue. I'm going to wait 48 hours for you to re-open it or provide me the means to do so on my own. Otherwise, I will open another issue with the same report.

feckert commented 1 month ago

Then you need to give us some more information.

Will the modem be detected if you restart the ModemManager?

After running service modemmanager restart and waiting for a minute to get past the error: couldn't find the ModemManager process in the bus response, I see the same output I've mentioned in the issue report.

Is the dbus service running?

Could use please execute the following command /etc/init.d/modemmanager restart and past the log message from syslog?

dangowrt commented 1 month ago

Looks like hotplug script may need to be extended to notify MM about the addition of the cdc_mbim as well as mbim network device. And maybe PCIe MHI devices...

feckert commented 3 weeks ago

What are the content of the files events.cache in /var/run/modemmanager/? Is the sysfs path of your Modem listed there? This file is used to simulate the udev events for the ModemManger in OpenWrt. This file is filled with the different subsystem events from hotplug.d.

dangowrt commented 3 weeks ago

I managed to reproduce this only on kernels which got all USB drivers for MBIM and QMI as well as USB serial devices exposed by modems built-in. With OpenWrt's default kernel config which builds all those things as modules I can not reproduce the issue.

The reason is simple: Especially on warm reboot, the modem is probed and devices are added before userspace is ready to process any hotplug events.

Hence I suggest to close this ticket as the bug is only present and directly caused by use of a custom kernel which has all those drivers built-in (having drivers built-in which usually aren't may obviously cause havoc for all sorts of applications relying on hotplug events).

arinc9 commented 3 weeks ago

Building a kernel with built-in drivers is standard practice. Whether or not they can be built as kernel modules for OpenWrt, I find it poor design for an operating system to depend on kernel modules to operate properly. Therefore, I see my issue as valid. With that said, I'm not here to argue the policy of the development of OpenWrt, so I'm fine with this issue being closed with a 'won't fix' resolution for this reason.

dangowrt commented 3 weeks ago

@arinc9 In that case it would be an issue to be opened for procd, as this is where handling coldplug is missing for built-in drivers, and modemmanager relies on that to receive hotplug events for every modem-related device.

"Standard practise" is obviously a matter of definition. In OpenWrt, it is not standard practise to have USB device drivers, or any drivers which are not required for failsafe mode, built-in. Often times this is not even an option as many devices come with proprietary bootloaders which impose a size limit on the size of the kernel image to be loaded, which is why we try to keep it is slim as possible.

brada4 commented 3 weeks ago

udev fires all events on start at the price of keeping device "database" in userspace.