search

opnsense / src

OPNsense operating system on top of FreeBSD
https://opnsense.org/
Other
354 stars 150 forks source link

Quectel LTE MODEM not working starting with 24.7 #214

Open pickonedev opened 1 month ago

pickonedev commented 1 month ago

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

The modem it is not connecting to the internet, I only get this in the log:

2024-08-12T14:18:23 Informational   ppp [opt1_link0] Link: reconnection attempt 148 in 3 seconds    
2024-08-12T14:18:23 Informational   ppp [opt1_link0] LCP: Down event    
2024-08-12T14:18:23 Informational   ppp [opt1_link0] Link: DOWN event   
2024-08-12T14:18:23 Informational   ppp [opt1_link0] MODEM: chat script failed  
2024-08-12T14:18:23 Informational   ppp [opt1_link0] CHAT: The modem is not responding to "AT" at MomCmd: labeell.

Seems that with the new 24.7 version there is a typo error as well "labeell" vs "label"

A clear and concise description of what the bug is, including last known working version (if any).

Tip: to validate your setup was working with the previous version, use opnsense-revert (https://docs.opnsense.org/manual/opnsense_tools.html#opnsense-revert)

To Reproduce

Steps to reproduce the behavior: Install the new 24.7 and the modem won't work anymore

Environment

Software version used and hardware type if relevant, e.g.:

OPNsense 24.7.1 (amd64). Intel{R} N100 (4 cores, 4 threads) Samsung 16GB DDR5 Intel I226-V Chip 2.5G

PUTTY SCREENSHOT

image

fichtner commented 1 month ago

Not sure where the error would be. Core repo:

% git grep -i labeell
%
pickonedev commented 1 month ago

The error is that the modem it is not working starting with 24.7 [opt1_link0] Link: reconnection attempt 148 in 3 seconds

Everything is configured by default, it is just not connecting...

I don't see more details in the log :-(

image

fichtner commented 1 month ago

Can you do a health audit first?

pickonedev commented 1 month ago

Sure, sorry.

***GOT REQUEST TO AUDIT HEALTH***
Currently running OPNsense 24.7.1 at Mon Aug 12 15:08:47 EEST 2024
>>> Root file system: zroot/ROOT/default
>>> Check installed kernel version
Version 24.7.1 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 24.7.1 is correct.
>>> Check for missing or altered base files
No problems detected.
>>> Check installed repositories
OPNsense
mimugmail
>>> Check installed plugins
os-ddclient 1.23
os-opnarp-maxit 1.0_4
>>> Check locked packages
No locks found.
>>> Check for missing package dependencies
Checking all packages: .......... done
>>> Check for missing or altered package files
Checking all packages: .......... done
>>> Check for core packages consistency
Core package "opnsense" has 68 dependencies to check.
Checking packages: ..................................................................... done
***DONE***
pickonedev commented 1 month ago

I am not the only one with this issue, there is at least one more person which I know about, the same issue... similar modem

fichtner commented 1 month ago

Looks clean. If it's the modem/driver I don't know where to look. It would perhaps be something in FreeBSD 14.1 that's not as it should be.

Cheers, Franco

pickonedev commented 1 month ago

The thing is that the modem response is ok, testing in ssh. More than that, on another machine, the modem is fine, working good... Only with this new version, I have this issue :-( I tried on all the ppp interfaces, even I know that only 2 and 3 is working

image

pickonedev commented 1 month ago

I have found something in System > Log Files > General. I don't know if is something or not...


2024-08-12T15:59:32 Warning opnsense    /interfaces_assign.php: ROUTING: refusing to set interface route on addressless opt1(ppp1)  
2024-08-12T15:56:23 Warning opnsense    /usr/local/etc/rc.routing_configure: ROUTING: refusing to set interface route on addressless opt1(ppp1) 
2024-08-12T15:56:17 Notice  kernel  <118> 4G (ppp1) ->  
2024-08-12T15:56:11 Warning opnsense    /usr/local/etc/rc.bootup: ROUTING: refusing to set interface route on addressless opt1(ppp1)    
2024-08-12T15:56:09 Notice  kernel  <6>ng0: changing name to 'ppp1' 
2024-08-12T15:52:46 Warning opnsense    /usr/local/etc/rc.routing_configure: ROUTING: refusing to set interface route on addressless opt1(ppp1) 
2024-08-12T15:52:37 Notice  kernel  <118> 4G (ppp1) ->
itn3rd77 commented 1 month ago

I can confirm the same problem with an Quectel EG25-G. Besides the message CHAT: The modem is not responding to "AT" at MomCmd: labeell. I also have found CHAT: line 358: label "MomIdentGeneriicc" not found.

pickonedev commented 1 month ago

No solution? Or at least a reason for why is not working anymore? :-(

Rhapsodyan commented 3 weeks ago

I can confirm the same problem with an Quectel EG25-G. Besides the message CHAT: The modem is not responding to "AT" at MomCmd: labeell. I also have found CHAT: line 358: label "MomIdentGeneriicc" not found.

Same problem here, with Quectel EG25-G and OPNsense 24.7.1.

pickonedev commented 3 weeks ago

Hope it will be fixed soon. I was using the quectel for backup internet line...

fichtner commented 3 weeks ago

You will have to do some research here because I have no idea where the change is coming from.

pickonedev commented 3 weeks ago

I don't know too much about programming, but if you want me to test something, please, don't hesitate to tell me, I will do all the needed tests and share all the needed logs :-D

fichtner commented 3 weeks ago

I'm going to need a dmesg output first. I checked the u3g driver and it doesn't deviate much between FreeBSD 13.2 and 14.1. Looking for clues...

fichtner commented 3 weeks ago

it's probably this change https://github.com/freebsd/freebsd-src/commit/d3a83456e1 changing "LTE modem" to "Quectel XYZ" ?

pickonedev commented 3 weeks ago

It this ok?

root@OPNsense:~ # dmesg | grep -i "usb"

usbus0: 5.0Gbps Super Speed USB v3.0
ugen0.1: <Intel XHCI root HUB> at usbus0
uhub0 on usbus0
uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
Root mount waiting for: usbus0
ugen0.2: <Quectel EP06-E> at usbus0
u3g0: <Quectel EP06-E, class 239/2, rev 2.00/3.10, addr 1> on usbus0
xhci0: <XHCI (generic) USB 3.0 controller> mem 0x6001100000-0x600110ffff at device 20.0 on pci0
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
ugen0.1: <Intel XHCI root HUB> at usbus0
uhub0 on usbus0
uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
Root mount waiting for: usbus0
ugen0.2: <Quectel EP06-E> at usbus0
u3g0: <Quectel EP06-E, class 239/2, rev 2.00/3.10, addr 1> on usbus0
xhci0: <XHCI (generic) USB 3.0 controller> mem 0x6001100000-0x600110ffff at device 20.0 on pci0
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
ugen0.1: <Intel XHCI root HUB> at usbus0
uhub0 on usbus0
uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
Root mount waiting for: usbus0
ugen0.2: <Quectel EP06-E> at usbus0
u3g0: <Quectel EP06-E, class 239/2, rev 2.00/3.10, addr 1> on usbus0
xhci0: <XHCI (generic) USB 3.0 controller> mem 0x6001100000-0x600110ffff at device 20.0 on pci0
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
ugen0.1: <Intel XHCI root HUB> at usbus0
uhub0 on usbus0
uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
Root mount waiting for: usbus0
ugen0.2: <Quectel EP06-E> at usbus0
u3g0: <Quectel EP06-E, class 239/2, rev 2.00/3.10, addr 1> on usbus0
xhci0: <XHCI (generic) USB 3.0 controller> mem 0x6001100000-0x600110ffff at device 20.0 on pci0
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
ugen0.1: <Intel XHCI root HUB> at usbus0
uhub0 on usbus0
uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
Root mount waiting for: usbus0
ugen0.2: <Quectel EP06-E> at usbus0
u3g0: <Quectel EP06-E, class 239/2, rev 2.00/3.10, addr 1> on usbus0
xhci0: <XHCI (generic) USB 3.0 controller> mem 0x6001100000-0x600110ffff at device 20.0 on pci0
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
ugen0.1: <Intel XHCI root HUB> at usbus0
uhub0 on usbus0
uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
Root mount waiting for: usbus0
ugen0.2: <Quectel EP06-E> at usbus0
u3g0: <Quectel EP06-E, class 239/2, rev 2.00/3.10, addr 1> on usbus0
xhci0: <XHCI (generic) USB 3.0 controller> mem 0x6001100000-0x600110ffff at device 20.0 on pci0
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
ugen0.1: <Intel XHCI root HUB> at usbus0
uhub0 on usbus0
uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
Root mount waiting for: usbus0
ugen0.2: <Quectel EP06-E> at usbus0
u3g0: <Quectel EP06-E, class 239/2, rev 2.00/3.10, addr 1> on usbus0
fichtner commented 3 weeks ago

I was looking at the code.... something silly could be happening but I need to be sure. For now let's test a full revert of said commit:

# opnsense-update -zkr 24.7.2_2

(and reboot)

Cheers, Franco

pickonedev commented 3 weeks ago

Done it, but the issue it is still here

kernel 24.7.2_2

2024-08-21T03:13:59 Informational   ppp [opt1_link0] Link: reconnection attempt 53 in 4 seconds 
2024-08-21T03:13:59 Informational   ppp [opt1_link0] LCP: Down event    
2024-08-21T03:13:59 Informational   ppp [opt1_link0] Link: DOWN event   
2024-08-21T03:13:59 Informational   ppp [opt1_link0] MODEM: chat script failed  
2024-08-21T03:13:59 Informational   ppp [opt1_link0] CHAT: The modem is not responding to "AT" at MomCmd: labeell.  
2024-08-21T03:13:58 Informational   ppp [opt1_link0] Link: reconnection attempt 52
fichtner commented 3 weeks ago

Ok, I'm entirely out of ideas then.

pickonedev commented 3 weeks ago

I have "upgraded" to 24.7.1 back, till new thoughts :-) Thank you!

pickonedev commented 3 weeks ago

Still, I have a question... This could not work because the modem I use, quectel, it it used through USB? If I add the modem through the Mini PCIe port, it could work? I need to know if the issue is from the connectivity with USB port, or the modem itself.

fichtner commented 3 weeks ago

If the problem is in the USB stack then yes. If it is elsewhere maybe.

pickonedev commented 3 weeks ago

Then... I will try the modem on Mini PCI port as well. Still, even if this will be the issue, there are other people which uses the modem with the USB adapter :-|

I will come back with results

pickonedev commented 3 weeks ago

I mounted the modem on Mini PCI port and still nothing... 

2024-08-21T19:41:00 Informational   ppp [opt1_link0] Link: reconnection attempt 13 in 4 seconds 
2024-08-21T19:41:00 Informational   ppp [opt1_link0] LCP: Down event    
2024-08-21T19:41:00 Informational   ppp [opt1_link0] Link: DOWN event   
2024-08-21T19:41:00 Informational   ppp [opt1_link0] MODEM: chat script failed  
2024-08-21T19:41:00 Informational   ppp [opt1_link0] CHAT: The modem is not responding to "AT" at MomCmd: labeell.  
2024-08-21T19:40:58 Informational   ppp [opt1_link0] Link: reconnection attempt 12  
2024-08-21T19:40:56 Informational   ppp [opt1_link0] Link: reconnection attempt 12 in 2 seconds 
2024-08-21T19:40:56 Informational   ppp [opt1_link0] LCP: Down event    
2024-08-21T19:40:56 Informational   ppp [opt1_link0] Link: DOWN event   
2024-08-21T19:40:56 Informational   ppp [opt1_link0] MODEM: chat script failed  
2024-08-21T19:40:56 Informational   ppp [opt1_link0] CHAT: The modem is not responding to "AT" at MomCmd: labeell.  
2024-08-21T19:40:54 Informational   ppp [opt1_link0] Link: reconnection attempt 11  
2024-08-21T19:40:52 Informational   ppp [opt1_link0] Link: reconnection attempt 11 in 2 seconds

A friend of mine made the same test, he have the same equipment as me, but older version of opnsense, everything is working flawless :-(

pickonedev commented 3 weeks ago

Just to be sure that I am doing everything good. Even if with the older opnsense version was working, maybe something changed with the new version... These are the steps I done to configure the PPP LTE Modem

Added a Point-to-Point device, type PPP, with the proper configuration (dialing the proper number, APN, and so on...) Assigned the PPP1 4G device to the opt1 interface (enabled after) Added a gateway for the 4G modem

All saved, opnsense reboot. Thats it. There is something more which I need to do, more than I did now? With the same actions, opnsense older version was working good.

jjabba commented 1 week ago

I'm having the same or at least a very similar issue running a 'Simcom-SIM7600G-H' pci-express LTE modem. Everything was working solid until i applied the 24.7 upgrade to the system.

Now all i'm getting in the PPP log section is:

2024-09-03T00:22:06 Informational ppp [wan_link0] Link: reconnection attempt 8 in 2 seconds
2024-09-03T00:22:06 Informational ppp [wan_link0] LCP: Down event
2024-09-03T00:22:06 Informational ppp [wan_link0] Link: DOWN event
2024-09-03T00:22:06 Informational ppp [wan_link0] MODEM: chat script failed
2024-09-03T00:22:06 Informational ppp [wan_link0] CHAT: line 391: label "MomIdentGeneriicc" not found
2024-09-03T00:22:06 Informational ppp Label 'MomIdentGeneriicc' not found
2024-09-03T00:22:06 Informational ppp [wan_link0] Link: reconnection attempt 7

fichtner commented 1 week ago

I'd recommend hitting https://bugs.freebsd.org at this point. Someone with more insight might tell us what's going on here. I don't see how we possibly broke this. Other modems still work fine.

pickonedev commented 1 week ago

The freebsd bugs platoform it is so bad... :-| I made a report right now, but I don't know if it was filled in a proper way. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=281218

fichtner commented 1 week ago

Thanks, I subscribed to the issue.

pickonedev commented 1 week ago

@fichtner You said that you tested with Sierra modules and all was good. By any mistake, have you tested with SIERRA MC7421 ? I am trying to buy a sierra module, if this is working alright....

fichtner commented 1 week ago

No we added a new one, see https://github.com/opnsense/src/commit/f8d2d48efc6b

pickonedev commented 1 week ago

Oh, I understand. So, could be this added in the future? SIERRA MC7421? It is the only one I can use. Just to know if I buy it or not :))

fichtner commented 1 week ago

MC7421 doesn't exist but if you can provide the device info we can make a similar kernel patch.

I'm not saying it will work, but there is also no apparent reason why it wouldn't work.

Cheers, Franco

pickonedev commented 1 week ago

This is only what I have till now, about this device https://www.sierrawireless.com/iot-modules/4g-modules/em7421/

fichtner commented 1 week ago

Can anyone affected try the following tunable?

"hw.usb.ucom.cons_baud" with value "9600".

Reboot and see...

Cheers, Franco

pickonedev commented 1 week ago

If you tell me step by step where do to this, I will do it right now. I'm newbie with this

fichtner commented 1 week ago

Simply go to System: Settings: Tunables and add the new tunable

hw.usb.ucom.cons_baud

and value

9600

Save and reboot.

pickonedev commented 1 week ago

Even if I use the modem on Mini PCI port? And not on USB?

I tried and it is not working, same errors in the log. But, as I said, I am using the modem on Mini PCI now... I don't have the USB adapter anymore

fichtner commented 1 week ago

As far as I can tell this tunable is only for USB serial.

pickonedev commented 1 week ago

Any tunable the modem itself? Even if is connected to Mini PCI port? :-(

jjabba commented 6 days ago

I'm in the same boat as @pickonedev, my system is currently disconnected and sitting on my lab-bench if there is any mods or patches I would be more than happy to try.

fichtner commented 6 days ago

I posted a thing to try:

https://github.com/opnsense/src/issues/214#issuecomment-2335189315

jjabba commented 6 days ago

@fichtner I should have mentioned that I'm also using mini-pci HW:

Hardware: APU6B4: 1 SFP, 3 LAN, quad core CPU, 4 GB RAM

Modem: Simcom SIM7600G-H LTE, mini-pci

Software OPNsense 24.7.3_1-amd64 FreeBSD 14.1-RELEASE-p3 OpenSSL 3.0.14

fichtner commented 6 days ago

@jjabba I can't find SIM7600G at a first glance. All of this feels stabbing in the dark. I have some knowledge with u3g driver and USB here but what to look for in mini-pci is beyond my abilities at the moment.

My latest theory is https://github.com/opnsense/src/commit/4722ceb7d5 caused this particular issue. I can try to revert and let you have a test kernel, but it's a bit of a stab in the dark at the moment with FreeBSD also not knowing what's going on.

Cheers, Franco

fichtner commented 6 days ago

Try this one then, but just a working theory:

# opnsense-updatge -zkr 24.7.3-serial

Cheers, Franco

jjabba commented 6 days ago

I gave it a go, installed the kernel update, but unfortunately the same issue as before:

2024-09-10T00:57:03 Informational ppp [wan_link0] Link: reconnection attempt 36
2024-09-10T00:57:02 Informational ppp [wan_link0] Link: reconnection attempt 36 in 1 seconds
2024-09-10T00:57:02 Informational ppp [wan_link0] LCP: Down event
2024-09-10T00:57:02 Informational ppp [wan_link0] Link: DOWN event
2024-09-10T00:57:02 Informational ppp [wan_link0] MODEM: chat script failed
2024-09-10T00:57:02 Informational ppp [wan_link0] CHAT: line 391: label "MomIdentGeneriicc" not found
2024-09-10T00:57:02 Informational ppp Label 'MomIdentGeneriicc' not found
2024-09-10T00:57:02 Informational ppp [wan_link0] Link: reconnection attempt 35
2024-09-10T00:56:58 Informational ppp [wan_link0] Link: reconnection attempt 35 in 4 seconds

Happy to do more testing, provide logs, whatever it takes.

Is there a way to verify the patched kernel installed correctly?

jjabba commented 6 days ago

I dug around and found this: https://docs.opnsense.org/manual/how-tos/cellular.html and https://docs.freebsd.org/en/books/handbook/serialcomms/#serial

which mentions /etc/remote

fichtner commented 6 days ago

Hmm, /etc/remote has been unmodified since 2008 and we're clearly looking for an immediate regression from FreeBSD 13.2 to 14.1. If it's not in the kernel/base it may be in mpd5 but I have no clue to be honest why it transforms a string from "MomIdentGeneric" to MomIdentGeneriicc". Here is the same issue reported with pfSense which may have some pointers for users with the issue ... https://forum.netgate.com/topic/175737/netgate-2100-micro-sim-4g-lte-5g

fichtner commented 6 days ago

I'm loading the mpd5 source code now via git-svn to sift through this. I'm afraid the code visibility isn't very good...