[Bug]: Unclear Issues with LR1110; "my T-1000-E is deaf"

fifieldt commented 2 weeks ago

Hardware

Seeed Card Tracker T1000-E

Firmware Version

2.5.0

Description

As discussed in Discord over the past 3 weeks, there are multiple instances of people reporting issues with LR1110 radios. This issue is being created to serve as a central place for recording this information and coordinate testing.

Reported issues include:

[x] Not being able to hear nodes more than 50m away
[ ] Not being able to communicate with SX1276 devices (eg tbeam)

fifieldt commented 2 weeks ago

@Talie5in has identified https://github.com/meshtastic/firmware/pull/4621 as the breakpoint.

2.5.0.d6dac17 OK
2.5.0.33eb073 OK
2.5.0.9ac0e26 - Talie5in: doesn't pick up anything other than WisMeshPocket (Rak4631)
v2.5.0.9ac0e26 - Fifield: can't talk with tbeam in the same room

caveman99 commented 2 weeks ago

@Talie5in has identified #4621 as the breakpoint.

Are the other devices on the same firmware level or older/newer? We have radiolib changes scattered in there somewhere.

fifieldt commented 2 weeks ago

Currently trying: 33eb073535c2bece2353b9fd76d0dda5d120ac05 with a later version of RadioLib...

fifieldt commented 2 weeks ago

OK, https://github.com/meshtastic/firmware/commit/33eb073535c2bece2353b9fd76d0dda5d120ac05 with RadioLib 7.0.0 results in the bad outcome.

By my logic, using the good meshtastic commit with just radiolib changes means the issue might be in RadioLib!

fifieldt commented 2 weeks ago

RadioLib 6.6.0 (working version) is May 29 https://github.com/jgromes/RadioLib/commit/954477b0c0589e6373692ce569ca117191667c30 .

Stepping forward through RadioLib with known-good Meshtastic version.

fifieldt commented 2 weeks ago

Radiolib w/ Meshtastic 33eb073: June 30 7a0b5bae9906905f32cd840ec79b2665f509f717 - OK Jul 30 6d0c226e65ca1835849c9b516255bee642d2ab21 - OK Aug 29 d3f9eaf30b438faa2406982a75f54be64f681909 - OK Sep 7 951bfc0625f966b8462be40318b9020558ed0a3e - FAIL

Looks like the breaking change was introduced between Aug 29 and Sep 7.

Aug 29 & prior - T1000E can receive messages from Tbeam. Sept 7 no messages get through.

Talie5in commented 2 weeks ago

@Talie5in has identified #4621 as the breakpoint.

Are the other devices on the same firmware level or older/newer? We have radiolib changes scattered in there somewhere.

The one example is my WisMeshPockt (Rak4631) on 2.5.0 Alpha (through the entire time I was testing different 2.5.0 TechPreview build - it was out in my car (50m away) and was the only device it would hear past https://github.com/meshtastic/firmware/commit/33eb073535c2bece2353b9fd76d0dda5d120ac05

Yes, RadioLib was changed in https://github.com/meshtastic/firmware/pull/4621 as @fifieldt has picked out of my Discord commentary

fifieldt commented 2 weeks ago

OK, it looks like the breaking change is

bc801c70043c2482af8cd81947b855d7325a41ac [PHY] Channel scan configuration

GUVWAF commented 2 weeks ago

That RadioLib commit introduces differences in the IRQ flags handling in startReceive(), so if using Meshtastic commit https://github.com/meshtastic/firmware/commit/33eb073535c2bece2353b9fd76d0dda5d120ac05 it is expected to break, but PR #4621 also should handle those API changes.

fifieldt commented 2 weeks ago

In my testing, I amended the meshtastic code appropriately to deal with the IRQ flag changes.

fifieldt commented 1 week ago

~~Potentially related: https://github.com/meshtastic/firmware/pull/4827~~ False alarm.

valzzu commented 1 week ago

works fine on long fast but doesent work on long moderate. wio tracker is the only node that can communicate with it.

GUVWAF commented 1 week ago

The incompatibility between SX127x and LR1110 really seems to be because of our non-standard sync word. If I change it on both boards to the default private network (0x12), I can send and receive between them.

GUVWAF commented 1 week ago

Confirmed LONG_MODERATE doesn't work between LR1110 and other radios, and it's likely due to low data rate optimization (LDRO), which is automatically enabled for SX126x and SX127x when the symbol length is longer than 16ms, but it's not yet there for LR11x0. I'll see if I can add it.

GUVWAF commented 1 week ago

@valzzu LONG_MODERATE should work after we update RadioLib to include https://github.com/jgromes/RadioLib/pull/1237.

valzzu commented 1 week ago

Awesome

GUVWAF commented 6 days ago

@Talie5in @fifieldt Can you try if #4885 fixes the bad receive performance for you?

In my testing, I amended the meshtastic code appropriately to deal with the IRQ flag changes.

Were you using lora.startReceive(RADIOLIB_LR11X0_RX_TIMEOUT_INF, RADIOLIB_IRQ_RX_DEFAULT_FLAGS, RADIOLIB_IRQ_RX_DEFAULT_MASK, 0)? If so, that only works from RadioLib commit https://github.com/jgromes/RadioLib/commit/3115fc2d6700a9aee05888791ac930a910f2628f (which is the one PR #4621 uses and is three commits after the commit you used).

Talie5in commented 6 days ago

Can you try if #4885 fixes the bad receive performance for you?

@GUVWAF Built against 699d81c in PR#4885 and I am getting nodes appearing after a nodeDB reset. 🎉

Some logs attached - of note one interesting bits i've not seen before but are likely normal...

WARN | 04:04:10 490 [Router] Alloc an err=5,to=0x850354e6,idFrom=0x1b7f84e4,id=0x25516740

meshtastic-log-2024-09-28T04-07-35.735Z.log Additional logging after a reboot

meshtastic-log-2024-09-28T04-15-59.764Z.log

GUVWAF commented 6 days ago

Great!

WARN | 04:04:10 490 [Router] Alloc an err=5 means "Max. retransmission reached".

meshtastic-bot commented 3 days ago

This issue has been mentioned on Meshtastic. There might be relevant details there:

https://meshtastic.discourse.group/t/sensecap-t1000-e-not-seeing-t-beams/14913/2

fifieldt commented 3 days ago

Many thanks @GUVWAF , coming back online a d should be able to test in the next few days.

meshtastic / firmware