[Bug]: Neighbor Info only being published as JSON via MQTT not as protobuf

sgtwilko commented 3 weeks ago

Hardware

T-Beam

Firmware Version

v2.4.2.5b45303 & 2.3.11.2740a56

Description

I can see the Neighbor info packets received from other nodes being published as JSON to MQTT on the /json/ topic. However the protobuf version that you'd expect to see in the /e/ topic is missing.

I've verified this on two version of the firmware. I temporarily increased the frequency of the reporting from one of my remote nodes, and I can see the JSON messages in MQTT, but there's no equivalent in the /e/ topic.

From the MQTT packets I can see the packet numbers of the decoded protobuf packets. These show the packet before and after the Neighbor info packet, with a missing packet between then.

Relevant log output

The logs do show an attempt to send to both the e and json topics:


DEBUG | 00:14:14 6183 [Router] MQTT onSend - Publishing encrypted message

DEBUG | 00:14:14 6183 [Router] MQTT Publish msh/EU_868/2/e/LongFast/!abf87f34, 98 bytes

INFO | 00:14:14 6183 [Router] serialized json message: {"channel":0,"from":1968580390,"hop_start":3,"hops_away":0,"id":1116071721,"payload":{"last_sent_by_id":2885189428,"neighbors":[{"nod

INFO | 00:14:14 6183 [Router] JSON publish message to msh/EU_868/2/json/LongFast/!abf87f34, 341 bytes: {"channel":0,"from":1968580390,"hop_start":3,"hops_away":0,"id":1116071721,"payload":

garthvh commented 3 weeks ago

Neighbor info is opt in do you have it enabled for your node? It may be that that JSON mode needs to be updated to properly manage the opt in.

GUVWAF commented 3 weeks ago

Are you using the public MQTT server? NeighborInfo packets are filtered out there: https://x.com/TheMeshtastic/status/1811082966283735317

sgtwilko commented 3 weeks ago

Are you using the public MQTT server? NeighborInfo packets are filtered out there: https://x.com/TheMeshtastic/status/1811082966283735317

Ah, yeah that'd be it.

Interesting that it's filtering out the protobuf but not filtering out the JSON version.

I must have spent a good couple of hours looking at the firmware source trying to work out why it wasn't sending 🤦‍♂️

Was that change listed anywhere other than twitter?

Edit: What's interesting is that I'm still seeing some neighbour packets from some nodes on the protobuf topics, probably about 1 in 10 of the neighbour messages are protobuf, and the other 9 are JSON.

GUVWAF commented 3 weeks ago

Was that change listed anywhere other than twitter?

Also on Discord, Reddit and Instagram, but I agree it would be good to document this on the website as well. I've opened https://github.com/meshtastic/meshtastic/issues/1384

What's interesting is that I'm still seeing some neighbour packets from some nodes on the protobuf topics, probably about 1 in 10 of the neighbour messages are protobuf, and the other 9 are JSON.

It only affects packets with the default key, maybe that's the difference.

sgtwilko commented 3 weeks ago

@GUVWAF with my limited understanding it looks like some are encrypted with the default, and some are unencrypted. Some seem to be sent both encrypted with the default and also unencrypted.

Weird.

Any plans on filtering out the json version? (asking as I was working on a PR for one of the map projects which uses the neighbour info)

GUVWAF commented 3 weeks ago

Any plans on filtering out the json version?

Not that I know of, but you can always host your own MQTT server.

meshtastic / firmware