search

monstrenyatko / butler-xbee-gateway

IoT Gateway between XBee ZigBee and TCP network (MQTT, etc...)
MIT License
12 stars 8 forks source link

Digimesh Support #12

Closed TGit-Tech closed 6 years ago

TGit-Tech commented 6 years ago

After installing the ArduinoMQTT and this library as a gateway to an MQTT broker; It seems like the libraries don't support straight Xbee Digimesh. Is that correct? If so, is there any plans on adding support for straight Digimesh instead of XBee Zigbee?

monstrenyatko commented 6 years ago

This application parses the Digi Xbee Zigbee messages received by Zigbee coordinator device and forwards payload to the TCP port. All devices must send external traffic through the coordinator. If Digimesh has different network structure or protocol It will not work. I never used the Digimesh devices to say for sure.

I decided to switch to the WiFi from Zigbee because of the cost. Sorry, I do not have a plan to add support for the Digimesh devices in near future.

TGit-Tech commented 6 years ago

Thanks for the quick response. And a big thanks for supplying and supporting these ready-to-go library packages - they are certainly a huge help versus trying to figure out the building confusion using paho.embedded-c with Arduino/xbee :).

I'm not really sure on the exact protocol differences between Zigbee and Digimesh myself but I'm thinking they're probably a little different as Digimesh doesn't require any "central" router/coordinator node. Then again they're probably pretty darn close being an Xbee firmware update can change between Zigbee and Digimesh protocols.

I'll see what else I can find that might support Digimesh and if all else fails will probably fork from paho.embedded-c and mimic your repository to some extent with the desired modifications. Let me know if you'd have any interest in a pull-request. If its possible to just add a little differences here and there with a pre-compiler directive to support Digimesh and Zigbee on this same client firmware.

As for myself; I'm entirely sold on Digimesh over Zigbee due to its De-centralized peer-2-peer and true ad-hock mesh network features. I'm actually not even sure if the Xbee 'Zigbee' radio firmware allows the radio to operate at full 24dbm transmission power for a 6-mile range which is something I'll have to have.

Anyways; I'm aiming for a smart-sensor/node peer-2-peer automation network w/optional IoT attachment. Due to the Zigbee protocols 'coordinator' requirement - it wouldn't be of any interest either way. I never quite understood the reason behind RF link routing (central router) since its my understanding that the only "routing" going on is the blocking of RF signals at the nodes only for the router to transmit the same data back out yet again on the same RF channel. I could be way off though in my thinking.

Anyways - sorry for the rambling. Just wanted to share those thoughts :) Thanks again.

monstrenyatko commented 6 years ago

paho.embedded-c has a lot of problem with implementation and sometimes with support. That is why I made my fork (ArduinoMqtt) to make it more friendly for the low-power devices and in parallel fixed a few issues.

In any case, the paho.embedded-c or ArduinoMqtt doesn't handle TCP network communication. Both just work with RAW MQTT messages. The TCP communication you should handle yourself.

If Digimesh is able to manage the TCP/IP stack You do not need this xbee-gateway.

TGit-Tech commented 6 years ago

I'm certainly glad you did - I did actually attempt to implement the paho.embedded-c libraries for about 20-minutes before I ran into yours and figured, "This is great!" - Sketch -> Add Library -> ArduinoMQTT done! :)

I haven't quite pegged if it's the Arduino sending or Gateway receiving or both that doesn't like Digimesh. Putting both Gateway and Arduino XBee in API-2 mode or both in AT-mode will entirely stop communications on the Gateway. But; running just the Arduino in AT mode and the Gateway in API-2 mode will cause the gateway to stack up error messages as shown below. Actually, once the message did get through to the MQTT broker but I still haven't determined how that 1-message got through short of dumb-luck with a bad mix of packet bytes.

XBee Digimesh from what I can tell is very similar to Zigbee - No TCP/IP stack just Digi's API packet format. I'd imagine there must be some packet decryption with your Zigbee implementation? Isn't there? I haven't looked into Zigbee hardly at all yet.

> >>>> 2018-01-02T01:51:34.566546 - [WARN ] : XBeeNetFromBuffer - push(), unexpected start of next frame
> >>>> 2018-01-02T01:51:34.567558 - [ERROR] : XBeeNet - onFrame(), error: XBeeFrame::XBeeFrame -> XBeeFrame::decode -> XBeeFrameApiId::XBeeFrameApiId -> XBeeFrameApiId::check -> Not implemented
> >>>> 2018-01-02T01:51:44.577196 - [WARN ] : XBeeNetFromBuffer - push(), unexpected start of next frame
> >>>> 2018-01-02T01:51:44.578201 - [ERROR] : XBeeNet - onFrame(), error: XBeeFrame::XBeeFrame -> XBeeFrame::decode -> XBeeFrameApiId::XBeeFrameApiId -> XBeeFrameApiId::check -> Not implemented
> >>>> 2018-01-02T01:51:44.605420 - [ERROR] : XBeeNet - onFrame(), error: XBeeFrame::XBeeFrame -> XBeeFrame::decode -> XBeeFrameApiId::XBeeFrameApiId -> XBeeFrameApiId::check -> Not implemented
> >>>> 2018-01-02T01:51:54.595017 - [WARN ] : XBeeNetFromBuffer - push(), bad Escape

Anyways; from the error messages - I'm gathering this function 'XBeeFrame::decode' may need a little changes since I'm thinking Digimesh packets may be a little different. Also; here's the packet I'm getting from the Arduino Xbee - using the Arduino MQTT example sketch 'PubSub.ino' to send "Hello"; In case you or anyone else reading gets interested in the changes before I get around to them :)

10 13 00 04 4D 51 54 54 04 02 00 0F 00 07 54 45 53 54 2D 49 44

And the log from the Arduino is;

Connecting
MQTT - Connect, clean-session: 1, ts: 39926757
MQTT - Wait for message, type: 2, tm: 9969 ms
MQTT - Connect ack is not received, rc: -6, ts: 39936766
Connection error: -6
monstrenyatko commented 6 years ago

Current implementation supports only two Digi's API packates:

Probably the Digimesh uses something else and in worst case scenario we just need to add the encoding/decoding of the new packet types to extract MQTT data or to send it back. In a bad case, the protocol might be completely different. Need to find some Digi's documentation and check.

monstrenyatko commented 6 years ago

Could you please find the Digi's documentation related to the protocol used in Digimesh?

once the message did get through to the MQTT broker but I still haven't determined how that 1-message got through short of dumb-luck with a bad mix of packet bytes'

Do you mean you got the ArduinoMqtt library connected to the broker once?

Try the next sequence with ArduinoMqtt example sketch PubSub.ino:

TGit-Tech commented 6 years ago

Links to all documents on the S3B 900HP modules I have are at:

https://www.digi.com/support/productdetail?pid=5576&type=documentation Packet information @ https://www.digi.com/resources/documentation/digidocs/90002173/Default.htm#reference/r_frame_0x10.htm%3FTocPath%3DOperate%2520in%2520API%2520mode%7CFrame%2520descriptions%7C_____3

Yes, the once-only time was indeed the full path from Arduino -> Xbee -> Xbee -> Gateway -> MQTT broker. I have mqtt-spy connected to the broker and it showed one message of "Hello" on it. but only that one message received has ever shown up with all the playing around.

Doing the sequence you suggested yield the following errors on the gateway

^C>>>> 2018-01-02T03:16:10.154972 - [INFO ] : Application - FINISHING
>>>> 2018-01-02T03:16:10.155332 - [INFO ] : Application - STOP
>>>> 2018-01-02T03:16:10.185034 - [INFO ] : Application - DESTROY
>>>> 2018-01-02T03:16:10.185612 - [INFO ] : Main - EXIT
pi@raspberrypi ~/butler-xbee-gateway $ butler-xbee-gateway --config ~/butler-xbee-gateway/config.json 
>>>> 2018-01-02T03:16:19.684039 - [INFO ] : Main - START
>>>> 2018-01-02T03:16:19.685016 - [INFO ] : Main - Version: 0.3.1
>>>> 2018-01-02T03:16:19.685728 - [INFO ] : Configuration - Loading from file, path: /home/pi/butler-xbee-gateway/config.json
>>>> 2018-01-02T03:16:19.687024 - [INFO ] : Configuration - Configuration:
>>>> 2018-01-02T03:16:19.687153 - [INFO ] : Configuration - logger.level = INFO
>>>> 2018-01-02T03:16:19.687255 - [INFO ] : Configuration - logger.file = /var/log/butler-xbee-gateway-my.log
>>>> 2018-01-02T03:16:19.687351 - [INFO ] : Configuration - serial.name = /dev/ttyUSB0
>>>> 2018-01-02T03:16:19.687446 - [INFO ] : Configuration - serial.boud = 9600
>>>> 2018-01-02T03:16:19.687540 - [INFO ] : Configuration - tcp.address = localhost
>>>> 2018-01-02T03:16:19.687632 - [INFO ] : Configuration - tcp.port    = 1883
>>>> 2018-01-02T03:16:19.689495 - [INFO ] : Application - INITIALIZATION
>>>> 2018-01-02T03:16:19.690205 - [INFO ] : Application - START
>>>> 2018-01-02T03:16:19.691704 - [INFO ] : SerialPort - Opening port: /dev/ttyUSB0 at 9600
>>>> 2018-01-02T03:16:19.692694 - [INFO ] : Application - PROCESSING
>>>> 2018-01-02T03:16:19.695391 - [INFO ] : SerialPort - Port is opened
>>>> 2018-01-02T03:16:46.496706 - [WARN ] : XBeeNetFromBuffer - push(), unexpected start of next frame
>>>> 2018-01-02T03:16:46.498332 - [ERROR] : XBeeNet - onFrame(), error: XBeeFrame::XBeeFrame -> XBeeFrame::decode -> XBeeFrameData::XBeeFrameData -> XBeeFrameData::decode -> Buffer is too short
>>>> 2018-01-02T03:16:56.498035 - [WARN ] : XBeeNetFromBuffer - push(), unexpected start of next frame
>>>> 2018-01-02T03:16:56.499064 - [ERROR] : XBeeNet - onFrame(), error: XBeeFrame::XBeeFrame -> XBeeFrame::decode -> XBeeFrameApiId::XBeeFrameApiId -> XBeeFrameApiId::check -> Not implemented
>>>> 2018-01-02T03:16:56.535033 - [WARN ] : XBeeNetFromBuffer - push(), frame-length is huge
>>>> 2018-01-02T03:17:06.508702 - [WARN ] : XBeeNetFromBuffer - push(), unexpected start of next frame
>>>> 2018-01-02T03:17:06.509787 - [ERROR] : XBeeNet - onFrame(), error: XBeeFrame::XBeeFrame -> XBeeFrame::decode -> XBeeFrameApiId::XBeeFrameApiId -> XBeeFrameApiId::check -> Not implemented
>>>> 2018-01-02T03:17:06.528631 - [WARN ] : XBeeNetFromBuffer - push(), frame-length is huge
monstrenyatko commented 6 years ago

Just pushed some changes that I made more than a year ago. This is related to MQTT optimization. It restarts the TCP connection if MQTT-CONNECT message suddenly received. Please try to get the latest code from GIT and recompile the gateway application. This may help in case your Arduino sends some garbage before MQTT-CONNECT message.

Is it possible to disable the low-power behavior on the Digimesh end-point device connected to Arduino? I remember the same problem with Digi Zigbee when low-power mode wrongly configured. Better to temporary disable it.

Please verify that the Serial speed is configured correctly on Arduino and Digi device connected to Arduino. The value must be equal.

TGit-Tech commented 6 years ago

I just bumped the Gateway-Baud rate to 57600 in the config file and the connected XBee - and for some reason when I first plugged it in. It worked for 9-messages and started to error out again.

I left the Arduino & connected XBee at their defaults

define SW_UART_SPEED 9600L

And XBee baud 9600

Arduino Log

MQTT - Connect, clean-session: 1, ts: 2021082
MQTT - Wait for message, type: 2, tm: 9970 ms
MQTT - Connect ack is not received, rc: -6, ts: 2031082
Connection error: -6
Connecting
MQTT - Connect, clean-session: 1, ts: 2031086
MQTT - Wait for message, type: 2, tm: 9969 ms
MQTT - Process message, type: 2
MQTT - Connect ack received
MQTT - Connect ack, code: 0
MQTT - Keepalive interval: 12 sec
MQTT - Session is not present => reset subscription
MQTT - Subscribe, to: test/TEST-ID/sub, qos: 0
MQTT - Wait for message, type: 9, tm: 9967 ms
MQTT - Process message, type: 9
MQTT - Subscribe ack received
MQTT - Publish, to: test/TEST-ID/pub, size: 6
MQTT - Yield for 30000 ms
MQTT - Keepalive, ts: 2046722
MQTT - Process message, type: 13
MQTT - Keepalive ack received, ts: 2046799
MQTT - Keepalive, ts: 2058727
MQTT - Process message, type: 13
MQTT - Keepalive ack received, ts: 2058806
MQTT - Publish, to: test/TEST-ID/pub, size: 6
MQTT - Yield for 30000 ms
MQTT - Keepalive, ts: 2074797
MQTT - Process message, type: 13
MQTT - Keepalive ack received, ts: 2074874
MQTT - Keepalive, ts: 2086802
MQTT - Process message, type: 13
MQTT - Keepalive ack received, ts: 2086881
MQTT - Publish, to: test/TEST-ID/pub, size: 6
MQTT - Yield for 30000 ms
MQTT - Keepalive, ts: 2104830
MQTT - Process message, type: 13
MQTT - Keepalive ack received, ts: 2104910
MQTT - Keepalive, ts: 2116839
MQTT - Process message, type: 13
MQTT - Keepalive ack received, ts: 2116917
MQTT - Publish, to: test/TEST-ID/pub, size: 6
MQTT - Yield for 30000 ms
MQTT - Keepalive, ts: 2134867
MQTT - Process message, type: 13
MQTT - Keepalive ack received, ts: 2134946
MQTT - Keepalive, ts: 2146874
MQTT - Process message, type: 13
MQTT - Keepalive ack received, ts: 2146953
MQTT - Publish, to: test/TEST-ID/pub, size: 6
MQTT - Yield for 30000 ms
MQTT - Keepalive, ts: 2164902
MQTT - Process message, type: 13
MQTT - Keepalive ack received, ts: 2164980
MQTT - Keepalive, ts: 2176908
MQTT - Process message, type: 13
MQTT - Keepalive ack received, ts: 2176988
MQTT - Publish, to: test/TEST-ID/pub, size: 6
MQTT - Yield for 30000 ms
MQTT - Keepalive, ts: 2194929
MQTT - Process message, type: 13
MQTT - Keepalive ack received, ts: 2195007
MQTT - Keepalive, ts: 2206936
MQTT - Process message, type: 13
MQTT - Keepalive ack received, ts: 2207013
MQTT - Publish, to: test/TEST-ID/pub, size: 6
MQTT - Yield for 30000 ms
MQTT - Keepalive, ts: 2224971
MQTT - Process message, type: 13
MQTT - Keepalive ack received, ts: 2225054
MQTT - Keepalive, ts: 2236973
MQTT - Process message, type: 13
MQTT - Keepalive ack received, ts: 2237051
MQTT - Publish, to: test/TEST-ID/pub, size: 6
MQTT - Yield for 30000 ms
MQTT - Keepalive, ts: 2255001
MQTT - Keepalive ack failure, ts: 2265003
Connecting
MQTT - Connect, clean-session: 1, ts: 2265003
MQTT - Wait for message, type: 2, tm: 9972 ms
MQTT - Connect ack is not received, rc: -6, ts: 2275009
Connection error: -6

Gateway Log

pi@raspberrypi ~/butler-xbee-gateway $ butler-xbee-gateway --config ~/butler-xbee-gateway/config.json 
>>>> 2018-01-02T03:57:52.478501 - [INFO ] : Main - START
>>>> 2018-01-02T03:57:52.479498 - [INFO ] : Main - Version: 0.3.1
>>>> 2018-01-02T03:57:52.480203 - [INFO ] : Configuration - Loading from file, path: /home/pi/butler-xbee-gateway/config.json
>>>> 2018-01-02T03:57:52.481485 - [INFO ] : Configuration - Configuration:
>>>> 2018-01-02T03:57:52.481621 - [INFO ] : Configuration - logger.level = INFO
>>>> 2018-01-02T03:57:52.481719 - [INFO ] : Configuration - logger.file = 
>>>> 2018-01-02T03:57:52.481813 - [INFO ] : Configuration - serial.name = /dev/ttyUSB0
>>>> 2018-01-02T03:57:52.481909 - [INFO ] : Configuration - serial.boud = 57600
>>>> 2018-01-02T03:57:52.482003 - [INFO ] : Configuration - tcp.address = localhost
>>>> 2018-01-02T03:57:52.482100 - [INFO ] : Configuration - tcp.port    = 1883
>>>> 2018-01-02T03:57:52.483843 - [INFO ] : Application - INITIALIZATION
>>>> 2018-01-02T03:57:52.484551 - [INFO ] : Application - START
>>>> 2018-01-02T03:57:52.486053 - [INFO ] : SerialPort - Opening port: /dev/ttyUSB0 at 57600
>>>> 2018-01-02T03:57:52.487062 - [INFO ] : Application - PROCESSING
>>>> 2018-01-02T03:57:52.489572 - [INFO ] : SerialPort - Port is opened
>>>> 2018-01-02T03:58:08.134415 - [WARN ] : XBeeNetFromBuffer - push(), unexpected start of next frame
>>>> 2018-01-02T03:58:08.135568 - [ERROR] : XBeeNet - onFrame(), error: XBeeFrame::XBeeFrame -> XBeeFrame::decode -> XBeeFrameLength::XBeeFrameLength -> XBeeFrameLength::decode -> Buffer is too short

I didn't have much luck compiling from source on the raspberry pi last time - ended up just using your .deb file; any chance you can build another one?

monstrenyatko commented 6 years ago

In accordance with the document (https://www.digi.com/resources/documentation/digidocs/pdfs/90001496.pdf)

The Digimesh should work similar to Digi Zigbee if you configure everything correctly. You need to be sure:

monstrenyatko commented 6 years ago

Good progress. Now you need to figure out why it stopped working after 9 messages. This might be because of wrongly configured low-power mode on the Digi devices.

TGit-Tech commented 6 years ago

According to the docs - I'm thinking there shouldn't be any sleeping going on. I don't see any other 'power' related items in the function set.

Set SM to 0 to enter Normal mode. Normal mode is the default sleep mode. If a device is in this mode, it does not sleep and is always awake.

monstrenyatko commented 6 years ago

I have uploaded the deb file to the version 0.3.2.

TGit-Tech commented 6 years ago

Omg.. This is embarrassing - I went to install your new .deb file and after shutting down the gateway in the terminal I thought I better make sure everything was off. When I noticed two instances of the gateway running.

Well, wouldn't you know it - Works great with only 1-running gateway..

I sure do appreciate the time you took to help out. Even though it ended up being a silly mistake on my end; I feel at least I ( maybe we ) found out it does work with digimesh as long as Arduino is AT-mode and Gateway is API-2. So sorry for the waste of time though.

monstrenyatko commented 6 years ago

Good news indeed :-) In any case, I pushed MQTT gateway changes stuck on my computer for a long time.