Exception in message handler for heartbeat

[dgorissen]

I found 3 threads about this but no authorative answer/solution:

https://groups.google.com/forum/#!topic/drones-discuss/tQQjc0_5Eng https://discuss.dronekit.io/t/issue-with-vehicle-state-py-and-rover/307 https://discuss.dronekit.io/t/missionplanner-and-dronekit-floods-console-with-errors/287

Im running dronekit 2.4.0 on an odroid UX4, connected with an FTDI cable to the Pixhawk Telem 2 port (dronekit is reading straight from /dev/ttyUSB0). Pixhawk is running APM copter 3.3.3 and the 3DR Radio is on Telem 1. Serial 4/5 is also in use (Taranis telemetry).

I notice the following messages being spammed to std err/out on the odroid. Otherwise it does seem to be working properly (both reading the data via dronekit and apm planner 2.0).

...
argument of type 'NoneType' is not iterable
Exception in message handler for HEARTBEAT
argument of type 'NoneType' is not iterable
Exception in message handler for HEARTBEAT
argument of type 'NoneType' is not iterable
Exception in message handler for HEARTBEAT
...

It seems the issue is something to do with the telem1 & telem2 streams interfering with each other somehow? Some guidance as to what the correct usage pattern is would be appreciated. Or confirm if this is a bug.

[PureRockets]

I am having the same error, however it is critical for me. I'm using an ErleBrain2 running APM. It always returns Not Armable even when it's armed. When I try to run vehicle_state, I get several of these heartbeat exceptions, then it crashes (same as reported in 307)

[peterbarker]

Attempting to iterate over NoneType is clearly a bug. We'll need more debug to try to track this.

Could you add the following lines:

            import traceback
            traceback.print_exc()

into mavlink.py, just under this line: errprinter('>>> ' + str(e))

please?

[dgorissen]

It was actually in a slightly different place that the edit was needed but none the less:

>>> Exception in message handler for HEARTBEAT
>>> argument of type 'NoneType' is not iterable
Traceback (most recent call last):
  File "/home/dgorissen/.virtualenvs/orangspy/local/lib/python2.7/site-packages/dronekit/mavlink.py", line 150, in mavlink_thread_in
    fn(self, msg)
  File "/home/dgorissen/.virtualenvs/orangspy/local/lib/python2.7/site-packages/dronekit/__init__.py", line 1011, in listener
    self.notify_message_listeners(msg.get_type(), msg)
  File "/home/dgorissen/.virtualenvs/orangspy/local/lib/python2.7/site-packages/dronekit/__init__.py", line 1467, in notify_message_listeners
    fn(self, name, msg)
  File "/home/dgorissen/.virtualenvs/orangspy/local/lib/python2.7/site-packages/dronekit/__init__.py", line 1169, in listener
    if self._is_mode_available(m.custom_mode) == False:
  File "/home/dgorissen/.virtualenvs/orangspy/local/lib/python2.7/site-packages/dronekit/__init__.py", line 1501, in _is_mode_available
    return mode_code in self._mode_mapping_bynumber
TypeError: argument of type 'NoneType' is not iterable

Edit: there does seem to be some cross talk going on between what Im reading over the FTDI cable from telem2 and what Apmplanner is receiving over the radios. The latter will drop and regain connection sporadically and the flight planner screen shows a blinking red GCS entry which I have no idea where it comes from:

[fnoop]

I've added a small PR to deal with the specific iterable problem, but it uncovers the underlying problem which I think is the GCS (or whatever) is using vehicle mode 0 which isn't in the mode mappings. This can be dealt with in several places:

>>> Exception in message handler for HEARTBEAT
>>> mode 0 not available on mavlink definition

This is possibly because from fw 3.3 the mavlink messages from different sources are routed or are sent if a non-specific source request is made so the dronekit script receives all mavlink messages on the line. The dronekit scripts should perhaps instead target a specific sysid/componentid to receive only what it wants?

[fnoop]

So the exception in the message handler should be dealt with better, but the underlying issue is almost certainly related to this: https://github.com/dronekit/dronekit-python/issues/516 The connection the dronekit script makes should be source filtered so it doesn't receive messages from the entire mavlink network.

[PureRockets]

I get the exception with a single vehicle, and just dronekit access (no Mission Planner). I'm dead in the water with dronekit, I can't arm my vehicle.

[peterbarker]

@fnoop source system filtering: https://github.com/dronekit/dronekit-python/pull/585

[dgorissen]

@fnoop, @peterbarker thanks. Looking at #585 it hasnt seen updates for a couple months (other than the CI failing). Im currently staying on the safe side and plugging in only a single telemetry consumer (my companion computer). Anything else and I get crosstalk. This means I cannot use my radios, taranis telemetry, etc. which is not great as Im flying blind.

From the issue it seems like a hairy one and will take some time to resolve (?). Any suggested workarounds or fixes?

[mrpollo]

@dgorissen are you running APM or PX4?

[dgorissen]

APM On 13 Apr 2016 02:58, "Ramon Roche" notifications@github.com wrote:

@dgorissen https://github.com/dgorissen are you running APM or PX4?

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/dronekit/dronekit-python/issues/610#issuecomment-209078790

[mrpollo]

@dgorissen this was fixed in #611 release to follow shortly

[dgorissen]

Awesome. Thanks @mrpollo

[hannahdn]

I am still getting this error (after #611 update):

Exception in message handler for HEARTBEAT mode 0 not available on mavlink definition

@fnoop You mentioned a fix for that, what do you use?

[doomedramen]

with everything up-to-date i still get:

Exception in message handler for HEARTBEAT mode 0 not available on mavlink definition Exception in message handler for HEARTBEAT mode 0 not available on mavlink definition Exception in message handler for HEARTBEAT mode 0 not available on mavlink definition Exception in message handler for HEARTBEAT mode 0 not available on mavlink definition Exception in message handler for HEARTBEAT mode 0 not available on mavlink definition Exception in message handler for HEARTBEAT mode 0 not available on mavlink definition Exception in message handler for HEARTBEAT mode 0 not available on mavlink definition Exception in message handler for HEARTBEAT mode 0 not available on mavlink definition Exception in message handler for HEARTBEAT mode 0 not available on mavlink definition Exception in message handler for HEARTBEAT mode 0 not available on mavlink definition Exception in message handler for HEARTBEAT mode 0 not available on mavlink definition c>>> Exception in message handler for HEARTBEAT mode 0 not available on mavlink definition Exception in message handler for HEARTBEAT mode 0 not available on mavlink definition

[squilter]

@wookoouk APM or px4?

[fnoop]

@hannahdn @wookoouk - the underlying issue is still there, dronekit doesn't handle multiple sources on a link properly. I'm wondering if dronekit has been somewhat given up on for things like this - I notice that recent 3dr swarm work has been done with ROS.

[peterbarker]

@fnoop @hannahdn @wookoouk fnoop is correct that dronekit-python doesn't handle multiple vehicles on one link properly at the moment. The problem is primarily in pymavlink, 'though, not dronekit-python.

I have had a great deal of success with my avoidance work and dronekit-python, however. Very reliable, multi-vehicle testing to the point I was happy to call the suite a regression suite.

https://github.com/peterbarker/dronekit-python/tree/source-system-filtering/examples/avoidance

(those instructions are a little out-of-date; the avoidance work in ardupilot is upstream now)

[expntly]

@peterbarker @LorenzMeier

Please kindly update on this issue. Here's all the info I have collected after a few hours of debugging, hope this helps and let me know if/how I can help.

I'm running a Pixhawk (HKPilot32 really) with the PX4 flight stack. It doesn't look like it's playing well with mavlink proto definitions. This is preventing me from using Dronekit at all. While debugging I should mention I tried ArduPlane 3.6.0 which works just fine. Using QGroundControl as a GCS.

$ mavproxy.py --master=/dev/serial0 --baudrate 57600 --out udp:127.0.0.1:14550
Connect /dev/serial0 source_system=255
Log Directory: 
Telemetry log: mav.tlog
MAV> Waiting for heartbeat from /dev/serial0
online system 1
ARMED
Mode(100925440)> Mode Mode(100925440)
fence breach
APM: data link #1 regained

Mode(100925440)> 
Mode(100925440)> mode stabilize
Mode(100925440)> Got MAVLink msg: COMMAND_ACK {command : 176, result : 0}
APM: [cmd] DISARMED by set mode command
APM: failsafe mode off
DISARMED
Mode(65536)> Mode Mode(65536)

Mode(65536)> 
Mode(65536)> APM: [blackbox] stopped (50146 drops)

Trying to connect from another console and run vehicle_state.py pointing at mavproxy:

$ python vehicle_state.py --connect "udp:127.0.0.1:14550"

Connecting to vehicle on: udp:127.0.0.1:14550
>>> Exception in message handler for HEARTBEAT
>>> mode 65536 not available on mavlink definition
>>> Exception in message handler for HEARTBEAT
>>> mode 65536 not available on mavlink definition
>>> Exception in message handler for HEARTBEAT
>>> mode 65536 not available on mavlink definition

Killing mavproxy (and QGroundControl) and running vehicle_state.py directly pointed at the serial port (this is a companion computer setup):

$ python vehicle_state.py --connect "/dev/serial0"

Connecting to vehicle on: /dev/serial0
>>> Link timeout, no heartbeat in last 5 seconds
>>> No heartbeat in 30 seconds, aborting.
Traceback (most recent call last):
  File "vehicle_state.py", line 37, in <module>
    vehicle = connect(connection_string, wait_ready=True, rate=57600)
  File "build/bdist.linux-armv7l/egg/dronekit/__init__.py", line 2827, in connect
  File "build/bdist.linux-armv7l/egg/dronekit/__init__.py", line 2100, in initialize
dronekit.APIException: Timeout in initializing connection.

Version info here:

$ pip list | grep mav
pymavlink (2.0.6)
$ pip list | grep drone
dronekit (2.8.0)
dronekit-sitl (3.1.0)

I tried multiple combinations of pymavlink and dronekit versions, without success.

Also tried all PX4 firmware versions (stable, beta, master as of today Aug 28, 2016), which doesn't help.

[peterbarker]

@expntly You will need to use the version of pymavlink found in the "mavlink" repo on github (https://github.com/mavlink/mavlink). This is typically done with "python setup.py build install --force --user" in the pymavlink directory.

[expntly]

@peterbarker Indeed that did the trick thanks! Any chance a mavlink release can be pushed? Or at least a warning message somewhere. That would have saved hours of frustration :-)

[LorenzMeier]

@mrpollo @peterbarker This has been fixed for PX4 a very long time ago and it would be nice to finally have a release addressing it.

[mrpollo]

Thanks @LorenzMeier I'll cut a new release asap.

[mrpollo]

I released 2.9.0 yesterday, can someone please verify this is fixed on that release?

[kostkar]

Hello all, I tried it with dronekit 2.9.0 and both pymavlink 2.0.3 and the mavlink/pymavlink(https://github.com/mavlink/mavlink) in both cases I see the >>> mode 0 not available on mavlink definition, as well as intermittent connection. Is there any plans for fixing this, I am available to test it, and flight test it as soon as it seems safe :).

[majorpr13]

Has there been any luck on this...I am having the same issue.

[paan2097]

I'm also getting the mode 0 heartbeat error that prints out in my dronekit terminal whenever I connect with Mission Planner.

[expntly]

Last time I had it a few days ago, it was with the following setup:

• RPi connected to Pixhawk via Telem 2
• Mission Planner connected to Pixhawk via 915Mhz telemetry Telem 1

I tried running dronekit from the Pi and got the "mode 0 not available on mavlink definition" exception message. The moment I disconnected MP, the issue went away.

So the bottom line is that there seems to be some weird interaction when multiple mavlink connections are made to the Pixhawk.

On Tue, Sep 20, 2016 at 1:00 PM, paan2097 notifications@github.com wrote:

I'm also getting the mode 0 heartbeat error that prints out in my dronekit terminal whenever I connect with Mission Planner.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dronekit/dronekit-python/issues/610#issuecomment-248416732, or mute the thread https://github.com/notifications/unsubscribe-auth/AGjVtGchGa_SOjSI6jGoXMuk2tEgcKW7ks5qsDtLgaJpZM4H-mq6 .

[Corusco]

I'm seeing the "mode 0 not available" error as well. I'm on PX4 flight stack, connecting my mac over FTDI cable to pixhawk Telem2.

I was getting a mixture of mode 0 and mode 65536 until I updated my pymavlink. I was able to get data down to populate some properties, but heartbeat was never consistent. Now it's the heartbeat error all the time.

Even if this isn't completely fixed in dronekit, I'm all about workarounds right now. I have some code I desperately need to ship.

[majorpr13]

@Corusco This was actually fixed in another github issue. First...run this command repeatedly until pip reports pymavlink is not found or no longer installed. pip uninstall pymavlink

Then follow these instructions 1) git clone https://github.com/Ardupilot/pymavlink 2) cd into the pymavlink directory 3) python setup.py build install --force --user

this should install the latest. There has been an issue when following multiple sets of instructions for the installs and how pymavlink installed from certain sources is updated with the latest.

[majorpr13]

@Corusco You can read more about it over here https://github.com/dronekit/dronekit-python/issues/632#event-797047749

[expntly]

Just curious, what's the difference between Ardupilot/pymavlink and mavlink/mavlink? Which one is preferred?

[paan2097]

@majorpr13 When I run those commands, after the third command it returns No XML message definitions found. This is suppose to fix the error when having a dronekit and mission planner both connected to a Pixhawk/SITL right??

[majorpr13]

@paan2097 Did you continue to execute pip uninstall pymavlink until it reported there were none left? Did you cd into the correct directory that was cloned from git? Which version of python are you running? It may be easiest if you direct message me a screenshot of your console on all of the steps.

[salonijain12]

We are trying to connect rpi with pixhawk serially on Telem2 using the link: http://plane.ardupilot.com/dev/docs/raspberry-pi-via-mavlink.html Able to connect MAVProxy on Pixhawk from Rpi but when using it with dronekit-python (running the python script), getting the following error : -Exception in message handler for HEARTBEAT -mode 65536 not available on mavlink definition

Following are the versions used: PX4 stack firmware 1.4.4 installed on pixhawk. Dronekit-python==2.9.0, installed from github source : https://github.com/dronekit/dronekit-python.git MAVProxy==1.5.4 (also tried with MAVProxy==1.5.1) pymavlink==2.0.6

Can anyone please suggest if something is being missed? What is the correct combination of dronekit-python, MAVProxy and pymavlink versions for rpi and pixhawk connection.

[lroberts99]

@majorpr13 I am also facing the same issue of Exception in message handler for HEARTBEAT mode 0 not available on mavlink definition

I followed all of your steps but I am facing the same No XML message definitions found as @paan2097 . Any ideas?

[majorpr13]

@paan2097 @salonijain12 @lroberts99 You need to be more clear when asking questions about what you are actually doing...settings firmware etc. Everybodys scenario may be entirely different and without a good background its hard to debug online. That being said, this thread was started via the PX4 running the ardupilot based firmware not the PX4 based firmware. I have not tested anything using the PX4 variant SW base. If I had to guess it cant find the directory containing the XML definitions because you didnt have permission for the install. Try the uninstall process again and then you could take a shortcut and try sudo pip install pymavlink.

[lroberts99]

Sure, sorry about that. I am using running the dronekit commands from an Edison board on a Solo drone. I have these connected via USB. I ended up installing dronekit this way http://stackoverflow.com/questions/35964326/raspberry-pi-proper-connection-to-pixhawk-with-dronekit. I still get the exception, but it only prints out twice and then moves on to the rest of my code. Thanks for your reply.

[Corusco]

@majorpr13 Apologies for not thanking you last week for your help. I no longer get the Mode 65536 error (though Mode 0 still pops up), so I've seen improvement there.

@lroberts99 @paan2097 I experienced the missing XML definitions problem when pymavlink was installed in a directory outside of the Mavlink/pymavlink directory, where it should sit. I would suggest reinstalling mavlink by cloning from https://github.com/mavlink/mavlink, and updating the submodules or navigating into the pymavlink directory (inside the mavlink directory) and cloning from https://github.com/ArduPilot/pymavlink. Running the setup should complete successfully here.

[majorpr13]

@Corusco when you connect are you connecting with mavproxy or dronekit?

[Corusco]

@majorpr13 That was with the dronekit vehicle_status example. Mavproxy wasn't returning anything at that point. I've since moved to using a Jetson TK1 as my companion computer. Now, the example doesn't return anything, but MavProxy is returning a lot of garbled data along with some of the message headers interspersed. I'm wondering if my problem there was a configuration problem or a Mac problem. I did fully reinstall mavlink, dronekit, and pymavlink on that setup. Working on debugging the TK1 now. So, my current status is not entirely related to this issue, but I could step back on to my mac to push through if we want to address a bug.

[majorpr13]

@Corusco what port are you connected on the TK1....are you using the correct voltage logic levels for comms between the devices?

[Corusco]

@majorpr13 I did start with the UART2 on the GPIO pins over a line level shifter. I had a lot of noise in that configuration, so I'm currently using USB-TTL FTDI cable.

[majorpr13]

@Corusco thats strange. I have my pixhawk running through the GPIO of my odroid and dont have a problem and thats a level shift from 1.8 to 3.3. Do you have the correct TTL level...of course if its the same used with your mac you should get the same results. i think about it let me know if you get it...curious problem

[Corusco]

@majorpr13 Yeah, I also had 1.8 to 3.3 with the TK1, using the TI TXB series shifters. I tried three chips to be sure it wasn't a chip problem, and all three exhibited the same noise, so it's entering somewhere else.

A couple other local guys are also using odroid problem-free, but we're locked in to the tegra chip, so a platform switch isn't an option. But I know several others have also used the TK1 successfully.

In terms of the USB-TTL cable right now, it's definitely a 3.3v.

Thanks again for being willing to help me out with this. I'll let you know if I find anything.

[lidia-toscano]

Hi, I am having the same issue and I've tried following the instructions: Then follow these instructions 1) git clone https://github.com/Ardupilot/pymavlink 2) cd into the pymavlink directory 3) python setup.py build install --force --user

However when I try to do step #3 I get the following error: root@beaglebone:~/pymavlink# python setup.py build install --force --user No XML message definitions found

Any ideas what I need to do to get the last step implemented?

Thanks in advance.

[lidia-toscano]

Now the problem is even worse because when I try to execute my program I get this:

root@beaglebone:~# python basic.py WARNING: Unable to load XML validator libraries. XML validation will not be performed Traceback (most recent call last): File "basic.py", line 6, in from dronekit import connect, VehicleMode, LocationGlobalRelative, LocationGlobal File "/usr/local/lib/python2.7/dist-packages/dronekit/init.py", line 42, in from pymavlink import mavutil, mavwp File "/root/pymavlink/mavutil.py", line 105, in set_dialect(os.environ['MAVLINK_DIALECT']) File "/root/pymavlink/mavutil.py", line 96, in set_dialect mavgen_python_dialect(dialect, wire_protocol) File "/root/pymavlink/generator/mavgen.py", line 197, in mavgen_python_dialect if not mavgen(opts, [xml]): File "/root/pymavlink/generator/mavgen.py", line 82, in mavgen xml.append(mavparse.MAVXML(fname, opts.wire_protocol)) File "/root/pymavlink/generator/mavparse.py", line 278, in init f = open(filename, mode='rb') IOError: [Errno 2] No such file or directory: 'message_definitions/v1.0/ardupilotmega.xml' root@beaglebone:~# ^C

Any ideas?

Thanks

[Corusco]

@lidia-toscano In my experience, the XML definitions pymavlink is looking for are located in the Mavlink library, unless you generate your own and give pymavlink a reference to their location. The easiest and most thorough solution that I have found is to install mavlink by cloning from https://github.com/mavlink/mavlink, and updating the submodules or navigating into the pymavlink directory (inside the mavlink directory) and cloning from https://github.com/ArduPilot/pymavlink. Running the setup should complete successfully here.

[peterbarker]

On Thu, 6 Oct 2016, lidia-toscano wrote:

Hi, I am having the same issue and I've tried following the instructions: Then follow these instructions 1) git clone https://github.com/Ardupilot/pymavlink 2) cd into the pymavlink directory 3) python setup.py build install --force --user

The pymavlink repo doesn't contain the message definitions, but does include the code to generate language binding from the message definitions. You may specify the location of the message definitions using the "MDEF" environment variable.

e.g. pbarker@bluebottle:~/rc/pymavlink(master)$ MDEF=$HOME/rc/ardupilot/modules/mavlink/message_definitions python setup.py build install --user --force

[lidia-toscano]

Thanks for your responses. Corrusco, I followed your directions and I think the clone was installed correctly. At any rate, now I have a directory for mavlink and a subdirectory within mavlink with a pymavlink clone full of files. I do see the setup.py file in there. Peter is right in that the XML files are not where they should be, during the cloning there were notices that they would not work. I then tried to follow the last instruction provided by Peter but it is giving me the following error:

root@beaglebone:~# /rc/pymavlink(master)$MDEF=$HOME/rc/ardupilot/modules/mavlink/message_definitions python setup.py build install --user --force -bash: syntax error near unexpected token `master' root@beaglebone:~#

Any other ideas on how to correct this error?

Thanks again,

Lidia

[majorpr13]

What is this command doing? /rc/pymavlink(master)$MDEF=