Error running Dronekit-Python with Multiprocessing

[MDB22]

I'm trying to have a multiprocessing Process as a container for running a Vehicle instance, and I get the following error:

Traceback (most recent call last): File "", line 1, in File "C:\Python27\lib\multiprocessing\forking.py", line 381, in main self = load(from_parent) File "C:\Python27\lib\pickle.py", line 1378, in load return Unpickler(file).load() File "C:\Python27\lib\pickle.py", line 858, in load dispatchkey File "C:\Python27\lib\pickle.py", line 1206, in load_setitems dict[stack[i]] = stack[i + 1] File "C:\Python27\lib\site-packages\dronekitinit.py", line 763, in setitem if self._readonly: AttributeError: 'Channels' object has no attribute '_readonly' Process Process-1: Traceback (most recent call last): File "C:\Python27\lib\multiprocessing\process.py", line 258, in _bootstrap self.run() File "C:\Python27\lib\multiprocessing\process.py", line 114, in run self._target(_self._args, *_self._kwargs) File "Z:\Repositories\MedExpress\scripts\autonomy.py", line 10, in main controller.start() File "C:\Python27\lib\multiprocessing\process.py", line 130, in start self._popen = Popen(self) File "C:\Python27\lib\multiprocessing\forking.py", line 280, in init to_child.close() IOError: [Errno 22] Invalid argument

I've attached a MWE to show the error. I'm testing on a Windows machine, so I'm not sure if it's the platform I'm using, or if Dronekit-Python is incompatible with multiprocessing, or something else entirely...

If I wrap the offending code in init.py in a try-except block, I get a whole bunch of other errors about failure to pickle. If I'm doing something ridiculous please let me know :)

scripts.zip

[hamishwillee]

@MDB22 We don't test in a multiprocess environment so we don't provide any guarantees that it will work. I suspect not.

Can you provide a test case (as a gist link) that would make this easy to test/reproduce?

[MDB22]

@hamishwillee sorry about that, I've never used Gist before, here's the link: https://gist.github.com/MDB22/582a73321cc616c4aed1

[hamishwillee]

THank you very much! @peterbarker - when you have time, we should have a look at this ...

[MDB22]

@hamishwillee @peterbarker have you had a chance to look into this?

I've been away from the code for a while and just ran into the problem again :(

I'm trying to create a Process for reading sensor data/responding to attribute callbacks, but it's still the same problem as in the original post.

Does it make sense to do this? If the callbacks happen asynchronously then I don't need to explicitly call functions to retrieve sensor information, but presumably when the callbacks are executed they will block the main process.

[peterbarker]

@MDB22 So pickling that object is going to need some work. Definitely a problem.

If you can get away with using threads, that might let you move forward until this is fixed.

[peterbarker]

Additionally, I'm unable to reproduce under Linux:

pbarker@bluebottle:~/rc/dronekit-python(source-system-filtering)$ python test.pyStarting copter simulator (SITL)
SITL already Downloaded and Extracted.
Ready to boot.
Connecting to tcp:127.0.0.1:5760
>>> Calibrating barometer
>>> APM:Copter V3.3 (d6053245)
>>> Frame: QUAD
>>> Initialising APM...
>>> barometer calibration complete
>>> GROUND START
mav recv error: invalid MAVLink prefix '73'
mav recv error: invalid MAVLink prefix '42'
mav recv error: invalid MAVLink prefix '13'
mav recv error: invalid MAVLink prefix '32'
Mission complete.
pbarker@bluebottle:~/rc/dronekit-python(source-system-filtering)$ 

That's going to make it difficult for me to attack this problem. Generally I was going to modify the Channel class's "setitem to check whether the object has a _readonly attribute before looking at it:

    def __setitem__(self, key, value):
        if hasattr(self, '_readonly') and self._readonly:
            raise TypeError('__setitem__ is not supported on Channels object')
        return dict.__setitem__(self, str(key), value)

[MDB22]

@peterbarker thanks for getting back to me so quickly! :)

I can try threading and see if that works, and use processes if that's too slow.

That's annoying that it's only a Windows problem... I could try and debug and submit a pull request but I don't really understand the problem.

[peterbarker]

@MDB22 This link describes the problem present in dronekit's "Channel" class: http://stackoverflow.com/questions/21144845/how-can-i-unpickle-a-subclass-of-dict-that-validates-with-setitem-in-pytho

Essentially, init isn't called when unpickling - and that's what makes _readonly a valid attribute on the instance. I'm assuming that unpickling the object will eventually would _readonly back into the instance, so only setitem should need fixing :-)

[MDB22]

@peterbarker thank you, I will take a look and see what I can do :)