Add support for w-iLab.t testbed

[twalcari]

Hi,

Working further on @bozidars27 pull-request, I've modified opentestbed to also support running on imec's w-iLab.t infrastructure.

I did the following:

• Made everything Python3.5 compatible
• created a setup.py to install opentestbed
• automatically detect motes
• created a Contiki-NG based firmware to emit the EUI64-address of each mote
• made the MQTT-server configurable when starting the testbed
• Created a basic CLI to send commands to the MQTT-server

The install scripts for berkeley opentestbed will need to be fixed, but I want to start a discussion on my work before going further.

To install opentestbed on a w-iLab.t:

• do a git clone of this repo
• run pip3 install -e . to install opentestbed with the setup.py (installs dependencies automatically)
• run opentestbed -t wilab -b <BROKER-IP-OR-HOSTNAME> -v

To use the cli:

• do a git clone of this repo
• run pip3 install -e . to install opentestbed with the setup.py (installs dependencies automatically)
• run otb-cli for more info

[malishav]

Regarding the CLI client, I think this is a good idea to have it as an example on how to instrument the testbed infrastructure side. At the moment, we have such client code spread out over different repos (e.g. OpenTestbed web interface, OpenVisualizer, OpenBenchmark web platform, w-ilab.t?,...) so it would be quite beneficial from the point of view of future development to keep it contained here and have other projects invoke the Python script. @changtengfei, what do you think?

[twatteyne]

Some quite note about the scope of the opentestbed. It's the smallest possible functional code that allows one to build a testbed which allows on to (1) reprogram the motes, (2) reset the motes (3) forward serial traffic. It's currently a single 668-line file, and we are very proud of that. It can we started on a rPi without any changes (i.e. not Python 3), perfect.

It adds a small number of simple features for manageability, in particular being able to update the Pi's software automatically.

It has been running for a number of weeks without any problem at Inria-Paris. It is NOT a research project: we just want to it to work and never have to look at it and do real things. It was put together in about 1 months by one intern with supervision from myself and Tengfei.

That's what the scope is, and what the scope will be. We believe that the minimalism of it is key to its success, and are excited to see it being used by half a dozen different groups here. We understand that this might mean some adaptation, which are limited to (1) additional bootloader, (2) additional reference firware to read the EUI64 and (3) additional serial connectivity. These are needed.

That being said, we cannot accept new features if they break the minimalistic nature of the code. On top of the simple addition of a new testbed, this PR adds a CLI interface, makefiles, C source code, restructuring of the code, setup integration and Python 3 support. None of these are needs.

[twalcari]

Hi @twatteyne et al,

I took your comments to heart, and I diff'ed my changes to find the minimal set of necessary changes to get wilab support going. (cfr. commit 53fabbc). I did however go a bit further and moved all testbed-specific code to separate classes. This to do away with the if self.testbed == "opentestbed" ... elif self.testbed == "iotlab" ... elif ... constructs. This was done in 9a862a4 .

Some remarks:

• Python 3 is installed by default on Raspbian, and recommended by the RPi community. So I'm a bit surprised by your remark.

• The otbox.py file also contains code for managing the picture on the screen attached to your RPi's. This code is not strictly necessary to reprogram/reset/serial coms with motes. I moved it to the "OpenTestbed"-class for clarity.

• I added the firmware source code for the EUI64 retrieval firmware because it is handy for other parties who want to use opentestbed with other hardware. I had to write my own because your OpenMote-B firmware bricked our Zolertia's, while I think that recompiling your firmware against our boards would have been sufficient. I'll move the firmware to a separate repo if needed, but I would recommend referencing it somewhere in the docs to help future parties.

[twalcari]

Hi @malishav ,

About your concern about crashing when an unknown message arrives: the _execute_command_safely function already contains the following:

            if cmd_handler is None:
                self.logger.debug("Ignoring unknown command '%s'" % cmd)
                return

This means that the program doesn't crash, but will log the unknown command.

[malishav]

Hi @twalcari !

Thanks, I overlooked that change! I would just ask for an addition of a comment on the topic subscription line to make this clear. Otherwise, it looks good to me.

[twalcari]

Done.

[bozidars27]

Hi @twalcari and @malishav,

Just want to let you know that I've just tested this PR on IoT-LAB, and it looks like all works fine.

[changtengfei]

@twalcari , I tested the PR on OpenTestbed and got the following errors, could you fix them and ping me to test it? Thanks!

Exception in thread image_thread:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "otbox.py", line 91, in _display_image
    pygame.init()
NameError: global name 'pygame' is not defined

Exception in thread mqtt_loop_thread:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/paho/mqtt/client.py", line 1481, in loop_forever
    rc = self.loop(timeout, max_packets)
  File "/usr/local/lib/python2.7/dist-packages/paho/mqtt/client.py", line 1003, in loop
    rc = self.loop_read(max_packets)
  File "/usr/local/lib/python2.7/dist-packages/paho/mqtt/client.py", line 1284, in loop_read
    rc = self._packet_read()
  File "/usr/local/lib/python2.7/dist-packages/paho/mqtt/client.py", line 1849, in _packet_read
    rc = self._packet_handle()
  File "/usr/local/lib/python2.7/dist-packages/paho/mqtt/client.py", line 2311, in _packet_handle
    return self._handle_connack()
  File "/usr/local/lib/python2.7/dist-packages/paho/mqtt/client.py", line 2372, in _handle_connack
    self.on_connect(self, self._userdata, flags_dict, result)
  File "otbox.py", line 305, in _on_mqtt_connect
    self.tb.on_mqtt_connect()
  File "otbox.py", line 116, in on_mqtt_connect
    self._otbox._excecute_commands('{0}/{1}'.format(self.mqtttopic_box_cmd_prefix, 'picturetoscreen'),
AttributeError: 'OpenTestbed' object has no attribute 'mqtttopic_box_cmd_prefix'

[wlcrs]

Hi Tengfei,

I fixed the two issues. Hopefully everything works as expected now.

wrt to pygame: I cannot find any reference in your install scripts to how you install it. So please check if it is installed if the error keeps popping up..

[changtengfei]

@twalcari it would be more efficient if you allow me to commit to your repository directly?

[changtengfei]

@twalcari I have added two commits which resolve the error I had in opentestbed side. For the fix UniCodeDecodeError commit, it resolves this error:

ERROR:OtBox:Exception while executing discovermotes
Traceback (most recent call last):
  File "/home/opentestbed/latest/otbox.py", line 376, in _excecute_command_safely
    returnVal['returnVal'] =  cmd_handler(deviceType, deviceId, payload)
  File "/home/opentestbed/latest/otbox.py", line 480, in _mqtt_handler_discovermotes
    line = line.decode("utf8")
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 1: invalid start byte

Could you check whether my changes work on your side as well? Once it works on both sides, I will merge it! Thanks a lot for the work involved!

[twalcari]

Hi @changtengfei ,

I tested your changes on wilab, and everything works fine here.

I assume that changing the encoding of the serial connection from line.decode('utf8') to line.decode('ascii') would also have done the trick, since that is the implicitly set encoding in sys.getdefaultencoding(), but that's only relevant when/if we would want to move to python3.

Also thanks a lot for the effort from your side!