EV3-to-EV3 protocol: client support for daisy-chaining

[WasabiFan]

We had a discussion in https://github.com/ev3dev/ev3dev/issues/92 a while back about support for daisy-chaining in ev3dev. The conclusion seems to have been that, if it were implemented, it would be implemented as custom logic in the individual client libraries. I received some questions about daisy-chaining with Python on ev3dev so I figured I'd open this issue to discuss. I don't expect much of anything to happen here short-term as we are in the middle of the updates for new APIs, new micropython, and Stretch.

There are two ways I could see this working. First, as I explained (although not very clearly) in the issue I mentioned above, I think it would be great to have a full client library which implements the protocol LEGO already uses for daisy-chaining via the same interfaces as they use, allowing you to control a stock EV3 from an ev3dev app without any software modifications to the second (or further) EV3s. However, this makes it odd if we have ev3dev-style APIs in the library which aren't supported via daisy-chaining. So, it might make more sense to implement our own generic server and support for one or more communication methods. This latter option would also enable interfacing with non-EV3 ev3dev platforms.

I could see some form of this being based on the new brickd service -- it could serve as the remote control server and our Python library would have to do relatively little work itself. That's something for @dlech to consider (although I might be able to make time to work on an implementation of that over the holidays if it were something we saw as valuable).

[dwalton76]

I mentioned SSDP in that discussion, I have a python implementation of that working. It could be used to discover the other EV3s on the network via IP multicast. Just one small piece of the puzzle

[dlech]

FYI, ev3dev-stretch already has avahi and ssdp daemons that allow autodiscovery.

[LorenzoBrualla]

I am doing a project that involves two daisy chained EV3 bricks. Is there any way to write a program in Python that allows daisy chaining with the currently available ev3dev software? If this is possible I would appreciate to know references to the relevant documentation. Thanks.

[dlech]

@LorenzoBrualla check out https://sites.google.com/site/ev3python/learn_ev3_python/rpyc

[dwalton76]

Is LEGO’s protocol for EV3 to EV3 communication documented somewhere?

[dlech]

There is an SDK with docs on the official MINDSTORMS website.

[dwalton76]

Found it here: https://education.lego.com/en-us/support/mindstorms-ev3/developer-kits

lego mindstorms ev3 communication developer kit-f691e7ad1e0c28a4cfb0835993d76ae3.pdf

[dwalton76]

A ton of awesome info (and code snippets) here on the UDP/TCP port numbers involved with setting up a socket with the EV3 to send the packets described in the pdf above: http://ev3directcommands.blogspot.com/2016/01/no-title-specified-page-table-border_94.html

Read some more...this person appears to have implemented most (maybe all) of the protocol and they did it in python and it is open source!! https://github.com/ChristophGaukel/ev3-python3

[dwalton76]

I updated the title to "EV3-to-EV3 protocol: client support for daisy-chaining" so that we can use this issue to track controlling an EV3 via the EV3-to-EV3 protocol. A huge amount of this code is already written in the https://github.com/ChristophGaukel/ev3-python3 repo.

We need to give some thought on what we want the API for this to look like. One option might be something like:

• create a LargeMotor object
• set some variables in the object that indicate this LargeMotor is part of a remote EV3
• have all future set_attr calls for this object use the EV3 protocol instead of writing to the local filesystem

But will all of our wait_for_XYZ methods work in this scenario? Probably not...to be sure there are going to be things we have implemented that are not supported by the EV3 protocol. I worry we may create a good bit of confusion with this approach.

Another option would be to create a RemoteEV3 class that would only have the exact API supported by the EV3 protocol. This shouldn't be too hard given how much of the protocol is already implemented in python. Long term maybe we could have a RemoteEV3DEV child class that would be used to control an ev3dev device? That would allow us to add to the API for all of the extra functionality we have.

[dwalton76]

@WasabiFan @ddemidov thoughts on what the API for this should look like?

[WasabiFan]

Note on the existing library: We have to be careful with it, as it's GPL-licensed... if we want to use that code I believe (?) we can at least sub-license that portion as GPL without it infecting the rest of our stuff, but I'm not a lawyer and haven't looked it up.

---

In general, I think either approach could work depending on how much implementation and maintenance effort they would turn out to need. Personally, I would really like to be able to support the entirety of our friendly API via daisy-chaining (e.g., maybe have special values you can pass for the port(s) which include remote device information). The main challenges I see with that approach are...

• Mapping of LMS2012 opcodes ¹ to the operations we support. For example, they see running a single motor as different from running a pair... obviously, you can emulate one with the other, but that might not have clean results -- we'd have to test it. So now perhaps the Move* classes also need to have special logic to handle remote operations. But I digress².
• Getting feedback on operation completion (as you mentioned)

If the control protocol doesn't provide any feedback, the dream of an opaque remote control layer is pretty much a non-starter... our APIs just wouldn't work. In that case. we'd need to have the design discussion for what a new control interface would look like -- I would suggest we seriously consider just recommending that external module itself.

---

¹: It's worth noting that, as far as I can tell from my 5 minutes of research, the wire protocol is just a carrier for sending standard lms2012 opcodes. David has an index of them here as part of lmsasm. So it's pretty easy to see what operations are available and how they work. I haven't looked into response values at all though.

²:

Scope creep (also called requirement creep, or kitchen sink syndrome) in project management refers to changes, continuous or uncontrolled growth in a project’s scope, at any point after the project begins.

[dwalton76]

I hear you on scope creep...we have the potential to create something that is pain to maintain. Thinking outloud on some of the various scenarios here:

No.	protocol	notes
(1)	ev3dev-to-ev3lego client	Could say "use the ChristophGaukel" repo. How useful will this be though? Are ev3dev users really still running the EV3 software on some bricks?
(2)	ev3lego-to-ev3dev server	Would allow the iPad MINDSTORMS app to control an ev3dev device (I think) which would be cool
(3)	ev3dev-to-ev3dev client & server	We could define our own protocol that provides our full API on a remote ev3dev device

(1) The more I think about this scenario the less interesting it seems. Personally I haven't run the EV3 software since the day I discovered ev3dev :)

(3) might not be a crazy amount of work, you basically have to make the low level places where we interact with the filesystem use the remote filesystem. Not sure how all of our event-driven code where we wait for a change to a particular file would work but worst case you could drop into loop and use a polling model. Would be API change be as simple as providing a way to indicate that a motor, sensor, etc is attached to a remote device?

[dlech]

FYI, iOS to ev3dev via Bluetooth is not currently possible for technical reasons (and possibly legal reasons). See https://github.com/ev3dev/ev3dev/issues/500.

[WasabiFan]

Personally, I don't particularly care about scenario (2). If you want to use the official LEGO control software, use the official LEGO OS too.

Regarding (1) or (3), I think we should pick one and support it -- either an ev3dev device controlling another ev3dev device or an ev3dev device controlling a stock EV3. The latter is attractive to me as it requires minimal configuration; personally, I find administering ev3dev devices to be a pain, especially when you have to confgure networking. The stock EV3 software makes it easy to connect a USB cable or pair via Bluetooth and everything "just works" without touching the slave device at all.

I think implementing a custom client/server pair for ev3dev devices is probably technically easier (for example, it would be almost trivial to implement a thin file system wrapper which took attribute read/write operations from a socket and handled them locally -- see my footnote). But that means that you have to flash and configure ev3dev on each device, persuade the various IP networking powers-that-be to play nice, and figure out how to address each device with your particular topology. This is already relatively difficult with only two devices... then what happens when you add another EV3? Or want it connected to the Wi-Fi network? I glossed over the possibility of not using IP, but I doubt that would be much easier.

---

Footnote on ev3dev-to-ev3dev control: I think it would make a lot of sense to find/implement a remote control server and configure it as an installable daemon. Perhaps even add it to Brickman or ev3dev-config so it's easy to enable. If we did it right, there's no reason the client side would be limited to our Python implementation either -- anyone could write an interface for it. And you could presumably run your Python code on a PC and use the remote control capacity to operate your tethered robot with minimal modification. I'm not aware of anything that exactly fits our use case already (in particular, we need poll events for things like state and latency is a real issue) but perhaps others are. Again, there's some technical investigation that would need to be done here -- e.g., how much latency is there over various transport mechanisms and how does that affect us?

[dwalton76]

For me (2) would only be interesting if I could use my ipad to control ev3dev and that sounds like a no-go :(

For (1) vs (3) to me (3) is more interesting mainly because I can't come up with a scenario where people are going to use ev3dev on one device and EV3 software on another.

ev3dev setup hurdles...my guess is among usb, bluetooth and wifi that wifi is the most commonly used so maybe we brainstorm some on how to make that easier? I only ever use wifi...I didn't think it was too crazy but my background is in networking so I am probably not a good reference point on this. SSDP could be used to discover other ev3dev devices on the same network easily enough.

Of the thin file system wrapper vs remote control server the former sounds much easier and seems like it would solve (3).

[WasabiFan]

For (1) vs (3) to me (3) is more interesting mainly because I can't come up with a scenario where people are going to use ev3dev on one device and EV3 software on another.

I would only install ev3dev on the other devices if necessary. As the user, if I want to have my code run on one device and then have it propagate control commands to the others, why would I care what software is running on the other devices as long as it works? The default OS requires no work to setup, whereas ev3dev does.

Nonetheless, it doesn't seem technically workable to implement support for a stock EV3 slave like we'd want, so the point is moot.

ev3dev setup hurdles...

I agree that Wi-Fi is generally the simplest to set up, but it's worth considering:

• Not everywhere has accessible or consistent Wi-Fi
  • e.g. I'm currently living in on-campus housing for school, and it's much easier to connect via USB rather than dealing with subnet isolation and various security rules on the provided Wi-Fi network.
• You must connect on every device inividually and give them unique names

If I were building a robot which had multiple EV3s built together, it would be nice to just string USB cables together and have the data "flow" refardless of whether the individual bricks are conneced to Wi-Fi. But perhaps that's a separate scenario which could be implemented after-the-fact if we could figure out how to consistently do, say, serial via USB or made it easy to set up IP tethering.

---

So, my vote would be to implement a network server which pretty much just forwarded commands to sysfs and then add the appropriate layer into this library so it can connect to it. @dlech do you have any comments on this that you'd like to get in before we go further with this idea?

[dlech]

What is the advantage of implementing something new vs. using RPyC for ev3dev-to-ev3dev or the existing python mentioned above library for ev3dev-to-official?

[WasabiFan]

RPyC is worth testing -- I've never used it before so I don't know how easy it is to work with or, more importantly, how efficient it is.

I think the existing library is still freely available for anyone who'd like to use it. The interface is completely different from ours, but I don't see a reason it wouldn't work. Personally, my goal with officially-supported daisy chaining would be to provide a uniform interface to make it easy to use multiple EV3-like devices as extensions of a single master.

[dwalton76]

Nonetheless, it doesn't seem technically workable to implement support for a stock EV3 slave like we'd want, so the point is moot

yeah to do (1) in a way that would allow you to use our current API to control a remote ev3lego (an EV3 running stock LEGO os) just seems like a mountain of work with lots of challenges.

So at this point I think we have tossed (1) and (2) which leaves us with (3) from an ev3dev device control another ev3dev device but that is doable today via rpyc. Maybe it is time to just declare that we support daisy chaining via rpyc? We have docs on it here: https://ev3dev-lang.readthedocs.io/projects/python-ev3dev/en/stable/rpyc.html

We could spend some cycles to make it easier to use:

• see if ev3dev can install the python3-rpyc debian package by default
• if so, create a service that runs via systemd that runs rpyc_classic.py

That would eliminate 2/3rds of the instructions on that page.

[dwalton76]

I just noticed https://github.com/ev3dev/ev3dev-lang-python/issues/677 so rpyc is already in ev3dev by default.

[dwalton76]

Another option here is MQTT. I’m trying to solicit some feedback (on a FB group) on it vs RPyC.

[WasabiFan]

My understanding is that MQTT is a message protocol whereas RPyC is a softtware implementation for remote Python interactions. So the former would require additional implementation work, I think, but an implementation based on it might be more performant.

If we wanted to "properly" support this, we'd have to:

• Test it (duh)
• Figure out how easy/diffficult it is to configure nettworking. Wi-Fi is naturally easy, but how about direct Bluetooth connection or USB? I suspect those are harder. It would be nice to have a script or Brickman option for it.
• We probably shouldn't auto-start the service, because every cycle is precious on these EV3 CPUs. But adding it as an enable-able service via Brickman would make sense.
• Ideally, write some tutorials.

[dwalton76]

To me RPyC feels a little too magical, there is a lot going on behind the scenes. MQTT seems much closer to a standard client/server model so me personally would probably use MQTT. I can see how the choice could easily vary depending on the user and the robot they are building.

How about we:

• close this issue as it is related to implementing the EV3g protocol which we are not going to do.
• close the issue for implementing an EV3g protocol server
• use the RPyC docs issue that David filed to make RPyC as easy to use as possible. I have a service file created that we could install so the user could easily enable it as a service. Agreed on not enabling this by default given our limited CPU.
• open an issue for making MQTT as easy to use as possible. I imagine a lot of this would be documentation.
• review networking docs to see if anything can be improved there. This falls under ev3dev though not ev3dev-lang-python.
• declare that daisy chaining via python is supported via RPyC or MQTT.

Thoughts?

[WasabiFan]

Most of that seems reasonable to me. However, I think we're conflating the roles of MQTT and RPyC a bit:

• MQTT is a generic communication protocol. You put bytes in, bytes come out the other end. It's then up to you to write your own code for doing something interesting with those bytes.
• RPyC provides a language-integrated interface which lets you operate on the remote end with calls that appear local. Depending on which features you use, you can either simply initiate a remote connection and import+use ev3dev2 like you would locally (keeping track of which objects are remote and which are local) or you can implement a server module which exposes specific methods and then use those methods. The former should work already as-is, while the latter would mean implementing something like a RemoteEv3devConnection class which lets you open motors, sensors, etc.

RPyC would work as-is, in that you can simply instantiate a LargeMotor, for example, and it will control the remote motor.

So, to me, RPyC is the clearly preferred approach due to its pre-integrated nature for the type of things we want to do, with a few potentially major caveats:

• It's probably somewhat easy to get confused about what objects are remote and what objects are local. I haven't yet tried it though.
• Speed and performance. There are a lot of potential issues here...
  • Sequential file writes might be spaced out too much, causing synchronization issues and potentially making it unusable
  • There might be lots of delay between trying to start an operation and having it actually happen
  • It might be too much extra Python dynamic "stuff" for the poor EV3 processor to handle
• It doesn't support Micropython and such support would be impossible.

That last caveat is perhaps the biggest.

The alternative, which would be using MQTT or another messaging protocol, would avoid many of those issues. The difference is that now there's no pleasant interface in Python and either we'd have to build a new one into the library or each user has to write their own.

[dwalton76]

I think these are all good points that we should mention in docs. At some point maybe we’ll be familiar enough with both to write a decent “RPyC vs MQTT” doc to help folks figure out which to use.

[dwalton76]

Closing this ticket as we have decided to not implement the EV3g protocol.