Phonebots

[DamonOehlman]

Figured this would a good place to continue discussion regarding the discussion @garrows and I had at CampJS. Will look to cover the how (this is probably best done by Glen), the why and the when.

With regards to the question regarding getting it going prior to the event, I think I'm in favour of that.

[garrows]

A bit of background, we are talking about running android/ios phone applications built by either Mobile Chrome Applications or just plain Cordova that connect to the Arduinos via bluetooth. This is still using the Johnny-five library but loaded via browserify and using browser-serialport instead of the normal node serialport library. I haven't published it yet but the browser-serialport library will wrap the Chrome Serial API, a Bluetooth Serial Plugin for PhoneGap and @DamonOehlman's WebRTC DataChannel library.

There are 3 approaches/options I see with Phonebots (not sure on the name btw).

1. Proxy Mode
2. Host Mode
3. Combo Mode

Proxy Mode

I see this as a pre-existing application possible available on Android Play and the App Store. It will connect via bluetooth to the device and connect to a WebRTC DataChannel negotiation server. A unique key will also be displayed on screen to identify itself.

On a PC or other device the roboticist Johnny-five is initialized using browser-serialport configured to connect to a WebRTC DataChannel (instead of a serialport). Johnny-five could be run here via node or a web application. The Firmata protocol is streamed from the Arduino to phone via bluetooth and repeated to the PC from the phone via WebRTC and everything should just work.

Host Mode

This is where I've done the most amount of work during Camp.js. Johnny-five is initialized on the phone using browser-serialport and connects to the Arduino via bluetooth. The roboticist builds the android/iOS application using cca or cordova. Most of the AI goes in the application. The application can also open up a WebRTC DataChannel so someone can control it from a PC but by using custom commands (i.e. Go forward, back, etc) and not using firmata.

Combo Mode

No idea if this is possible yet but it's a combination of the proxy and host mode but there would have to be 2 instances of johnny-five running and it would switch between proxy and host modes depending on if anyone is connected to it.

Bonus Features

• Both approaches should include a 2 Way video stream using WebRTC too.
• Phone sensors (i.e. accellerometer, proximity sensor, gps, etc) should be implemented in the Johnny-five library. Not sure what Rick would think about this though.

I'm looking forward to hearing thoughts.

[sidorares]

I'd like to try to use phone on a mobile device (possibly 2 phones) and do 3d stereo reconstruction from 2 moving cameras.

Also would be great to have easily pluggable "remote firmata" without requiring BT shield - with usb cable using android device as usb host ( and transparently proxy firmata protocol with phone networking)

[garrows]

@sidorares cool idea. There is the IOIO board which uses ADB. Not sure about the Arduino support/implementation for ADB though.

[sidorares]

adb protocol is quite simple, I have node client for adb over tcp - https://github.com/sidorares/node-adbhost/

[garrows]

Awesome. I dont suppose you have ever hooked up an arduino as the ADB server before have you?

[sidorares]

oh, you are talking about arduino side. I was suggesting to use usb as a serial device from phone ( requires Android 3.1+ - http://developer.android.com/guide/topics/connectivity/usb/host.html ). ADB though will allow to map incoming traffic to local tcp port in the phone, so no need for compiled app to be installed.

[garrows]

I see. But how would you communicate with the Arduino then? Can you map TCP to a bluetooth device?

[sidorares]

From the phone using usb cable ( connected to both phone and Arduino), need micro usb to usb otg adapter, without bluetooth. at usb wire level communication is exactly the same as when computer is connected to arduino and talks firmata protocol ( But I like idea of using ADB layer. In that case arduino is a client, not server )

[garrows]

Thought I should give an update on this...

TL;DR: I've previously made good progress but haven't looked for a while. Debugging is needed.

I had to make some modifications to the cordova bluetooth library to support raw data rather than just strings. Changes are in my fork here https://github.com/garrows/BluetoothSerial/compare/don:master...master

Things are now connecting however firmata errors pretty quickly and debugging is hard since I have to debug the state of the JS on the device and the state of the arduino. Currently I'm doing console.log() on the phone and listening over ADB while doing Serial.println() on a leonardo (has 2 serial ports). Anyone got better suggestions?

I've come to the conclusion that firmata is very fragile and does not handle errors or flaky connections very well. Even bluetooth connections from OSX is flaky at best. I've previously had to make stability fixes to firmata but its code is hard to follow. I think the success of this project hinges on making firmata more robust but I'm committed to see it through.

[geekscape]

Whenever I've used Android Bluetooth to Arduino (via HC-05) ... I checksum every message (and throw away the bad ones). It isn't reliable otherwise.

Andrew and I have been discussing improvements to the Firmata protocol, particularly for use over mesh-networking (where radio reliability is an issue). Looks like we'd need to put in a small reliability feature.

[nog3]

With the advent of those wifi232-t modules I wonder how reliable they'd be in comparison to the bluetooth. (It might be easier to implement than the cordova serial bluetooth module too.)

On Thu, Jul 31, 2014 at 11:51 AM, Andy Gelme notifications@github.com wrote:

Whenever I've used Android Bluetooth to Arduino (via HC-05) ... I checksum every message (and throw away the bad ones). It isn't reliable otherwise.

Andrew and I have been discussing improvements to the Firmata protocol, particularly for use over mesh-networking (where radio reliability is an issue). Looks like we'd need to put in a small reliability feature.

— Reply to this email directly or view it on GitHub https://github.com/nodebotsau/nbdau/issues/12#issuecomment-50704097.

[geekscape]

Given that you're running TCP/IP over those Wi-Fi modules, you'd expect to either get the messages (correctly) or not. And, would not have to deal with corrupted messages. So, if you are within radio range, I'd expect Wi-Fi / TCP IP to be more reliable than Bluetooth / Serial (without some custom message checksum).

[garrows]

@geekscape what do you think the signal to noise ratio is? I didn't find it too bad coming from my mac but I did have to lower the frequency when I was reading from a sensor. I guess the chip isn't fast enough.

I'm keen to hear your thoughts on making firmata more reliable. A big problem with adding reliability features is testing. Perhaps we can write a unit test with varying levels of packet loss and delays.

@nog3 I did consider switching over to using those wifi modules but having to reconfigure them every time you change locations is a bit impractical in my opinion. They would be much more reliable though.

[ajfisher]

@garrows - don't forget you can flip those wifi modules into AP mode so you can just connect and go to them and then use the web interface to connect to the local wifi. I more or less only use mine in AP mode for my SimpleBot and don't even worry about connecting to the station - just like using a Parrot Drone.

Andrew Fisher http://about.me/ajfisher/bio

On Thu, Jul 31, 2014 at 1:59 PM, Glen Arrowsmith notifications@github.com wrote:

@geekscape https://github.com/geekscape what do you think the signal to noise ratio is? I didn't find it too bad coming from my mac but I did have to lower the frequency when I was reading from a sensor. I guess the chip isn't fast enough.

I'm keen to hear your thoughts on making firmata more reliable. A big problem with adding reliability features is testing. Perhaps we can write a unit test with varying levels of packet loss and delays.

@nog3 https://github.com/nog3 I did consider switching over to using those wifi modules but having to reconfigure them every time you change locations is a bit impractical in my opinion. They would be much more reliable though.

— Reply to this email directly or view it on GitHub https://github.com/nodebotsau/nbdau/issues/12#issuecomment-50710176.

[garrows]

@ajfisher true but then you lose your internet connection.

[ajfisher]

True that... but you're driving a ROBOT not looking at cat videos on youtube ;P

Andrew Fisher http://about.me/ajfisher/bio

On Thu, Jul 31, 2014 at 2:07 PM, Glen Arrowsmith notifications@github.com wrote:

@ajfisher https://github.com/ajfisher true but then you lose your internet connection.

— Reply to this email directly or view it on GitHub https://github.com/nodebotsau/nbdau/issues/12#issuecomment-50710529.

[ajfisher]

Unless of course you want a robot that drives around looking at cat videos on youtube... at which point I agree - you'd have a problem. ;)

Andrew Fisher http://about.me/ajfisher/bio

On Thu, Jul 31, 2014 at 2:20 PM, Andrew Fisher ajfisher.td@gmail.com wrote:

True that... but you're driving a ROBOT not looking at cat videos on youtube ;P

Andrew Fisher http://about.me/ajfisher/bio

On Thu, Jul 31, 2014 at 2:07 PM, Glen Arrowsmith <notifications@github.com

wrote:

@ajfisher https://github.com/ajfisher true but then you lose your internet connection.

— Reply to this email directly or view it on GitHub https://github.com/nodebotsau/nbdau/issues/12#issuecomment-50710529.

[garrows]

lol. @ajfisher my dream is to have the phone permanently attached to the robot and connect to the cloud using p2p tech to control it over the internet. So yeah, internet is important.

Side project: a robot that chases cats, records them and uploads the videos to youtube.

[ajfisher]

That is a great project idea... a laser pointer to control the cat to do silly things, chase it and upload it to the Internet.... I know what my wednesday night project is going to be now.... now I just need a cat...

Have totally hijacked this Phonebots thread - sorry all, as you were.

Andrew Fisher http://about.me/ajfisher/bio

On Thu, Jul 31, 2014 at 2:24 PM, Glen Arrowsmith notifications@github.com wrote:

lol. @ajfisher https://github.com/ajfisher my dream is to have the phone permanently attached to the robot and connect to the cloud using p2p tech to control it over the internet. So yeah, internet is important.

Side project: a robot that chases cats, records them and uploads the videos to youtube.

— Reply to this email directly or view it on GitHub https://github.com/nodebotsau/nbdau/issues/12#issuecomment-50711242.

[DamonOehlman]

http://www.gumtree.com.au/s-cats-kittens/melbourne/cat/k0c18435l3001317

You're welcome :)

[ajfisher]

Thanks man.

Andrew Fisher http://about.me/ajfisher/bio

On Thu, Jul 31, 2014 at 2:36 PM, Damon Oehlman notifications@github.com wrote:

http://www.gumtree.com.au/s-cats-kittens/melbourne/cat/k0c18435l3001317

You're welcome :)

— Reply to this email directly or view it on GitHub https://github.com/nodebotsau/nbdau/issues/12#issuecomment-50711749.

[geekscape]

None of those cats support Firmata or Bluetooth :(

I believe that the WiFi module can run simultaneously both as an AP (for your phone, etc) ... and as a client (STA) wirelessly hanging off your Internet connected Wi-Fi router ... also whilst maintaining a serial connection to your robot ! See section 2.1.2, page 18 and 19, figure 8 ...

http://en.usr.cn/download/USR-WIFI232-T.pdf