Better execution of generated code

[makenai]

Currently, the way we run generated code is to write it to the local filesystem and execute it as a child process. Two issues:

• Right now it just writes into the current directory - this sometimes fails due to permissions. I tried getting this into a tmp directory, but it wasn't working, so I left it like this so I could finish in time for the workshop.
• There is probably a nicer, safer way of executing this code. Maybe we could do so in a sandbox in the current process?

[BrianGenisio]

Here is what I was thinking while dreaming about this IDE:

I think that there should be a generic REST interface to J5. This thing is robotnik/scratch/etc agnostic and exposes board objects via REST. Other apps can be built upon this service as well.

Because of this, I don't think sending code down to the node process is the cleanest way to go about this. Instead, I think we have a REST service that allows the creation of objects (servos, motors, LEDs, etc) as well as actions on the objects:

POST /api/components
POST /api/components/leftwheel/ccw
POST /api/components/rightwheel/stop

Then all the code should be executed in the browser, and the custom blocks make REST calls back to the server. Blockly recommends using their interpreter to run the generated code.

My Scratch POC took a similar approach with a lot of success.

What do you think of this approach?

[makenai]

I'll need to look into your POC, but one of my goals is for the generated code to work outside of the IDE - ie, you drop it in a standalone file, install Johnny-Five and it runs just fine. I think generating re-usable an non-ide dependent code will be useful for people trying to learn. I'm hoping to avoid creating a pseudo-language dsl or shim to make the code nicer to generate and execute but which won't be useful when they do more complicated stuff outside of the IDE.

I'm going to look into your POC to understand this approach better, but maybe you can clarify if that's what you meant.

[makenai]

OK, looked into the code. It's kind of as you described, a REST interface into calling J5 methods.

I think that's a pretty cool idea, but I am pretty big about the code generation aspect.

What if we went about a different way of doing this? Currently a number of J5 adapters like SparkIO and Electric Imp already rely on a REST interface. What if we created a REST interface similar to those for other boards like a standard Uno that used a backend server. THEN we could write a J5 IO adapter to talk to the REST interface and still generate real code. The user would only have to remove one line to talk to a local board, or we could set up the adapter to determine if it should talk locally or remotely.

I think that J5 could run under browserify - possibly might have to do something about the node-serial requirement.

@Resseguie has done a lot of work with VoodooSpark, so he might also have some insights.

[makenai]

And to clarify my comment original comment: I don't mind a shim as long as it's a fairly transparent one.

[BrianGenisio]

I hadn't considered the idea of taking a project and running it outside of the IDE. That makes a lot of sense. Need to stew with that thought.

[Resseguie]

Apologies, I've been AFK for a few days (at least for this stuff).

@makenai on initial thought, I like the idea of an IO Plugin layer to accomplish both goals (REST interface and also easy migration to running directly on a board). This is where Remote-IO by @monteslu might come into play and save us some work? https://github.com/monteslu/remote-io

[makenai]

That's OK! I've been pretty busy on unrelated things too. One of them is behind me now, and I'm excited about diving into some of this over the weekend. That plugin looks perfect for it too.

[monteslu]

Thanks for the mention, @Resseguie This project looks really cool.

The concept with remote-io is that we can have many different types of node-serialport implementations socket.io-serial, mqtt-serial, skynet-serial, browser-serialport, etc.

Then a johnny-five app (in node or in a browser) can initialize with the default firamata IO class using this network-based serial device. Another process (possibly on another machine) can operate in a couple of ways to talk to the physical hardware:

• If its an arduino with firmata, it can just bind the physical serial port to the same type of network serialport that the johnny-five app is using. bindPhysical
• If its something that doesn't speak firmata like raspberry Pi, it can use the Remote-IO class to wrap the Raspi-IO class and plug the network serialport into the Remote-IO instance so it can translate firmata to IO class calls.

I put up a longer description of remote-io here

A couple more things to note:

When building chromebots, I found that you need a new process for each execution of johnny-five. In the browser for a new DOM, i just reload an iframe for it to execute in and do some window.postMessage things to kick off scripts. I haven't tracked it down, but it seems J5 keeps some state around that could be causing issues if you don't shell out a new node or new DOM each time.

I like @BrianGenisio 's idea of REST apis. I think having a declarative syntax on top of both generic IO calls as well as higher level J5 apis would be extremely useful. However, we have a problem. Some J5 methods require bi-directional communication. Things like sensors and i2c send data back to the client. Unless the client can setup a its own http server, the REST calls would be one way. Since we already have firmata.js as a binary encoder, and remote-io as a decoder, I've just been using socket.io/mqtt/skynet up to this point.

[makenai]

We now now longer write to the filesystem, and instead fork off a process and send code to be executed in a vm sandbox via an IPC message.

I'll reference this ticket from the Chrome app ticket since @monteslu's notes are so good.

[lynnaloo]

I'm also executing the code inside of a vm sandbox in one of our projects and I'm really happy with that solution. I'm using a REST API as well, but since I don't have a lot of bi-directional communication I just poll the server at an interval and get the latest from the sensors.

[makenai]

Definitely got that idea from talking to you @lynnaloo. Would love to see how you're doing it as well if it's something you can share.

I have some silly ideas including injecting a fake 'console' object into the VM sandbox to hijack the output selectively, as well as injecting a fake 'require' or something that can hijack the board on ready event and give some feedback to the browser that it's running your code properly.