techninja / cncserver

A RESTful API server for driving serial based CNC devices
133 stars 40 forks source link

Add interface for Scratch extension #50

Closed oskay closed 10 years ago

oskay commented 10 years ago

Not sure if this is the right place (or way) to suggest this.

With a few modifications to cncserver, it might be possible to write an extension for the Scratch programming language ( http://scratch.mit.edu ), that could interface to cncserver. This could give the ability to drive the WaterColorBot or EggBot directly from within Scratch, which is a popular graphical programming language that features LOGO-like turtle graphics. (It could be a magnificent addition!)

Scratch extensions are documented here: http://wiki.scratch.mit.edu/wiki/Scratch_Extension_Protocol_(2.0) http://wiki.scratch.mit.edu/w/images/ExtensionsDoc.HTTP-9-11.pdf

The extension interface is quite primitive, and (if I understand correctly) allows Scratch to communicate via HTTP GET requests only. It also requires that a "crossdomain.xml" file be returned upon request.

It may (A) be reasonable to add a few equivalent "non-restful" command aliases to the cncserver API, to enable interfaces like Scratch or requests from a web browser address bar to drive the robot. (Personal opinion: that would be really neat.)

Or perhaps (B) it may be better to add those as a separate helper application, or to add those functions within RoboPaint, which then sends the properly formatted command to cncserver.

Note added: A good example of a Scratch extension and interface can be found here: https://github.com/damellis/A4S

techninja commented 10 years ago

Oh yea, totally. That interface certainly smells of early XMLRPC/Flash interfaces, but hey, why not. It's certainly worth it. Probably open it up as an interface entirely separate from the versioned REST api, maybe something like /scratch or perhaps /simpleapi ? I'll look into it, awesome suggestion :)

oskay commented 10 years ago

Sweet! Yes, it's "primitive" in a sense, but it's also accessible. ;)

I'd suggest versioning it like "/s1", so that it could have parallel versions with the "/v1" that we already use. This could be really awesome.

techninja commented 10 years ago

Just got back from a quick attempt, and I've basically fallen flat with a big fat "bleh": Scratch restricts HTTP interaction in some very strange and awkward ways;

Regardless of the difficulty, I've fully implemented the very first portions to that spec, and have tested against Scratch 2 offline v404, with some success, except for the fact that apparently all "wait" commands make no requests at all (verified with wireshark), so I'd say this is essentially dead till this gets some kind update. Apparently the actionscript version is live on their github page, so hope is not all lost. FWIW, I'll open up an issue.

The general excuse is that it's a "experimental" feature, though it seems making even the "implemented" feature set stable enough to test is out of the question. Forum posts here, and here, and here for continuation.

Another option that seems possibly in-line with the intent of the ticket, is to use Snap, an HTML5/JS native re-implementation of scratch, which would likely be able to run the CNCServer API directly in a custom block. Of course it's also buggy and quite a bit more complex than Scratch :P

oskay commented 10 years ago

Neat! Can you please upload your Scratch example as well? I wonder if we might be able to do the waiting on the Scratch side, and just communicate "open loop" to the CNC server.

techninja commented 10 years ago

I actually never got scratch working well enough to make it worth it. The procedure for use/testing is as follows:

Reading pen values also apparently has some kind of issues, not sure why.

Switching the process blocking "w" commands to async " " commands does technically work, though I've mostly written it off as bad form. Truth to tell cncserver has a full and complete queuing system that could simply take whatever flood of commands are sent from Scratch without waiting, of course scratch would never have a clue about what the bot was doing. Perhaps we'd also need some level of interaction with the buffer, and to properly clear the work queue and park on reset, and maybe even blocks for pausing and clearing the buffer.

What do you think? It's certainly a way forward, even if it's a little questionable. Also, what other block types might you want? Setting speed or other settings perhaps?

oskay commented 10 years ago

Ah -- sorry -- didn't see the .s2e file there.

oskay commented 10 years ago

I think that "open loop" control is the most appropriate approach given the current state of scratch. My guess is that they'll eventually fix up the interface. I'll give it a try, and see what can be done.

techninja commented 10 years ago

Sorry neglected to mention this is in the scratch branch. I'll look into making it open loop and if it's feasible and actually works in scratch I'll wrap it up and commit to master.

techninja commented 10 years ago

Open loop works! Looking around at how these things are usually applied, I figured to run with the "turtle" method of having a separate pointer that has an angle and gets a distance to walk. This applies well to the loop blocks. I've also implemented a number of different methods of doing the same thing, which allows for ease of use in different situations. Like direct text setting of the tool, or using an integer that allows for a random color to be grabbed.

The extendability of this is huge, at least as far as what can be done on the CNCServer side with the limited trigger data coming from Scratch. I'd like you (or others) to beta-test this a bit, and see if there's any more useful blocks you can think of adding/abstracting. Once I get approval, I'll merge the feature to master and this will be an option to enable in RoboPaint.

example cncserver scratch program

My test Scratch program:

Get it here: TestCNCServer.sb2. This project file includes its own copy of the s2e extension definition, you should just be able to open and run it with a copy of the scratch branch of CNC server running (with scratchSupport on).

oskay commented 10 years ago

Looking nice so far! I'm re-installing node right now (long story...) but I do have some first impressions: (I've edited this several times with additional notes.)

I can see that we need:

Likely also want:

Suggestions:

Harder, maybe not possible:

Questions:

Note to self: node cncserver --botType=watercolorbot --scratchSupport=true

oskay commented 10 years ago

Everything seems to be working except for pen raising and lowering... not sure why.

oskay commented 10 years ago

I've added a example program to the branch, with a slightly better sprite.

techninja commented 10 years ago

I'll reply to the larger commentary in a moment, but I have a feeling I know what's up with Pen lift/drop. When creating the wireless WCB with Brian, his newst firmware update apparently flopped the variable used for going up and down. What I think is needed is a switch based on firmware version so both of these firmwares can exist together. Whats your/the standard fw version?

oskay commented 10 years ago

I'm using 2.0.1.

[I've edited my larger commentary about ten times-- I'll move to adding new comments from this point forward.]

oskay commented 10 years ago

I've added another example, showing basic subroutines ("custom blocks") for moving and turning the brush and sprite simultaneously.

techninja commented 10 years ago

I've implemented almost everything you suggested (click the ... above for details), the following are what didn't get in/up for discussion:

Some method of previewing what will be drawn, without actually drawing it. Perhaps an "offline" command? "Turn WaterColorBot[On/Off]" with a boolean/menu option?

  • Not sure what that would entail. The modifications a Scratch extension can do are 100% encapsulated within that s2e file in JSON, and they only seem to be drop down menus and the blocks. No other tie-ins exist, and there's no way for CNCServer to display anything.
  • If this was hooked up to RoboPaint in a mode made for it, you could quite likely see it work. Course at that point we could likely just embed the whole dang Scratch interface inside RoboPaint and auto-load the blocks. Hmm.

It would be nice to have some obvious scaling between the Scratch draw window and the paper, to serve as a guide to the Scratch programmer, on how big of a thing to draw, and where to draw it. One method of "limit checking" would be to have the turtle draw the paper's outline in the scratch window.

  • Currently the "steps" are 20x the stepper motor steps, so.. they're still somewhat small, but not too small.
  • X/Y coords are the same, but this multiplier is just a guess at what could be the most useful for scale. Seeing as there's no visible turtle powered by the commands, getting a 1:1 ratio or even some kind of usevul visual queue is going to be trickier than the tools we have immediately available to us.
  • Limits are handled by a sanity checker that simply hard limits to the bots max/mins. I could have had the pointer continue off the edge and retain that value, but as of right now, it hard limits and the stays at that limit.
  • I was actually able to add the X, Y and Angle Readers, and they re-convert to the dumbed down resolution so should be usable, can add these for any of the other vars as well.

Should the tool name be "Brush" not "pen" everywhere?

  • My intent with CNCserver, and in turn, RoboPaint, is to allow them to be a tool for far more than just one or two bots, so keeping it generic makes sense. Of course because of the locked down nature of this tool, as soon as you add the color tools, it's now WCB specific, so I figure why not.
  • If anything, this appeals greatly to the exact target audience just for the WCB, so having this on and ready to go in the next version of RoboPaint would be a big plus. It never fails to surprise me how well the Scratch interface lends itself to very quickly controlling complex movements on the bot.

Should there be an option to use an actual pen? (A "no water, no paint" mode)? It could be called with a "PenMode(true)" kind of operation.

  • This makes sense in RoboPaint, but here, it's might just be another toggle to keep people from messing up, because if you have a pen in, just make sure not to run color or water switch blocks!
  • Dunno, can still add this between /reset_all calls

We could have a "high level" setup routine... one that washes the brush, and moves to the center, along with a "high level" cleanup routine, that washes the brush and goes home. Would it be better to do this?

  • We have a grouped routine for washing now, though it only just does that. Though that somewhat assumes a use case, and adds to the "block bloat" a bit. Up to you.

Scratch has some sort of "color" variable. Is there a way to map that to paint color?

  • CNC Server knows nothing about colors. This is all part of the client implementation in RoboPaint, and entirely depends on properly selecting a colorset.
  • I have in fact found a way to make the color selector work on a custom extension block, but about the only way I could imagine having this select a color is if the colorsets existed in cncserver, were pulled from RoboPaint, or were configured by the user in scratch (somehow)
oskay commented 10 years ago

Just looking at the new version-- this is looking better all the time!

Simple changes to suggest:

  1. "turn to X degrees" -> "point brush in direction X" (For consistency with other motion commands)

Questions:

  1. Is the "move to center center" block doing what it is supposed to (in scratch, not in terms of cncserver)? It looks like the popups are not working correctly. I'm not sure that this block is necessary given that we're now centered to begin with.
  2. Is there a way to add a "point brush towards X, Y" command? In Scratch's motion menu, there is a "point towards (mouse-pointer)" command. Adding a "point brush towards" command would make it pretty easy to build a real-time mousing app. :)

Other comments

On the issue of previewing:

My comment was that we need a way for someone to test what the scratch program will do, without moving the robot. In other words, an "offline simulation mode" supported by cncserver. When it receives the "simulateOn" command, it will ignore all other commands until restarted or it receives a "simulateOff" command. As far as visualizing goes, that's up to the Scratch programmer.

Possible method to implement: [" ", "begin playing dead", "pen/simulateOn"], [" ", "stop playing dead", "pen/simulateOn"], or [" ", "go to sleep", "pen/simulateOn"], [" ", "wake up", "pen/simulateOff"],

On the issue of the size scaling:

The Scratch "stage" is 480x360 pixels, in the normal mode. At 450 steps per inch. The WaterColorBot paper "printable" area is 8 x 11, or 4950 x 3600 @450 DPI. If we fill the paper vertically, that would give an even 10 WaterColorBot steps per screen pixel, not 20.

My example file showed how to make a visible turtle that leaves tracks on the screen-- I'll update it for the new version.

On the name of "brush" or "pen"

I agree with the sentiment about keeping it general. However, Scratch already has a "Pen", and saying "turn WaterColorBot" doesn't quite make as much sense. :P

On the color sets:

I don't understand how scratch organizes colors very well. It would be nice if there were some way (in Scratch or CNCserver) to relate the visual color to paint number, but I'm not seeing it yet.

oskay commented 10 years ago

I've committed a couple of small changes, including a change form 20 to 10 in the scaling factor. (This works very nicely.) I've also noticed that Scratch has positive Y towards the TOP of the page.

I added one change to CNCserver that seems to fix this (so that you can build an example where the WCB carriage follows the mouse) but there may be other contexts that are still broken. (For example, the "if (op == 'x' || op == 'y')" part has a similar issue, but I didn't see where it's called from.)

Added updated example with turtle graphics blocks. Looking pretty darned neat, now!

techninja commented 10 years ago

...why didn't we do this sooner? :P

oskay commented 10 years ago

Without looking into it, it wasn't obvious how well suited it would be.

oskay commented 10 years ago

Suggestion: change name in extension from "CNCServer" to "WaterColorBot" Edit: camelcasing.

oskay commented 10 years ago

Adding screenshots!

ss1 ss2 ss3

oskay commented 10 years ago

This is still "pre-alpha" but here is a mostly-working testing procedure for Mac:

  1. If you haven't already installed it, install RoboPaint from
    https://github.com/evil-mad/robopaint/releases

    If you already have it and it happens to be running, quit RoboPaint.

  2. Install the Scratch 2.0 offline editor, from: http://scratch.mit.edu/scratch2download/
  3. Update the RoboPaint application:

(a) Pick from the menu: Finder>Go menu >Go to folder (b) enter: /Applications/RoboPaint.app/Contents/Resources/app.nw/node_modules/cncserver (c) Replace the older cncserver.js file in that folder with that from this Scratch branch of this repository. ( https://github.com/techninja/cncserver/archive/scratch.zip )

  1. Open RoboPaint. In the preferences, set the speeds (e.g., 70%) and other things that you would like to set. :) Leave RoboPaint running.
  2. Open example scratch file from the zip file (test3-who.sb2), in the offline editor, and run.

I would suggest "running dry" -- without paint or water -- until things are working.

Things that don't work right now:

oskay commented 10 years ago

It looks like Scratch compatibility will also give us easy compatibility with Snap! ( http://snap.berkeley.edu )!

I've got some basic examples built in Snap! now, and it is far more elegant, at least in the sense of remaining "tidy" with the addition of extensions.

One function that needs to be added: Check for connection, return true or false, maybe?

oskay commented 10 years ago

Current list of suggested additions to CNCserver endpoints, after building a working Snap! interface:

techninja commented 10 years ago

Will add what I can tonight before we leave for New Hampshire. Hopefully can meet up with someone at Scratch HQ in the MIT Media lab this Tuesday.

techninja commented 10 years ago

FWIW: Apparently the "wait" problem is fixed in master and they're just behind on releasing: https://github.com/LLK/scratch-flash/issues/320

Also: Would we even still want to replace our current functions with "closed loop" commands that hold for every block? The commands would likely not be at all as fluid. Don't know of course, it hasn't been tested.

oskay commented 10 years ago

My instinct is to stick with the open-loop commands for the time being, especially since the Snap integration is looking so nice-- I'll post examples soon.

It would be very nice to get a repackaged version of RoboPaint with the scratch integration soon-- we have several people itching to try it. :D

techninja commented 10 years ago

Repacking RP is going to be a little bit tricky considering the version offset. I'll see what I can do. Might be able to swap out the version of CNCServer for current builds and just update the current release (as there's a heavy amount of not quite ready for prime time features currently in RP master). Though I think we might need to hold off right now as this feature is still in flux and hasn't been tested enough to not need to change often.

I'd recommend anyone interested in dev testing this to simply run it through command line CNCserver following the instructions in the readme with the additional command line argument --scratchSupport=true, and once we get it solid I'll make a RoboPaint release ASAP with this feature on by default (and hopefully clearly visible.)

RE: new additions:

Also added distanceCounter getting so if you're clever you could duplicate RoboPaints auto paint refreshing (though that's not something CNCServer will do for you at this time).

Alright, off to get ~2 hrs sleep before the trip to get to the airport for the plane ride to Manchester, MIT bound on Tuesday.

oskay commented 10 years ago

Thank you!

It is a lot of blocks.... the additional motion commands are in parallel with those found in Snap.

Thus far, haven't gotten distanceCounter to work properly. Could well be user error.

techninja commented 10 years ago

Yea, it's broken. I seem to have messed something up :P Will fix once I finish my work here at the airport :airplane:

techninja commented 10 years ago

Fixed. Turns out it was WAY off from just being added. Also made sure the amounts were correct.

This seems to have outgrown its initial planned implementation overhead a bit, I'm thinking this will need to become a "plugin" of some sort and move out of the central cncserver codebase and into perhaps a "tpi" folder for third party integration where we can then store its various example files. Though this is really just a refactor/cleanup discussion.

I've finally arrived at CMK 2014, and holy cow there's nearly 150 people here. Hopefully during the trip to the media lab today I can flag someone down and show it off.

oskay commented 10 years ago

I agree that there's some room for refactor and cleanup. But overall, it feels like a much bigger step forward that I was expecting. :)

techninja commented 10 years ago

Field trip to MIT went great! Unfortunately the scratch "team" wasn't exactly around, but we got a great and very successful demo running and hackable very quickly, and even got to show off the whole setup to the original Scratch designer and lead Mitchel Resnick, who was quite pleased and impressed to se eit going. He says if we get the integration to a point we're really happy with he'd love for us to get some video/photos of demos for them to publicize.

Not sure what's left to do between here and there, but honestly it's already quite usable. I had someone who's never used a WaterColorBot write a program to have it paint random colored circles in random places across a page in less than 5 minutes.

oskay commented 10 years ago

The next step is to get this "baked in" to a standalone program that's easy to launch, so that we can a few other people to beta test the integrations. (That could be a new release of RoboPaint, or a standalone launcher for cncserver.)

techninja commented 10 years ago

As part of wrap up, we need some things cleared up:

If we get all this managed I'll merge the feature into master and get the ball rolling for the now long belated RP release, which may as well be a beta with intents for beta point releases when issues come up.

oskay commented 10 years ago

Current up/down code is working perfectly.

Reset should not change the movement speed, nor park (I think). I haven't really tested the current functionality, to see how it "feels" -- but my instinct is that this part is at least ready for our beta testers.

techninja commented 10 years ago

Forgot, one last question: the .s2e file that you need to import the extensions in with is going to need to be in a better place. One idea is to place it (and the other files) into the webserver root dir so that they can be linked and downloaded as parts of CNCServer directly from RoboPaint.

Also I'd imagine we should probably just create an incredibly simply example app that can be opened in the offline editor that simply contains the extension blocks in it instead of walking users through Shift+click for the file menu. Also this brings up the question of exactly how we intend to integrate in RP.

oskay commented 10 years ago

I would suggest that the examples should be available for download separately from RoboPaint. They could be available as part of the RoboPaint or CNCserver repository. But, I don't think that it's helpful to require people to do something "funny" like ask RoboPaint to download the example code.

Very importantly, keeping them separate will let the examples be updated easily, without recompiling robopaint.

techninja commented 10 years ago

Ah, not exactly what I intended to get across, but I see what you're after. These links can simply be to a wiki page that lays out any special instructions and links to file downloads on github (or elsewhere). I'll be opening a new ticket up on RP to discuss the lengths of integration both beta and future, which should cover the rest. Till, then. I call this feature request closed and will merge in the branch. Further work on this should be in new more specific tickets :)

oskay commented 10 years ago

I wonder if the scratch/snap examples should be in a separate repository?

techninja commented 10 years ago

That's certainly and sounds reasonable and very doable. Should have links from this repo's readme, etc. My one worry is that the capabilities of the released version of CNCServer and the extensions are very much linked... of course if they never really change that much that's a moot point.

oskay commented 10 years ago

Then we'll do it that way. :+1:

I'm not too worried about the issue of matching versions. If we add new API features/hooks that require a certain version of CNCserver, we can always say "note this feature requires CNCserver version X.YZ (or newer) or RoboPaint version Q.RS (or newer)." So long as we label new features as being such in the documentation (and don't remove endpoints) everything should work out quite well.

Question: Name of the project/repository? Some possibilities:

oskay commented 10 years ago

https://github.com/evil-mad/WaterColorBlocks

techninja commented 10 years ago

I was getting the RoboPaint version ready (yay!), but.. during testing I realized with the latest build of Scratch2 offline.... the extension seems crazy broken (only a few blocks actually trigger anything), and the documented API works great so... something big may have changed in the extension format :/

Can anyone else verify this is broken in the latest update/build of Scratch2 Offline?

techninja commented 10 years ago

Just verified in Windows with a brand new fresh install, things are not working as they should. There's a new button to "Add an Extension", which pops up a new UI for selecting hardware to interface to.. which is super cool! But.. It seems to be part of another update that seems to kill or seriously modify extension support :'(

https://github.com/LLK/scratch-flash/compare/v423c...v423d

oskay commented 10 years ago

Crap. Looks like they've turned off the ability to use extensions for a while... Have you tried opening an existing file from within the online version?

techninja commented 10 years ago

Tried that too, also acting up and not at all working well. I guess they did say "experimental"! :/

The is thing is that the "move to" command still works (while most others don't), which really makes me wonder why. I'm going to continue to work towards release regardless of this as it seems to be continually changing. In the end this may just require a bit of adjustment to the config file and we'll be good to go.

Still looking for more testing. If we can pinpoint the exact version number and people can still download previous releases of scratch 2 then we should be able to steer users to install an older release for the time being.

oskay commented 10 years ago

My best guess on this is that the CNCserver part of the interface is still fine, but that we may need to rewrite the .s2e file later, depending on what "security measures" they decide to take. I'd be very happy to have Snap support right now, even without Scratch support.

oskay commented 10 years ago

The park command works, in addition to the XY move. Possible issue due to presence of '/' in other commands?