WebView window for HTML user interfaces like pie menus to Slate.

[SimHacker]

Slate is great, and I'm starting to learn my way around the code so I can work on it.

I would like to implement pie menus for Slate in JavaScript/HTML. It would be great to be able to do something like this:

http://www.donhopkins.com/home/movies/TabWindowDemo.mov

To that end, I am playing around with exposing the WebView that Slate uses for its JavaScript interpreter.

Instead of making a hidden WebView in JSController.m, I am making a window in MainWindow.xib, plugging it into SlateAppDelegate, and grabbing it from the JSController.m initialization code.

So Slate's JavaScript interpreter can create HTML user interfaces in the WebView in the window.

I need to write some wrappers to make it possible for JavaScript to control the window, show, hide, position, bring to top, etc.

And also it would be great to figure out how to make a transparent window that only shows through where HTML draws, leaving the background of the web page transparent.

I'm not sure how easy that is, but it's worth doing if possible.

It would also be great to flesh out the accessibility and speech recognition APIs, and make it possible to write all kinds of intelligent application automation and integration scripts, bots, with nice HTML user interfaces in JavaScript. Take a look at what Dragon Naturally Speaking has done with Python:

https://code.google.com/p/dragonfly/

Here are some thoughts and opinion I wrote about window managers, stuff I've done in the past with older systems, and the direction I would like to take Slate, from a discussion on Hacker News:

https://news.ycombinator.com/item?id=5861229

X-Windows is a dead end (and has been for a long time), and X window managers are as complex and dug into a pit of pointless complexity as the Space Shuttle.

There are a lot of good reasons to have a scriptable window manager, with a scripting language tightly integrated with the window system.

When I started hacking X10, I wanted to implement pie menus for it. There was a cheezy little library called "XMenu" that was intended for other programs to use to make menus, which I adapted for pie menus, but no programs actually used it, and it was not very configurable (no menu definition language, just an api), plus its model and api was not well suited for pie menus. And once you have menus, you need a way to make selecting their items actually do something!

So I adapted the "uwm" window manager to support pie menus, so you could define your own pie menus and linear menus in your .uwmrc file. Because a window manager could really benefit from pie menus: lots of the commands are spatially oriented and can be arranged in appropriate mnemonic directions, and they have a fixed set of common window management commands that can be thoughtfully arranged into a set of efficient pie menus, as well as a menu definition language to enable users to create custom menus for running apps, connecting with remote hosts, etc.

I wanted to experiment with different styles of menu tracking, mouse ahead, display pre-emption, feedback, screen edge handling, rolling between pie menus and linear menus, invoking window management commands, and stuff like that, and uwm's C code was not very flexible in that regard.

We also wanted to perform a controlled experiment comparing the selection time and error rate of pie menus and linear menus, so I needed to program the window manager to administer the experiment, define and randomize the menus, prompt the user to select from pie and linear menus, time the menu selections, report the statistics, etc. I had been using Mitch Bradley's "Forthmacs", which was a very nice Forth system for the 68k. (It eventually evolved into the Sun 4 Forth boot ROMS, OpenFirmware, and the OLPC boot ROMs). It was capable of dynamically linking in C code (well not dll's or shared libraries, but it would actually call the linker to relocate the code to the appropriate place in Forth's address space, read in the relocated code, and write Forth wrappers, so you could call C code from Forth, pass parameters, etc -- SunOS 4.2 didn't have relocatable shared libraries or light weight threads back then, so Forth had to do a lot of the heavy lifting to plug in C code. But Forth was a really great way to integrate a library into an interactive extension language, call it from the Forth command line, build on top of C code in Forth, call back and forth between C and Forth, play around with it from the Forth command line, etc).

So I refactored UWM a bit so I could plug it into Forth and rewrite the main loop and event tracking code in Forth, and write higher level menu tracking and window management code in Forth, and administer the experiment in Forth. For example, you could pick up a window and throw it, and it would fork off a light weight Forth task to make it bounce around on the screen! See the "fling" code:

http://www.donhopkins.com/home/archive/piemenu/uwm1/hacks.f

I didn't realize it at the time, but it foreshadowed what I would later do with NeWS and PostScript.

It was very nice to have a window manager that I could program in Forth, because that was my scripting language of choice at the time, and I could integrate it with C code and Xlib. But then I learned about SunDew aka NeWS (I read the SunDew article in Methodology of Window Management: http://www.chilton-computing.org.uk/inf/literature/books/wm/... and saw Gosling give a talk about it at SUG).

I visited James Gosling, David Rosenthal, Owen Densmore and Mitch Bradley at Sun, who showed me NeWS, and gave me a beta copy to play around with. I learned to program PostScript, and switched from Forth to PostScript for programming windows managers, pie menus and user interfaces. PostScript is a much better language for that than Forth, not only because it has a nice graphics model, but because it's a lot more like Lisp or Smalltalk than Forth, and NeWS extended it with an object oriented programming system that used the dictionary stack to implement inheritance, with which we built user interface toolkits and entire apps, like a visual PostScript programming environment and debugger:

http://www.donhopkins.com/drupal/?q=node/97

NeWS was architecturally similar to what is now called AJAX, except that NeWS coherently:

• used PostScript code instead of JavaScript for programming.
• used PostScript graphics instead of DHTML and CSS for rendering.
• used PostScript data instead of XML and JSON for data representation.

Some years passed, NeWS was canceled, some other stuff happened, the web came along, the Mac stopped sucking so badly and adopted Unix and NeXT Step, the term AJAX was coined to describe what NeWS was doing all along and what X refused to do for political reasons, and JavaScript took over the world.

So back to real-time: I just recently installed and read the source code of a nice Mac tiling window manager called "Slate": https://github.com/jigish/slate . It look very clean and well designed and even simple, and I think it has a lot of potential! You can program it in JavaScript (it makes an invisible web browser and injects a JavaScript API to the application into it, and delivers events to JavaScript when windows move or change state, calling JavaScript handlers that can call back and manipulate the windows to achieve tiling window layout, and so on and so forth), and it exposes some of the Mac's accessibility API to JavaScript, which brings a lot of power to the table.

Slate is very powerful and flexible, and I'd like to teach it to support pie menus (and other user interface widgetry, maybe by simply popping a web browser up in a window), so you can define pie menus in JSON for window and application management or whatever you want, and script their behavior in JavaScript, and manipulate windows and apps via the accessibility API from JavaScript. Maybe throw in an http server so your window manager can surface web pages and interactive user interfaces in web browsers (that would be a good way to implement pie menus in JavaScript and html, instead of coding them up in Objective C).

I think that Slate takes exactly the right approach with JavaScript, and I hope to get my head around it and do some work with it, like making pie menus for window management, app launching and controlling the accessibility API.

[SimHacker]

Well I got it to use the web browser in the window of the ui file, so I can display html in it! I had to comment out the line: [[webView mainFrame] loadHTMLString:@"" baseURL:NULL]; from JSController.m because that was happening asynchronously and stomping on the stuff I put into the window. And I was able to make the web view and window float on top and be transparent except for the html content like this:

SlateAppDelegate appDelegate = (SlateAppDelegate )[NSApplication sharedApplication].delegate; NSWindow *browserWindow = appDelegate.browserWindow; browserWindow.opaque = NO; browserWindow.level = kCGPopUpMenuWindowLevel; browserWindow.backgroundColor = [NSColor clearColor];

webView = appDelegate.browserWebView; webView.drawsBackground = NO;

So now I can start coding up some pie menus in javascript, and fill in the rest of the api I need to show and hide and resize and position the browser window, and capture the mouse clicks of course, to implement pop-up pie menus for Slate! And whatever else is convenient. Maybe it would work to make the browser window big enough and positioned over the entire set of screens, and draw stuff in absolute screen coordinates, to implement any kind of overlay feedback (window dragging rubber bands, etc). Don't know if it would perform well, but that would make it easier to do a bunch of different user interface widgetry at the same time all in the same webview without them interfering with each other.

[SimHacker]

Making the transparent browser window big enough to cover both screens works like a charm!

One thing I don't know if it's currently possible is to bind mouse clicks to Slate functions -- it only seems to use the names of keys, not mouse buttons.

But even without that feature, I think an easy way to pop up pie menus in response to click events is to draw some html over or adjacent to the window that you can click on to get the pie menu, and handle all the input tracking in the web browser itself.

Mouse events that happen over non-transparent parts of the web browser are delivered to the browser, so it should work just fine.

But it would be nice for the purposes of pie menus to be able to capture mouse clicks (with shift modifiers or without) on windows, and even conditionally pass them on to the app if they're not interesting. Any idea how to do that?

I've been looking at the accessibility stuff, to figure out what kind of stuff Slate can make applications do, but I haven't dug really deep yet.

Does the JavaScript/Objective C bridging magic automatically let you get and set properties and call methods on the Objective C Accessibility objects from JavaScript, and drill down to their children?

Or would more wrappers need to be written to support that?

There's a lot of surface area to the accessibility API, so it would be great if the bridging was automatically handled, so I didn't have to write millions of wrappers.

Although it might be necessary to define some of the constants from the .h files in JavaScript at least.

I am excited about the possibilities of being able to implement Slate user interfaces in HTML/JavaScript!

[SimHacker]

The windowMoved and windowResized notifications to JavaScript were passing in an app wrapper instead of a window wrapper, so I fixed RunningApplications windowChanged to check for kAXWindowCreatedNotification, kAXWindowMovedNotification, kAXWindowResizedNotification, kAXWindowMiniaturizedNotification, kAXWindowDeminiaturizedNotification, kAXMovedNotification and kAXResizedNotification and in those cases call runWindowJSCallbacks (which sends a JSWindowWrapper parameter instead of a JSApplicationWrapper parameter) instead of calling JSController runCallbacks:payload: (which sends a JSApplication parameter), and also I added notification for kAXWindowMiniaturizedNotification and kAXWindowDeminiaturizedNotification.

I have punched holes in JSWindowWrapper to expose its AccessibilityWrapper, and in AccessibilityWrapper to support manipulating its AXUIElementRef window, including the methods actionNames, performAction:, attributeNames, getAttributeValue:, getAttributeValueCount:, getParameterizedAttributeValue:parameter:, isAttributeSettable:, setAttributeValue:value:, postKeyboardEventKeyChar:virtualKey:keyDown:, and implemented webScriptNameForSelector: and webScriptNameForKey: so they replace :'s in selectors with _'s, so you can call the methods easily from JavaScript. The basic stuff seems to work but there is some more glue code to write, but I think I'm well on the road to bridging a pretty useful subset of the Accessibility API to JavaScript!

[SimHacker]

Do windows have some kind of constant ID I can use that does not change over time? It would be nice to expose that, so I can keep track of them in JavaScript structures without hanging onto wrapper references. Using window titles is kind of sketchy.