proposal: golang.org/x/exp/shiny: an experimental GUI library.

[nigeltao]

Purpose.

First and foremost, this is an experiment. This is not 'the official Go GUI (Graphical User Interface) library' or the only blessed way to do things. Unlike a well-understood problem like decoding a JPEG image into an in-memory buffer, there are many possible approaches to GUIs, including existing code both under golang.org/x/mobile and elsewhere. This is simply an exploration. I've reached the point where I have an interesting if minimal demo, and I'd like to share and discuss the code, design and ideas.

Name.

There's nothing particularly deep about the "shiny" name. I just like shiny things.

The code would live at golang.org/x/exp/shiny. To emphasize what I've said above, this is an experiment, and this is a GUI library, not the GUI library, so it's not e.g. golang.org/x/gui.

It is under golang.org/x because I want to use the same code review process as everything else under golang.org/x, a process I use every working day.

Overview.

There are two layers: a lower-level window layer, and a higher-level widget layer.

For the lower layer, the primary concept is a Window. On the input side, a shiny.Window looks similar to what the golang.org/x/mobile/app package currently calls an App: you get a channel of events, such as key and mouse events.

The output side is where things diverge. x/mobile/app assumes that you paint pixels with OpenGL, either directly via the x/mobile/gl package, or indirectly via e.g. x/mobile/exp/sprite uses x/mobile/exp/gl/glutil uses x/mobile/gl. Instead, shiny has twin concepts of Buffers and Textures which map naturally to an OpenGL implementation, but also allow other implementations such as one that speaks a pure X11 protocol, without the need for cgo and OpenGL shared libraries. I assume that you could do somewhat similar things on Windows and Mac.

A Buffer is an in-memory, in-process bucket of bytes. A Texture is an opaque handle to something 'out-of-process'. A Buffer's contents can be Uploaded to a Texture, and Textures can be Drawn to the Window. As an optimization, it may be possible to Upload a Buffer directly to a Window.

For OpenGL, you can think of a Buffer as the result of a simple make([]byte, etc) call, and a Texture as an OpenGL texture. For X11, you can think of a Buffer as MIT-SHM shared memory (if applicable) and a Texture as a (server-side) Pixmap.

Uploads are 1:1 between source (Buffer) and destination (Texture) pixels. Draws can be subject to an arbitrary affine transformation (including simpler copies and scales) between source (Texture) and destination (Window) pixels. For OpenGL, affine transforms are simple shader programs. For X11, they are provided by the XRender extension.

For the higher layer, you construct a tree of widgets like buttons, text areas and containers. This shouldn't be overly surprising, except for the fact that the Go language doesn't have traditional object-oriented inheritance. The details of how that works is best explained with actual code, so I'll save it for a CL, if this proposal is approved.

The widget set is pure Go, like Swing instead of AWT or SWT. Sweating the details of a production-quality widget set is an enormous amount of work, and hitting that level of quality is not a short term goal. The primary goal is to explore what's possible in the GUI space with Go, and not necessarily to crank out a production-ready classic Win32 or Cocoa app any time soon.

A secondary goal is to drive development of the Freetype Go port - it has some basic functionality but also some long-standing known deficiencies. Writing a quality GUI text editor is again a substantial amount of work, and I don't intend to build one, but I intend that working on shiny will push the Freetype Go library to be good enough to let someone or some team do exactly that, if they had the time and inclination.

Relationship to golang.org/x/mobile.

There is a lot of overlap between shiny and x/mobile, and obviously I have contributed to both. The key difference, I think, is that the x/mobile code is rightly focused on mobile apps, where the dominant model is each app has only one (full-screen) window, whereas shiny starts more from a desktop perspective. To mix some metaphors, triangulating from different starting points (mobile vs desktop GUIs) can help avoid overfitting to local optima. In the medium term, the two should end up sharing a lot of code, but it's not obvious to me right now whether x/mobile should depend on shiny or vice versa, and it would be premature to pick a winner now. As I keep saying, there are many possible approaches, and I think it's valid to explore more than one of them concurrently.

There is also the operational concern that I would like the freedom for shiny to depend on third party libraries, such as github.com/golang/freetype and github.com/BurntSushi/xgb, but I'd also like to keep the golang.org/x/mobile tree free of such dependencies, at least for now.

Relationship to github.com/google/gxui.

I'm obviously aware of their work, but the two projects aren't strongly linked. As I've said, there are many possible approaches that are worth trying here. Longer term, we might discuss what we can take from each other's codebase, but it's premature to do that now.

Closing Remarks.

I'll repeat that this is experimental and exploratory. I just have a demo and I'd like to share it. I know that there could be a lot of excitement in the community for a 'Go GUI library', but please don't set your expectations too high at this stage, and the existance of this project should not invalidate any other Go GUI projects.

[robpike]

SGTM

[taruti]

What is the license policy of the x/ tree? Probably the FTL (or GPL) requirement is fine for most uses, but to avoid clients accidentally breaking the attribution requirement it should be clearly documented if there is a freetype dependency.

[ajstarks]

Interesting proposal. Can you sketch what a simple client program would look like?

[eaburns]

Looks very interesting. I also prototyped a UI library in Go. It shares many of the ideas that you propose.

It has two halves: the ui package (described below) and the gui package (not written) for simple widgets.

The ui package deals with Windows and Events. It's currently implemented on top of SDL2 using cgo. With ui, one can draw Buffers to Textures and Textures to Windows. However, I call Buffers image.Images and drawing them to Textures happens via draw.Draw. It also has an interesting (I think) solution to the annoying OpenGL thread context issue. In short, one calls Window.Draw with a drawing function that accepts a ui.Canvas. The Canvas has methods for actually drawing to the Window. Window.Draw ships the draw function and Canvas to the thread with the OpenGL context in order to perform all the drawing. This obviates the need for channel operations and context switching for each call to a draw primitive.

Perhaps you'd like to take a peek: http://godoc.org/github.com/eaburns/T/ui.

Also, I'd be interested in ditching my package in order to help out with Shiny. Particularly because the motivation for my ui package is to write a text editor, and you explicitly mention as a target use for Shiny.

[andrewchambers]

My vote for first demo app is an acme clone :)

[crawshaw]

SGTM

Note that x/mobile/app's OpenGL dependency will eventually be generalized to ES2, ES3, Metal, and Vulkan. But all of these share a similar GL style and shiny will need different machinery.

I would like to see the code in x/mobile/app converge in the medium term. There's a lot of subtly in cross-platform window and event management I would like to combine in one place.

[mattn]

Sounds good to me. Are you talking about mobile limited?

[aclements]

In the medium term, the two should end up sharing a lot of code, but it's not obvious to me right now whether x/mobile should depend on shiny or vice versa, and it would be premature to pick a winner now.

Rather than either depending on the other, would it make sense for them to share a common third dependency? I don't have a strong sense of exactly what would be shared between them, but, as an example, the OpenGL bindings seem like a common dependency that is neither mobile- nor GUI-specific, but is required by both.

Put another way, x/mobile is focused on mobile; it needs things like windows and OpenGL, but doesn't (itself) need a GUI toolkit. shiny is focused on GUIs; it needs things like windows and OpenGL, but isn't specific to mobile.

[owenthereal]

:+1: Is there any more info on what it is like?

[theGeekPirate]

I've reached the point where I have an interesting if minimal demo, and I'd like to share and discuss the code, design and ideas.

Mind sharing?

[crawshaw]

@aclements I believe that is what Nigel and I are both trying to say, there are just enough moving parts that's it is difficult to describe. Another attempt:

There should ultimately be one shared piece of code for creating windows and delivering events. It may have two separate interfaces, one for mobile and one for desktop, or maybe not. To some extent, it will have to know about the underlying drawing technology the user wishes to use: GL (probably all varieties can be collapsed at this level), Metal, Vulkan, or a *image.RGBA that is fed into the underlying system some other way (the X11 protocol). This is the work mostly being done in the x/mobile/app package right now, and needs to be generalized a little bit and live somewhere other than x/mobile. Both mobile and shiny will use it. In some ways this window+event package is like glfw with two differences: firstly that it's written in Go, and secondly that it doesn't necessarily require GL.

Then there are the packages for accessing the graphics system. Right now we have x/mobile/gl for GL ES 2. It has no mobile dependencies, and should live somewhere else, and can be optionally used by shiny. There are other GL packages out there that expose more features (and a fork of x/mobile/gl that works on GopherJS.) The x/mobile/app and its future generalizations will depend, probably in a "pluggable" way, on these graphics system packages.

[skelterjohn]

Consider treating the input/output (keyboard and mouse/pixels on the screen) as a completely separate issue from GUI widgets.

As an example, I created github.com/skelterjohn/go.wde to do the input/output, and github.com/skelterjohn/go.uik to do GUI widgets using go.wde.

[phonkee]

Sounds interesting, is there anything we can help with?

[andlabs]

Oh boy, this is quite a long thread with a lot to talk about. here goes!

---

@nigeltao original post

Your design is in good company: this is pretty much how GTK+ works; the name of its lower layer is GDK.

What I learned from my own UI package attempt is that the way operating systems give you events and the way Go cahnnels work don't line up fully. With a Go channel, a transaction is considered complete once the message is received, and the code that produces the message can continue operating regardless of what the receiver does. But with a traditional GUI system, the system expeccts that you handle a message fully before asking for another one, and will generate input, painting, and timer events according to that design. In practice, things may seem to work at first, but unless you have an "okay, done with this one" channel for everything, you're eventually going to fall into the trap I fell in with the first version of my package ui: needing to introduce such a channel for painting events and everything deadlocking as a result. I also don't know how x/mobile handles it.

Of course, I could be wrong and this could be a non-issue if done right; only experimentation will tell. (Until then, I've decided to separate my own package ui from Go as much as possible, with the intention of avoiding conflicts between the Go runtime and what the C world expects.) There is definitely room for experimentation here (experimentation I am very interested in).

Next, here is a very very important question. Or rather, two questions whose importance is mutual. Using "shiny.low" to refer to the lower layer and "shiny.high" to refer to the higher layer:

1. Will a shiny.low.Window map solely to a top-level window (such as the one your web browser runs in) or to an arbitrary region of the screen that has its own independent event handling?
2. Will each shiny.high.Widget have its own shiny.low.Window?

If you answer "the latter" to 1 and "yes" to 2, the GTK+ project has done this in the past and found that it doesn't really work well on X11. The link is to a video that shows a window being resized in GTK+ 2 and the massive flicker that resulted as the widgets were redrawn. (Windows and OS X don't have this problem as much because of heavy optimization and API design tricks in the former's case and a similar design to what I'm about to say in the latter.) You may find it more performant to chop up a shiny.low.Window into each shiny.high.Widget's parts and mux out events that way. GTK+ 3 has switched to this model.

Another thing is that for the widget library, you would need more than just a text rendering library; you would also need a full vector graphics library. I know the old code.google.com freetype package had one built in, but I'm not sure how comprehensive it is for the purposes of a GUI package. That also leaves open the question of theming.

Threading is another issue. Windows and OS X (I don't know about the other Unix display servers) assume that all your UI operations exist on the same thread; OS X goes even further by saying that it must be the very first thread the OS creates (though there is an undocumented function to change it; I have yet to list the full side effects). I'm aware of runtime.LockOSThread() and that might be the solution you'll want to go for. (The future of my package ui will lie in having the GUI thread be a C thread that Go knows nothing about, and that uses other methods of communicating with other threads. In my early experiments this seems to work fine as well.)

That being said, I would be happy to contribute to this project if it did exist; I could even possiby (no promises, just a thought) prototype the low level part this weekend using code that I already have in my ui and libui projects. skelterjohn has also tried something like this with his go.wde (the low layer) and go.uik (the high layer) projects; possibly using some of that code might also help.

One big question that looms in my mind is how cgo, moving garbage collection, and arrays and slices will work, since it isn't very well defined (as far as I'm aware, anyway). At some point we're going to have to feed Go data blobs to C. I should have stated this in that one issue; I'm really not against moving garbage collectors so long as the ability to safely feed large amounts of data to the C world in a well-defined manner is provided (or the rules for doing so defined).

---

@crawshaw

See what I said above about how to map a low-level window to a high-level widget. I'm not sure how this will conflict with golang.org/x/mobile's needs... And in fact, wouldn't having a single end-all-be-all OpenGL package be the logical conclusion of this?

[rsc]

SGTM.

[ajstarks]

I agree with @andlabs on having a robust 2D vector API. In fact this is more important to me than standard "widgets" like buttons; I'd like to have the ability to build my own interaction objects without being constrained to one set. Would the vector stuff go in shiny.Widget or shiny.Window?

[nigeltao]

@taruti. The golang.org/x code is licensed the same as Go itself. For example, see https://go.googlesource.com/exp/+/master/LICENSE

IANAL, but I think you're right that shiny packages should document that an (implicit) Freetype dependency, or other third party dependency, could have license implications. Such a dependency would be in the high-level layer, not the low-level one.

[nigeltao]

@ajstarks, it looks like the consensus is to go ahead with this, so the best place to show example code will be in what we call a CL or change list (a git-codereview change).

[nigeltao]

@eaburns thanks for the link. I'll check it out.

[nigeltao]

@mattn, I'm sorry but I don't understand the question. What does "mobile limited" mean?

[nigeltao]

@aclements, yes, there might be a common third dependency in the future, and packages might move, but I don't know what the ideal package 'org chart' should be yet, and part of landing code now and iterating will be to learn what that should be.

[nigeltao]

@jingweno, @theGeekPirate, yes, I will start sending out some code, as it looks like the consensus is to go ahead with this. Code reviews will be CC'ed to the https://groups.google.com/forum/#!forum/golang-codereviews mailing list. Grep for "shiny".

[nigeltao]

@skelterjohn, yes, I think we're in agreement here. There will be separate shiny packages for the lower level (key/mouse/pixels) and higher level (widgets) as I originally described.

[nigeltao]

@phonkee, not anything specific right now, but constructive feedback is always welcome.

One general issue is that I'd like shiny to work on Windows, whether via OpenGL or otherwise, but I don't really run Windows any more, and certainly haven't written any Go code for Windows. So, after I submit what I've got so far, I'd appreciate any Windows expertise, from you or any other gopher.

[nigeltao]

@andlabs, the x/mobile code tries hard not deadlock around two-way channel communication. For example, see the pump function in https://go.googlesource.com/mobile/+/master/app/app.go.

A lowshiny.Window is always a top-level window, and highshiny.Widgets will not have their own 'window'. Client-side widgets, a la GTK+3.

Re vector graphics, that would indeed be nice-to-have, and the Go Freetype port does have a Bezier curve rasterizer, but I'm not convinced that it is a must-have yet. I know that @ajstarks is keen on SVG, but e.g. I'm not aware of a lot of web-app that use SVG for their GUIs, and to echo @andrewchambers' comment, we could possibly write an acme clone without the Go equivalent of a Java2D. I'll repeat that a Go2D would be nice-to-have, but I think it's a separate package than either lowshiny (which gives you an *image.RGBA that you can draw vectors on) or highshiny (which would be irrelevant; affine transformations of buttons and text areas are out of its scope).

Re threading, we already have this concern with golang.org/x/mobile/app and OpenGL, and I think the same approach can apply here.

Re cgo, blobs of bytes and a potentially moving GC, yes, this is an open question, but not one I think limited to shiny.

[andlabs]

Fair points. I'm not familiar with how x/mobile deals with threading and two-way communication; apart from pump(), where else can I look to learn what it's doing?

What exactly do you have ready now, since you seem to be implying that you do have something? (Not asking to upload anything; just wondering.)

[nigeltao]

I don't have a better suggestion than reading the x/mobile/app source code, although it's not the simplest Go package, as it has a lot of OS-specific parts.

As for what I have ready now, I'll start sending out code reviews shortly.

[mattn]

@nigeltao I expect to use this GUI library on Windows, OSX, Linux.

[adg]

SGTM

[nigeltao]

@mattn, so do I. :-)

[ngrilly]

As you wrote in the issue description, there are multiple possible approaches to GUIs. Do you intend to draw some inspiration from the "immediate mode" (versus "retained mode") recently popularized in some way by React and now React Native?

In my experience, this is really to not think anymore about the differences between the initial rendering and the subsequent updates.

References: Immediate-Mode Graphical User Interfaces Removing User Interface Complexity, or Why React is Awesome

[egonelbre]

@ngrilly both "Immediate" and "Retained" mode rely on stateful widget/layout representation (with the exception of games, which can allow updating screen at 30 times per second). Application UI-s have a lot of pixels that can carry over directly from the previous frame, hence it's beneficial to keep state and avoid redraw. Note, that I do agree that using "transient" representation in applications and let the GUI framework worry about the stateful part makes things much easier to manage.

[ngrilly]

@egonelbre I'm not sure to understand.

Immediate mode doesn't necessarily rely on a stateful widget/layout representation, according to the usual definition of "immediate mode". Wikipedia states that in immediate mode, in contrast to retained mode, lists of objects to be rendered are not saved by the API library. The whole UI is re-rendered for each update.

I agree that the underlying implementation can maintain data structures as a kind of cache for optimization purposes. And I agree that an immediate mode API can be built above a retained mode API (this is what React does), and reciprocally.

I was wondering if this new GUI toolkit for Go could be an opportunity to explore a more "transient" API and let the app manage the state, as you wrote at the end of your comment :-)

[egonelbre]

@ngrilly - sure... I simply meant to say that for complicated regular applications you need stateful part to keep the CPU usage low. Also, even if @nigeltao implements the stateful part, the immediate mode can be implemented on top of it - although code-wise it might not be as clear as pure immediate mode.

[anacrolix]

k

[anlhord]

this made my day. ACKed by important figures and deemed consistent with go philosophy (decision of go people that duck typing an interface{} is superior to everything) is all that's needed. when it's merged to the main go repo. I'll be laughing for 2 days at such a spectacular failure.

why would we build atop let's say wayland protocol, created by those who actually understand how a modern windowing system app should operate on lowest level when we can go Not Invented Here...

[andlabs]

Um, what exactly are you suggesting @anlhord? That the idea here is to write a window system? That some vague design decision of Wayland conflicts with this? Can you elaborate that? You should also look up GDK, because it works really similarly to this (a GdkWindow provides a window system object and you use cairo to draw to it instead of a pixel buffer (even though there are pixel buffers if you want them) but other than that), has been around for 20+ years, and is used in lots and lots and lots of applications, then try to argue that it failed :) And how does Wayland do things differently from Windows, X11, and the OS X window manager anyway? At the end of the day you have a window and a pixel buffer. Also, who are you suggesting is saying duck-typing with interface{} is a good idea? Russ? Rob?!

[anlhord]

1. type Buffer interface is useless. there should be a decent 2d slice in the language by now.
2. Release() in Buffer ? Really? I thought this was a GC'd language
3. type Window interface is too high level and mixes input stuff with drawing. there is nothing to review here really

yawn

perhaps somebody can review my proposed language change at gccgo in the meantime

https://groups.google.com/forum/#!topic/gofrontend-dev/VOtbIKclOPo

[andlabs]

And Wayland doesn't mix input and drawing? Because protip: every window system I am aware of does.

Where do you see Release() in Buffer? Do you mean Release() in Texture, the OS handle wrapper? Because that needs releasing... Or did Nigel upload his proposal and I don't see it?

I have no comment on 2D slices. image.RGBA gets along fine without it. I'm still not sure what Buffer does differently.

As for your proposal, have you described it in written form anywhere (not just a code sample)?

[tucnak]

Idea itself is really promising! Imo you should definitely think of QML and how it could be applied here. My experience with QML on Qt stack was really great and I think it can be fairly integrated with Golang.

[andlabs]

@tucnak you are aware of this, right?

[tucnak]

@andlabs I obv heard about it, but haven't got enough time to deeply look into. My colleagues reacted poorly about it though.

[andlabs]

Right; I haven't used it myself, but as it stands it's the most stable and well-supported GUI library for Go, so it must have some value :S I see you (and thus your colleagues) might have specialized needs, however, so I would ask them what the current version doesn't do.

[tucnak]

@andlabs Thanks for a clue, I gonna dive into the whole thing this weekend and give it a close look.

[niemeyer]

For the record, despite being the author of Go QML, I'm in full support of a native toolkit for Go that the community gets strongly behind. The reason Go QML exists is because it is a cheap way to go fast forward towards GUI applications in Go, leveraging a complete and modern toolkit. Conversely, implementing a full graphical toolkit in Go is a major undertaking, and to be honest pushing such a project naively will most likely do more damage to the ecosystem than otherwise, as it splits attention across several half baked offerings (x/mobile? google/xgui? x/shiny? etc). I wish someone that really cares for GUIs and knows how to organize a community could take the lead on this, and stick with it for a longer while.

[ajstarks]

@nigeltao just to clarify, I'm not expecting SVG, but as you put it "Go2D". If that can cleanly ride on shiny, cool. I'll restate that doing transforms on GUI elements is useful. OpenVG is an experiment with what is possible with a simple 2D API, but it lacks things like input support and animation that I would look to something like shiny to provide.

[ianremmler]

This looks very promising. I've considered trying my hand at making a simple GUI toolkit (if it even deserves to be called that), possibly even abstracted over x/mobile and GLFW or similar. It's way on the back burner at this point, but shiny seems to be in the same spirit.

There are a few things that I hope will be at least possible, hopefully easy, and perhaps even baked in.

• Non-rectangular widgets: The painted and event-receiving areas (which may or may not be identical) need not be the bounding rectangle of a widget. I have done this in a double buffered OpenGL application by drawing a "footprint" in the back buffer in a color coded to correspond to the widget, for example.
• Widget-local coordinate system: It sounds like the proposed affine transformations would be post-render. I would like a natural way for a widget to have it's own extents, origin, (maybe rotation and shear) that apply when rendering to the buffer, as well as the ability to convert to and from other widget, window, and screen coordinate systems.
• Aspect ratio constraints: A widget may have a fixed aspect ratio and weighted values specifying where to position the widget along the less constrained axis.

I'm really looking forward to see where this goes.

[anlhord]

@andlabs this is the shiny I found:

https://go-review.googlesource.com/#/c/12568/1/shiny/shiny.go

[nigeltao]

@ngrilly, @egonelbre, "immediate mode" is an interesting concept, and worth exploring, but the shiny widget library will take a traditional "retained mode" approach. As I said, there are many possible approaches, and I'm more than happy for somebody else to try an "immediate mode" substitute for the higher-level, that re-uses the lower-level. Or it could re-use the higher-level. I haven't used React myself, but IIUC, it re-uses the browser's DOM, and that DOM takes a traditional "retained mode" approach.

@anlhord, if you feel that there is nothing to review here, then you are free not to review it. As for your proposed language change, please file a separate proposal for that, and we can discuss it there.

@anlhord, @andlabs, as for Release, yes, some resources can be explicitly released. Just like the io.Closer interface exists for a reason, and we don't rely on garbage collection to close your database connection, io.Pipe, or gzip.Writer.

@andlabs, @anlhord's proposal was previously discussed at https://groups.google.com/forum/#!topic/golang-dev/_V7_Be-AA38, but this is off-topic here, and I have asked @anlhord to file a separate proposal.

@tucnak, by QML I presume you mean a declarative way to specify a widget hierarchy. It's an interesting idea, especially for people wanting to build GUIs visually, in IDEs, but we're a long way from there yet. In any case, it seems like something that could be a separate package that used (high-level) shiny, rather than necessarily being part of shiny.

@niemeyer, yes, I would like to repeat that shiny does not aim to be 'the official GUI package'. It is an experiment, although it will hopefully influence golang.org/x/mobile/app in the medium term.

@ianremmler, I would phrase it a potentially transparent widgets, instead of non-rectangular widgets, as I'd still like the concept of a bounding box (a rectangle). Each widget will also have its own relative coordinate system that is a translation of the window coordinate system, but not an arbitrary affine transformation (sorry @ajstarks), at least not in the near term. As for fixed aspect ratios, layout will indeed be an interesting problem.

[andlabs]

All right then. Argumentative posts aside, glad to see that a CL is up now; I'll look through it sometime in the next day or two and make my own comments as well :)