Animations

[georgefst]

In #1164 we add proof-of-concept support for creating animations with Primer. We have a new ADT, Picture, representing 2D images, and a primitive function animate : Integer → (Integer → Picture) → Animation, where Animation is a new primitive type. The arguments are a duration, in seconds, and a description of how to render each frame. This is inspired by the API of pure functional animation libraries such as Haskell's Gloss.

This issue outlines the work that remains. Some of these points will be expanding in to their own issues in due course.

(I've folded in one or two important points from @brprice's notes on this topic, but there's more there on inspirations, alternative APIs and example animations we'd like to support)

Implementation

The one major current limitation is around the kinds of functions which we can pass as the second argument to animate without the evaluator getting stuck. This is explained further in #1169.

We implement Animation internally as a base-64 encoding of a GIF, for ease of use by clients. The conversion from List Picture to GIF is implemented via the diagrams library with the Rasterific backend. This is probably not what we want to use beyond the prototyping stage:

• It's likely to be relatively slow. While we haven't stress-tested thoroughly, we have hardcoded the output to low resolution and frame rate in order to keep the GIF small for quick feedback. The official diagrams manual actually warns against using the library for real-time processing for performance reasons.
• This sort of one-shot processing won't be suitable when we want to extend from simple animations to outputs which depend on real-time interaction such as mouse and keyboard input (see below).
  • While we could still use diagrams to generate one frame at a time, performance is likely to become even more of a concern with the need for constant back-and-forth between client and server, though this part would be mitigated by compiling the Haskell code to WASM, as we intend to eventually for other parts of the library (#1104).
• Given that our main goal is to render these graphics in a web app, it might be preferable to stick to vector graphics and leave rasterization up to the browser. This should allow us to support more complex animations.

We may well be better off leaving interpretation up to clients, e.g. implement a renderer in JS for primer-app. This slightly contradicts our "dumb frontend" philosophy, but it shouldn't be too bad, since mapping from Primer's Picture ADT to SVG is very straightforward in principle.

Some more minor issues have been left out of the initial PR due its proof-of-concept nature (and particularly the hope that we might ditch the new primitives completely in favour of the projections-based approach discussed below):

• Testing could be better. In particular, we haven't added Animation to the Hedgehog generators, i.e. genPrimCon in both Raw and Typed modules.
• The module structure is awkward (primarily Primer's own standard library, though this also affects the structure of the Haskell library implementing it). Picture is in the Primitives module despite being a non-primitive type. This is because the animate primitive function needs Picture, which in turn depends on the primitive type Int.
  • Ultimately, we need to split things up so that primitives are not all in the same module. Otherwise we'll likely have similar issues in future.
  • This should probably wait until we have given our standard library more thought in general (#344, #766).
• We might want to factor the support in to a separate package, since diagrams is quite a heavy dependency. Though this could be difficult, since our primitives aren't really extensible.
• For now, there are a lot of lossy fromInteger conversions going on in exprToDiagram due to diagrams' use of Double. Ultimately we'll want to use floats, as discussed below.

Student-facing API

• The Animation type is currently unique in that we can't directly construct it. It only appears as an output of a function. This works fine, but we should at least consider disabling pattern matching for it (currently we insert a wildcard pattern and don't allow any others).
• The Picture type doesn't have all the flexibility students are likely to want:
  • We'd benefit from having named fields (#1108) for clarity, e.g. red, width, radius.
  • Some constructors (Compound) could have friendlier names.
  • Colours:
  • Making Colour a separate type might be easier to work with.
  • Field values are expected to be out of 255, which isn't obvious. One would usually use (products of) bounded integer types, but we've chosen not to have any in Primer.
  • More constructors:
  • Polygon : [(Int, Int)] -> Picture
    • Rectangle, Triangle etc. could be implemented in terms of this.
  • RegularPolygon : Int -> Picture
  • Sprite? This would probably have to be a primitive with it's own logic for editing.
  • Lines, curves...
  • The whole API uses integers because that's what we have to hand. But floats (#1171) would be more conventional once those are available. Using integers would quickly become unusable, due to not supporting square roots, trigonometric functions and others likely to be useful in creating pictures.
  • Eventually, it would be cool to have other more domain-specific APIs built on top of the core one, such as a chart-drawing API like (Number -> Number) -> Opts -> Picture.
  • We should be able to set line colours as well as fill colours. We have chosen not to show shape outlines at all for now, even though diagrams makes it very easy, in order to keep the API simple.
• We hardcode certain values right now for API simplicity (as well as performance):
  • Canvas size is 160x90. It would be friendlier to base this on the largest frame, but then we can't easily tell students that rendering is taking ages because their GIF is too big. That might be less of an issue if we change our implementation. Alternatively we could just scale down to something sensible.
  • Output is at 10 frames per second. It might be acceptable to have this remain fixed but at a higher value like 30 or 60. And once we have floats, we'll likely wish for the time input to be in seconds rather than frame numbers, so students wouldn't have to think about this.
• We should possibly insert an outline around the animation. With it currently being transparent, it's impossible to see the boundaries.
• It might be nice to include functions like beside (from diagrams) for easier composition. While we're using diagrams ourselves, this would be trivial to implement as a constructor of Picture. But implementing it as a Primer function would allow students to see how it works.
• Currently all our GIFs have the "looping forever" property set, since this is the only option that really works for our frontend. Ideally we'd have play/pause/fast-forward/rewind controls instead.
  • Note that if we do drop this, we may wish to add an extra frame at the end of animations: see https://github.com/hackworthltd/primer/pull/1164#discussion_r1392898708.
• We may wish to output sound alongside images.
• We'll want to add some sort of interactivity, such as via keyboard and mouse. In Gloss terms this means adding a play function, rather than just animate.
  • Note that play (or even the intermediate simulate) would be a polymorphic primitive (in the world/state type), which is something which may take a little work to support, since we had previously assumed we'd never want such a thing. See source comment "Since no primitive functions are polymorphic [...]".
  • It might be that a Gloss-esque API isn't the right fit here. Gloss can be awkward to use for complex UIs, since interaction and display are too decoupled, making it difficult to break out components.

Alternative approach via projections

After the initial work on #1164 was completed, we discussed the possibility of dropping the primitive Animation type in favour of a projections system, which we have long wanted anyway for lists and text. Our output would then just be an alternative view of List Picture. It would also be useful to be able to project a single Picture as a static output.

This feels more principled than the current, rather ad-hoc approach where we have a primitive type with no support for directly creating it (i.e. it only appears as the result of a primitive function), and special support for displaying it (whereas every other type at least looks like a normal tree node).

The fact that we'd require no primitive function or type at all to support animations would also actually solve several implementation issues, particularly #1169.

A few open questions:

• Where should we then put information like frame rate and looping?
• What about interaction/games? Maybe a projection over some new record type, which would have the same signature as the equivalent play primitive?
• What is the backend/frontend split here? To what extent would supposedly-dumb clients need to have knowledge that animations (and other project-able types) require special handling?

[dhess]

Thanks for writing this up. I do feel pretty strongly that leaving Primer as a pure language and writing interpreters in frontends is probably the best approach of the ideas discussed thus far. The only drawback is that we'd likely need to implement those interpreters in TypeScript, but on the other hand, that probably makes sense anyway, as there are many amazing libraries for writing animations, games, etc. in the JavaScript/web ecosystem.

[dhess]

See #1186, which removes diagrams, as that is blocking Wasm support.

I think the next step in this process is to change the Animation type from a primitive to just an ordinary algebraic type, which presumably will obviate any need to solve #1169.

[dhess]

I just found out about this project, which may provide some inspiration:

https://gitlab.com/TristanCacqueray/animation-fractal