Open jimbojw opened 1 year ago
Some alternative API ideas.
Expose the SpriteView object and allow direct manipulation, including the ability to make duplicates.
const sprite = scene.createSprite();
// Expose the Sprite's underlying SpriteView.
const spriteView = sprite.view;
// Changes to the SpriteView trigger eventual synchronization and
// drawing. No need to set a callback.
spriteView.FillColor = d3.color('red');
// The SpriteView's duplicate() method returns a new view with identical values.
const duplicateView = spriteView.duplicate();
// Changes to the duplicate have no immediate effect.
duplicateView.FillColor = d3.color('navy');
// To apply the duplicateView attributes, assign them from the duplicate.
spriteView.assignFrom(duplicateView);
// To apply the duplicateView attributes later, hang them off of a callback.
sprite.nextSync(() => {
spriteView.assignFrom(duplicateView);
});
sprite.transitionFinished(() => {
spriteView.assignFrom(duplicateView);
});
Optional: For convenience, have the Sprite implement the SpriteView API and pass through any calls.
const sprite = scene.createSprite();
// These two are identical in effect.
sprite.TransitionTimeMs = 10;
sprite.view.TransitionTimeMs = 10;
Or, get rid of the Sprite/SpriteView dynamic entirely:
const sprite = scene.createSprite();
// Changes to the attributes trigger eventual synchronization and drawing.
sprite.FillColor = d3.color('red');
// The Sprite's duplicate() method returns a detatched Sprite with all the same
// attribute values.
const duplicateSprite = sprite.duplicate();
// Changes to the duplicate have no immediate effect, because it's detatched.
duplicateSprite.FillColor = d3.color('navy');
// To copy values from one Sprite to another, use assignFrom().
sprite.assignFrom(duplicateSprite);
// The copying of values can be delayed by hanging it off a callback like
// nextSync() or transitionFinished().
sprite.nextSync(() => {
sprite.assignFrom(duplicateSprite);
});
sprite.transitionFinished(() => {
sprite.assignFrom(duplicateSprite);
});
Since the duplicated Sprite would not be attached immediately, maybe it would make sense for the Scene's createSprite() method also to not attach sprites by default. In that case, the API user would have a chance to make any changes before attaching.
const sprite = scene.createSprite();
// Sprite is NOT attached, so these changes trigger no action.
sprite.FillColor = d3.color('salmon');
// Then, when ready, have the Scene attempt to attach the Sprite.
scene.attachSprite(sprite);
The above code seems to imply that a Sprite could be attached to one Scene or another. To support that, maybe we should get rid of the factory constructor and use the new
operator instead.
const firstScene = new Scene();
const secondScene = new Scene();
const sprite = new Sprite(); // Detached.
firstScene.attach(sprite);
// What should happen here?
secondScene.attach(sprite);
This raises the question of what should happen if a sprite is attached to a different scene while already attached to a scene. In the DOM, when you appendChild()
, the target element is removed from its previous parent if any. That would be reasonable here. It could also be reasonable to throw an error, thereby making the API user decide whether they want to detatch from the first scene, and signaling their error if that was not intended.
Status: DRAFT
This design document describes a new, simplified API for creating and modifying Megaplot sprites.
Sprites can be created using the new
operator. These will not be automatically attached to a scene.
const sprite = new Sprite(); // Unattached.
Using a scene's createSprite()
factory method, the created sprite will be automatically attached to the scene asynchronously.
const sprite = scene.createSprite(); // Auto-attach.
sprite.FillColor = [255, 255, 0, 1]; // Will be included in initial render.
Note here that the asynchronous attachment may be in the same animation frame if the creation was done in a callback handled by Megaplot. If the creation code was executed outside of a Megaplot context, then the earliest that the sprite will be auto-attached is the next animation frame.
The next section discusses explicit and automatic attachment in more detail.
A sprite can be attached to a scene. A sprite that is not attached to a scene is unattached. This is comparable to how DOM elements may be connected. (See Node.isConnected)
When a sprite becomes attached to a scene with capacity to receive it, the sprite's attributes will be flashed to the scene's data texture as-is, with no transition.
const sprite = new Sprite(); // Unattached.
sprite.FillColor = [0, 255, 0, 1]; // Initially green.
sprite.Sides = 3;
sprite.SizeWorld = [1, 1];
sprite.TransitionTimeMs = 250; // Ignored for initial attachment.
scene.attachSprite(sprite); // Values captured immediately.
sprite.FillColor = [0, 0, 0, 1]; // Fade to black over 250 ms.
As long as a sprite is attached to a scene, changes to the sprite's attributes will be reflected in the rendered scene eventually. Transitions between attribute values will take the amount of time specified in TransitionTimeMs
. (See Transition times section)
A sprite can be attached to more than one scene simultaneously. This feature can be used to implement mini-maps. For example:
const scene = new Scene({container: someBigDiv});
const minimap = new Scene({container: someSmallDiv});
minimap.scale.x *= 10; // Zoom out 10X.
minimap.scale.y *= 10;
const sprite = new Sprite(); // Unattached.
sprite.FillColor = [0, 128, 128, 1]; // Teal.
sprite.Sides = 2; // Square.
sprite.SizeWorld = [1, 1];
scene.attachSprite(sprite);
minimap.attachSprite(sprite);
Calling attachSprite()
to attach a sprite that is already attached to that scene with throw an error:
scene.attachSprite(sprite);
scene.attachSprite(sprite); // Throws.
To detach a sprite from a scene, call the scene's detachSprite()
method.
scene.detachSprite(sprite);
Calling detachSprite()
with a sprite that is not attached to that scene will throw an error.
You can check whether a sprite is attached to a given scene using the sprite's isAttached()
method.
const scene = new Scene({/*...*/});
const minimap = new Scene({/*...*/});
scene.attachSprite(sprite);
sprite.isAttached(scene); // True.
sprite.isAttached(minimap); // False.
Use a sprite's assignFrom()
method to copy all attribute values from another sprite.
sprite.assignFrom(someOtherSprite);
This feature can be used for templating. The following example shows how to fade sprites in over time.
const template = new Sprite(); // Unattached.
template.FillColor = [64, 0 ,0, 0]; // Crimson, zero opacity (transparent).
template.Sides = 2; // Square.
const sprites: Sprite[] = [];
for (let x = 0; x < 4; x++) {
const sprite = new Sprite(); // Unattached.
sprites.push(sprite);
sprite.PositionWorldX = x;
sprite.assignFrom(template); // Does NOT overwrite PositionWorldX.
scene.attachSprite(sprite); // Explicit attachment captures values.
sprite.TransitionTimeMs = 250; // Triggers update.
sprite.FillColorOpacity = 1; // Fade in over 250ms.
}
Note that only attributes that have been set on the template sprite are assigned during the assignFrom()
operation. In the preceding example, PositionWorldX
is not overwritten.
Care must be taken when using the scene's createSprite()
method to ensure that attribute values are captured at the intended time.
for (let x = 0; x < 4; x++) {
const sprite = scene.createSprite(); // Will auto-attach on assignment.
sprites.push(sprite);
sprite.PositionWorldX = x; // Triggers attach next frame.
sprite.assignFrom(template);
sprite.TransitionTimeMs = 250; // Ignored on attachment.
sprite.FillColorOpacity = 1; // UNINTENDED: Will start opaque!
}
To produce the intended behavior, use the scene's attachSprite()
method to capture values synchronously:
for (let x = 0; x < 4; x++) {
const sprite = scene.createSprite(); // Will auto-attach next frame.
sprites.push(sprite);
sprite.PositionWorldX = x;
sprite.assignFrom(template);
scene.attachSprite(sprite); // Explicit attachment captures values.
sprite.TransitionTimeMs = 250; // Ignored on attachment.
sprite.FillColorOpacity = 1; // UNINTENDED: Will be opaque!
}
The next section discusses the effects of constrained memory on sprite attachment.
Due to memory constraints, scenes have limited sprite capacity. Once the scene's capacity has been reached, it may or may not allow additional sprites to be attached. Whether a scene accepts additional sprites is controlled by the acceptWaitingSprites
scene setting.
const scene = new Scene({
container: someDiv,
acceptWaitingSprites: true,
});
When acceptWaitingSprites
is false
(the default), attempts to attach additional sprites beyond capacity will throw errors. This default avoids the situation where the developer expected a sprite to be visible, but it's not due to capacity.
When acceptWaitingSprites
is true
, additional attached sprites will be accepted, and will wait until memory is available to render.
The rationale for allowing scenes to queue waiting sprites is transitions. It's common for exiting sprites to have an exit transition, for example fading out. At scale, using an abstraction like a selection may cause a surplus of sprites during binding, as some sprites are leaving the scene and others being initialized.
After a sprite attribute is set, its rendered representation will transition to the new value over time. Attributes are grouped into categories, each with their own transition time:
BorderTransitionTimeMs
BorderPlacement
BorderRadiusPixel
BorderRadiusRelative
ColorTransitionTimeMs
BorderColor
FillColor
PositionTransitionTimeMs
OrderZ
PositionPixel
PositionRelative
PositionWorld
SizeTransitionTimeMs
GeometricZoom
MaxSizePixel
MinSizePixel
SizePixel
SizeWorld
ShapeTransitionTimeMs
Sides
ShapeTexture
As a shorthand, setting a sprite's TransitionTimeMs
will set all of the attribute group transition times to the same value.
When TransitionTimeMs
is read, it will return the value of PositionTransitionTimeMs
.
I think this makes sense overall. Two questions:
Sorry for the delayed response. These are good questions.
Can you say more about when, after setting an attribute, the attributes actually start being represented in the visualization? Is it always the next animation frame?
I take your question to mean, "how soon after an attribute is set will the rendered canvas reflect this change?".
Short answer: in practice it will rarely be the next frame. It will typically be within the next 2-3 frames. It will occasionally take 4 or more frames.
Full answer: let's consider what should be a typical example usage of the new API once implemented. Using this example we can gauge how many frames it will take before pixels reflect API invocations.
Say we use a scene to make a sprite:
// 1. Create sprite from Scene's createSprite() factory.
const sprite = scene.createSprite();
If you were to do nothing else, no pixels would change as a result of this. The sprite initially has zeros for all attributes. No size. No opacity.
The scene is content to wait indefinitely. Only once you've set at least one attribute (or explicitly attach the sprite) will the scene take any action regarding the rendering of the sprite.
So let's set some attributes:
// 2. Set one or more sprite attribute.
sprite.FillColor = [0, 128 ,128, 1]; // Teal, opaque.
sprite.Sides = 2; // Square.
sprite.SizeWorld = 1; // 1x1 world unit in size.
Once you set any sprite attribute (FillColor
in the example above), the Scene will schedule an attachment task for the next frame. The attachment task will lock in any attribute values that you've specified up to that point. Those values will be the initial state of the Sprite on its first render.
From that point, it takes two more internal steps to get the data values drawn to the canvas. First, the captured attribute values need to be flashed to the data texture (sent to the GPU) in a task called a texture sync. Second, the Scene has to be rendered in a draw task.
Internally, the scene carries out these internal steps in batches. The scene will attempt to perform as much work as possible each animation frame, running up to one instance of each of these batched operations, with the caveat that work will cease once maxWorkTimeMs
is exceeded (which defaults to 20ms).
So in an ideal situation, the scene will be able to carry out the texture sync and draw steps in the same frame. But if a draw task is already scheduled, then the two steps will be out of order. The draw task will run first, then sync the texture. The next draw will have to wait for the next frame.
In this example so far, not including the execution context frame, it will have taken 1, 2 or 3 frames before a teal pixel is rendered to the canvas. Exactly how many is a function of whether and which tasks may already be scheduled in the pipeline and their order.
It gets sligtly more complicated when we introduce transitions. Say we wanted to transition our teal square to a red circle over a 1/4 of a second:
// 3. Transition to red circle.
sprite.TransitionTimeMs = 250; // 1/4 second transition.
sprite.FillColor = [255, 0, 0, 1]; // Red, opaque.
sprite.Sides = 1; // Circle.
Before we do this, either we need to have explicitly called scene.attachSprite(sprite)
to capture the values, or a frame needs to have elapsed so that this would be called automatically. If neither of these have occurred, then the previous FillColor
and Sides
values will be lost (having never been captured, sync'd and drawn).
Irrespective of whether the attachment was explicit or automatic, the code above sets up a cascade of necessary steps:
As before, in an ideal case, these steps could all be accomplished in the same animation frame. In practice, a draw call has probably already been scheduled, and the rebase and texture sync may happen out of order. So again it could take 1, 2, or 3 frames before the rendered canvas first shows a pixel transitioning from teal to red.
As the number of sprites in the scene increases, the more likely it becomes that the whole pipeline takes longer than maxWorkTimeMs
to complete. When this happens, queued tasks are re-queued to run again in a future animation frame. The result is that it could take more than 3 frames for your attributes to be rendered if there's contention.
Are there any performance implications, good or bad, for this new API?
I expect that using the new API:
In most cases, I expect API users will be able to perform their updates synchronously, even for large data sets (hundreds of thousands of data points). If API users find that they need to chunk work, the existing .enter()
/.update()
callback mechanism offers scheduling.
So overall, I think this will probably be faster, and will probably use less memory overall (although more of the memory will be handled by Megaplot and less by the API user).
The objective of this issue is to simplify the Sprite API, making it easier to use and less likely to produce unexpeted or erroneous behavior.
Sections:
Summary of proposed Sprite API
Set attributes directly on Sprite object without having to schedule a callback (replaces the
.enter()
method):Set a singleton callback to be invoked after the next time values are flashed to the GPU (supersedes
.enter()
/.update()
dynamic):Set a singleton callback to be invoked after the Sprite has finished its transition (supersedes
.exit()
):A detatched sprite can be attached to the scene by calling the
.attach()
method. Attaching an attached sprite or detatching a detatched sprite has no effect. User can checkisAttached
to see its current state.If there is insufficient capacity to accommodate attachcing a sprite, then it waits in an intermediate state until space is freed up. During this time,
isAttached
will be false, butisWaiting
will be true.If one or more attributes have been changed, but the data has not been flashed to the GPU, the
isDirty
property will be true.Background: Issues with the current Sprite API
In this section, we'll walk through a fairly basic usage of the current Sprite API. This will demonstrate the interplay between memory management and timing which lead to unexpected and unintuitive behavior.
The current Sprite API requires users to use the
.enter()
,.update()
and.exit()
methods to schedule callbacks. Each callback receives aSpriteView
object through which the user can specify attributes likePositionWorld
,FillColor
, etc. Here's an example:To reduce the amount of code needed to set attribute values, the attributes offer destructuring assignment. The above code can be rewritten as follows:
The downside of this approach is that there are now two, short-lived array objects which are created:
[0, 0]
and[255, 127, 0, 1]
. For a single sprite, this is not a problem, but at scale, creating these kinds of objects causes noticeable performance degradation, and precipitates more frequent garbage collection.To reduce this memory thrashing, users can move their objects outside of the callbacks for reuse, like this:
This works, but the next problem is that the values are often dynamic in a visualization context. So rather than constants, the position and color would change over time:
Notice here that we're using
.update()
for the second set of changes. Megaplot guarantees that the.update()
callback will be called after the.enter()
callback. But both callbacks are scheduled and invoked asynchronously.In our running example, this means that the
.enter()
callback may not have been run by the time later that theproperties
values are changed. That is, the sprite may never have had itsPositionWorld
set to[0, 0]
and itsFillColor
set to[255, 127, 0, 1]
.To solve this problem, the API user must explicitly capture the intended values for each callback. This can be done in any of several different ways. The following example promotes the initial state back to constants.
Overall, the behavior and subtle interaction between memory and timing makes the existing Sprite API difficult to work with and reason about.
Walkthrough of proposed Sprite API
The proposed Sprite API would be much simpler. Rather than requiring the API user to make changes inside of callbacks, the user sets properties directly on the sprite object:
This solves the problem of unexpected late execution. As soon as the setter is called, the values are read and stashed for eventual rendering. This is illutrated by adding a second instance of setting the values:
While this solves the problem of unexpected late execution, there's another issue. Rendering is necessarily asynchronous, so it may be some time between when the attributes are set and the sprite is actually drawn to the screen. So while the sprite had its position and color set right away, it may have never rendered in that state before having its values updated.
To solve this issue, the API user needs to delay performing the update until after the values have been flashed to the GPU (at the earliest). We can make this possible by offering a
nextSync()
method, which takes a callback to invoke after the next time the values are flashed:NOTE: The callback provided to
nextSync()
could be invoked as soon as the next frame, but will not be called immediately in the same execution pass. As a general principle, an API that takes a callback should either always execute the callback immediately, or always wait for a future turn of the event loop. API methods that differentially delay execution are difficult to reason about and cause hard to predict behavior. For this reason,nextSync()
must always delay invoking the callback, even if the sprite'sisDirty
flag currently reads false.Rather than waiting simply for the next sync, which will often be the next frame, the API user may want to wait until the sprite has finished its transition. A method like
transitionFinished()
could make this easier:transitionFinished()
enables chaining for more complex animations:One could be tempted to unroll these nested callbacks with Promises:
While this approach does reduce the nesting observed in the nested callbacks, it introduces the possibility of forever-suspended functions. Each call to
transitionFinished()
supersedes the previous, overwriting the callback previously set (if any). So if, say, the above code was suspended on the firstawait
, waiting for the sprite to finish transitioning to red, and some other callback callstransitionFinished()
, then this function's execution context will hang forever.Megaplot could overcome this by returning Promises which are rejected when superceded. In that case, the API user should surround the transition chain with
try
/catch
to handle the rejected Promise:The downside to providing this kind of aid is that it encourages writing
async
functions, which, in the interactive data visualization context, may make code more difficult to reason about. A lot can happen in between frames.For more discussion of the tradeoffs between a callback-based and a Promise-based API, see the following section.
Discussion of callbacks vs Promises
As a design alternative, rather than taking callbacks (or perahps in addition), methods like
nextSync()
could return Promises. In the case ofnextSync()
, the returned Promise is fulfilled the next time values are flashed. For example:There are several downsides to this approach. The first is that creating a Promise requires allocating a potentially short-lived object.
Another downside is that promises encourage the recipient to write
async
functions, which, in this context, may have lingering unpredictable behavior. A lot can happen in between frames.But perhaps the best reason to prefer a callback API to a Promise one is that only a single callback function will be saved, while any number of async functions could be hanging on the same Promise.
For example, consider this code, which attempts to update the visualzitaion based on an end-user triggered event:
Since
mousemove
events can be prolific, this implementation could spawn many hangingasync
functions all waiting for the sprite to finish its last transition. This style would exacerbate memory thrashing if the post-await code does any allocation. Consider:d3.color()
allocates an object withr
,g
,b
andopacity
properties. So this seemingly innocuous, Promise-based code suspends a number of callback handles, each of which, when unsuspended, allocates a short-lived object.To mitigate the potential for many hanging methods, it would be prudent to reject the previous Promise when a method like
transitionFinished()
is called again. In that case, the API user would want to surround theawait
with atry
/catch
block:Compare that to the equivalent, callback-based implementation:
Because of the additional complexity of the
try
/catch
in the Promise-based code, the level of indentation here is identical. Plus, this code better handles a barrage ofmousemove
events. Each timesprite.transitionFinished()
is called, the callback passed in supplants the previous callback, without triggering atry
/catch
flow.Strictly speaking, these two APIs are not mutually exclusive. The
transitionFinished()
implementation could return a Promise only when called without a callback. This would mitigate the creation of short-lived objects.However, since it's always possible to add the Promise-based features on later, for an initial implementation, the callback implementation is sufficient.