Sparse frame animation and other enhancements (on the way)

[m-7761]

I'm about to add some new logic to frame animation, not directly to this code, but to my fork work.

What I have in mind is to add a flag to the animation to enable this. How it works is not every vertex must be defined by a frame, in which case it inherits the previous frame's. It could skip entire frames, which is necessary, or frames can be timestamped to avoid dummy frames and add the ability to use non-integral key frame timestamps.

Because of this dilemma I'm inclined to package these together into one flag that involves more than omitting vertices.

I think that what this can look like is a control packet that defines a block of vertices, and how to interpolate them, and one mode is to skip/inherit them.

The timestamp is for the entire frame, so it would be additional data at the top. The number of frames then doesn't define the length of the animation, which may need to be non-integral. But I think instead of making that a new field, the "Frames" field itself can be made to be non-integral, so that instead of working in "seconds" it counts frames. That avoids changing the UI radically for legacy animation formats. The reason non-integral is important is because many formats use non-integral timestamps for animation. They can't be represented right now, and so they can't be imported...

The same goes for skeletal animation. So they should receive an upgrade too.

I think that you need this to actually edit animations competently. At the moment Model can really just display frame animations, but not really make/edit them. When you design an animation it's too burdensome to input every frame, obviously. It's unlikely the editors used to make the models that MM3D imports were so saddled by limited facilities.

In any case, this is my roadmap, and I have to implement this, so I thought I should share my plans.

Ultimately I would like to somehow be able to scroll through the animation frames that have data and skip over those that don't. I'm thinking that the simplest way to achieve this is to make the animation tools do this by default and let them do otherwise if a modifier key is held down. Another way to go is to add a toggle feature somewhere, but I think that would be a waste of space in the UI. One way is possibly to let the play button change into a pause button, and maybe use the stop button to go back to normal scrolling mode, and while it's paused let the animation be scrolled freely, as if you're manipulating the animation time independent of frame selection. Double-clicking the play button could then put it into this state, as if that is the toggle feature.

[m-7761]

So far to implement this functionality I've had to add the following to the Model API and add an interpolation mode enum type to some methods. I think this is nearly the extent of it.

set/getCurrentAnimationFrameTime set/getAnimTimeFrame

It gets kind of confusing because the notion of a frame means two things under this system... the old sense is the actual data, whereas the new sense is higher-level meaning time keeping in frame units. That means the UI is identical, which is important because I don't want to add new UI elements.

The gist is you set a time frame for the animation, which doesn't have to be an integral value (mainly to be lossless.) At that point no actual frame exists until you edit a vertex at a new time for the first time. That's probably going to require manually inserting a data frame with a new API.

Since the timestamps aren't in seconds it remains compatible with the existing import/export formats.

getCurrentAnimationFrameTime gets the timestamp for the current frame. These are initialized to the frame number.

getAnimTimeFrame is similar to the loop setting. It sets the time when animation ends or loops but is needed to so to not artificially create a data frame to establish the running time. With interpolated animation modes it may be desirable to set this beyond the final data frame since it will interpolate between itself and the initial data frame.

To change the interpolation mode a new "property" will have to be added to the sidebar after the Dimensions field. Current modes are InterpolateStep and InterpolateLerp. There are two other pseudo values for the enum type. One is to clear the vertex and the other is only for a default parameter to retain the existing interpolation.

[m-7761]

FWIW it turns out the "frame" animation system seems to be unmaintained code or something, since the skeleton frame system has a way to redirect the vertex data and use that to generate normals, whereas the frame system doesn't use that infrastructure and so its normals aren't even correct. It's easier to treat the frame animations just like the skeleton animations are, and that allows all or most of the frame animation specific code to be removed.

I assume just implementing setCurrentAnimationTime for frame animation should permit for all of the frame paths to be delete/removed from the other places where they are considered outside of the frame animation editing API.

[m-7761]

Progress report: I'm over the hill with this project. (The rest is just tidying up.)

This is an image link. This animation luckily covers all of the hard cases I think.

http://www.swordofmoonlight.net/bbs2/index.php?action=dlattach;topic=286.0;attach=1009;image

With this approach it's possible to color animated and unanimated vertices differently. I added this texture quickly today to share a progress screen. I ran into all the problems with everything being disabled for animations (a major source of UX bugs) so I removed everything except for modifying the geometry. I would like to work on that also, and will probably need to ASAP, but it's not something I'm planning for the next demo I drop.

I am not sure about what version to put in the MM3D file. I changed it to 0xF7 for the minor version, but how the reader works doesn't make sense. It rejects mismatches on the major version (fine I guess if there's only one) but it should support earlier versions. It ignores the minor version. I think it should refuse to read files that are newer than itself, just to be safe. It could put up a confirmation screen too.

There needs to be some logic to safeguard cross version opening of files. For this system I've changed it to always write this new system's animation. I've added this flag:

enum MisfitFrameAnimFlagsE
{
    MFAF_ANIM_LOOP = 0x0001,

    //https://github.com/zturtleman/mm3d/issues/106
    MFAF_MODE_2020 = 0x1000
};

I think this system should be adopted since the code I worked on to implement it greatly improves the general Model code around animation. It removes virtually all branching on animation modes, so really only model_anim.cc deals with that. It's really very simple. The existing code was already set up to do that.

What could be discussed is a need add anything else into this mode, while the iron is hot.

I changed setCurrentAnimationTime to stop at the end of the animation instead of rotating back around to the beginning give the final time. I found that nicer on the user so that they don't have to do their own clamping. In the demo when an animation is selected it plays itself once. I left it to stop at the end, now, which I found more natural in the sidebar since it positions the slider closer to the sidebar and tells how long the animation is by virtue of putting the final frame in the Frame textbox.

This system fixes the existing normals issues for frame animations. (They can only show generic vertex normals right now.) I think it will take me quite a while to finalize everything. One or two weeks minimum if I don't get distracted.

[m-7761]

For the record, what I came up with for scrolling the animation timeline came down to how to phrase it for the menu version of the feature.

I couldn't come up with a way to word the concept, so what I ended up instead was to call it "Scroll Lock", which I think is probably the most straightforward way to intimate it. Scroll Lock is on the top-right of my keyboard but I think it's not always so easily accessible to the right/mouse hand.

How it works (on by default) is to snap to data frames. I wanted to use Ctrl to invert the function, and I recalled Blender seems to use Ctrl I think to enable vertex/grid snapping even when it's not toggled (probably to invert it more or less.) So I wanted to use Ctrl to invert snapping universally, and programmed something like this (also simplifying the Tool interfaces in the process to not take parameters since they're rarely used) but ultimately I realized that Ctrl couldn't snap the cursor when first pressed as it currently does because it does the view-rotation thing. So for the moment I made Ctrl+Alt to do this, as every other modifier turns on the window menu. I also made Shift to constrain the timeline to integer frames while not restricting it to existing data frames.

I think that the view rotation combination would be better bound to the space bar to free up the modifiers, but I can't make a major change like that without setting up a Preferences screen to opt into it. I think Space+LMB to rotate and Space+RMB to translate would be good. Plus I've added a new tool that doesn't require the keyboard to do that.

(I feel very strongly now that selection should union by default and maybe Shift with selection should be changed to starting a fresh selection. That just makes sense with how 3D apps do selection, plus it's confusing that subtraction in apps never requires a modifier because it doesn't make sense to use it any other way. If you think about working in 3D it's much more common to want to union your selection than to start a new selection. Double-clicking could also be used instead of shift, but is a bit more cryptic. Both can be supported. Other than losing the option to use shift for something else.)

How I ended up rendering the tick-marks was to draw the lines on the scrollbar itself from top to bottom. That made it to look like the icon I made for the default (None) tool that uses a filmstrip.

Adding timestamps to the key-frames is important for editing animations since it means that you can easily imagine a feature to time-shift the frames (scale, translate, etc?) that could be done easily with the TextureWidget objects except that the timeline is 1D and so may not be so easily rendered.

Also the skeleton keyframes already use interpolation, and in fact they require their own step-mode option because I know of particular animations that don't render correctly when using interpolation with a skeleton. That is they look fine using step but interpolated the movements were never designed to do that and so appear to wobble to the eye.

In this system it's not obvious how to add a frame to the animation, but your options are to input the frame in the text box that isn't subject to scroll-lock (obviously) and begin editing, or to disable scroll-lock either with the toggle or temporarily with Ctrl or Shift.

[m-7761]

Update: For this to work I've added the following two enums:

        //RENAME US
        //https://github.com/zturtleman/mm3d/issues/106
        enum Interp2020E
        {
            InterpolateKeep =-2, // HACK: Default argument.
            InterpolateVoid =-1, // HACK: Return value.
            InterpolateCopy = 0, // Enable sparse animation.
            InterpolateStep = 1, // Old way before 2020 mode.
            InterpolateLerp = 2, // Linear interpolate.
        };
        enum SkelKey2020E
        {
            SkelKey = -1,
            SkelTranslate = 1,
            SkelRotate = 2,
            SkelScale = 4, //UNIMPLEMENTED
        };

The second replaces the "isRotation" idiom and lets the types be combined in theory, but really just the pseudo SkelKey is supported without casting.

For the interpolation modes I changed a bunch of the API that returns bool to return these codes instead, so that they're easily incorporated into the getter methods.

A wrinkle in that is the getters that default to the unanimated data, which I think should simply be made to not do that at some point, but anyway, in that case they return the InterpolateVoid value.

For each getter there is a new method like hasSkelAnimKeyframe that doesn't take xyz coordinates and doesn't return the InterpolateVoid value, so are safer/easier to use to set the interpolation menu in the Properties sidebar...

I labeled that widget "Motion" instead of Interpolation or whatever. I found that the more natural wording, but I used names like "Step Function" and "Lerp Function" for the values. The other value is called "<None>" and if there is more than one value I've made it to show nothing in the dropdown selection... which isn't a value in the menu. (I think this is what "<Mixed>" means in the joint weights system. If so I think having nothing would be more clear.)

EDITED: Currently frame animations default to Step mode, but skeletal defaults to Lerp. Because that's how they were implemented I suppose. But I don't know if that's ideal, but it's compatible with old files without any code change.

[m-7761]

Update: I knuckled down and did the hard part of the editing side of the job the other day and did another pass at it yesterday to the point I have a clear view of the ups and downs of it all...

Part of my takeaway is that it probably would be good to be able to control interpolation modes independently, except it's pretty weird to express in sidebar layout terms.

One problem (which I don't think is a new one) is there isn't a way to remove keyframes from skeletal animations. Now there is a way to do that, but it can't distinguish between translation and rotation (later scale) so one way would be to commit to independent settings, which makes more sense from an API point-of-view, but less so from the sidebar view. So, I think what I will do instead is try to add a checkbox to Position and Rotation panels, off to the side of the labels. I don't know if that's convenient with Qt but I imagine it's possible. That can aid in giving a clear view of what's present as well. By unchecking you may delete the keyframe data.

Long term I feel like the sidebar could be more complicated (for better or worse) by adding a collapse widget to the panels that can switch between input models that display one field per component or all in one field like the Dimensions box (which for Scale could double as a uniform scale representation.)

I thought about adding the interpolation mode (Motion) menus as a 4th "T" component (time or tween) but feel as though it would probably be better to make the panel collapsible and represent each independent mode with R S T components (Rotation, Scale, Translation) which is a little clumsy but I think better organized.

My sense is as soon as anyone needs that feature it can be implemented, and will be eventually, but for now I think, whatever is less work makes sense. I think those checkboxes are a fine idea either way, even though with independent control you could accomplish removal by setting them to <None>, which is how it's currently done in frame-animation mode.

When I complete this round of work there won't be a huge amount of control available, but combined with the copy/paste function it should be enough to get by. But I think the Animation menu's copy/paste features should probably be removed in favor of universal copy/paste (especially since they don't have hotkey bindings... I will probably add Ctrl+Shift+C/P if nothing else) if it did not preclude the ability to use copy/paste normally while doing animation work... I mean ideally I'd like to be able to do modeling and animation at the same time, but it can prove challenging too.

P.S. I've switched the Play function to using the Pause key by default, and being a checkbox in the menu. I got the idea from adopting "Scroll Lock" for animation work, which sits beside Pause on my keyboard.

[m-7761]

EDITED: So yesterday I tried the checkbox way and didn't like how it broke up the division between the properties in the sidebar, so I tried the other ways and ended up going with the tripartite Motions settings way, which is probably for the best, but more work, yay!

What I ended up with was to not use the per component style labels for this, and instead changed "Lerp Function" to "Lerp Position" and "Lerp Rotation" and so on, so that when the item is visible in the menu you can tell which it affects.

I worry that with Scale added the properties will run a little too long for small windows so am going to investigate a scrolling option. I have an unorthodox idea of rotating the properties instead of scrolling. That way you can move what you're most interested in to the most opportune place without hiding anything offscreen. I've never seen a UI do that, but it doesn't seem that crazy.

[m-7761]

Briefly here is a screenshot preview of the changes described above. I'm going to assume it's self-explanatory and comment some on ideas I have that are not made evident by this image.

First though, I've done some work on changing how joints are displayed. You can see that. The basic idea is joints are black and wires are white. These neutral colors tend to look best but can disappear in the texture view like anything theoretically can. When using colors with blending I think monitors play tricks on the eye that can make it hard to tell what color is what.

When editing the joint animation I couldn't come up with a way to highlight the selected joint tree (I think the current release may have a problem with ambiguity owing to coloring selected joints purple but also unselected ones in this mode #120) until the other day it occurred to me that "unhilighting" the unselected tree was another way that could be just the trick. That's what's shown in this image in the very faint bone in the back... that's not the default way bones appear but rather a way they are obscured when selecting bones in animation mode. They just receive a fraction of the supplied opacity (I left it to render the old way if the opacity is 100%. I think what's shown is 33%.)

For the copy/paste problem I mentioned before I'm considering strongly using the Insert key to toggle the semantics in animation mode. Insert is kind of copy/paste adjacent since it can also be used to do copy/paste (which should be implemented too) in an alphabet neutral way with modifiers. For sanity sake that would be a global state that's off by default. (When on copy/paste would not copy keyframes but regular data instead, permitting editing in animation mode. I think some keys are needed for moving the timeline and animation but am not sure what. I have A assigned to popping up the animation window which can be quickly changed and closed with Esc.)

I think that the status bar needs to display the vertex/grid snapping statuses so I'm thinking about how to do that simply, likely with some additional text down there that should be underlined when toggled. I think though it doesn't look clickable I will endeavor to make it so clicking the text in the status bar is an alternative way to toggle those things.

In the image you can see little notches in the timeline that today I added to old version files that don't use the sparse representation. I think that the joint vertices should be colored in rainbow colors where red means a translation keyframe is present, green means rotation, and blue means scale. And so if all are present you get white. You can't see it in this image but the default color is black, which is not surprising seeing as how the joints are here colored black. That just leaves the question of how to indicate they are selected. I don't know if they should be colored in the white or gray range if selected or unselected or what, but I think it's a good idea nevertheless.

[m-7761]

Since working on skeletal animation (#125) I realized things are more complicated than I thought...

Since this system doesn't require an initial frame to be present I decided that the keyframe interpolation modes need to go backward to fill in the mode for the initial missing frame's. That means the final period's mode is left undefined but that's more reasonable than leaving out the first, and without wrapping there is no obvious interpretation of the end-condition anyway.

Even if you were to define the behavior at the beginning and end... it might need to be different for every channel, which is every vertex in the case of the frame animation, so this seems like the best decision even if it might be different from other modeling formats.

ALSO there is more than one interpretation of <None>. For skeletal (joint) keyframes it means there is no keyframe present, so the interpolation is done from the two present frames. For the frame animations I originally imagined it meant that the vertex did not move. But that's different from the skeletal semantics, regardless of their data-structures.

For this I decided to add an "InterpolateHalt" mode that is another pseudo value like "none" (InterpolateCopy) that acts like slipping in a duplicate keyframe.

In "frame" (vertex) animation mode this "halt" mode is how I originally implemented <None>.

For the end-user facing UI I decided to call this knot mode <Position> <Rotation> and <Scale>. It's a pseudo mode that means to use the existing parameters.

(EDITED: I was calling this "InterpolateKnot" in the previous version of this post. But by the end I'd convinced myself it would be better to find another phrasing because a "knot" in path-parameterization algorithms is constant in all dimensions, whereas in this case the time dimension isn't.)

[m-7761]

_CORRECTION: After a night's rest I changed the InterpolateCopy enum (from code above) to "InterpolateNone" (0) and renamed (again) "InterpolateHalt" to InterpolateCopy (1) and adjusted the enum (accordingly and patched my test file.)

I still have a ways to go to complete this feature. I have to rewrite the (new) frame interpolation code I wrote and double-check the skeletal routine, and then finish file read/write, then reevaluate if the UI is functional enough. I've sat on it long enough._

[m-7761]

Update: Today I implemented the Insert feature and removed from deleteSelected the behavior of removing keyframe data in skeletal mode. It's useful to be able to do that but it shouldn't be done that way, and it didn't implement the same feature for frame-animation mode.

In the new system there is an interpolateSelected API that can do what it did but it's not ideal for deleting, so I would add another API maybe. From the editor perspective you change the keyframe modes to <None> and that has the same result. The only difference is there are three menus to set for rotation and scale.

I called the new Animation->Insert menu check item "Clipboard Mode" which I think helps to highlight that the animation system doesn't use the clipboard proper. It needs to be per-window because it depends on if an animation is open. I made it so the Geometry->Copy & Paste & Delete items are disabled when an animation is open.

So if you don't want that you press Insert or toggle this item and you can use them in animation mode. In that case the Copy, etc. items are disabled in the Animation menu. I think this is reasonable since Copy/Paste is so ingrained. I can't imagine reassigning the hotkeys but I suppose technically that's possible, in which case this could get in somebody's way.

I removed the ability to clear out the keyframes because the sidebar can do that now, and assigned Delete to deleting the frame because that's hard to do otherwise. I removed the things that are easy to do since there aren't enough hotkeys to go around. I use Shift+Ctrl+D for Delete but make the Delete key work too by artificial alias. It's harder to use Delete since it's on the right hand side.

I made Shift+Ctrl+C to Copy the entire frame. I've yet to add Ctrl+A to select-all because the semantics are a little iffy, but that would be equivalent.

Clipboard Mode (checkbox) Copy Frame Copy Keyframes Paste Delete Frame

If these don't have hotkeys they're not practical to use them and I've already assigned all of the left-hand side so it only makes sense to use Ctrl+C etc.

I think it would be best if frames are deleted automatically (when the last keyframe is removed) but that's currently not implemented, It would require better bookkeeping data-structures than currently exist. Still I think it makes sense to lump Delete in with (Cut?) Copy/Paste.

(Also I've said so elsewhere but as of now "animation mode" doesn't interfere with editing and frame-animations don't prevent editing. I have to work on the calculateNormals routine and a few more things before I will feel like it's at a good place for an evaluation release. I'm losing a lot of work days this year on this just to get it to a well-rounded stopping point that I regret.)