search

brianchirls / Seriously.js

A real-time, node-based video effects compositor for the web built with HTML5, Javascript and WebGL
MIT License
3.88k stars 354 forks source link

Jitter when animating via transformNode.scale #126

Open tencircles opened 8 years ago

tencircles commented 8 years ago

Hey!

I'm currently trying to put together a project using masking and we've run across an issue wherein setting scale on a transformNode causes a visible jitter.

This behavior seems to become worse the more nodes are added to the graph.

Essentially we are running an requestAnimationFrame loop which sets a new value using

transformNode.scale(value)

on every frame. What we observe is either stale values, or conflicting values within the transform node. I've attached a video of the observed bug. This doesn't seem to be performance related as the page runs at a solid 60fps in the test.

I'd be happy to try to contribute a fix if you can point to me in the right direction in the source. Any help on this one is greatly appreciated as I've got a delivery coming up very soon.

jitter_capture.mp4.zip

brianchirls commented 8 years ago

Hey @tencircles. That's interesting. Do you have a reduced code sample you can share? Maybe a jsbin, codepen or something live like that?

Does it only happen with whatever effects you're using for masking?

Have you tried it in Firefox as well?

Does it make a difference if you change the resolution of the source image/video?

tencircles commented 8 years ago

Hey @brianchirls, thanks for the response!

I've set up a test repo here https://github.com/tencircles/seriously_mask_test

The javascript code is here: https://github.com/tencircles/seriously_mask_test/blob/master/js/index.js

And you can find the code live here: http://45.55.37.142/seriously_mask_test/

The jitter seems to be most prominent when using the translate method of the transform2d node.

Steps to reproduce:

  1. Check do_translate
  2. Set start/end_translate to any values
  3. Click 'tween'

There seems to be an observable disparity between the numbers we send, and the result on screen. You can see both the scale and translate input values on screen.

You can find a screen cap showing the disparity between input values on and on screen result here. If you step through frame by frame it's really easy to see.

http://45.55.37.142/seriously_mask_test/captures/chrome_counter.mp4

The behavior seems to be the same in firefox and in safari. Tried changing images, didn't have any effect. Seems to be limited to the transform node, not sure if the behavior is present in other nodes.

Let me know if I can dig anything up!

brianchirls commented 8 years ago

Okay, first of all, this is a pretty cool-looking composition. I'm looking forward to seeing the finished product.

This is a very strange one. But there is definitely something going wrong here - you're not imagining things. ;-) I'm not able to replicate your problem on my machine, but I can see in your video that the "A" is moving backwards, which it obviously should not do. I've run CPU profiles, timeline recordings and screen captures; I even tried CPU throttling. Apart from a few janky frames, this does not appear to be a performance issue. That wouldn't explain backwards movement anyway.

I wonder if something weird might be going on with TweenMax. It's hard for me to tell for sure, because the code is minified and I'm not familiar with the API or behavior of that library. In theory, it might be possible that two requestAnimationFrame cycles are running and fighting over setting different values on the same target object. A later value might be smaller than the previous one, and the difference might be < 1/1000, in which case you would not see it show up in your scale value text. But the difference might be magnified enough to be visible in the canvas by rounding of pixel positions and/or floating point quirks. I know it sounds like a bit of a stretch, but it's all I got at the moment.

Are you sure you're cleaning up the timeline properly?

Is it possible that this only happens after you've gone through the animation at least once? Or does it happen on the first time you play through it?

Since I can't replicate, I can't do the debugging for you, but I can point you in the right direction. I suggest two things:

1) Set a conditional breakpoint at line 5933 of seriously.js with this code: translateX < 499 && x < translateX. If it breaks, you'll know that the incoming value is less than the one before it. (This will only work if you let the animation play all the way to the end.) You can see in the call stack where that happened. If it never breaks and you still see the A going backwards, that will tell us we need to look deeper into Seriously.

2) Try making a reduced test without TweenMax or even dat.gui. Write your own code in the callback passed to seriously.go() to determine the x value of the translation. If you still have the problem, then we'll know for sure the bug is in Seriously.js. If not, then it's more likely that something is going on with TweenMax.

Can you report the results back here?

tencircles commented 8 years ago

1) Set a conditional breakpoint at line 5933 of seriously.js with this code: translateX < 499 && x < translateX. If it breaks, you'll know that the incoming value is less than the one before it. (This will only work if you let the animation play all the way to the end.) You can see in the call stack where that happened. If it never breaks and you still see the A going backwards, that will tell us we need to look deeper into Seriously.

Just tried this out, but the breakpoint doesn't trigger. You can also seek frame by frame in the video above to see the values being passed to seriously. You'll see the values on screen going up constantly, but the result rendered jumps back.

It's very difficult to see with the naked eye, so this might be occurring for you just not quite as noticeable perhaps? For me most of the time it looks like slight performance jank, but at a 60fps screen cap you can actually see it's moving backwards.

2) Try making a reduced test without TweenMax or even dat.gui. Write your own code in the callback passed to seriously.go() to determine the x value of the translation. If you still have the problem, then we'll know for sure the bug is in Seriously.js. If not, then it's more likely that something is going on with TweenMax.

Rather than coding the manually, I just added all values sent to seriously to an array with timestamp + value. Since all values we send are send through a single point (line 112 in the example above), we can be 100% sure that nothing else is setting values or calling methods within seriously. Checking the values sent both manually and programmatically it looks like we are sending values which steadily increase over time. If you want to have a look at the number let me know.

Any ideas on where to look within seriously to try to catch this? Stale matrix values maybe?

tencircles commented 8 years ago

Just pinging this. Any clue where we could start looking?

brianchirls commented 8 years ago

I was only just able to replicate this. Earlier efforts, even with screen recording, didn't show the problem.

It's not jank, because that wouldn't be moving backwards. Also seems unlikely that stale matrix values would cause it to go backwards. I checked most of the matrix values of the different, and they all seem correct.

Another hunch I had is that something weird is happening with ping-ponging textures. Some effects do that, but not the ones you're using. I suppose it's possible that something weird is going on with buffering in the GPU driver, though unlikely. What machines have you tested this on?

I did notice that you're using a lot of "layer" effects that don't seem necessary, either because they only have a single source input or they have 2 sources but only one of them is in use. Maybe there's something going on there? Is there a reason for that? Maybe if you can eliminate those nodes and it fixes the problem, that might get you far enough to deliver your product and it'll give me/us a starting point to hunt down whatever bug.

johanbelin commented 8 years ago

Hello Brian,

We've ran some stress tests to try and see what nodes specifically could be causing the issue. For each of these tests, we animate the letter by calling the letter transform node's translate method within the callback passed to seriously.go().

All tests have jittering when animating. The jittering is sometimes limited, sometimes erratic, seemingly without taking into account the level of complexity. The video captures in the zip have all been made on Chrome OS X, the behavior is exactly the same in Safari. On Chrome Windows, the jittering is still present but much more subtle.

We haven't been able to identify what worsens or betters the jittering as the behavior seems to be random. To clear any possible external causes, we are not updating the translate method's x position with TweenMax anymore.

To answer your question, we are using layer nodes so that we can control each layer's opacity level independently.

You can download the zip and run the directory on a local server. Here's the routing for each of these tests, in order of complexity.

simple_test_step1_noReformat:

simple_test_step1:

simple_test_step2_noReformat:

simple_test_step2:

simple_testBug:

Do you have any suggestions as to what we could do? Thank you very much

seriously_stressTests.zip

brianchirls commented 8 years ago

Okay, thanks for these reduced test cases. I can work with this. I have some ideas of where to start and will dive into it as soon as I can.

tencircles commented 8 years ago

@brianchirls Any way I can help out with this one? I've poked around in the source, but it's really just guesswork.

johanbelin commented 8 years ago

Hi Brian! Have you had any chance to look into this? Can we be of any help?