Thoughts on zoom strategy - continuous zoom and long audio files

[lowpitch]

Having played around with peaks.js zooming for a couple of days, using data generated from long audio files (2 - 3 hours), I've made the following observations (ignoring the bug reported in #90).

• Peaks.js works well for stepping between zoom levels
• Resampling data when a new zoom level is selected is quite slow
• Drawing waveforms at various zoom levels is very fast
• The method for animating between zoom levels could be improved I think - currently there's a lot of resampling going on (slow), and (ignoring the bug) a lot of redrawing (fast).

I think it would be nice to support continuous zooming - rather than stepping up and down between pre-defined zoom levels, being able to zoom to any level between the minimum and maximum without a significant performance hit. For example, moving a zoom slider, or using the trackpad to zoom in or out. The existing behaviour would still be supported - jumping up or down to the next zoom level would see peaks animating the current zoom from one level to the other, just as if the user had altered the position of a slider from one level to the other.

Some thoughts on how this could be achieved (just a starting point...):

• Pre-generate resampled data at fixed zoom levels (for example, [512, 1024, 2048, 4096]). Cache these resampled data sets so they don't need to be regenerated at a later point. Resampling is expensive, and doing it on-demand causes noticeable rendering delays.
• When rendering the waveform at certain zoom level, peaks would retrieve the nearest (rounded down) data set from the cache. So in the example above, if the zoom level request by the user was 768, peaks would use the cached data for the 512 zoom level. If the zoom level was 1600, peaks would use the cached data for the 1024 zoom level.
• Peaks would then render the waveform at the zoom level taken from the cache, but would render a waveform that is wider than the current zoom view (current zoom view width * (actualZoomLevel / cacheZoomLevel)). The waveform layer would then be scaled down (KineticJS pseudo code - setScaleX( cacheZoomLevel / actualZoomLevel) ) to fit the zoom view window. In the example above, assuming a window of 1000px wide and a requested zoom level of 768..
  • Peaks would use the pre-generated resampled data for the 512 zoom level
  • It would use this to render a waveform that is (1000 X (768/512)) = 1500px wide
  • The layer would then be scaled in the X axis to (512/768) = 0.66667, bring it down in size to 1000px to neatly fit the window.
• It's a bit like Google Earth - there's a continuous zoom, but at fixed points in the zoom the images are swapped out for the next level of detail.
• This approach would allow data to be pregenerated at fixed zoom points, but would allow the display of any zoom level between the minimum and maximum without regenerating any data. Drawing is cheap, calculation is expensive here.

Concerns / assumptions / things to check:

• Memory usage if caching several sets of large data
• Visual differences when drawing at 100% scale vs stretched / compress scale. How noticeable would it be when switching from one set of data for the next.

Just a starting point for discussion.

[thom4parisot]

Yep what you explain is right, even with partial resampling, zooming from an overview level will demand a lot of CPU to run the animation, especially for a long programme ;-( In term of scaling, the more atop we are, and the longer the programme is, the more intense is the computation.

Some options to consider for the processing:

• ServiceWorker/WebWorker to compute that outside of the main thread;
• swappable storage of memoized data: IndexedDB, localStorage.

I know that on the original zooming work, avoiding the resampling has been considered but the visual quality of the outcome was poor enough to opt in for the partial resampling. Have you read the paper written by Manvir? If not, I can send it to you or Rob can do it.

Maybe the Scale feature of Kinetic has not been tried so it could eventually be interesting to investigate that way.

Thanks for sparking the conversation on that tough topic :-)

[lowpitch]

Thanks for this! I haven't read the paper - I will ask Rob for a copy. Sounds interesting...

I should get some time over the next couple of days to knock up a quick test of the zooming version. I do think there will be some visual differences as it switches out from one level to another, but if it's quick enough it might work quite nicely. Back in the Flash days, this is how I would have tackled a similar problem.

ServiceWorker / memoized data would be cool - or even passing in pre-generated resampled data in external files (to keep basic browser support).

I'll think about this some more and keep adding thoughts here...!

[thom4parisot]

Thinking again about it, things we can consider:

• what we want is a feeling of zooming/stretching/shrinking but an accurate frame when the animation ends;
• memoizing the next/previous zoom level is going to be useful in any case;
• the zooming could be a transition from a level to another one, by adding/removing samples rather than by partially resampling at every animation frame.

If this is visually good looking, it would be hard to be faster than a simple filter/loop over thousands of elements.

[lowpitch]

an accurate frame when the animation ends;

Agree with this!

I had another thought too. What if we...

• Just had one master dataset (no resampling when zooming or animating)
• In the draw function, pick the first line to draw based on the current offset of the zoom window
• Iterate through the data, starting from this first position, then 'jumping' a particular number of samples based on the zoom level. Then drawing the next line, then jumping again.
• When enough lines have been drawn to fill the window, stop the loop

Or something...

[thom4parisot]

@lowpitch yep your proposal is even cheaper in term of computation!

Your solution is a good starting point to see if it's worth trying in order to see if skipping n-frames is consistent across the zoom operation.