Feasibility of adding webp support

[bscott711]

This has such potential for sharing volumetric movies and I was curious how difficult it would be do add support for loading webp files. With the proper settings there is ~100x compression achieved with 'nearly indistinguishable' images.

[Twinklebear]

Playing volumetric videos is a cool idea, I think it should be doable. I think all browsers support WebP now, so the Image API should be able to decode it, so supporting WebP should be no problem. Supporting an animated WebP file might need a bit more work, but if I can get access to the "image" of each frame, then that can be passed to WebGL easily.

I have some follow up questions:

• WebP is a 2D 8-bit color image format, if you have volumetric microscopy/CT/(whatever your data source is), I'd assume you have an image per slice. For animation, do you store animated slices?
• Is the volume data color (RGB?) or single channel/intensity (like the microscopy data I have in the demo)? Essentially, how would you view a slice of the volume today, by drawing the colors directly or applying some colormapping like the app does now?
• Would you be able to share some test data? I can't promise anything about how much free time I'd have but it might be interesting to try out. It might not be too bad to get something running.

[bscott711]

Thanks for the reply, I'll start at the end and work back. The tiff version in your viewer looks beautiful, it will just never load quick enough to have a smooth playback.

• Yes! Loads and loads of data I can share: Starting simple with a single volume stored as a series of files and then the combined animated as proof of principle, but have more if it works.
• The data are colored like you do now, and ultimately enable additive blending of the channels.
• The data are currently multipage tiff for each color and timepoint, and I just tried to follow a similar format in making the animated (don't actually need that), just the multipage nature, and I didn't know how else to achieve that.

[Twinklebear]

Cool! Thanks for the test data, that will help a lot with getting something running, and seeing the example tiff helps to understand the data type.

So I think data type/rendering wise it should be fine since it can be rendered the same way I do now (later I could maybe add multi-channel volume rendering support since it sounds like you have multiple channels too).

I was looking into this a bit and I think if we wanted to have the slices stored as an "animation" in a single file we'd need to use webm instead of webp so that the renderer can get access to the individual frames of the "animation" (the slices), it seems like that would be harder to do with webp. With webm the video can be "played" by the browser and I can grab the frames (essentially like this demo). With webp I think I'd need a separate library to get each frame, which may also be slower than the native browser video playback implementation since it'd be a Javascript library.

However, when using the lossless encoding mode the webm file is also a bit bigger (5.4MB vs 1.7MB), but looks the same as the webp you shared: https://drive.google.com/file/d/1jLm4Hmulhso6nqcdHshsV3wSyvrYX8Fi/view?usp=sharing .

I made the webm with ffmpeg using the webp stack you had on dropbox:

ffmpeg -i test%03d.webp -c:v libvpx-vp9 -lossless 1 output.webm

In looking at the original webp stack on dropbox, the zipped set of all the test*.webp files is 1.7MB, which is the same size as output.webp. So what would probably work best is to just have each timestep stored as a zip file of webp slices, since stacking them to an animation doesn't seem to reduce the size further. That will be easier to load into the renderer since there's no need to play back a video and grab frames, and probably easier for you on the data preparation side too since you don't need to encode a video for each timestep.

Then depending on the size of the animation, for sharing a whole animation we could zip a bigger directory structure like:

dataset/
    - t0/
        - slice000.webp
        - ...
        - slice###.webp
    - t1/
        - slice000.webp
        - ...
        - slice###.webp

Or if it's quite large the individual time steps can be zipped individually and streamed on demand as the animation plays.

[bscott711]

Your zip solution for each timepoint sounds like the right way to go! That would actually match the current folder structure that we use for tiffs exactly. By essentially swapping out tiff for zipped webp we could then take advantage of simultaneous requests and preloading adjacent time points, but it seems like that would be lost if the entire time series were zipped in one file. I can drop a few zipped timepoints in that dropbox for future testing.

[Twinklebear]

Thanks for the test data! I was messing around a bit with this over here: https://github.com/Twinklebear/webgl-volume-animation , it can load multiple zip files containing the webp stacks and step through them as an animation. It shows each frame for a fixed amount of time so the animation doesn't look as "smooth" as it can be (later the page can have some playback controls/etc. kind of stuff):

https://user-images.githubusercontent.com/1522476/166184036-8eee2888-a4f2-4260-ad68-ac9014f161b2.mp4

However I'm seeing some very strange and extreme memory consumption on Windows in Chrome and Firefox, where the app uses an order of magnitude more memory than it should. This results in the app not being able to load more than 3 of those webp stacks in Chrome (on FF it barely fits all 5). On macOS I don't see this issue, that's where I recorded the video.

I was trying to find out what's causing this, but WebGL is pretty opaque (and the mapping to the platform API etc) so it's hard to see why it's happening or what I can do to address it (on a desktop app this would fine). I'll try porting the app to WebGPU next, which is a lower level and more modern API (https://web.dev/gpu/), so there the behavior should be consistent and controllable. The downside is that WebGPU is still in development, so the app would only work in Chrome (or Chrome Canary w/ a flag set) for now. WebGPU should be getting released later this year, and implementations are also underway in Firefox and Safari.

Edit: Here's an animation stepping through faster

https://user-images.githubusercontent.com/1522476/166184959-07da0134-6d1b-4183-b84c-c0e429469f72.mp4

[Twinklebear]

I also remembered you can change the backend used by ANGLE (Chrome's WebGL -> native graphics API mapping), and after changing it to OpenGL the app runs how I expect on Windows/Chrome as well. You can try that out by going to chrome://flags/#use-angle in Chrome if you're on Windows.

[bscott711]

I cannot find the right words at the moment so there's gonna be a lot of emojis here. This is me for the next month: :exploding_head: :exploding_head: :exploding_head:

It loads so fast and the playback looks amazing :star_struck:

This is seriously so amazing!

[Twinklebear]

That's awesome to hear 😄 ! Then I'll stick with WebGL for now since it's possible to work around the issue on windows and is a bit more accessible at the moment vs WebGPU, and sounds like it's working well on your end. I reported a bug about it https://bugs.chromium.org/p/angleproject/issues/detail?id=7258 so hopefully it'll be fixed eventually too for the default config of Chrome on Windows.

When I get some time later this week/next I'll add in remote timestep streaming so you can give it a text file with URLs or a URL to a text file (second option for link sharing) and it'll stream those on demand to play them back.

[Twinklebear]

I had a bit more time to play around, I added support for remote timestep streaming (but not on-demand, so it may crash if you give a huge time series), and support for providing the files to load as a list of URLs. You can upload the text file or provide a URL to them so it's easy to share a time series with a link: https://www.willusher.io/webgl-volume-animation/#url=https://lab.wushernet.com/data/urllist.txt . Both the zip files and text file (if fetched by URL) need to support CORS (so Dropbox should work). There's also some basic playback controls now.

Is that time series something I could share as a live demo of the volume animation app?

[Twinklebear]

I'll also transfer this issue to the webgl-volume-animation repo