Wanted: a good JPEG 2000 decoder for Rust.

[John-Nagle]

I'm writing a new client for Second Life/Open Simulator, using Rend3->WGPU->Vulkan, and need a fast JPEG 2000 decoder.

Demo videos:

• https://vimeo.com/553030168 [From 2021, rather slow]
• https://vimeo.com/user28693218 [From 2022, faster]
• https://video.hardlimit.com/w/peBesyAgtzfRWS5FnDQQtn [Latest video, faster with better rendering]

So now that things are starting to work, I have performance problems. A big bottleneck is JPEG 2000 decoding. All Second Life content is encoded with JPEG 2000, multi-resolution, highest resolution lossless. JPEG 2000 decoding is the major bottleneck on content loading, even with several CPUs working in parallel. (All the content comes from servers; it's a metaverse. The world is the size of Los Angeles, and you can drive or fly, so you do a lot of content decoding.)

There are several JPEG 2000 decoders available:

Free:

• OpenJPEG - C, reference implementation, very slow. (I'm using that now). There are three Rust crates which link to this: jpeg2k, bevy-jpeg2k, and openjpeg.sys. The C code underneath is notorious for buffer overflows and has been the subject of three CERT security advisories.

• JPEG 2000 on Github - pure Rust implementation, underway for a year, stalled, developer needs help. Read the issues for a status report. It's a promising start, but right now it just decodes the header.

Commercial:

• Kakadu - C++, No Rust linkage yet, about 10x faster than OpenJPEG, very expensive to license. (The Firestorm viewer for Second Life uses it, and that license is their biggest expense.)

• Grok - C++. either expensive to license or limited to Affero GPL use. About as fast as Kakadu. Open source, so you can look and see how they do it. Rust bindings in grokj2k-sys Currently won't build on Ubuntu 20.04 LTS due to a dependency that should be fixed in Ubuntu 22.04 LTS.

• nvJPEG2000 - from NVidia. Requires an NVidia GPU. (of course).

• https://www.j2k-codec.com/ - Commercial, opaque binary blob from Russia, Windows only.

So, if anyone is interested, the pure Rust implementation on Github needs developers. You can look at Grok and see how they do it.

Ideally, JPEG 2000 would be implemented as another codec for Rust's "image" crate.

[AlexEne]

Would it work to have a pre-processing step that transforms the images from jpeg into something that's a bit more well-supported (even DXT/ETCx textures?). Jpeg is quite unusual as an asset for games.

[John-Nagle]

The servers that hold Second Life assets have over a petabyte of content in JPEG 2000 format. It's a world the size of Los Angeles, with user-created content. 115875 different chairs are available for sale, for example.

So, no.

This is a difference between a game and a metaverse. Huge amounts of user-created content created by hundreds of thousands of different people over many years. Very little instancing. Really big world. Clients have to deal with huge amounts of incoming content as the user moves around.

[AlexEne]

I had no idea what second life is :). Also that's a lot of chairs.

[siana]

Kakadu isn't 10x faster than OpenJPEG. It's at most 2x faster than modern OpenJPEG versions compiled with all optimisations, it's been a while and OpenJPEG team has actually been diligently working on performance. Kakadu however is pretty well internally multithreaded but i don't recommend using that feature if you have a choice, as it's actually wasting away raw performance due to synchronisation overhead. When you have high image decoder pressure and actually need all the decoder performance you can get, the situation is that you have hundreds of images queued, but each of them isn't very large and most outright tiny, taking individually just milliseconds with just a handful taking more. You can better make use of the processor resources by running worker threads where each decodes an image in single threaded mode. Current SL application just doesn't do this and decodes images sequentially, and then it just depends whether the corresponding decoder has multithreading internally implemented or not, which gives an advantage to Kakadu - an advantage in hardware utilisation, not that much in raw performance. You can gain most of that performance back in your client by running your own decoder pool, which will be more efficient anyway since you're not pressed by the effects of Amdahl's Law.

One problem you've got is that starting to work on a new implementation won't solve your issue in any foreseeable future, it will take several years for any new implementation to even just reach OpenJPEG performance. JPEG2000 is an exceptionally unwieldy and ill formed standard. It gets worse if you were to demand that one were to write such an implementation without borrowing from existing software because of their license terms. And without borrowing also likely means without looking, due to potential liability. Also to note i have had a good look into OpenJPEG codebase on many occasions.

Oh BTW OpenJPEG 1.3/1.4 makes for a terrible comparison point, being now one and a half decades out of date.

It's so unfortunate that we're stuck with this unwieldy image format. I think new grids should foresee JPEG XS and XL compression instead, maybe Basis Universal could be evaluated as well. With client and grid support, this can be something the system or the individual creators can decide upon during upload.

[John-Nagle]

It's unwieldy, but JPEG 2000 has two major advantages:

• You can read lower resolutions without reading the whole file.
• The highest resolution can be uncompressed, so you can use it for normals, elevation maps, and other non-image data.

Agree about multi-threaded. I don't need per-image multithreaded decoding and already have five images running in parallel.

I've so far avoided linking to C/C++ from Rust. Builds become far more complicated cross-platform. Too many safety issues, especially with OpenJPEG's history of buffer overflow security bugs. If I have to use C/C++ code, it runs in a subprocess.