Open lgritz opened 5 years ago
Updated, correct link:
In April 2021 KTX 2.0 was released. Its support for Basis Universal supercompression makes it very appealing for many real-time graphics use-cases. One of particular interest is support for it in Blender (uses OIIO) to import/export glTF 3D assets with KTX textures.
Thanks
Hi! I'm a maintainer of https://threejs.org/ and wanted to voice support here. We view KTX2 as a more modern and portable GPU texture container format, compared to older formats like DDS. These types of formats are critical for realtime 3D applications. KTX2 has several internal formats with different benefits that complement those currently available in OpenImageIO:
Basis Universal is by far the most common format used in KTX2 today, and supporting that alone would be amazing, but I did want to mention the other options in case they are relevant to others.
For illustration, here's a demo loading a glTF 3D file containing KTX2 compressed textures:
https://threejs.org/examples/?q=gltf#webgl_loader_gltf_compressed
How would KTX in it's primary use case ("textures compressed for GPU usage") actually work within OIIO context? To me it seems that OIIO is primarily targeted at applications that need to "read images and get their pixel values" (the images can be in multiple formats, scanline or tiled, deep etc., but in all cases you want to get pixel values in the end). And then a bunch of other OIIO functionality (image cache, image algorithms) operate on these actual pixel values.
However, KTX (and especially Basis Universal) use case is very much different - you don't want to get the pixel values; you want to get "data ready to use for the GPU" (in one of GPU supported block-compressed formats). Decoding the KTX compressed image into raw uncompressed form kinda nullifies all advantages of using KTX in the first place?
To some extent this affects other formats like DDS too (#2338). Current DDS implementation in OIIO always decompresses the block-compressed texture data. Which is fine for CPU pixel access, but not really efficient if you want to use the loaded image on the GPU.
@aras-p I think that's an interesting question. If it's a widely used image format, I still think there's a use case for being able to get to the pixels, if for no other reason than interchange to other file formats, or other pixel-by-pixel operations.
But indeed, as we push forward to (a) using GPUs to accelerate OIIO operations, and (b) having the OIIO texture system or its analogue working for GPU-based renderers, we will sooner or later need to grapple with this very issue. That probably means thinking about extending the current ImageInput API to have new entry points for "load tile/image into a GPU buffer" with an implied desire to keep it in a compact GPU-friendly format if it is already in that state.
Yeah, so basically it's several questions:
1) "short term" ("today"), how would KTX support in OIIO look like? Would it always just do decoding to r8g8b8a8 pixel values upon reading the files?
2) "long term", is a question for OIIO in general. If it wants to enable use cases like "the data is meant for the GPU only", then it has to support block-compressed GPU formats (BCn on PC, ASTC/ETC on mobile), and also possibly enable usage of DirectStorage (on Windows), "Metal 3 fast resource loading" (on Mac), and <I've no idea whether something similar exists on Linux>. But in any case, this would need extending OIIO public API to deal with this new concept.
1. "short term" ("today"), how would KTX support in OIIO look like? Would it always just do decoding to r8g8b8a8 pixel values upon reading the files?
This is what I'd be primarily interested in, but with even more emphasis on encoding KTX 2.0. It kicks open the door for authoring (e.g. Blender) and format conversion workflows. Feeding it r8g8b8(a8) would be a great start. A floating point cube map format for IBL could be up next.
2. "long term", is a question for OIIO in general. If it wants to enable use cases like "the data is meant for the GPU only", then it has to support block-compressed GPU formats (BCn on PC, ASTC/ETC on mobile), and also possibly enable usage of DirectStorage (on Windows), "Metal 3 fast resource loading" (on Mac), and <I've no idea whether something similar exists on Linux>. But in any case, this would need extending OIIO public API to deal with this new concept.
KTX-Software/libktx would support direct upload to GPU (for OpenGL and Vulkan). I assume you mean something like this, right?
I see this useful in "final" applications (games, RT3D apps; only displaying those textures) for which I personally wouldn't use OIIO. I'd rather convert to KTX and use libktx for loading. Then again, I don't know what OIIO users use it for/require in the future.
Ah, so basically the expectation of what "KTX" would mean in OIIO today is, that it would follow the current OIIO design - reading an image produces uncompressed pixel values for CPU consumption, and writing an image does uncompressed->compressed process. i.e. completely ignoring "GPU" bits, right?
And then if/when at some point OIIO gets functionality/API to target GPU usable textures directly, then all of this would need to be rethought for KTX, DDS etc.?
Ah, so basically the expectation of what "KTX" would mean in OIIO today is, that it would follow the current OIIO design - reading an image produces uncompressed pixel values for CPU consumption, and writing an image does uncompressed->compressed process. i.e. completely ignoring "GPU" bits, right?
Yes, that aligns with my understanding (and requirements)
And then if/when at some point OIIO gets functionality/API to target GPU usable textures directly, then all of this would need to be rethought for KTX, DDS etc.?
I assume yes, but I don't know OIIO's internals nor plans.
That sounds right to me -- a near-term implementation of the current APIs (calls to retrieve uncompressed scanlines or tiles into an application buffer in whatever data type is most appropriate from among those we support), and also start thinking about a long-term extension of the APIs to ask for raw compressed data (including, possibly, all the way into handle to a GPU buffer).
I haven't speced out what would be necessary or helpful for the latter, and my experience with compressed GPU texture formats is very limited. I would be interested in somebody sketching out exactly would would be desired in this set of features. Open questions that come to mind (and which I don't claim to know the answer to):
I assume yes, but I don't know OIIO's internals nor plans.
This is it, right here. These discussions are how the plans are made.
Any updates on KTX2 support in OIIO? Seems like quite an important feature seeing how much is moving to Vulkan these days.
I think we're just waiting for somebody who knows about KTX2 and needs this feature to come forward and make a PR with the implementation. Any of you can be the hero of this story!
I don't know anything about KTX2 nor do I program for Vulcan, so I'm certainly not the right person to do it.
@Silverlan requested (on a different ticket):
Support for the KTX-format would be great! https://www.khronos.org/opengles/sdk/tools/KTX/file_format_spec/