Support for filtering for all texture formats

[almarklein]

Context

I've asked about this in the element.io chat and gotten some answers already, but I'm still left with some concerns.

As I understand it:

• The xx8xnorm and xx16float formats support filtering samplers. (docs on texture-format-cap)
• The xxfloat32 formats support samplers, but only non-filtering.
• All other (unsigned/signed non-norm) formats do not support a sampler.

Problem

In short: in our use-cases, polyfilling is too expensive and converting to another format not always possible.

Our eventual goal with wgpu-py is visualization of scientific (e.g. medical) data, which come in a wide range of formats, and can be very large (e.g. 3D MRI data).

Ideally, we'd upload the data in it's original format (i.e. data type), and apply filtered sampling to create smooth graphics. Otherwise, we'd have to polyfill linear filtering, which is probably (much?) less efficient. One particular use case where this would be a pain point is 3D volume rendering, where we'd need trilinear interpolation to avoid a range of artifacts, which is particularly costly (27 lookups).

One possible solution/workaround is to convert the data to one of the formats that does support filtered sampling, but for some formats this would mean a loss of precision, which is unacceptable in certain use-cases.

This limitation feels frustrating because the hardware of the devices that this kind of stuff would run on would be capable of doing what we need.

Way forward

I hope that we can discuss possible avenues to resolve this. For our particular use-case we are mostly interested in the filtering (interpolation) aspect of the sampler, and care less about levels and address-mode.

Some ideas:

• Support filtered samplers for all formats and polyfill for HSL.
• An extension/feature to enable samplers for all formats that would only be available where it works.
• A simpler solution that I am overlooking?

cc @korijn

[Korijn]

For my understanding; where exactly lies the limitation? This would all work just fine if we were targeting the underlying layers (metal, DirectX, vulkan) directly right?

[kvark]

Moving to wgpu: this isn't Naga issue. Naga doesn't know what the exact texture format is it, even.

Good news is, in wgpu there is already a solution for you! It's called TEXTURE_ADAPTER_SPECIFIC_FORMAT_FEATURES feature. It's a native-only feature that's guaranteed to be exposed (on native). With it, you can get TextureFormatFeatures for a format and check filterable, which will turn out to be true for a good deal of platforms.

[kvark]

For my understanding; where exactly lies the limitation? This would all work just fine if we were targeting the underlying layers (metal, DirectX, vulkan) directly right?

No it wouldn't. WebGPU is basing off an intersection of these APIs. They don't guarantee 32-bit formats to be filterable.

[Korijn]

Thanks for explaining!

[almarklein]

It's called TEXTURE_ADAPTER_SPECIFIC_FORMAT_FEATURES

Ah, I checked for existing features, but only in WebGPU :) This is good news, thanks!

[almarklein]

It's a native-only feature that's guaranteed to be exposed (on native).

I see these lines https://github.com/gfx-rs/wgpu/blob/master/wgpu-core/src/instance.rs#L217-L220, but it looks like there'd need to be some work to expose this to wgpu-native.

[almarklein]

@kvark would it be a matter of applying TEXTURE_ADAPTER_SPECIFIC_FORMAT_FEATURES here (and a few lines below)? An initial try did not work. I'm double-checking whether anything else would be needed (e.g. use TextureFormatFeatures somewhere), before I dive deeper.

[kvark]

Yes, that should be enough.

[almarklein]

Ok, I can confirm that I can now create a filtering sampler for f32 textures. However I still run into problems for integer textures, because Naga freaks out when it sees this shader code:

[[group(0), binding(0)]]
var r_tex: texture_2d<i32>;
[[group(0), binding(1)]]
var r_sampler: sampler;

[[stage(fragment)]]
fn fs_main(texcoord: vec2<f32>) -> [[location(0)]] vec4<f32> {
    let sample = textureSample(r_tex, r_sampler, texcoord);
    return vec4<f32>(sample) / 255.0;
}

Running:

λ cargo run --features wgsl-in -- c:\dev\py\tmp.wgsl
    Finished dev [unoptimized + debuginfo] target(s) in 0.06s
     Running `target\debug\naga.exe c:\dev\py\tmp.wgsl`
Entry point fs_main at Fragment is invalid:
        Expression [4] is invalid
        Unable to operate on image class Sampled { kind: Sint, multi: false }
error: process didn't exit successfully: `target\debug\naga.exe c:\dev\py\tmp.wgsl` (exit code: 0xffffffff)

My best guess is that the error originates here: https://github.com/gfx-rs/naga/blob/350ceb383d4c85e5ae05e5f9bd2b2881917dac2d/src/valid/expression.rs#L536-L554

[kvark]

@almarklein seems like a different issue? The original one is about filtering, and integer formats are not filterable in any API at all. In WGSL (as well as HLSL), they are not even sample-able. You can only do textureLoad on them.

[almarklein]

Thanks for the clarification!

It feels the same issue for us; the "filtering" in the title was meant more generic. I meant the hardware doing the linear interpolation for us as we access the texture.

One cause (the samplers-for-float32-not-being-filterable) can be solved with TEXTURE_ADAPTER_SPECIFIC_FORMAT_FEATURES. From you comment, I derive that the other cause (integer texture formats cannot be accessed via a sampler) is not solvable.

[kvark]

I meant the hardware doing the linear interpolation for us as we access the texture.

Ok, looks like we are still in confusion. Integer textures can not be linearly interpolated in any of the APIs. So it's not reasonable to expect wgpu to do this.

[almarklein]

Interesting. In wgpu-py we have a series of tests that draw a simple shape using different texture formats, including integer textures. With the version of wgpu-native of a few weeks ago all these tests worked with a filtering sampler. Back then our shaders were provided as SpirV and using OpImageSampleExplicitLod). Now that we're transitioning to WGSL, things have become more strict.

I believe you :) but am wondering how Vulkan accepted our shader code earlier.

[Korijn]

Is this table on page 130 from the GLES spec relevant here?

https://www.khronos.org/registry/OpenGL/specs/es/3.0/es_spec_3.0.pdf

[kvark]

@Korijn yes, thank you for linking this! All the formats ending with I or UI are not filterable. @almarklein low level APIs are quite unportable and unsafe. The fact that nothing yelled at you for filtering integer textures is sad, but also we've seen cases where Vulkan validation was silent. I even wrote a relevant piece in http://kvark.github.io/api/2021/04/12/api-validation.html.

[Korijn]

@almarklein we should probably look at the table in the spec and adjust the test suite to match

[almarklein]

@kvark thanks for the link, interesting read (and so are your other posts)!