disable sampling filter for textures

[kushalkolar]

Hi, not sure what's the best place to put this so feel free to transfer/redirect me.

I was trying to figure out if it's possible to disable the filter set in pygfx TextureView and found that it eventually ends up calling wgpu::FilterMode which seems to be directly from the WGPU API. It only has options for nearest and linear. I was wondering if there's any way to bypass the sampling filter or turn it off? This would make it easy to display scientific heatmaps and rasterplots without interpolation.

Thanks!

[almarklein]

I think that nearest is what you mean with off :)

Longer answer: strictly speaking a pixel is an (infinitely small) point. To fill in the space between the pixels you need some interpolation method. With nearest you pick the value of the nearest pixel. With linear you linearly combine the values of the 2 (1D), 4 (2D) or 8 (3D) nearest pixels. And then there are a variety of cubic interpolation methods that take more pixels into account.

So with nearest you get that typical square "pixelated" appearance.

[rob-the-bot]

I'm looking at this issue as well. What's the best way to set the filter to be nearest in a pygfx.Image object? The code below is draws a 20x20 diagonal matrix. I think it should happen in the gfx.ImageBasicMaterial function but I couldn't get it to work.

import numpy as np
from wgpu.gui.auto import WgpuCanvas, run
import pygfx as gfx

N = 20
img_data1 = np.eye(N, N).astype(np.float32)

canvas = WgpuCanvas(size=(2*N, 2*N))
renderer = gfx.renderers.WgpuRenderer(canvas)
scene = gfx.Scene()
camera = gfx.OrthographicCamera(2*N, 2*N)
camera.scale.y = -1

image2 = gfx.Image(
    gfx.Geometry(grid=gfx.Texture(img_data1, dim=2)),
    gfx.ImageBasicMaterial(clim=(0, 1))
)

scene.add(image2)

def animate():
    renderer.render(scene, camera)
    canvas.request_draw()

canvas.request_draw(animate)
run()

The code below worked, but this is setting a meshgrid in gfx.Geometry and setting the actual image data (colormap2 = gfx.Texture(img_data1, dim=2).get_view(filter="nearest")) in gfx.ImageBasicMaterial. I don't feel this is the correct way.

canvas = WgpuCanvas(size=(2*N, 2*N))
renderer = gfx.renderers.WgpuRenderer(canvas)
scene = gfx.Scene()
camera = gfx.OrthographicCamera(2*N, 2*N)
camera.scale.y = -1

xx, yy = np.meshgrid(np.linspace(0, 1, N), np.linspace(0, 1, N))
img_data2 = np.stack([xx, yy], 2).astype(np.float32)

colormap2 = gfx.Texture(img_data1, dim=2).get_view(filter="nearest")
image2 = gfx.Image(
    gfx.Geometry(grid=gfx.Texture(img_data2, dim=2)),
    gfx.ImageBasicMaterial(map=colormap2),
)

print(f"{image2.material.map.filter = }")

scene.add(image2)

def animate():
    renderer.render(scene, camera)
    canvas.request_draw()

canvas.request_draw(animate)
run()

[kushalkolar]

@rob-the-bot thanks for looking into it further!

I think I found where it's actually creating the TextureView for the image data that is set to the Geometry grid, https://github.com/pygfx/pygfx/blob/d6941399fc887f2a26879f64169000a53afdfaa1/pygfx/renderers/wgpu/imagerender.py#L70

Setting this to nearest works for me, note that this is running as a desktop window so you don't see the jpeg smoothing the edges as seen in notebooks via jupyter_rfb

[kushalkolar]

I'm wondering what's the best way to customize the filter (and other Texture) settings for the displaying Images:

1. Always use a TextureView, so then:

   pygfx.Image(
   pygfx.Geometry(grid=TextureView(<data>, filter=<filter>)
   ...

Is there any downside to using TextureView rather than Texture for setting grid?

2. The other possibility is adding filter as a kwarg to Texture, which it can pass to TextureView instances when it creates them. I'm thinking that the first idea is better since it doesn't modify the API.

[almarklein]

Is there any downside to using TextureView rather than Texture for setting grid?

Pygfx sometimes allows specifying a texture when it really needs a textureview, and then creates the view itself. I think it would be a good idea to make this more consistent and simply require a view where it needs one.

[almarklein]

@rob-the-bot I think you want to set the textureview for the grid to nearest, and can probably leave the colormap to linear.

[kushalkolar]

Pygfx sometimes allows specifying a texture when it really needs a textureview, and then creates the view itself. I think it would be a good idea to make this more consistent and simply require a view where it needs one.

Thanks! It would be very useful (for performance) to set thekwargs that are used by Texture.get_view() when creating a Texture instance. Because the image_renderer just calls it with filter="linear".

If you think this addition is appropriate I can implement it :)

To elaborate:

This is slow

texture_view = pygfx.Texture(<data>, dim=2).get_view(filter="nearest")

image = pygfx.Image = pygfx.Image(
    pygfx.Geometry(grid=texture_view),
    pygfx.ImageBasicMaterial(clim=<minmax>, map=<some cmap>)
)

def animate(new_data):
    image.geometry.grid = pygfx.Texture(data, dim=2).get_view(filter="nearest")

This is fast but uses the default filter="linear":

texture = pygfx.Texture(<data>, dim=2)

image = pygfx.Image = pygfx.Image(
    pygfx.Geometry(grid=texture),
    pygfx.ImageBasicMaterial(clim=<minmax>, map=<some cmap>)
)

def animate(new_data):
    image.geometry.grid.data[:] = data
    image.geometry.grid.update_range((0, 0, 0), image.geometry.grid.size)

If you think modifying pygfx.Texture to accept and use kwargs like filter is detrimental I can probably just subclass it downstream for fastplotlib :)

[almarklein]

I think the correct approach would be:

texture = pygfx.Texture(<data>, dim=2)

image = pygfx.Image(
    pygfx.Geometry(grid=texture.get_view(filter="nearest")),
    pygfx.ImageBasicMaterial(clim=<minmax>, map=<some cmap>)
)

def animate(new_data):
    image.geometry.grid.data[:] = data
    image.geometry.grid.update_range((0, 0, 0), image.geometry.grid.size)

That said, the textureview is an abstraction that is extra cognitive load if you're just interested in an image. So we could think about making it a bit simpler. Some options, just to list some ideas:

• Let the texture define the default kwargs for the view (what you suggested).
• Let a texture have an internal view that is used when the texture is passed, and in the texture's init you can specify the filtering for it.
• Define a SimpleTexture that combines a texture and view in one.

As a side-note: in pygfx we don't support using part of a buffer yet (using offset and size). It might be that a future API for that includes something like a buffer view. If that happens, it would be good if we could keep the texture/textureview and buffer/bufferview API's similar.

[Korijn]

My vote is on the second option!

I don't think the texture/view abstraction is all that complicated. We could perhaps document it more clearly.

My simplified understanding is that the texture is just a reference to the data, and the view is the sampling configuration that you want to use when reading from a texture. So that's really what the material needs to have a reference to, in order to know what data to sample and how to sample that data.

It's a pretty powerful concept in the sense that you can sample the same data with different views!

[almarklein]

Another option that I mentioned in an earlier comment:

• Disallow passing a texture where a textureview is required. We'd have to adjust some of our examples, and we could document the distinction better.

[kushalkolar]

I think the correct approach would be:

texture = pygfx.Texture(<data>, dim=2)

image = pygfx.Image(
    pygfx.Geometry(grid=texture.get_view(filter="nearest")),
    pygfx.ImageBasicMaterial(clim=<minmax>, map=<some cmap>)
)

def animate(new_data):
    image.geometry.grid.data[:] = data
    image.geometry.grid.update_range((0, 0, 0), image.geometry.grid.size)

This does not work,

    image.geometry.grid.data[:] = np.random.rand(512, 512).astype(np.float32) * 255
AttributeError: 'TextureView' object has no attribute 'data'
Draw error: 'TextureView' object has no attribute 'data' (2)

• Let a texture have an internal view that is used when the texture is passed, and in the texture's init you can specify the filtering for it.

Do you mean a TextureView instance is always present in a Texture instead of a new instance created each time get_view() is called?

[almarklein]

Try:

image.geometry.grid.texture.data[:] = np.random.rand(512, 512).astype(np.float32) * 255
                     /\

Do you mean a TextureView instance is always present in a Texture instead of a new instance created each time get_view() is called?

That would be the idea of that option, yeah.

[almarklein]

Closing. Following up in https://github.com/pygfx/pygfx/issues/350

[kushalkolar]

Try:

image.geometry.grid.texture.data[:] = np.random.rand(512, 512).astype(np.float32) * 255
                     /\

Do you mean a TextureView instance is always present in a Texture instead of a new instance created each time get_view() is called?

That would be the idea of that option, yeah.

just want to followup because I finally got around to trying this and it works:

world_object.geometry.grid.texture.data[:] = np.random.rand(512, 512)
world_object.geometry.grid.texture.update_range((0, 0, 0), size=world_object.geometry.grid.texture.size)