Multi-texturing with `filter_mode='linear'` and `boundary_mode='zero'` broken (?)

[mworchel]

Hi,

I have encountered unexpected behavior when I tried to sample multiple textures (dr.texture) with filter_mode='linear' and boundary_mode='zero': the sampling succeeds for the first texture in the minibatch but outputs zero images for the remaining textures. I have verified that this only occurs for this combination of filter and boundary mode. My minimal reproducer attached at the end generates the following output:

I assume this is not the expected behavior?

I am using nvdiffrast version 0.3.0, pytorch 1.12.1 on Windows 10.

Minimal reproducer:

import matplotlib.pyplot as plt
import nvdiffrast.torch as dr
import torch

device = torch.device('cuda:0')

def create_dummy_texture(width, height, device):
    dummy = torch.zeros((height, width, 3), dtype=torch.float32, device=device)
    dummy[:dummy.shape[0]//2, :dummy.shape[1]//2, :] = torch.tensor([1.0, 0.0, 0.0], device=device)
    dummy[:dummy.shape[0]//2, dummy.shape[1]//2:, :] = torch.tensor([0.0, 1.0, 0.0], device=device)
    dummy[dummy.shape[0]//2:, :dummy.shape[1]//2, :] = torch.tensor([1.0, 1.0, 0.0], device=device)
    dummy[dummy.shape[0]//2:, dummy.shape[1]//2:, :] = torch.tensor([0.0, 0.0, 1.0], device=device)
    return dummy

h, w = 16, 16
uv = torch.stack(torch.meshgrid([
    (torch.arange(0, h, device=device) + 0.5) / h,
    (torch.arange(0, w, device=device) + 0.5) / w
], indexing='ij')[::-1], dim=-1)

uv      = torch.stack([uv, uv-0.2, uv+0.2])
texture = create_dummy_texture(32, 32, device=device)[None].repeat((uv.shape[0], 1, 1, 1))

results = []
for filter_mode in ['nearest', 'linear']:
    for boundary_mode in ['zero', 'clamp', 'wrap']:
        results += [ (f"{filter_mode}-{boundary_mode}", dr.texture(texture, uv, filter_mode=filter_mode, boundary_mode=boundary_mode)) ]

fig, axs = plt.subplots(len(results), 3, figsize=(0.3*len(results), 1.5*3), constrained_layout=True)
for ax in axs.flatten():
    ax.set_xticks([])
    ax.set_yticks([])

for i, (name, images) in enumerate(results):
    axs[i, 1].set_title(name)
    for j, image in enumerate(images):
        axs[i, j].imshow(image.cpu())
plt.show()

[s-laine]

Thanks for the bug report, and sorry for the delayed answer. You're correct, this is not the intended behavior and there is a bug in the zero boundary code.

The bug occurs only when the texture has minibatch dimension greater than 1, i.e., you don't rely on broadcasting the texture tensor along the minibatch axis. That is probably why it hasn't been noticed before. For example, if you removed the .repeat((uv.shape[0], 1, 1, 1)) in the reproducer, the bug doesn't trigger.

As a quick fix, you can replace the following block in nvdiffrast/common/texture.cu, lines 450–455, in your installation:

    iu0 += tz * w * h;
    iu1 += tz * w * h;
    tcOut.x = iu0 + w * iv0;
    tcOut.y = iu1 + w * iv0;
    tcOut.z = iu0 + w * iv1;
    tcOut.w = iu1 + w * iv1;

with

    int iu0z = iu0 + tz * w * h;
    int iu1z = iu1 + tz * w * h;
    tcOut.x = iu0z + w * iv0;
    tcOut.y = iu1z + w * iv0;
    tcOut.z = iu0z + w * iv1;
    tcOut.w = iu1z + w * iv1;

I'll include a fix in the next release and leave this issue open until then. Again, big thanks for the report and the repro.

[mworchel]

Awesome, looking forward to the next release and thanks for the intermediate fix.

Agreed, this setting is probably very uncommon. I need the functionality for (differentiable) shadow mapping with a batch of lights, so at least this fix will be useful in a real scenario (replacing a workaround using torch's grid_sample function).

Thanks again for looking into it.

[s-laine]

Fixed in v0.3.1, closing the issue.