Weird artifacts when using restore shaders with a LUT filter

[SpongeBed81]

When using restore shaders (GANUUL, CNNUL) with a LUT filter, weird red artifacts appear. However, this issue does not occur with the "Original" shader. Also, I noticed the same artifacts in various anime videos with CNNUL or GANUUL shader is enabled with a LUT filter.

Here is some information about my setup to help you guys better understand:

GPU: AMD Radeon RX6650XT (Driver version: 24.5.1) CPU: AMD Ryzen 5 5500 Ram: 16GB Operating system: Windows 11 Chrome version: 124.0.6367.210 (Official Build) (64 bit)

Also here is the chrome://gpu report:

about-gpu-2024-05-24T10-12-12-514Z.txt

Frieren: Beyond Journey's End:

Bocchi The Rock! (Turkish translation is baked into the video)

And here is the LUT filter which I inspired from this repo

// ../shaders/lutFilter.wgsl

struct VertexOut {
  @builtin(position) position : vec4<f32>,
  @location(1) texelCoords : vec2<f32>
}

@group(0) @binding(0) var linearSampler: sampler;
@group(0) @binding(1) var imageTexture: texture_2d<f32>;

//using the third binding is essential. I am passing the texture through the third binding in the WebGPU side.
@group(0) @binding(3) var filterImageTexture: texture_2d<f32>;

@vertex
fn vertex_main(@location(0) position: vec4<f32>, @location(1) texelCoords: vec2<f32>) -> VertexOut {
    var output: VertexOut;
    output.position = position;
    output.texelCoords = texelCoords;
    output.texelCoords.y = 1. - output.texelCoords.y;
    return output;
}

@fragment
fn fragment_main(fragData: VertexOut) -> @location(0) vec4<f32>
{
    var textureColor = textureSample(imageTexture, linearSampler, fragData.texelCoords).rgba;
    var blueColor = textureColor.b * 63.0;

    var quad1: vec2<f32>;
    quad1.y = floor(floor(blueColor) * 0.125);
    quad1.x = floor(blueColor) - (quad1.y * 8.0);

    var quad2: vec2<f32>;
    quad2.y = floor(ceil(blueColor) * 0.125);
    quad2.x = ceil(blueColor) - (quad2.y * 8.0);

    var texPos1: vec2<f32>;
    texPos1.x = ((quad1.x * 64.0) +  textureColor.r * 63.0 + 0.5)/512.0;
    texPos1.y = ((quad1.y * 64.0) +  textureColor.g * 63.0 + 0.5)/512.0;

    var texPos2: vec2<f32>;
    texPos2.x = ((quad2.x * 64.0) +  textureColor.r * 63.0 + 0.5)/512.0;
    texPos2.y = ((quad2.y * 64.0) +  textureColor.g * 63.0 + 0.5)/512.0;

    var newColor1 = textureSample(filterImageTexture, linearSampler, texPos1);
    var newColor2 = textureSample(filterImageTexture, linearSampler, texPos2);

    return mix(newColor1, newColor2, fract(blueColor));
}

And here is the overall code(click to expand)

## Key changes and notes 1. I always use the presentation format as "rgba16float" (can be seen in "configureWebGPU" function) 2. Removed fullscreenTexturedQuad.wgsl, sampleExternalTexture.frag.wgsl and their usages to use the LUT filter 3. Moved some variables inside "init" function to the top of the file as a global variable 4. If LUT texture hasn't been selected yet, it will just use the LUT texture from "/LUTs/default.png" for both "Original" and CNNUL modes(Also tested with different kinds of LUT textures which there was no difference about red artifcats) ```ts import Stats from './stats'; import { info, success } from '../utils/logger'; import * as anime4k from 'anime4k-webgpu'; import lutFilter from '../shaders/lutFilter.wgsl?raw'; // contains both vertex and fragment shaders for LUT filters const CNNUL = anime4k.CNNUL; const Original = anime4k.Original; let filterImageTexture: GPUTexture; let device: GPUDevice; let context: GPUCanvasContext | null; let updateRenderBindGroup: () => void; let presentationFormat: GPUTextureFormat; let oneFrame: () => void; let WIDTH: number; let HEIGHT: number; const vertices = new Float32Array([ -1.0, 1.0, 0.0, 1.0, 0.0, 1.0, -1.0, -1.0, 0.0, 1.0, 0.0, 0.0, 1.0, -1.0, 0.0, 1.0, 1.0, 0.0, 1.0, -1.0, 0.0, 1.0, 1.0, 0.0, 1.0, 1.0, 0.0, 1.0, 1.0, 1.0, -1.0, 1.0, 0.0, 1.0, 0.0, 1.0, ]); let currentStrategy: 'original' | '4k' = 'original'; function colorAttachments() { return [ { view: context!.getCurrentTexture().createView(), clearValue: { r: 0.0, g: 0.0, b: 0.0, a: 1.0 }, loadOp: 'clear', storeOp: 'store', }, ] as Iterable; } async function configureWebGPU(canvas: HTMLCanvasElement) { const adapter = await navigator.gpu.requestAdapter({ powerPreference: 'high-performance', }); if (!adapter) { throw new Error('WebGPU is not supported on this device'); } device = await adapter.requestDevice(); context = canvas.getContext('webgpu'); //const presentationFormat = navigator.gpu.getPreferredCanvasFormat(); presentationFormat = 'rgba16float'; context!.configure({ device, format: presentationFormat, alphaMode: 'premultiplied', }); return true; } async function selectLutTexture(texturePath = '/LUTs/default.png') { const filterImage = new Image(); filterImage.src = texturePath; await new Promise((resolve) => (filterImage.onload = resolve)); await filterImage.decode(); const filterImageSize = { width: filterImage.width, height: filterImage.height, }; filterImageTexture = device.createTexture({ size: filterImageSize, dimension: '2d', format: presentationFormat, usage: GPUTextureUsage.COPY_DST | GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.TEXTURE_BINDING, }); device.queue.copyExternalImageToTexture( { source: await createImageBitmap(filterImage), }, { texture: filterImageTexture, mipLevel: 0, }, filterImageSize, ); if (updateRenderBindGroup) { updateRenderBindGroup(); oneFrame(); } info('LUT filter texture created'); return true; } const init = async ({ canvas, video, lowPerformance, }: { canvas: HTMLCanvasElement; video: HTMLVideoElement; lowPerformance: () => void; }) => { const stats = new Stats(); const lastTwoSecondPerformance: number[] = []; const intv = setInterval(() => { const frameRate = stats.getFrameRate(); if ( !document.hidden && frameRate != undefined && !video.paused && currentStrategy == '4k' ) { lastTwoSecondPerformance.push(frameRate / video.playbackRate); } if (lastTwoSecondPerformance.length > 20) { if (lastTwoSecondPerformance.every((v) => v > 0 && v <= 10)) { updateRenderingStrategy('original'); updateRenderBindGroup(); oneFrame(); lowPerformance(); } lastTwoSecondPerformance.shift(); } }, 100); await configureWebGPU(canvas); const sampler = device.createSampler({ magFilter: 'linear', minFilter: 'linear', }); const lutModule = device.createShaderModule({ code: lutFilter, }); const verticesBuffer = device.createBuffer({ size: vertices.byteLength, usage: GPUBufferUsage.VERTEX | GPUBufferUsage.COPY_DST, mappedAtCreation: true, }); new Float32Array(verticesBuffer.getMappedRange()).set(vertices); verticesBuffer.unmap(); await selectLutTexture(); let videoFrameTexture: GPUTexture; function createTexture() { videoFrameTexture = device.createTexture({ size: [WIDTH, HEIGHT, 1], format: presentationFormat, usage: GPUTextureUsage.TEXTURE_BINDING | GPUTextureUsage.COPY_DST | GPUTextureUsage.RENDER_ATTACHMENT, }); } function updateSize() { WIDTH = video.videoWidth; HEIGHT = video.videoHeight; createTexture(); updateRenderingStrategy(currentStrategy); updateRenderBindGroup(); } function updateVideoFrameTexture() { device.queue.copyExternalImageToTexture({ source: video }, { texture: videoFrameTexture }, [ WIDTH, HEIGHT, ]); } let customPipeline: anime4k.Anime4KPipeline; const updateRenderingStrategy = (target: 'original' | '4k') => { currentStrategy = target; if (target == 'original') { customPipeline = new Original(videoFrameTexture); } else if (target == '4k') { customPipeline = new CNNUL(device, videoFrameTexture); } }; const renderBindGroupLayout = device.createBindGroupLayout({ label: 'Render Bind Group Layout', entries: [ { binding: 0, visibility: GPUShaderStage.FRAGMENT, sampler: {}, }, { binding: 1, visibility: GPUShaderStage.FRAGMENT, texture: {}, }, { binding: 2, visibility: GPUShaderStage.FRAGMENT, texture: {}, }, { binding: 3, visibility: GPUShaderStage.FRAGMENT, texture: {}, }, ], }); const renderPipelineLayout = device.createPipelineLayout({ label: 'Render Pipeline Layout', bindGroupLayouts: [renderBindGroupLayout], }); const vertexBufferLayout: GPUVertexBufferLayout = { attributes: [ { format: 'float32x4', offset: 0, shaderLocation: 0, }, { format: 'float32x2', offset: 16, shaderLocation: 1, }, ], arrayStride: 24, stepMode: 'vertex', }; const renderPipeline = device.createRenderPipeline({ layout: renderPipelineLayout, vertex: { module: lutModule, entryPoint: 'vertex_main', buffers: [vertexBufferLayout], }, fragment: { module: lutModule, entryPoint: 'fragment_main', targets: [ { format: presentationFormat, }, ], }, primitive: { topology: 'triangle-list', }, }); let renderBindGroup: GPUBindGroup; updateRenderBindGroup = () => { renderBindGroup = device.createBindGroup({ layout: renderBindGroupLayout, entries: [ { binding: 0, resource: sampler, }, { binding: 1, resource: customPipeline.getOutputTexture().createView(), }, { binding: 2, resource: videoFrameTexture.createView(), }, { binding: 3, resource: filterImageTexture.createView(), }, ], }); success('WebGPU: Update render bind group'); }; oneFrame = () => { updateSize(); if (!video.paused) { return; } updateVideoFrameTexture(); const commandEncoder = device.createCommandEncoder(); customPipeline.pass(commandEncoder); const passEncoder = commandEncoder.beginRenderPass({ colorAttachments: colorAttachments(), }); passEncoder.setPipeline(renderPipeline); passEncoder.setVertexBuffer(0, verticesBuffer); passEncoder.setBindGroup(0, renderBindGroup); passEncoder.draw(6); passEncoder.end(); device.queue.submit([commandEncoder.finish()]); success('WebGPU: Begin render pass'); }; function frame() { stats.begin(); if (!video.paused) { updateVideoFrameTexture(); } const commandEncoder = device.createCommandEncoder(); customPipeline.pass(commandEncoder); const passEncoder = commandEncoder.beginRenderPass({ colorAttachments: colorAttachments(), }); passEncoder.setPipeline(renderPipeline); passEncoder.setVertexBuffer(0, verticesBuffer); passEncoder.setBindGroup(0, renderBindGroup); passEncoder.draw(6); passEncoder.end(); device.queue.submit([commandEncoder.finish()]); stats.end(); video.requestVideoFrameCallback(frame); } video.requestVideoFrameCallback(() => { if (!customPipeline) oneFrame(); frame(); }); return { switchResolution(target: '4k' | 'original') { if (target == currentStrategy) return; updateRenderingStrategy(target); updateRenderBindGroup(); oneFrame(); }, getCurrentStrategy() { return currentStrategy; }, video, canvas, destroy() { clearInterval(intv); video.removeEventListener('canplaythrough', oneFrame); }, oneFrame, frame, selectLutTexture, }; }; export default init; ```

[plasmas]

Thanks for opening the issue and the detailed description. I used your code and the LUT filter in the repo you mentioned but didn't find any red artifacts. I'm currently testing on M1 Pro but will use NVIDIA cards to test this later. The artifact seem to appear around black edge boarders, which might be due to some kind of underflow/overflow. We will try to reproduce and look into it.

[SpongeBed81]

Thanks for letting me know! As you mentioned, the artifacts appear around black edges. I used the LUT filter from the GitHub repository, and while the artifacts are still present, they are now white because the LUT filter applies a grayscale effect.

As I noted earlier, these artifacts do not appear when the "Original" shader is enabled. It seems the restore shaders are causing this issue.

LUT filter & CNNUL (I recommend opening this in a new tab for a closer look):

LUT filter & Original

The color of the artifacts depends significantly on the LUT filter used. From now on, I'll refer to them simply as "artifacts."

I am looking forward to seeing the results on NVIDIA cards!

[SpongeBed81]

I am also including all of my LUT textures in a zip file to help you better debug the issue.

LUTs.zip (default.png should display the video as is)

[plasmas]

Using your default LUT image, I can confirm the artifact exists on both M1 Pro and NVIDIA cards (RTX 40 series), and the issue is with out-of-bound floating point numbers. Also, the artifact exists on upscale pipelines.

To be specific, the texture output by our pipeline contains colors not in range [0, 1]. We have never noticed this artifact before both on webGPU and MPV so I suspect the LUT shader might have magnified the effect a bit.

For a simple temporary fix, you can clamp the color after it is read (in the fragment shader):

textureColor = clamp(textureColor, vec4<f32>(0., 0., 0., 0.), vec4<f32>(1., 1., 1., 1.));

This has resolved the artifacts on my end.

The issue might be inherent to the weights of the network themselves, so we plan to include the clamp at the end of every pipeline in the next release. Thanks for putting it under our radar.

[SpongeBed81]

Thanks for the quick response and the fix. Clamping the color values in the fragment shader worked perfectly, and the artifacts are gone on my end too. Appreciate you looking into this and planning to include the fix in the next release.

[plasmas]

Shaders updated to fix this issue in v1.0.0. Note that v1.0.0 contains some minor API changes (see readme for details). Closing this issue for now.