Anime4K shaders only seem to work in preview but fail during encoding

[Selur]

Not sure where the problem is. Using the shaders from https://github.com/bloc97/Anime4K/releases and for example:

# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/libvs_placebo.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/FFMS2/ffms2.dll")
# source: 'G:\TestClips&Co\files\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading source using FFMS2
clip = core.ffms2.Source(source="G:/TestClips&Co/files/test.avi",cachefile="E:/Temp/avi_6c441f37d9750b62d59f16ecdbd59393_853323747.ffindex",format=vs.YUV420P8,alpha=False)
# making sure input color matrix is set as 470bg
clip = core.resize.Bicubic(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# GLSL filter: Anime4K_Darken_VeryFast.glsl
# adjusting color space from YUV420P8 to YUV444P16 for VsGLSLFilter
clip = core.resize.Bicubic(clip=clip, format=vs.YUV444P16, range_s="limited")
clip = core.placebo.Shader(clip=clip, shader="C:/Users/Selur/Desktop/Anime4k/Anime4K_Darken_VeryFast.glsl", width=640, height=352)
# adjusting output color from: YUV444P16 to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, range_s="limited")
# set output frame rate to 25.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()

that the preview works fine, vspipe --info works fine, but encoding using: [code]vspipe "E:\Temp\encodingTempSynthSkript_2021-12-26@07_38_12_9010.vpy" - -c y4m | x264 --preset veryfast --crf 18.00 --profile high --level 4.1 --ref 3 --direct auto --b-adapt 0 --sync-lookahead 48 --qcomp 0.50 --rc-lookahead 40 --qpmax 51 --partitions i4x4,p8x8,b8x8 --no-fast-pskip --subme 5 --aq-mode 0 --vbv-maxrate 62500 --vbv-bufsize 78125 --sar 1:1 --non-deterministic --range tv --colormatrix bt470bg --demuxer y4m --input-range tv --fps 25/1 --output-depth 8 --output "E:\Temp\2021-12-26@07_38_12_9010_03.264" -

fails with: `y4m [error]: bad frame header magic`

Did I make a mistake somewhere?
Is this a known limitation?
Is this the glsl filter somewho requiring an active preview to work and can't be used during encoding?

Cu Selur

[Lypheo]

Looks like a vspipe bug to me

[Selur]

Any idea why this only happens with the Anime4k models? (tons of other models work fine)

[quietvoid]

There does seem to be some different behaviour in previewing vs requesting a frame manually. I'm not sure what but I'm investigating since placebo.Deband doesn't work at all when previewing.

For example, running a script with python using clip.get_frame(0) processes the shader properly. But nothing happens when previewing with vsedit..

[Selur]

Just tested core.placebo.Shader again with the Anime4k model now and it seems like it's working. Also tried placebo.Deband

# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/libvs_placebo.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/FFMS2/ffms2.dll")
# source: 'G:\TestClips&Co\files\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading source using FFMS2
clip = core.ffms2.Source(source="G:/TestClips&Co/files/test.avi",cachefile="E:/Temp/avi_6c441f37d9750b62d59f16ecdbd59393_853323747.ffindex",format=vs.YUV420P8,alpha=False)
# making sure input color matrix is set as 470bg
clip = core.resize.Bicubic(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
original = clip
# applying debanding using vs-placebo.Deband
clip = core.placebo.Deband(clip=clip, threshold=100.00, planes=1)
# adjusting output color from: YUV420P8 to YUV422P10 for ProResModel
original = core.resize.Bicubic(clip=original, format=vs.YUV422P10, range_s="limited")
clip = core.resize.Bicubic(clip=clip, format=vs.YUV422P10, range_s="limited")
original = core.text.Text(clip=original,text="Original",scale=1)
clip = core.text.Text(clip=clip,text="Filtered",scale=1)
stacked = core.std.StackHorizontal([original,clip])
# set output frame rate to 25.000fps
stacked = core.std.AssumeFPS(clip=stacked, fpsnum=25, fpsden=1)
# Output
stacked.set_output()

and it works fine here (used threshold=100 to be sure one can see a difference ;)) (I'm using https://github.com/Selur/vsViewer which is basically vsedit)

[quietvoid]

Yeah it might be platform dependent as well.

Here's the output from your script for me when previewing:

Directly writing with imwri.Write:

[Selur]

Argh, that does look strange. May be a driver issue? (using nvidia drivers 511.65 here) Sometimes rebooting helps. :)

[quietvoid]

No, it's most likely related to how frame requests are done by previewers compared to how the plugin gets them. Like I said, the frames never reach libplacebo when previewing with Deband.

[Selur]

Okay, correction: Some of the Anime4k shaders still only work inside the preview and not during encoding. Anime4K_Darken_HQ for example does work properly during preview and encoding here. (Anime4K_Deblur_Original does only work during preview) Deband however does work here in preview and encoding.

So you are probably right that depending on some unkown sometimes libplacebo gets triggered and sometimes not.

[quietvoid]

On my side I discovered that the function names in the code are conflicting with some existing libraries on my system. For example, filter in deband.c was being replaced by some filter function in libncursesw.so.

So I guess I'll rename everything.

[quietvoid]

Anime4K_Darken_HQ for example does work properly during preview and encoding here. (Anime4K_Deblur_Original does only work during preview)

I tried reproducing this on my Windows machine but I'm not able to get x264 to fail encoding.

[quietvoid]

You could try this build (from 01f2378) to see if it makes any difference: https://0x0.st/oXMG.zip

[Selur]

I tried reproducing this on my Windows machine but I'm not able to get x264 to fail encoding. Encoding doesn't fail! It's just that the filtering that should take place doesn't. You see the efffect during preview, but when looking at the encode it's not there. (I made sure to take ridiculously high value to be sure this couldn't be an effect of x264/x265 (I usually use x265).

Tried the liked version: placebo.Deband -> works too for preview and encoding, but to be frank I also found a bug in Hybrid which could lead to the filter not being added to the call properly so I'm unsure whether that was the cause of my issue. (I guess it was) => at least my issue seems to be fixed. :)

[quietvoid]

I made a 1.4.1 release which should solve this. It's pretty much the same as the previously linked build.

[Selur]

Thanks! I can confirm that 1.4.1 works. :)