search

CrendKing / avisynth_filter

DirectShow filters that put AviSynth and VapourSynth into video playing
MIT License
108 stars 8 forks source link

AVSF with the latest Avisynth+ using ConvertToYUV420 or ConvertBits, the player does not respond. #66

Closed BlackMickey closed 2 years ago

BlackMickey commented 2 years ago

Environment

Describe the bug

AVSF with Avisynth+ released earlier than November 3, 2021, the player works fine. AVSF with Avisynth+ released later than November 5, 2021, the player does not respond. However, SVP's scripts using these two functions, the player has no playback problems.

AviSynth+ Version AVSF [ConvertToYUV420] AVSF [ConvertBits(16)] ffdshow [ConvertToYUV420] AVSF [SVP]
v3.7.0 OK OK OK OK
v3.7.1_20211103_test23 OK OK OK OK
v3.7.1_20211104_test24 No response OK OK OK
v3.7.1_20211105_test25 No response No response OK OK
v3.7.1 No response No response OK OK
v3.7.2 No response No response OK OK

no_response

CrendKing commented 2 years ago

Yeah. I think it is the same problem as vapoursynth-mvtools, which tries to get frame 0 to access properties. Unfortunately, they decided to not change that.

The problem is introduced in this commit, which as you observed, is around v3.7.1. AviSynth+ has a related issue. And as the release note of their v3.7.2 says, if you specifies fulls argument, it should work.

SVP works because it uses AviSynth+ 3.5.1.

Unless AviSynth+ and VapourSynth introduce the concept of clip properties, there is nothing this filter can do to fix this (i.e. there is no frame to get during initialization, since the playback hasn't started). You have to stay at an earlier version of AviSynth+.

CrendKing commented 2 years ago

@BlackMickey You could try https://github.com/CrendKing/avisynth_filter/actions/runs/2193657724 to see if AVS 3.7.1/2 works now. Also https://github.com/CrendKing/avisynth_filter/issues/70.

BlackMickey commented 2 years ago

The player will no longer hang. However, when the script contains 2 functions (ConvertBits or ConvertToYUV420) to play YUV444 format video, the first frame is green. Other than that, everything is fine.

Avisynth: v3.7.2 Avisynth Filter: https://github.com/CrendKing/avisynth_filter/actions/runs/2207139952

Script:

input_m = AvsFilterSource()    # YUV444P10
input_m = Is420(input_m) ? input_m : ConvertToYUV420(input_m)    # YUV420P10
input_m = input_m.ConvertToYUV420()    # YUV420P10
input_m.Subtitle(
    \   "\nPixelType = " + String(PixelType(input_m))
    \ + "\nIsPlanar = " + String(IsPlanar(input_m))
    \ + "\nHasAlpha = " + String(HasAlpha(input_m))
    \ + "\nBitsPerComponent = " + String(BitsPerComponent(input_m))
    \ , font="courier", text_color=$ffffff, size=32, align=9, lsp=0) # YUV420P10

image

CrendKing commented 2 years ago

Tried your script on a YUV444P10 video and can't reproduce. Could you share the log? Potentially I will also need your video to hopefully repro.

BlackMickey commented 2 years ago

Link: https://drive.google.com/file/d/19wowhkJZiOcTz1irKh6CaB9KslUzHOEH/view?usp=sharing

global threads = 4

input_m = AvsFilterSource()    # YUV444P10 (Y410)
input_m = Is420(input_m) ? input_m : ConvertToYUV420(input_m)    # YUV420P10
input_m = input_m.ConvertBits(8)    # YUV420P8 (YV12)
input_m.Subtitle(
    \   "\nPixelType = " + String(PixelType(input_m))
    \ + "\nIsPlanar = " + String(IsPlanar(input_m))
    \ + "\nHasAlpha = " + String(HasAlpha(input_m))
    \ + "\nBitsPerComponent = " + String(BitsPerComponent(input_m))
    \ , font="courier", text_color=$ffffff, size=32, align=9, lsp=0) # YUV420P8 (YV12)
CrendKing commented 2 years ago

Just to confirm, does the first green frame stay forever until you seek, or it's just a flash?

BlackMickey commented 2 years ago

just a flash.

CrendKing commented 2 years ago

Same. I think it's the best we can do, because at the time ConvertBits requests that first frame, we don't have frame to give it. So I created dummy frame (with no data, which shows as green by default). If that filter actually passes that frame downstream, instead of request frame 0 later again, then you see green frame.

If you really don't like that one green frame, you still can change ConvertBits(8) to something like ConvertBits(8, fulls=true). But it is specific to certain filters like ConvertBits. Not every filter has that kind of workaround. The best we can do, still, is either hope AviSynth+ stop doing this, or request them to formally introduce clip property.

BlackMickey commented 2 years ago

I understand. Thank you for all your hard work.

CrendKing commented 2 years ago

I took some time to rethink about the filter's architecture, and I thought a way to actually fix the frame 0 problem instead of just working around it. You can find a test build at https://github.com/CrendKing/avisynth_filter/actions/runs/2211396382. This is experimental, so expect to have bug. Let me know if you still see green frame.

Also, if you can, also test the VapourSynth version. VapourSynth mvtools also requests frame 0.

BlackMickey commented 2 years ago

If AviSynth Filter disable the logging, the player crashes when it is played.

CrendKing commented 2 years ago

Very interesting finding. The cause is that logging makes worker thread slightly slower, just enough to avoid a race condition. Because I never disable logging during development, I couldn't find this.

https://github.com/CrendKing/avisynth_filter/actions/runs/2212279321 should fix this.

BlackMickey commented 2 years ago

It works fine.

CrendKing commented 2 years ago

Great. I'll give it some time. If nobody has found any problem, I'll merge and release this approach in the next version.

Just to document for future readers: The reason why I think the frame 0 is a problem was that because there used to be one frameserver, which is both used for media type negotiation and frame serving. Later we broke that into main and aux frameserver, where ideally the former is purely used for frame serving and the latter purely for negotiation. However, until now, the implementation doesn't follow that principle perfectly, i.e. some negotiation is still done in main, which happens earlier than the receiving of the first source frame. Because of that, I used to think this is a chicken-egg problem.

By moving all negotiation logic to aux, followed by delaying activation of the main until certain number of source frames have been stored, we can then properly solve the issue. The number of pre-buffered frames is currently hardcoded as 3. If needed, this can be increased, at the cost of delaying the first frame after start/seeking.

Disaer commented 2 years ago

Hello.

Looks like 3 pre-buffered frames is not enough for some temporal filters. TemporalDegrain2() works fine but TemporalDegrain2(degrainTR=2) cause AVSF to stuck.

T  25792 @        0: Filter version: 1.4.1 # 5cbac67
T  25792 @        0: Configured script file: test.avs
T  25792 @        0: Configured input format  NV12: 1
T  25792 @        0: Configured input format  YV12: 1
T  25792 @        0: Configured input format  I420: 1
T  25792 @        0: Configured input format  IYUV: 1
T  25792 @        0: Configured input format  P010: 1
T  25792 @        0: Configured input format  P016: 1
T  25792 @        0: Configured input format  YUY2: 1
T  25792 @        0: Configured input format  P210: 1
T  25792 @        0: Configured input format  P216: 1
T  25792 @        0: Configured input format  YV24: 1
T  25792 @        0: Configured input format  Y410: 1
T  25792 @        0: Configured input format  Y416: 1
T  25792 @        0: Configured input format RGB24: 0
T  25792 @        0: Configured input format RGB32: 0
T  25792 @        0: Loading process: mpc-hc64.exe
T  25792 @        0: FrameServerCommon()
T  25792 @        3: AviSynth version: AviSynth+ 3.7.2 (r3661, 3.7, x86_64)
T  25792 @        3: AviSynth supports frame properties
T  25792 @        5: CSynthFilter(): 0000017B78451610
T  25792 @        5: ReloadScript from auxiliary frameserver
T  25792 @      172: Source frame      0 is requested without the frame handler being linked
T  25792 @      557: Source frame      0 is requested without the frame handler being linked
T  25792 @      562: Source frame      0 is requested without the frame handler being linked
T  25792 @      570: Source frame      1 is requested without the frame handler being linked
T  25792 @      592: Source frame      2 is requested without the frame handler being linked
T  25792 @      842: Source frame      3 is requested without the frame handler being linked
T  25792 @      858: Source frame      0 is requested without the frame handler being linked
T  25792 @      860: Source frame      2 is requested without the frame handler being linked
T  25792 @      960: Source frame      4 is requested without the frame handler being linked
T  25792 @     1400: New script clip: 0000017B7B1513C0
T  25792 @     1400: Release script clip: 0000017B7B1513C0
T  25792 @     1448: Add compatible formats: input  NV12 output  NV12
T  25792 @     1448: Add compatible formats: input  NV12 output  YV12
T  25792 @     1448: Add compatible formats: input  NV12 output  I420
T  25792 @     1448: Add compatible formats: input  NV12 output  IYUV
T  25792 @     1448: ReloadScript from auxiliary frameserver
T  25792 @     1613: Source frame      0 is requested without the frame handler being linked
T  25792 @     1989: Source frame      0 is requested without the frame handler being linked
T  25792 @     1994: Source frame      0 is requested without the frame handler being linked
T  25792 @     2002: Source frame      1 is requested without the frame handler being linked
T  25792 @     2024: Source frame      2 is requested without the frame handler being linked
T  25792 @     2312: Source frame      3 is requested without the frame handler being linked
T  25792 @     2328: Source frame      0 is requested without the frame handler being linked
T  25792 @     2331: Source frame      2 is requested without the frame handler being linked
T  25792 @     2433: Source frame      4 is requested without the frame handler being linked
T  25792 @     2860: New script clip: 0000017B4AD40400
T  25792 @     2860: Release script clip: 0000017B4AD40400
T  25792 @     2903: Add compatible formats: input  YV12 output  NV12
T  25792 @     2903: Add compatible formats: input  YV12 output  YV12
T  25792 @     2903: Add compatible formats: input  YV12 output  I420
T  25792 @     2903: Add compatible formats: input  YV12 output  IYUV
T  25792 @     2903: ReloadScript from auxiliary frameserver
T  25792 @     3062: Source frame      0 is requested without the frame handler being linked
T  25792 @     3062: Source frame      0 is requested without the frame handler being linked
T  25792 @     3681: New script clip: 0000017B4CED1A10
T  25792 @     3681: Release script clip: 0000017B4CED1A10
T  25792 @     3711: Add compatible formats: input  P010 output  P010
T  25792 @     3711: ReloadScript from auxiliary frameserver
T  25792 @     3867: Source frame      0 is requested without the frame handler being linked
T  25792 @     3868: Source frame      0 is requested without the frame handler being linked
T  25792 @     4347: New script clip: 0000017B53F16EA0
T  25792 @     4347: Release script clip: 0000017B53F16EA0
T  25792 @     4375: Add compatible formats: input  P016 output  P016
T  25792 @     4375: ReloadScript from auxiliary frameserver
T  25792 @     4532: Source frame      0 is requested without the frame handler being linked
T  25792 @     4533: Source frame      0 is requested without the frame handler being linked
T  25792 @     5058: New script clip: 0000017B7AC11FF0
T  25792 @     5058: Release script clip: 0000017B7AC11FF0
T  25792 @     5087: Add compatible formats: input  P216 output  P216
T  25792 @     5087: ReloadScript from auxiliary frameserver
T  25792 @     5268: Source frame      0 is requested without the frame handler being linked
T  25792 @     5268: Source frame      0 is requested without the frame handler being linked
T  25792 @     5953: New script clip: 0000017B7AF17C00
T  25792 @     5953: Release script clip: 0000017B7AF17C00
T  25792 @     5981: Add compatible formats: input  P210 output  P210
T  25792 @     5981: ReloadScript from auxiliary frameserver
T  25792 @     6158: Source frame      0 is requested without the frame handler being linked
T  25792 @     6159: New script clip: 0000017B790AFE20
T  25792 @     6159: Release script clip: 0000017B790AFE20
T  25792 @     6170: Add compatible formats: input  YUY2 output  YUY2
T  25792 @     6170: Reject input format due to settings: RGB32
T  25792 @     6170: Reject input format due to settings: RGB32
T  25792 @     6170: Reject input format due to settings: RGB24
T  25792 @     6170: Reject input format due to settings: RGB24
T  25792 @     6170: ReloadScript from auxiliary frameserver
T  25792 @     6349: Source frame      0 is requested without the frame handler being linked
T  25792 @     6350: Source frame      0 is requested without the frame handler being linked
T  25792 @     6759: New script clip: 0000017B79055DD0
T  25792 @     6759: Release script clip: 0000017B79055DD0
T  25792 @     6789: Add compatible formats: input  Y416 output  Y416
T  25792 @     6789: ReloadScript from auxiliary frameserver
T  25792 @     6950: Source frame      0 is requested without the frame handler being linked
T  25792 @     6950: Source frame      0 is requested without the frame handler being linked
T  25792 @     7496: New script clip: 0000017B7B34A8F0
T  25792 @     7496: Release script clip: 0000017B7B34A8F0
T  25792 @     7526: Add compatible formats: input  Y410 output  Y410
T  25792 @     7526: ReloadScript from auxiliary frameserver
T  25792 @     7690: Source frame      0 is requested without the frame handler being linked
T  25792 @     8011: Source frame      0 is requested without the frame handler being linked
T  25792 @     8020: Source frame      0 is requested without the frame handler being linked
T  25792 @     8036: Source frame      1 is requested without the frame handler being linked
T  25792 @     8075: Source frame      2 is requested without the frame handler being linked
T  25792 @     8566: Source frame      3 is requested without the frame handler being linked
T  25792 @     8592: Source frame      0 is requested without the frame handler being linked
T  25792 @     8597: Source frame      2 is requested without the frame handler being linked
T  25792 @     8783: Source frame      4 is requested without the frame handler being linked
T  25792 @     9481: New script clip: 0000017B4B3505A0
T  25792 @     9481: Release script clip: 0000017B4B3505A0
T  25792 @     9539: Add compatible formats: input  YV24 output  YV24
T  25792 @     9539: Pre pin connection CheckInputType(): input  NV12 result 1
T  25792 @     9618: GetMediaType() offers media type  0 with  NV12
T  25792 @     9618: GetMediaType() offers media type  1 with  YV12
T  25792 @     9618: GetMediaType() offers media type  2 with  I420
T  25792 @     9618: GetMediaType() offers media type  3 with  IYUV
T  25792 @     9618: GetMediaType() offers media type  4 with  P010
T  25792 @     9618: GetMediaType() offers media type  5 with  P016
T  25792 @     9618: GetMediaType() offers media type  6 with  P216
T  25792 @     9618: GetMediaType() offers media type  7 with  P210
T  25792 @     9618: GetMediaType() offers media type  8 with  YUY2
T  25792 @     9618: GetMediaType() offers media type  9 with  Y416
T  25792 @     9618: GetMediaType() offers media type 10 with  Y410
T  25792 @     9618: GetMediaType() offers media type 11 with  YV24
T  25792 @     9762: GetMediaType() offers media type  0 with  NV12
T  25792 @     9770: Pins are connected with media types:  NV12 ->  NV12
T  25792 @     9770: Pin connections are settled
T  25792 @     9770: Filter in graph: LAV Splitter Source (internal)
T  25792 @     9770: Filter in graph: LAV Video Decoder (internal)
T  25792 @     9770: Filter in graph: AviSynth Filter
T  25792 @     9770: Filter in graph: madVR
T  25792 @     9770: GetMediaType() offers media type  0 with  NV12
T  27128 @     9770: Start worker thread
T  25792 @    10268: ReloadScript from auxiliary frameserver
T  25792 @    10450: Source frame      0 is requested without the frame handler being linked
T  25792 @    10830: Source frame      0 is requested without the frame handler being linked
T  25792 @    10835: Source frame      0 is requested without the frame handler being linked
T  25792 @    10843: Source frame      1 is requested without the frame handler being linked
T  25792 @    10865: Source frame      2 is requested without the frame handler being linked
T  25792 @    11118: Source frame      3 is requested without the frame handler being linked
T  25792 @    11135: Source frame      0 is requested without the frame handler being linked
T  25792 @    11138: Source frame      2 is requested without the frame handler being linked
T  25792 @    11242: Source frame      4 is requested without the frame handler being linked
T  25792 @    11681: New script clip: 0000017B56453140
T  25792 @    11681: Release script clip: 0000017B56453140
T  18456 @    11761: Stored source frame:      0 at          0 ~     400000 duration(literal)     400000 max_requested      0 extra_buffer      0
T  18456 @    11762: Stored source frame:      1 at     400000 ~     800000 duration(literal)     400000 max_requested      0 extra_buffer      0
T  18456 @    11764: Stored source frame:      2 at     800000 ~    1200000 duration(literal)     400000 max_requested      0 extra_buffer      0
T  18456 @    11764: ReloadScript from main frameserver
T  18456 @    11940: Get source frame: frameNb      0 input queue size  3
T  18456 @    11940: Return source frame      0
T  18456 @    12350: Get source frame: frameNb      0 input queue size  3
T  18456 @    12351: Return source frame      0
T  18456 @    12355: Get source frame: frameNb      0 input queue size  3
T  18456 @    12355: Return source frame      0
T  18456 @    12363: Get source frame: frameNb      1 input queue size  3
T  18456 @    12363: Return source frame      1
T  18456 @    12387: Get source frame: frameNb      2 input queue size  3
T  18456 @    12387: Return source frame      2
T  18456 @    12658: Get source frame: frameNb      3 input queue size  3
CrendKing commented 2 years ago

Looks like 5 frames were requested. Since I can't test every single use case out there, I can either make the number really big, like 20, hoping nothing requests that much, or make it configurable.

Disaer commented 2 years ago

I'm fine with either option, but making it configurable looks more versatile and flexible.

Nuihc88 commented 2 years ago

I would recommend making it configurable to make diagnosing of possible future issues easier; also would recommend defaulting to '5' for now and then if later someone reports a problem with a reasonably common filter requesting (for example) 7 frames, i think that should become new default (and so on).

CrendKing commented 2 years ago

I can make it configurable. But know that the trade off is that bigger pre-buffer equals longer seek time, meaning your player will only start to update after receiving all the pre-buffer frames. I certainly don't want to impose the (possibly unnecessary) 2-frame delay to EVERY user just because some of you use TemporalDegrain2(degrainTR=2).

UPDATE: Test build: https://github.com/CrendKing/avisynth_filter/actions/runs/2305758529 Wiki update: https://github.com/CrendKing/avisynth_filter/wiki/Configuration

Nuihc88 commented 2 years ago

I ran some quick tests on my over 15 years old hardware and the difference in seek times between InitialSrcBuffer of 2 & 128 is less than a second with FullHD footage being fed through SVPflow, leading to an average delay of <8ms/frame at most. My guess is that since the actual processing steps take so much longer than pre-buffering, there isn't going to be an observable difference while using sane values; not that i mind the current defaults.

Disaer commented 2 years ago

Thanks for the new option, now temporal filters like TemporalDegrain2 or BM3D work much better!

CrendKing commented 2 years ago

actual processing steps take so much longer than pre-buffering

I'm not sure what you mean here. Pre-buffer is nothing special. It is the same processing, just with all the processed frames withheld without being showed. The delay is heavily influenced with the content of the script. I ran 128 with a VPSF mvtools script (which is less efficient than SVP) on a large 4:4:4 video, and it hangs for 10 seconds and almost killed my system because of the spike of memory usage. I still think responsiveness is important, because one of the original key factors I hate ffdshow about is its long seek delay.

Nuihc88 commented 2 years ago

I'm not sure what you mean here. Pre-buffer is nothing special. It is the same processing, just with all the processed frames withheld without being showed.

Ok, due to the naming i thought this buffer was withholding frames in between Source Filter and AviSynth, rather than AviSynth and Video Renderer.

The delay is heavily influenced with the content of the script. I ran 128 with a VPSF mvtools script (which is less efficient than SVP) on a large 4:4:4 video, and it hangs for 10 seconds and almost killed my system because of the spike of memory usage.

I guess the main bottle-neck then is host system's memory bandwidth... While some filters regulate buffer by size instead of frame count, i think that in this case that approach would only add unnecessary overhead as this buffer is only a workaround for technical problems in the first place.

Cache size of 128 was just for calculating a more reliable average; in real world i wouldn't recommend using a value higher than used renderer's buffer size (usually within 5-16 range); from what i've read, most implementations of EVR default to buffer size of 5 (...7 in PP's Custom Presenter), while MadVR can default to various values depending on the rendering mode used.

I still think responsiveness is important, because one of the original key factors I hate ffdshow about is its long seek delay.

I fully agree that responsiveness is important and that the value should be kept minimal; what i'm trying to figure out is whether there is a perfect balance between reliability and performance, as well as where that would be.

For example, most people who encounter stuttering issues when loading a new video or seeking, solve it by slightly increasing their renderer's buffer size, so after recalling these comments 1 & 2, i started wondering if the same phenomena was at play with AVSF.....

While testing this buffer i did notice some weak correlations with specific clips on EVR, but am still having trouble creating a minimal set of conditions for reliably replicating the issue to then tune InitialSrcBuffer for getting around it, likely due to my PC not being suitable for 4K+ playback where this problem would be more readily observable. All i can say with reasonable certainty is that increasing InitialSrcBuffer, appears to reduce the likelyhood of stutters occurring, at least up to used renderer's configured buffer size.

The way i see it, any value that avoids most usability and compatibility problems that people are likely to encounter while still minimizing overhead would be a good default to use. Whether that is something that can be to some extent optimized universally or has to be manually configured for each system, i'm not really sure.