analog capture card jitter and sync offset issues?

[Usulyre]

Environment

• Filter version: 0.81
• AviSynth+ version: [ v3.6.1]
• OS: [e.g. Windows 10 x64]
• Video player: [ MPC-HC x86 1.9.8]
• Splitter filter and video decoder: [default decoder]
• Video renderer: [ EVR]
• Video format: yuy2
• AviSynth script:

  SetMemoryMax(6000)
  global threads=4
  AvsFilterSource()
  ConvertToYV12()
  FDecimate2(24)
  SetFilterMTMode("DEFAULT_MT_MODE",2)
  global crop_params=""
  global resize_string=""
  global super_params     = "{gpu:1}"
  global analyse_params   = "{gpu:1}"
  global smoothfps_params = "{rate:{num:144,den:1,abs:true},algo:21,scene:{mode:0,blend:true}}"
  global demo_mode=0
  stereo_type=0
  stereo_left_selection=""
  stereo_right_selection=""
  ########## BEGIN OF MSMoothFps.avs ##########
  # This file is a part of SmoothVideo Project (SVP) 3.1
  # This is NOT the full AVS script, all used variables are defined via 
  # JavaScript code that generates the full script text.
  function interpolate(clip src)
  {
  input = crop_params=="" ? src : eval("src.crop("+crop_params+")")
  input = resize_string=="" ? input : eval("input."+resize_string)
  super=SVSuper(input, super_params)
  vectors=SVAnalyse(super, analyse_params, src=input)
  smooth=SVSmoothFps(input, super, vectors, smoothfps_params, mt=threads, url="www.svp-team.com")
  return demo_mode==0 ? smooth : demo(input,smooth)
  }
  input=last
  stereo_type==0 ? eval(""" interpolate(input)
  """) :  stereo_type==1 || stereo_type==3 ? eval("""
      lf = interpolate(input.crop(0,0,input.width/2,0))"""+stereo_left_selection+"""
      rf = interpolate(input.crop(input.width/2,0,0,0))"""+stereo_right_selection+"""
      StackHorizontal(lf, rf)
  """) :  stereo_type==2 || stereo_type==4  ? Eval("""
      lf = interpolate(input.crop(0,0,0,input.height/2))"""+stereo_left_selection+"""
      rf = interpolate(input.crop(0,input.height/2,0,0))"""+stereo_right_selection+"""
      StackVertical(lf, rf)""") : input
  ########### END OF MSMoothFps.avs ###########
  Prefetch(4)

• Do you use other DirectShow filters? Specify name and version: [none]

Describe the bug

Jitter and sync offset not 0 when using avisynth filter with svp script on an analog capture card-ImpactVCBe model 1381- but when just using ffdshow and it's avisynth function, jitter and sync offset is 0.

To Reproduce

1.analog capture card and avisynth filter and my script and evr-cp 2. 3.

Checklist

• Did you try a different video player and check if the issue persist? For example, if your main video player is MPC-HC, try MPC-BE. Answer: no

• Did you try a different video renderer and check if the issue persist? For example, if issue exists with madVR, try EVR. Answer: no

• Did you try a different video file and check if the issue persist? Try another file with different format, dimension and frame rate. Answer: no

[Usulyre]

You could enable logging and upload the log file to help diagnosis. Follow wiki https://github.com/CrendKing/avisynth_filter/wiki/How-to-enable-logging

https://gist.github.com/Usulyre/350472df40f2ce43d02fce132b799338

You could upload screenshots if the issue is related to visual.

https://ibb.co/sH94tct

You could upload the video file if it is small enough, or you know how to cut and crop it. If you do, be sure to test the cut file before uploading.

[CrendKing]

I tested both avsf and ffdshow with my webcam. Both has constant sync offset drift (keep minus-ing) and no jitter. The 10ms jitter in your screenshot is most likely within error range. Could be even from the capture device itself.

Most importantly, ignoring the numbers, is there any visual problem that you experience? Lag? Glitch? Freeze?

Also, observe these numbers without either avsf and ffdshow, see if they are strictly 0.

Try the bare minimum script with avsf, see if problem persists:

AvsFilterSource()

# optionally, uncomment this line
# Prefetch(4)

A little background

Unlike normal video file which has fixed amount of frames, capture stream could theoretically last forever. The capture device usually does not provide the "offset time" of when each frame should be rendered by the video renderer. Without AviSynth, this is fine. Each intermediate filter just process each frame without caring about the offset time, and the renderer just renders frames as they arrives. And if a filter can't function without an offset time, it most likely won't connect in the filter graph.

AviSynth can't work without offset time (more detail in the next section). By this standard, avsf should not support use cases without offset time and infinite stream. However, I did come up with some hacks to simulate finite stream. Some imitates what ffdshow does, some on my own.

I think the "sync offset" is the "Late" metric in DirectShow, but I think it is only meaningful if there is frame time. I don't know how it is calculated internally in DirectShow.

Why AviSynth needs frame offset time

AviSynth could insert or delete frames in the stream. Imagine you have the original video stream that consists of frames:

frame 1 ... frame 2 ... frame 3

When we feed frame 1 to AviSynth, the underlying script could replace it with arbitrary number of new frames 1.1, 1.2, 1.3 etc. Effectively the output stream would look like

frame 1.1 ... frame 1.2 ... frame 1.3 ... frame 2.1 ... frame 2.2 ... frame 2.3 ... frame 3.1 ...

Problem is, how do we know how many frames are inserted or deleted by avs? Avs does not tell (and can't tell) us this information. User could install arbitrary number of avs plugins, each doing the insertion and deletion by themselves. We overcome the problem by observing the offset time of each frame, and calculate the duration of each frame by subtracting adjacent frame's offset time. Then we know the mapping between original frames and avs output frames. Without offset time, this won't work.

[Usulyre]

Ignoring the numbers, the framerate when using avsf doesn't reach 144 fps as specified in my script. However when using ffdshow, it does.

FFdshow is pretty much 0 for the sync offset and jitter.

Here is what I got with bare minimum script and avsf

https://ibb.co/KDsxtt1

with no script and no ffdshow or avsf

https://ibb.co/8YG8Dd8

ffdshow

https://ibb.co/mctHMHr

ffdshow with my script

https://ibb.co/8mtTgZw

[Usulyre]

3rd link above is ffdshow and bare minimum script

[CrendKing]

The problem is, even if I use fresh installed MPC-HC / MPC-BE, my webcam would give me huge sync offset and non-zero jitter. Without a hardware like your capture card, I can't know how the video frames are produced, and how to change the logic.

I mean, sure, you can install Visual Studio 2019 and set up a remote desktop (and maybe virtual machine for privacy/security purposes) so that I remote to your computer and debug through your VS2019, but that's a little crazy, isn't it?

[Usulyre]

I guess your not planning on getting any capture card, (pci, pci-e, or usb 3.0) anytime soon right?

[CrendKing]

No, I don't.

What I can do is to modify the filter to dump extensive amount of detailed logs about every frame, so that I can get a grasp of what's going on. I need to you run that special version and send the log back, much like our old debug session days. Are you willing to do that?

[Usulyre]

Sure.

[CrendKing]

So here is what I find. Unlike regular video file where all media samples are already available on disk, capture card is a Live Source. For video files, as long as your disk can read as fast as the video stream goes, which any modern hard disk is capable of (unless we are talking about a 16K video on a old cylinder-based disk), live source's data is available in real time. Once a sample is produced by the device, it assigns the timestamp when it was created and push down to the filter chain. This array of samples can't be prefetched and buffered, meaning the longer the filter chain takes to process these samples, the more delay or lag you get when rendered. Some renderers, such as the "Sync Video Renderer" could even just drop samples if they arrive too late (more than a VSync window). This is the "sync offset" metric (with unit of millisecond) you see on the player. Negative number indicates how late each sample is.

So, without any trick, the more tasks you add to AviSynth script, the more delayed it will be. This is hard limit and I don't believe ffdshow can bypass. However, what it can do, and also what I can do, is reassign the sample time a bit late to render so that the video renderer thinks they arrive on time.

For example, one sample is captured at time 100, when the stream time is at 80. If this sample is immediately delivered to the video renderer, it will be scheduled to render 20 ticks later. However, suppose avsf adds 40 ticks to the process, making it deliver at stream time 120. The video renderer sees that the stream already passes the point to present the sample, so it simply drop it. If avsf change the sample time to 120, the video renderer will just render it.

One note though, artificially change the sample time doesn't alter its content. A 40 ticks late sample is 40 ticks late no matter how we lie about it's capture time. So if you adds huge amount of AviSynth tasks to the filter, and your CPU is super busy at other processes and background tasks, you will still feel the lag. The only way to relieve this is to decrease AviSynth task load and acquire more powerful CPU.

With all that said, here is the test build AviSynthFilter.zip. The answer I'm seeking is, does lying about the number actually solve any problem. When you test, first make sure only use the bare minimum script

return AvsFilterSource()

and turn on logging. Do the following tests:

1) Use EVR (not with custom presenter) on capture. 2) Use EVR CP on capture. 3) Use Sync Video Renderer on capture. 4) Use madVR on capture. 5) Turn off both ffdshow and avsf. Check if you have the same FPS (I want to confirm that bare minimum script does not decrease your FPS)

Watch the sync offset. They all should be 0 or near 0. If yes, add your original script and try again. Let me know the result.

[Usulyre]

https://gist.github.com/Usulyre/c8e17e0521efa434fd3d65c3e55ba084

https://gist.github.com/Usulyre/78784b2d81f5d7a4cf287a5cafe99e64

https://gist.github.com/Usulyre/e617554aff8936cd350cd0886e663bc8

https://gist.github.com/Usulyre/2cf39bebd9a0484e28e47f2de5970ceb

Results for ffdshow and avsf both off the same pretty much.

[Usulyre]

I didn't use ffdshow video at all actually.

Also here are the other results with my script and there was a problem with mpc-hc not closing completely (had to use task manager) when using my script with avsf test build and the renderers.

[Usulyre]

Log files are big wouldn't fit so I had to upload them in zip file.

https://drive.google.com/file/d/12u5PzhMM-FI8B5xearO94wVwJx4zzy8s/view?usp=sharing

[CrendKing]

Thanks for the log files, but you didn't answer my question. Does this version improve anything? The only difference for me is that Sync Video Renderer used to drop every frame due to delay, and now it behaves the same as other renderers. If you don't use that renderer, I don't think this changes anything.

[Usulyre]

Using evr-cp, this version doesn't bring the sync values and jitter to 0 no.

[CrendKing]

Again, not answering my question. What's so important you obsess with having sync offset being 0? What's your REAL problem? Is there screen tear? Is there visible jitter all over the place? Is the image freezing or all green? If everything works perfectly, why do you care about a stat number being bajillion big?

Also, I checked all 4 log files and they all show sync offset become 0 (the last number of each "Delivered frame" line). Do you have screenshot?

[Usulyre]

I'm not sure I mentioned this already, but I'm not getting 144 fps with avsf and my script, but getting 144 fps with ffdshow

[Usulyre]

ffdshow has no jitter and 0 sync offset, avsf had which was my original screenshot.

[CrendKing]

I know you mention the 144fps thing. I asked do you get 144fps without anything filter. For example, my webcam is supposed to produce 30fps but in reality it's usually 25 or 20fps. With bare minimum script it shouldn't affect the fps.

Does your capture device output 144fps video stream? If not, is it 30fps? If so, does avsf with minimum script also produce 30fps output? If still yes, then does the problem only occur when you add the complex script?

You see how I approach this? You just straightly jump to the end of the road and tell me problem, where I'm trying to get the picture step by step. So can you do step by step diagnosis and tell me details one by one?

[Usulyre]

Capture card outputs 29.97 fps, avsf with minimum script produces 29.97 fps output. Yes, my complex script is to output 144 fps and it's a problem when using avsf, evr-cp, and my capture card. But it's not a problem with ffdshow, my script, evr-cp and my capture card.

[CrendKing]

We are make some progress. It would nice if you opened the issue with this as the description, because we have no idea what causes this symptom. Don't jump into conclusion that the sync offset issue is the cause as you have no evidence to show the causality between them.

Several tests to run:

1) Starting from minimum script, add the SVP block. Leave out FDecimate2(24). Change num:144,den:1,abs:true to num:2,den:1. Do you get 60 fps? 2) From the previous script, change that to num:60,den:1,abs:true. Do you get 60 fps? 3) Change to num:144,den:1,abs:true. Do you get 144fps? 4) Start from minimum script, add FDecimate2(24). Do you get 24fps? 5) Change to FDecimate2(24, 0). Do you get 24fps? 6) From script of 1), add FDecimate2(24, 0). Do you get 60fps? 7) From script of 2), add FDecimate2(24, 0). Do you get 60fps? 8) From script of 3), add FDecimate2(24, 0). Do you get 144fps?

If any of these test does not get expected result, attach log file.

[Usulyre]

No. Yes. No. Yes. Yes. No. Yes. No.

Log files

https://drive.google.com/file/d/1R4lycZwiyC-xqj0hVkkeZ76TElMnz6Af/view?usp=sharing

[CrendKing]

Sorry I made a mistake at instruction 6. You are supposed to get 48fps, not 60fps. I also see that there's the freezing problem at shutdown. I'll tend to that later.

But apart from that, I see all frame time are as expected. Of course the times were adjusted by the test build. Can you do the tests with the published v0.8.1?

BTW, why 1. and 5. are so short?

[CrendKing]

From log file I see that the freeze is caused by malformed frames from upstream. Your capture device, or whatever filters before avsf, send frame to avsf, whose start times go backward in time. The code currently does not handle this exception. I'll correct that.

EDIT: Try this. This version has no "sync offset adjustment": AviSynthFilter.zip. Use this to do the retests.

[CrendKing]

Once you've done all the retests, please do one more set of tests:

1) Set up the order so that ffdshow comes before avsf. Put them both in the graph. Let ffdshow do your full complex script (which outputs 144fps), and use minimum script in avsf. Make sure you still have 144fps. Collect log. 2) Remove ffdshow, put your complex script to avsf. Collect log.

The theory is, since you confirmed avsf with minimum script won't affect fps, 1) should still have 144fps. I want to compare the log to see what in the frame sequence causes the fps drop.

[Usulyre]

Not sure why 1 and 5 were so short.

No. No. No. Yes. Yes. No. No. No.

[Usulyre]

https://drive.google.com/file/d/1QfBFSSD7NJSJ6ZkhHXkCWvAEI8WqGpNi/view?usp=sharing

[Usulyre]

More like 129 fps.

https://drive.google.com/file/d/1ehtDxV2NOJaW6FqD1fN6AGGGT3dOQ_Sh/view?usp=sharing

[CrendKing]

Did you use the build from https://github.com/CrendKing/avisynth_filter/issues/26#issuecomment-732080945 to run these? If yes, then I don't know how to proceed anymore. You probably have to use ffdshow for the use case. Sorry.