Vertical padding setting causes crash using FFDShow if renderer doesn't support initial color output format

[GoogleCodeExporter]

What steps will reproduce the problem?
1. create a graph where video filter outputs YV12 (e.g. as primary color space 
in ffdshow)
2. connect video filter to xy-vsfilter
3. connect xy-vsfilter to a VMR9 or Haali's Renderer (both do not support YV12)

What is the expected output? What do you see instead?
Expected: fallback to another color space or pin connection error
Instead: crash in vsfilter.dll

What version of the product are you using? On what operating system?
xy_VSFilter_test_20111220.7z, Windows Vista 64bit

Please provide related subtitle script (if any). And the less characters
the script is, the less effort I have to make to fix the issue, so if you
could make the script as simple as possible, that would be appreciated.
Crashes even with no subtitles loaded into xy-vsfilter

Please provide any additional information below.
It seems to crash on any combination where the renderer does not accept the 
initial color space selected by the video filter. It's easiest reproduced in 
Haali's Renderer, since it only supports YUY2 and RGB32. It should happen in 
the default ffdshow configuration where color space is set to auto. It works 
fine with old plain vsfilter versions.

Original issue reported on code.google.com by OV2.gcode on 5 Jan 2012 at 10:03

[GoogleCodeExporter]

Sorry, I've pasted the wrong filename - I've used the latest build, so it's 
xy_VSFilter_stable_20111226.7z

Original comment by OV2.gcode on 5 Jan 2012 at 10:07

[GoogleCodeExporter]

I don't have a Vista computer handy, but I'm unable to reproduce the problem in 
either WinXP x86 or Win7 x64.

Please try the following:

1) Open regedit, navigate to HKEY_CURRENT_USER\Software\Gabest\ and delete any 
keys with VSFilter in their name.

2) Open an Administrative Command Prompt and navigate to where your 
VSFilter.dll is located

3) regsvr32 /u VSFilter.dll

4) regsvr32 VSFilter.dll

5) Check to see if you still get a crash

6) If it still crashes, uninstall Haali Media Splitter + FFDShow and reboot

7) Install the latest Haali from 
https://customers.corecodec.com/dl.php?type=d&id=12  , latest FFDShow x86 from 
http://www.xvidvideo.ru/ffdshow-tryouts-project-x86-x64/

8) Check to see if you still get a crash, and report back.

Original comment by cyber.sp...@gmail.com on 6 Jan 2012 at 12:04

[GoogleCodeExporter]

Deleting the Registry Key and registering again fixed it on the next try. This 
caused me to investigate a bit further and compare my settings to the default 
settings, and it seems that the crash also requires the "Vertical Padding" 
option to be set to anything but "Original Height".

I can reliably reproduce the crash by using xy-vsfilter default settings, 
changing vertical padding and then using the steps listed above.

Original comment by OV2.gcode on 8 Jan 2012 at 5:16

[GoogleCodeExporter]

I can reproduce the vertical padding crash. 

From the looks of it this only happens with FFDShow, so this is something 
outside of our control to fix. Report the bug on in the FFDShow-tryouts thread 
on Doom9 forums and hope someone fixes it. It would also appears FFDShow has 
another bug which causes a green bar to be shown when vertical padding is used 
with YUY2 output.

Workaround:

Use LAV Video from the LAV Filters package http://code.google.com/p/lavfilters/

Original comment by cyber.sp...@gmail.com on 9 Jan 2012 at 8:47

• Changed title: Vertical padding setting causes crash using FFDShow if renderer doesn't support initial color output format
• Changed state: WontFix
• Added labels: Priority-Low
• Removed labels: Priority-Medium

[GoogleCodeExporter]

I can also reproduce the crash with the xvid video decoder instead of ffdshow:
Xvid Settings: http://i.imgur.com/S4N7U.jpg
Graph: http://i.imgur.com/B7sCE.jpg

Same with the green bar issue.

Original comment by OV2.gcode on 9 Jan 2012 at 5:59

[GoogleCodeExporter]

Both the crash and the green bar likely have to do with image stride and stride 
alignment during reconnection attempts. VSFilter requests/expects one thing, 
the decoder sends another. All these legacy padding settings from VSFilter 2.x 
are things a decoder would need to explicitly support, and it wouldn't be 
surprising if most don't, especially during a reconnection.

Original comment by cyber.sp...@gmail.com on 9 Jan 2012 at 8:20

[GoogleCodeExporter]

It's not that big an issue for me, I only really brought it up since both do 
not occur in plain 2.39 vsfilter.

If you say it's definitely an upstream issue I'll post it on the ffdshow thread.

Original comment by OV2.gcode on 9 Jan 2012 at 8:30

[GoogleCodeExporter]

VSFilter 2.39 never reconnects. In cases like Haali, VSFilter will accept YV12 
and do a color conversion to YUY2. xy-VSFilter on the other hand would reject 
YV12 and reconnect with YUY2. 

Since works fine with LAV Video and to some extent CoreAVC as well, it would 
suggest this is a limitation in FFDShow & XVID, not a xy-VSFilter bug. If you 
were able to identify a regression when both xy-VSFilter and VSFilter 2.39 are 
using identical input + output formats (renderer accepts initial color format), 
that may be something we could fix.

Original comment by cyber.sp...@gmail.com on 9 Jan 2012 at 9:54