interlaced SD&HD PAL&NTSC 4:2:0 mpeg2/dv/h264 content has chroma upsample error on decklink consumer 422 +workaround

[walterav1984]

Problem: During the playback of Red and or Blue high saturated colors and or contrast within interlaced motion area's in SD PAL mpeg2/h264/dv video content which uses yuv interlaced 4:2:0 chroma sampling, there is visible artifacts/distortion like jaggy edges especially moving Red/Orange color contrast on blacks during playback on a SDL/Decklink consumer connected to CRT SD monitor. This might is also the case for NTSC & HD content, but I don't have interlaced HD broadcast CRT monitor to check (HDMI LCD).

melt -profile dv_pal sdinterlaced420mpeg2dvh264examplevideofile -consumer decklink #plays incorrect with chroma artifacts

Analyses: From video file to decklink output the following has to take place, detect field order of file and if necessary change it to field order of SDI decklink output. Followed by the detection of chroma sampling level 4:2:0 and upscale it to decklink native 4:2:2 (possible a lot more routines happen and maybe in different order). The interlacing and field order is detected correctly for (mpeg2dvd upper field & DV lower field), because motion with less saturation/contrast is playback correct and smooth. But the problem is chroma upscaling probably happens "progressive" instead of "interlaced". Its indeed C.U.E. (chroma upsampling error) because it is mostly visible in the motion affected areas of footage, on contrary (ICP) interlaced chroma problem is mostly visible in static areas of the footage, see here for explanation:

http://web.archive.org/web/20060719025947/http://www.hometheaterhifi.com/volume_8_2/dvd-benchmark-special-report-chroma-bug-4-2001.html https://trac.ffmpeg.org/ticket/4168

FIX/Workaround? As of my understanding "melt" is build around ffmpeg, the following ffmpeg filter libswscale which is also part of melt can fix this in ffmpeg with "-vf scale=interl=1" and choosing a 422 or 4:4:4 output video codec . Investigation by ddennedy / max-verem shows that this interlaced aware code in the melt scaler equivalent is missing.

Workaround PRECONVERT footage: ffmpeg -f mpeg -i bug.vob -vf scale=interl=1 -c:v prores test.mov melt -profile dv_pal test.mov -consumer decklink #plays correct on CRT monitor no chroma errors

Workaround LIVE/REALTIME streams: ffmpeg -re -i pipe:0 -vf scale=interl=1 -vcodec rawvideo -pix_fmt uyvy422 -acodec copy -f avi - | melt -profile dv_pal pipe:0 -consumer decklink As a example source to ffmpeg pipe:0 I used "dvsink-command" which gets a DV 4:2:0 PAL video stream from "dvswitch" video mixer on the network. I do have to call this above mentioned workaround command in a seperate bashscript to "dvswitch-command -h ipadress -p port ffmpeg420i422.sh" otherwise execvp or getopt errors from dvsink-command when parsing the arguments off the command line.

Concluding&Fix: FFmpeg needs to be told/forced with "-vf scale=interl=1" and choosing a 422 or higher pixelformat/videocodec to fix correct upscaling of interlaced chroma 4:2:0 to 4:2:2. For m(e)lt there is a workaround for now until interlaced aware chroma scaling code is ported and a working patch is available @max-verem commit 934744c branch :max-verem:sliced_pixfmt3 https://github.com/mltframework/mlt/pull/191/commits/934744c0fdf07da2322ebeaf5193bbfde967bb79

Software: OS ubuntu: 16.04.1 amd64 kernel 4.4.0.xxx M(e)LT version: ubuntu repository 6.0.0 & Shotcut 17.01 binaries 6.5.0 BMD Desktop Video Driver: 10.8.4 and 10.6.2/10.7.0

Hardware: Intensity Pro 4K, analog SD component to SONY PVM20ME4 crt monitor Decklink SDI 4K, SDI SD to Barco crt monitor

Example Video: intro-clean.m2v "dvd copy tff example" intro-clean.m2v.zip

clean.dv "original lff example" https://github.com/walterav1984/dvpause/blob/master/videos/clean.dv

[ddennedy]

I doubt I and many others will be able to reproduce and test it. Does it require a CRT, or can it be observed with a 1080p panel as well? Maybe @max-verem can help. The source code is here so you can take a crack at it yourself: https://github.com/mltframework/mlt/tree/master/src/modules/avformat

The chroma upsampling is done in the avformat producer: https://github.com/mltframework/mlt/blob/master/src/modules/avformat/producer_avformat.c#L1250

You can see flags such as SWS_ACCURATE_RND and SWS_FULL_CHR_H_INT being used.

In more complex MLT compositions, the avcolour_space filter is used for image format conversion: https://github.com/mltframework/mlt/blob/master/src/modules/avformat/filter_avcolour_space.c#L71

And the swscale filter is used for scaling: https://github.com/mltframework/mlt/blob/master/src/modules/avformat/filter_swscale.c#L69

I suggest tinkering with these flags. I believe ffmpeg defaults to only SWS_BICUBIC, but it may automatically inject other flags based on the conversion and dithering requested. Here is what x264 command line uses: https://github.com/mirror/x264/blob/master/filters/video/resize.c#L461

SWS_FULL_CHR_H_INT means full chroma horizontal interpolation SWS_FULL_CHR_H_INP means full chroma input

Not entirely sure what "full" means without thoroughly reading libswscale source code, but those seem most likely to experiment with for your problem.

[ddennedy]

When reading the ffmpeg vf_scale.c source code for the flags it uses in your command line, I found none; but I did find this interesting code: https://github.com/FFmpeg/FFmpeg/blob/master/libavfilter/vf_scale.c#L405

[walterav1984]

Thanks for replying and taking this serious, since allot of content is created using consumer and prosumer camera's in the 4:2:0 chroma sampling, not only interlaced SD PAL but also NTSC DVD and HD sources.

As of replication of this issue it may be very easy to replicate, therefor I will try to record the SDI output from Decklink using the melt playback with "broken" upscaling and the "correct" ffmpeg preconverted file with the "swscale interlaced" flag. Maybe enhanced stills of the recording will show the distortion without special CRT monitors. Any user/developer with a Decklink card and the right high saturated blue/red coloured demo footage may be able to replicate it, since all decklink cards are PAL/NTSC switchable, I might try NTSC tomorrow too since all devices are compatible.

Having a setup with native interlaced and high resolution real CRT monitors is easier, both are 800lines in resolution and both use different signal path so it cannot be related to the hardware. I do think its limited to interlaced only, I will try the Blackmagic intensity output to a HDMI monitor tomorrow both SD and upscaled to HD with the melt ATSC_1080i50 profile to see if its visible on LCD monitor.

The strange thing is that like you discovered (one post earlier @L405) some ffmpeg developers also don't know why calling the "swscale interlaced" option is effective even though swscale itself is not the active filter for chroma upscaling...

I will try to record the effect with and without this filter and upload some effective example footage. Although we may not find the real fix is it possible finding the equavalent of the ffmpeg swscale=interlaced option in the melt commandline translated ffmpeg parameters as a workaround?

[ddennedy]

I do not recommend using the atsc_1080i_50 profile for this bug and testing as that will invoke automatic deinterlacing since MLT does not have a field-aware scaler.

[walterav1984]

Good tip about avoiding de-interlacing when using the atsc_1080i_50, I will try to make a composite of multiple SD clips horizontal/vertical repeated with lines/fields alligned for HD 1080i50 footage.

[walterav1984]

Recording of the SD PAL Decklink SDI signal output of the melt playback was successful and reveals the distortion in the recordings.

https://we.tl/0PmY0AaDhj

I uploaded the recorded uncompressed 8bit yuv 422 videos to wetransfer. There are 3 clips, clip03 is native mpeg2 playback in melt, clip04 is melt playback of ffmpeg preconverted mpeg2 to prores with vscale=interl=1 flag and clip05 melt playback with preconverted prores without the interlace flag also a screenshot is added to the zip and uploaded here.

Only clip 04 is correct, the visual error is prone on interlaced(moving) red edges/contrast. Looks like the interlaced resolution is only half?

PS: Monitoring on lower resolution CRT monitor through analog Composite/CVBS signal won't show any distortion. HD testing and LCD monitors will follow.

[max-verem]

could you provide a source mpeg2 file and command lines that was used to create clip04 and clip05.

does showinfo filter for your original mpeg2 files mention that source is interlaced?

[walterav1984]

Source mpeg2 file (original was DV lowerfield somewhere on tape): https://we.tl/VcjsQY8VS4

ffmpeg -i intro.m2v -vf scale=interl=1 -c:v prores untitled04.mov ffmpeg -i intro.m2v -c:v prores untitled05.mov

ffmpeg -i intro.m2v -vf showinfo -f null - Input #0, mpegvideo, from 'intro.m2v': Duration: N/A, bitrate: N/A Stream #0:0: Video: mpeg2video (Main), yuv420p(tv), 720x576 [SAR 16:15 DAR 4:3], max. 7250 kb/s, 25 fps, 25 tbr, 1200k tbn, 50 tbc [Parsed_showinfo_0 @ 0x13b9aa0] n: 205 pts:9888000 pts_time:8.24 pos: 5871159 fmt:yuv420p sar:16/15 s:720x576 i:T iskey:0 type:P checksum:B15E9D1B plane_checksum:[37804E71 E2CC4C08 CC9302A2] mean:[171 126 134] stdev:[58.3 13.1 21.1] [Parsed_showinfo_0 @ 0x13b9aa0] side data - pan/scan frame= 206 fps=0.0 q=-0.0 Lsize=N/A time=00:00:08.28 bitrate=N/A
video:19kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

ffmpeg version 2.8.10-0ubuntu0.16.04.1 Copyright (c) 2000-2016 the FFmpeg developers Ubunturepo ffmpeg version n3.0.5-6-g76961f4 Copyright (c) 2000-2016 the FFmpeg developers Shotcut

[walterav1984]

ddennedy

When reading the ffmpeg vf_scale.c source code for the flags it uses in your command line , I found none

You might have found one, its the exact flag I'm using in ffmpeg! { "interl", "set interlacing", OFFSET(interlaced), AV_OPT_TYPE_BOOL, {.i64 = 0 }, -1, 1, FLAGS }, https://github.com/FFmpeg/FFmpeg/blob/master/libavfilter/vf_scale.c#L664

In comparing mlt code of swscale to ffmpeg I cannot find references/context to interlaced chroma scaling like "isws" only "sws" is used, also AVCOL_RANGE_MPEG is missing which is used by isws while AVCOL_RANGE_JPEG is available for sws.

struct SwsContext *isws[2]; ///< software scaler context for interlaced material https://github.com/FFmpeg/FFmpeg/blob/master/libavfilter/vf_scale.c#L84

On the otherhand AVCOL_RANGE_MPEG is available in mltframework through avfilter colorspace av.range. https://github.com/mltframework/mlt/blob/master/src/modules/avformat/filter_avfilter.c#L534 https://www.mltframework.org/plugins/FilterAvfilter-colorspace/#avrange

[walterav1984]

NTSC is also affected.

For creating similar NTSC chroma upscale error exposing demo footage I used ffmpeg to speedup+crop+reversed-playback the PAL intro.m2v and maintain 1:1 pixel/chroma/field interlace information. The NTSC footage is therefor a little bit vertical stretched and playback order is reversed as a side affect, but most important motion and sharpness are identical to the PAL version to determine chroma error!

ffmpeg PAL>NTSC complete 420interl422 conversion: ffmpeg -r 29.97 -i intro.m2v -filter:v "crop=in_w:480:0:48,scale=interl=1,reverse,setdar=4:3" -c:v rawvideo -pix_fmt uyvy422 intro-ntsc-speedup-crop-reversed-scaleinterl422.avi

This one plays correct in "melt -profile dv_ntsc video -consumer decklink" & BMD mediaexpress since it contains 422 chroma interlace upscaled with -vf "swscale=interl=1" by ffmpeg.

ffmpeg NTSC 420 test footage: ffmpeg -r 29.97 -i intro.m2v -filter:v "crop=in_w:480:0:48,reverse,setdar=4:3" -c:v rawvideo -pix_fmt yuv420p intro-ntsc-speedup-crop-reversed420.avi

This NTSC example plays incorrect in melt, just like PAL with chroma errors probably since it is 4:2:0 and it needs interlaced chroma upscaling to 4:2:2 to avoid artifacts. It is comparable to intro.m2v was for PAL.

ffmpeg interlaced upscale from 420 test footage to 422: ffmpeg -i intro-ntsc-speedup-crop-reversed420.avi -vf scale=interl=1 -c:v rawvideo -pix_fmt uyvy422 intro-ntsc-speedup-crop-reversed420scaleinterl422.avi

As a closing test to show that the second example footage is not broken from its origin, we correctly chroma upscale it from 420 to 422 interlaced with ffmpeg and playback the resulting file correctly in "melt -profile dv_ntsc video -consumer decklink" as well as BMD mediaexpress with no chroma upscale artifacts.

[walterav1984]

LCD monitors also affected.

No CRT monitors nor Decklink SDI equipment is necessary, just using SDL output on pc desktop and making screenshots/using well trained eye gives similar chroma artifacts, just like the VLC screenshots of the Decklink output captured footage posted earlier. Like I mentioned earlier it is just better noticable on a CRT monitor.

[ddennedy]

I think this comes down to the fact that MLT's scaler is known to not be field-aware, which I have stated here and elsewhere such as on mailing list. Probably, the chroma interpolation during pixel format conversion (chroma upsampling) does not have field isolation/affinity. We probably need to make all usage of libswscale interlace aware using vf_scale.c as a reference.

[walterav1984]

You're right about the missing field awareness of the scaler is a known fact but I always thought it was doing different resolution scaling only, not chroma upscaling as we know now.

Well there is a workaround for non-realtime sources, just upconvert material to 422 mpeg2/h264/prores in ffmpeg. For live/realtime playout I have to find a way to pipe ffmpeg output to melt(ed). Found one, just need to know how to write in xml/mcvp.

melt -profile dv_pal_wide pipe: -consumer decklink

Since the wetransfer examples will disappear, I can upload a definitive version of the chroma upscale example footage. I own the copyright since I made it as a kid 15 years ago during a school trip, but I can just trim the name of the school at the end to make licensing easier/compatible with MLT.

Do you have a prefered license so this footage may be distributed by mlt github and used by anyone?

[max-verem]

I think this comes down to the fact that MLT's scaler is known to not be field-aware, which I have stated here and elsewhere such as on mailing list. Probably, the chroma interpolation during pixel format conversion (chroma upsampling) does not have field isolation/affinity. We probably need to make all usage of libswscale interlace aware using vf_scale.c as a reference.

i completely agree with it

it would not be difficult to rework producer_avformat.c (for start) to use interlaced aware colorspace conversion with newer sws_alloc_context instead of depricated sws_getContext but i rather interesting in making it slicable over threads....

another things that warried me is a deinterlacing approach currently used - deinterlace-as-possible... it should be reviewed

[max-verem]

please have a look on https://github.com/mltframework/mlt/pull/177 it should fix your issue (i hope)

[walterav1984]

177 build doesn't fix it on the SDL consumer, testing Decklink output now.

PS: I fixed this dependency by downloading melt from the ubuntu 16.04 repo, could this be an issue? melt: error while loading shared libraries: libmlt.so.6: cannot open shared object file: No such file or directory

[ddennedy]

@walterav1984 please explain your "PS" more. It sounds like you might not really understand how to build it and test the build.

If you are able to compile successfully, you can try it without having to make install and deal with multiple installed versions conflicting. Try this: cd to the top level mlt source directory, enter command source setenv, now run melt. That source command will set environment variables to make your terminal session use the locally built melt and MLT instead of anything installed.

[walterav1984]

It works now on SDL and Decklink seems no more upsample artifacts!!!

But with errors in console: sliced_pix_fmt_conv_proc:1285: [id=0, idx=4, jobs=10], interlaced=1, field=1, slices=5, mul=2, h=288, slice_h=56, slice_y=232 ctx->desc=[log2_chroma_h=1 Current Position: 112 [consumer decklink] renderVideo:648: ScheduleVideoFrame failed, hr=80000009 sliced_pix_fmt_conv_proc:1285: [id=0, idx=0, jobs=10], interlaced=1, field=0, slices=5, mul=2, h=288, slice_h=58, slice_y=0 ctx->desc=[log2_chroma_h=1 sliced_pix_fmt_conv_proc:1285: [id=0, idx=0, jobs=10], interlaced=1, field=1, slices=5, mul=2, h=288, slice_h=58, slice_y=0 ctx->desc=[log2_chroma_h=1

@ddennedy

Normally I do run the ./melt builds without "sudo make install" but forget that I also have the melt from the ubuntu repository installed already. But this time I didn't had it installed and came to the libmlt.so.6 error. I will try your advice source setenv on my next clean build system.

I did the following and now it works but don't know why: sudo apt-get remove melt sudo apt-get autoremove #get rid of dependencies libmlt++3 libmlt-data libmlt6 sudo make install #already had built it sudo apt-get install libmlt-dev

[walterav1984]

The chroma upsampling seems to be fixed (tried mpeg2/dv), but a new artifact arises some kind of field sync tearing/judder?

I have recorded it from the decklink output. https://we.tl/axbRS6KadB

This sync issue is a lot harder to spot even on a CRT. The prores file recorded as Untitled 06.avi "left image" (correctly upsampled with ffmpeg) doesn't show this scanline issue during melt playback on decklink, but the Untitled 08.avi "right image" recording (upsampled m2v by melt) does show it. Both video examples used the same melt build, although the prores file doesn't flood the console with debug slice info like the mpeg2 m2v 420 file.

I will try to describe how it looks, its seems like a invisible horizontal line is messing with the red circle logo movement around inner north-west to inner north east its also suggesting its moving very slow from top to bottom (like plain old analoge scanline interference).

Try to look at the mouse pointer just above it the red looks more squared in the screenshot, and at 180 degrees mirrored from the center of the circle at the opposite side of the mouse pointer call it south east side its also visible. This time you need to deinterlace and zoom 2x to view the error on a LCD monitor.

[max-verem]

does it (intro.m2v) has correct field order? was it captured in DV originally? I will review your source soon at NLE with CRT monitor (Premiere or FCP) and MLT output by decklink output...

Maybe some synthetic reference video files could helps with it...

[walterav1984]

The video file intro(-clean).m2v both have correct field order tff/upper. It was indeed mastered in DV(canopus) bff/lower, I just have found the original DV tape and captured the original. The animation was done in 3D Studio Max with lower field aware rendering...

The original DV file is uploaded at the end of the first topic it is called wetransfer "clean.dv". Both the m2v and dv file show the field distortion on playback with melt doing the 420-422 upsampling.

[max-verem]

I did a check on CRT monitor (it is old JVC but picture is fine) - shame on me, but i see no difference - maybe it is bluring something...

original issue (interlaced 4:2:0) seems fixed, but i am worry about additional artifacts you seen...

i split Untitled 08.avi into image sequence with command:

ffmpeg -i "Untitled 08.avi" "Untitled 08/%5d.png"

but i see no artifacts, could you do the same and point me with resulting frame for deeper investigation?

i will try to create synthetic tests to estimate that

[max-verem]

another one example for dumping resulted files:

ffmpeg -i "Untitled 08.avi" -vf "field=0" "Untitled 08.f/%5d_0.png"
ffmpeg -i "Untitled 08.avi" -vf "field=1" "Untitled 08.f/%5d_1.png"

that will output by fields and you can compare them (fields) more adequate..

[walterav1984]

Remember original DV file is uploaded, it has less compression artifacts.

Does the JVC monitor uses YUV or Composite, only the first shows artifacts (s-video is broken on my decklink)? If you use Blackmagic IO make sure "Remove Field Jitter On Pause" is unchecked in desktop video settings.

I will make a difference mapping with Adobe After Effects of the original DV playback from default melt and custom melt build to show the strange artifact sync line.

[walterav1984]

Enhanced version of the difference mapping of ffmpeg upscaled 420>422 and melt upscaled 420>422 both playedback via mlt and recorder uncompressed. The artifact is very good vissible on CRT within the moving O charachters.

https://we.tl/Tganjo13kA

[max-verem]

it is Composite monitor. But...

i built a script that process your intro.m2v by ffmpeg and patched melt, next it (script) calculate difference, please download and analize it.

archive with results: http://downloads.m1stereo.tv/4ff69756a1300a8b29c270eeeb5ae5e2/

in comparing r1.avi and r2.avi:

[Parsed_ssim_0 @ 0xa49460] SSIM Y:1.000000 (inf) U:0.999588 (33.846839) V:0.999617 (34.166582) All:0.999801 (37.014100)
[Parsed_psnr_1 @ 0xa4a1a0] PSNR y:inf u:59.129336 v:59.086175 average:62.118002 min:59.336775 max:73.455448

[walterav1984]

Keep in mind CRT monitor 20inch composite connection won't even show the Chroma Upsample Error itself, probably won't show this field drift at all.

Your example "diffB-r1-r2.avi" shows the most prominent place of the errors just as in my difference blend example following the moving white O characters:

https://we.tl/Tganjo13kA

[max-verem]

I did more researches and found the reason of issue.

First the artifacts you seen on a CRT monitor related to DAC overloading/clipping

Here is a new script that should prepare more appropriate video for study:

#!/bin/bash

ffmpeg -i intro.m2v -vf "scale=720:576:interl=1" -pix_fmt yuv422p -f avi -vcodec rawvideo -y r1.avi
ffmpeg -i intro.m2v -vf "scale=720:576:interl=0" -pix_fmt yuv422p -f avi -vcodec rawvideo -y r0.avi
melt -profile dv_pal intro.m2v in=1 out=205 -consumer avformat:r2.avi vcodec=rawvideo top_field_first=1

FILTER="[0:v]extractplanes=y+u+v[y0][u0][v0];[u0][v0]hstack=2[uv0];[y0][uv0]vstack=2[yuv0]"

# extract to planes

ffmpeg -i r0.avi \
    -filter_complex "$FILTER" \
    -map "[yuv0]" -an -vcodec rawvideo -y r0_planes.avi

ffmpeg -i r1.avi \
    -filter_complex "$FILTER" \
    -map "[yuv0]" -an -vcodec rawvideo -y r1_planes.avi

ffmpeg -i r2.avi \
    -filter_complex "$FILTER" \
    -map "[yuv0]" -an -vcodec rawvideo -y r2_planes.avi

# build diff

ffmpeg -i r1_planes.avi -i r2_planes.avi \
    -filter_complex "blend=all_mode=difference" \
    -an -vcodec rawvideo \
    -y diffB-r1-r2.avi

ffmpeg -i r2_planes.avi -i r1_planes.avi \
    -filter_complex "blend=all_mode=difference" \
    -an -vcodec rawvideo \
    -y diffB-r2-r1.avi

and found (i have to use VirtualDub to rise brighness/contrast) that artifacts present related to slices count used in new method. I did a changing TST_SLICES_CNT value and resulting picture changed: this mean artifacts appearing on a edge of slices. I dont know reason of that but i will try to play with a adjusting height of slices to be even to 2pow....

[walterav1984]

Good to know that artifacts arise on edges of slices, could we try slice count=1 or 0?

By the way how can it be the DAC if the recorderd HDMI output shows the artifacts on a PC screen even in VLC (needs deinterlacing to see the line judder). I also tried your patch on a Decklink SDI to Barco SDI 14 inch monitor and it also shows the line judder artifacts.

Since "C.U.E." Chroma Upsample Error(visible in moving parts) is fixed, is this the "I.C.P." interlaced chroma problem (visible in static part of image needs low pas filter)as the last interlaced 420 inherent problem to deal with when up converting to 422?

[max-verem]

if you specify 2 slices you will get good picture. there is some noice but it very low: next i did a check more close: as you can see, problem at top and bottom of slice related to filter with 3 pixel window. i suspect it is related to SWS_BICUBIC, because switching it to SWS_BILINEAR gives very funny results - everything correct but some noise present. try to download: http://downloads.m1stereo.tv/4ff69756a1300a8b29c270eeeb5ae5e2/log3-bilinear.png http://downloads.m1stereo.tv/4ff69756a1300a8b29c270eeeb5ae5e2/log3-fast_bilinear.png and use levels in GIMP to see artifacts.

So there are two solutions:

• use SWS_BILINEAR (or SWS_FASTBILINEAR) instead of SWS_BICUBIC
• rework slicing scaling to operate with 3 parts: fake top (3 lines), working area (X lines), fake bottom (3 lines) - this will make bicubic interolator working properly...

@walterav1984 could you do a simple check results after changing SWS_BICUBIC or SWS_BILINEAR (or SWS_FASTBILINEAR)?

[walterav1984]

Where do I have to change SWS_BICUBIC/BILINEAR/FASTBILINEAR in sourcecode to test these changes?

I now understand that because of a slice, each slice has its own filter area "within" with its own TOP edge pixel row and BOTTOM edge pixel row. To have correct filtering "among" instead of "within" slices, each slice indeed needs to overlap at-least the amount of pixels that the Bicubic/(bi)linear filter takes into account. Than two slices can have the same filtered pixels on the edges?

BTW what is the CPU/GPU load improvement if we use sliced rendering/threading instead of single slice when chroma upsampling?

[max-verem]

Where do I have to change SWS_BICUBIC/BILINEAR/FASTBILINEAR in sourcecode to test these changes?

clone https://github.com/max-verem/mlt/tree/sliced_interlaced_pixfmt_conv_7 and replace SWS_BICUBIC in producer_avformat.c to SWS_BILINEAR or SWS_FASTBILINEAR

BTW what is the CPU/GPU load win if we use sliced rendering instead of single slice when chroma upsampling?

please elaborate

[walterav1984]

Result: Bicubic>Bilinear= makes field/slice artifacts worse Bicubic>Fastbilinear= compile error undefined ... first time usage

@max-verem

i rather interesting in making it slicable over threads....

Optimizing the code using slices will improve CPU multithreading performance as I understand, instead of not using slices, how big is performance impact if chroma upsampling is done without slice cpu multithreading?

[max-verem]

Bicubic>Bilinear= makes field/slice artifacts worse Bicubic>Fastbilinear= compile error undefined ... first time usage

i wonder why it gives such visible artifacts... according to my tests: http://downloads.m1stereo.tv/4ff69756a1300a8b29c270eeeb5ae5e2/log3-bilinear.png http://downloads.m1stereo.tv/4ff69756a1300a8b29c270eeeb5ae5e2/log3-fast_bilinear.png noise level not such high...

Optimizing the code using slices will improve CPU multithreading performance as I understand, instead of not using slices, how big is performance impact if chroma upsampling is done without slice cpu multithreading?

month or two ago i faced very strange issue then 1080i with yuv420 takes up 15miliseconds to convert it to interleaved YUYV

implementing sliced processing could spread slices processing over CPU cores and will increase performance of system

[max-verem]

idea to use fake buffer for paddings has failed, it seem sws_scale stores into destination buffer data not sync with supplied data but from temporal buffers used.

for now method works fine without artifacts for ONE slice for progressive frame and TWO slices for interlaced frame.

sliced parallel processing could be implemented, but would require:

• temporal buffer for storing encoded padded slices
• 6 lines per slice will be processed for nothing...
• additional data copying

on multiprocessor system it could gain serious performance boost,

@ddennedy what do you think? would not it makes producer_avformat.c compleatelly unreadable?

[max-verem]

current working version comes in pull request https://github.com/mltframework/mlt/pull/181

[walterav1984]

Great it looks #181 Fixed it for PAL! NTSC still shows the initial Chroma Upsampling Errors.

I tried most of the affected footage including native mpeg2 transport streams from our Public Broadcasting (https://github.com/tvheadend/tvheadend DVB-C recording) like Sport and News items including high saturated animations and orange on black moving clothing sports. All looks fine!

Probably there are more scope or difference blend mode tests but visually this looks as good as the ffmpeg 422 interlaced up sampling.

[max-verem]

i would not close this issue...

i still want to check if vertical slicing works. if bicubic for vertical scaling only does not affect horizontal-neighbor pixels that is could be used as main solution.

[walterav1984]

Well I have to admit I must also test NTSC footage (my apology for forgetting) besides the personal PAL interest.

[max-verem]

Please tests against pull request https://github.com/mltframework/mlt/pull/182

[walterav1984]

Good to know your are still progressing, I will test pr #182 or newer tonight ~7 hours from now.

BTW are the horizontal slice quantities related to PAL 576/18=32, 576/9=64 and is it NTSC 480/18=26,6666666667 480/9=53,3333333333 compatible or problem?

Using vertical slices both PAL&NTSC have 720 pixels can use same subdivision? What about 704/352 and other non D1 video resolutions?

[max-verem]

BTW are the horizontal slice quantities related to PAL 576/18=32, 576/9=64 and is it NTSC 480/18=26,6666666667 480/9=53,3333333333 compatible or problem?

that release (https://github.com/mltframework/mlt/pull/182) working with vertical slices - i.e. they located horizontally (picture splitted horizontally and processed vertically). It will use 64 x 576 (for progressive pal) and 64x288 (for field)... for NTSC it will use 64x480

using horizontal slices failed - i did a check against all kind of slice height, but artifacts were present...

[walterav1984]

I tested #182 also successful for PAL (mpeg2/dv/rawyuv420 tff&bff)!

But NTSC is broken, its showing the initial Chroma Upsampling Errors again see some mlt debug output:

sliced_420_pix_fmt_conv_proc:1282: [id=9, idx=9, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=576 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=10, idx=10, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=640 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=11, idx=11, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=16, slice_x=704 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=0, idx=0, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=0 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=1, idx=1, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=64 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=2, idx=2, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=128 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=3, idx=3, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=192 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=4, idx=4, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=256 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=5, idx=5, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=320 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=6, idx=6, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=384 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=7, idx=7, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=448 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=8, idx=8, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=512 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=9, idx=9, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=576 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=10, idx=10, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=640 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=11, idx=11, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=16, slice_x=704 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=0, idx=0, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=0 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=1, idx=1, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=64 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=2, idx=2, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=128 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=3, idx=3, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=192 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=4, idx=4, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=256 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=5, idx=5, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=320 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=6, idx=6, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=384 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=7, idx=7, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=448 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=8, idx=8, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=512 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=9, idx=9, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=576 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=10, idx=10, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=640 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=11, idx=11, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=16, slice_x=704 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=0, idx=0, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=0 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=1, idx=1, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=64 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=2, idx=2, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=128 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=3, idx=3, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=192 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=4, idx=4, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=256 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=5, idx=5, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=320 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=6, idx=6, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=384 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=7, idx=7, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=448 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=8, idx=8, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=512 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=9, idx=9, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=576 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=10, idx=10, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=640 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=11, idx=11, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=16, slice_x=704 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1]

[walterav1984]

I also retried your original fix #177 and found that NTSC is broken, but I thought I have seen NTSC working in some of your versions... maybe it was on a other build machine.

Or could NTSC really need other approach?

[max-verem]

that happens because of ffmpeg does not properly detected interlacing of this video. as you notices from:

sliced_420_pix_fmt_conv_proc:1282: [id=9, idx=9, jobs=12], interlaced=0, field=0, slices=12, mul=1, h=480, slice_w=64, slice_x=576 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1]

mul=1 that is comes from:

mul = ( interlaced ) ? 2 : 1;

you can try manually update flag to force it interlaced:

diff --git a/src/modules/avformat/producer_avformat.c b/src/modules/avformat/producer_avformat.c
index 414955e..b023ab7 100644
--- a/src/modules/avformat/producer_avformat.c
+++ b/src/modules/avformat/producer_avformat.c
@@ -1422,6 +1422,8 @@ static int convert_image( producer_avformat self, AVFrame *frame, uint8_t *buffe

                avpicture_fill( ctx.output, buffer, ctx.dstFormat, width, height );

+               frame->interlaced_frame = 1;
+
                c = ( width + SLICED_420_SLICE_WIDTH - 1 ) / SLICED_420_SLICE_WIDTH;
                c *= frame->interlaced_frame ? 2 : 1;
                for ( i = 0 ; i < c; i++ )

you can check it also with showinfo (https://ffmpeg.org/ffmpeg-filters.html#showinfo) information if the decoded frame is interlaced...

[walterav1984]

Forcing + frame->interlaced_frame = 1; on #182 fixes NTSC rawyuv420 interlaced Chroma 422 Upsampling!

1, slices=12, mul=2, h=240, slice_w=64, slice_x=512 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=18, idx=9, jobs=24], interlaced=1, field=0, slices=12, mul=2, h=240, slice_w=64, slice_x=576 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=19, idx=9, jobs=24], interlaced=1, field=1, slices=12, mul=2, h=240, slice_w=64, slice_x=576 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=20, idx=10, jobs=24], interlaced=1, field=0, slices=12, mul=2, h=240, slice_w=64, slice_x=640 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=21, idx=10, jobs=24], interlaced=1, field=1, slices=12, mul=2, h=240, slice_w=64, slice_x=640 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=22, idx=11, jobs=24], interlaced=1, field=0, slices=12, mul=2, h=240, slice_w=16, slice_x=704 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1] sliced_420_pix_fmt_conv_proc:1282: [id=23, idx=11, jobs=24], interlaced=1, field=1, slices=12, mul=2, h=240, slice_w=16, slice_x=704 ctx->desc=[log2_chroma_h=1, log2_chroma_w=1]

Still one issue for NTSC dv codec. This is 4:1:1 compared to 4:2:0, is it possible to hack your code to also detect 4:1:1? BTW I wasn't able to make ffmpeg itself capable of correctly interlace upsampling 4:1:1 dv to 4:2:2 without chroma upsample errors.

[walterav1984]

NTSC 411 dirty hack I tried this in producer_avformat.c still with forced interlaced option: + else if ( AV_PIX_FMT_YUV420P == pix_fmt && *format == mlt_image_yuv422 ) Changed 420>411:

+ else if ( AV_PIX_FMT_YUV411P == pix_fmt && *format == mlt_image_yuv422 )

0, slices=12, mul=2, h=240, slice_w=64, slice_x=576 ctx->desc=[log2_chroma_h=0, log2_chroma_w=2] sliced_420_pix_fmt_conv_proc:1282: [id=19, idx=9, jobs=24], interlaced=1, field=1, slices=12, mul=2, h=240, slice_w=64, slice_x=576 ctx->desc=[log2_chroma_h=0, log2_chroma_w=2] sliced_420_pix_fmt_conv_proc:1282: [id=20, idx=10, jobs=24], interlaced=1, field=0, slices=12, mul=2, h=240, slice_w=64, slice_x=640 ctx->desc=[log2_chroma_h=0, log2_chroma_w=2] sliced_420_pix_fmt_conv_proc:1282: [id=21, idx=10, jobs=24], interlaced=1, field=1, slices=12, mul=2, h=240, slice_w=64, slice_x=640 ctx->desc=[log2_chroma_h=0, log2_chroma_w=2] sliced_420_pix_fmt_conv_proc:1282: [id=22, idx=11, jobs=24], interlaced=1, field=0, slices=12, mul=2, h=240, slice_w=16, slice_x=704 ctx->desc=[log2_chroma_h=0, log2_chroma_w=2] sliced_420_pix_fmt_conv_proc:1282: [id=23, idx=11, jobs=24], interlaced=1, field=1, slices=12, mul=2, h=240, slice_w=16, slice_x=704 ctx->desc=[log2_chroma_h=0, log2_chroma_w=2]

Funny looking but broken result, seeing 11.25 vertical slice instead of 12 do they overlap?

[walterav1984]

NTSC 4:1:1 DV "uff" already works fine with original "melt" code without the patches addressed in this issue! But DV footage is lowerfield first by spec, but playback of NTSC lowerfield first has field jitter on decklink output, reverse playback of the file reverses field and corrects playback without chroma artifacts!

As long as the Chroma Down Sampling with ffmpeg from correct NTSC 422 footage to 411 NTSC-DV has been done and "melt" detects the footage as interlaced and correct field order! ffmpeg -i intro-ntsc-speedup-crop-reversed-scaleinterl422.avi -vf scale=interl=1 -target ntsc-dv ntsc422correct411dv.dv #might add reverse depending on field order

(probably PAL DVCPRO which is also 4:1:1 might work too)

[max-verem]

Still one issue for NTSC dv codec.

could you clarify? is it still present?

[walterav1984]

Sorry still investigating NTSC issues, I think your code is fine in handling yuv420 PAL and NTSC footage!

But my last concern is NTSC needed sometimes "reverse" filter to be played back correctly without chroma upsampling error. So melt is sometimes not able to detect interlacing or use field order correctly for NTSC that might be the different bug.

Native DVcodec Testcase: I've dig up a SONY DV camcorder (PAL&NTSC) with firewire DV input and S-video output to CRT monitor. Therefor I can skip "melt+decklink output to CRT monitor" from pc instead and use firewire camcorder playback as DAC converter for "dvvideopalorntscfile.dv to camera dv-in" natively and monitor via camera S-Video out to CRT monitor! This flow will only work if PAL or NTSC clips are lowerfield first DV files.

clean.dv #pal example plays correct using firewire and your slice code in melt ntsc422correct411dv.dv #ntsc example plays !!!incorrect!!! on firewire but correct using melt (which needed + frame->interlaced_frame = 1 so something is not totally correct...)