Strange quantization error in waveform

[iamdamosuzuki]

I found a file that creates a strange waveform in QCTools. It looks like the values are quantized, instead of being evenly spread out across all of the available values. Videos containing similar content from similar materials do not display this issue. I thought maybe it was form an 8-bit to 10-bit upres, but the Bit Plane Noise Slice filter ruled that out. Any ideas?

[kieranjol]

Was this a betasp captured on a digibeta deck by any chance? I've noticed this and am writing a blog post about it actually! I usually see weird bit plane info in ninth and tenth bits.not black as you might expect,but very similar detail in both planes. And bits seven and eight are just noise, bit nine and ten both have the same detail!

On 27 Sep 2017 20:14, "Morgan Oscar Morel" notifications@github.com wrote:

I found a file that creates a strange waveform in QCTools. It looks like the values are quantized, instead of being evenly spread out across all of the available values. Videos containing similar content from similar materials do not display this issue. I thought maybe it was form an 8-bit to 10-bit upres, but the Bit Plane Noise Slice filter ruled that out. Any ideas?

[image: dance_01] https://user-images.githubusercontent.com/3967767/30932760-4da52b3c-a37d-11e7-8b5b-93f2391fa046.png

[image: dance_02] https://user-images.githubusercontent.com/3967767/30932759-4da50260-a37d-11e7-8d17-3b3d5d80a801.png

[image: dance_03] https://user-images.githubusercontent.com/3967767/30932761-4db037ca-a37d-11e7-9dac-40c99e07677d.png

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bavc/qctools/issues/451, or mute the thread https://github.com/notifications/unsubscribe-auth/ABEyvjihfiF9JdnjrMT5nQs8urhDGXsAks5smp6DgaJpZM4PmQpI .

[kieranjol]

P.s you can also see weird values in the histogram!

Btw,ive seen this when capturing betasp using a j30,510p,m2000p.all on different workstations with different brands of capture card. When I play back digibeta in the same decks,waveforms,histograms and bit planes all look grand.

On 27 Sep 2017 21:57, "Kieran O Leary" kieran.o.leary@gmail.com wrote:

Was this a betasp captured on a digibeta deck by any chance? I've noticed this and am writing a blog post about it actually! I usually see weird bit plane info in ninth and tenth bits.not black as you might expect,but very similar detail in both planes. And bits seven and eight are just noise, bit nine and ten both have the same detail!

On 27 Sep 2017 20:14, "Morgan Oscar Morel" notifications@github.com wrote:

I found a file that creates a strange waveform in QCTools. It looks like the values are quantized, instead of being evenly spread out across all of the available values. Videos containing similar content from similar materials do not display this issue. I thought maybe it was form an 8-bit to 10-bit upres, but the Bit Plane Noise Slice filter ruled that out. Any ideas?

[image: dance_01] https://user-images.githubusercontent.com/3967767/30932760-4da52b3c-a37d-11e7-8b5b-93f2391fa046.png

[image: dance_02] https://user-images.githubusercontent.com/3967767/30932759-4da50260-a37d-11e7-8d17-3b3d5d80a801.png

[image: dance_03] https://user-images.githubusercontent.com/3967767/30932761-4db037ca-a37d-11e7-9dac-40c99e07677d.png

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bavc/qctools/issues/451, or mute the thread https://github.com/notifications/unsubscribe-auth/ABEyvjihfiF9JdnjrMT5nQs8urhDGXsAks5smp6DgaJpZM4PmQpI .

[richardpl]

Typical for no real 10bit content but just badly upconverted one.

[kieranjol]

@richardpl I'm glad that you verified this.i didn't know if a dodgy 8bit to 10bit upscaling was a cause of this. I think what puzzled me was that the ninth and tenth bits didn't just have padded zeroes.

[kieranjol]

Ppppps I ingested roughly the same scene in a betasp deck and then a digi deck. Here's four videos showing the scopes. I see your pattern replicated in the digi decks,but the betasp ingests show roughly expected values. https://archive.org/details/betasp_510p_waveform.mkv_silent

On 27 Sep 2017 23:07, "Paul B Mahol" notifications@github.com wrote:

Typical for no real 10bit content but just badly upconverted one.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/bavc/qctools/issues/451#issuecomment-332669761, or mute the thread https://github.com/notifications/unsubscribe-auth/ABEyvjLDP16n6m9bWrsPVtYd2a9W8RCXks5smscqgaJpZM4PmQpI .

[iamdamosuzuki]

Crazy! This was a VHS transfer going S-Video into a BlackMagic 4K Extreme card

[kieranjol]

Was it a direct connection or was there a TBC in between.. Or did the deck have TBC built in?I've never digitised vhs at work actually.. very interested to see this same issue here!

On 27 Sep 2017 23:28, "Morgan Oscar Morel" notifications@github.com wrote:

Crazy! This was a VHS transfer going S-Video into a BlackMagic 4K Extreme card

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/bavc/qctools/issues/451#issuecomment-332673802, or mute the thread https://github.com/notifications/unsubscribe-auth/ABEyvvw9E8exO0ZdEjbFhh311zyd2M2Fks5smsv6gaJpZM4PmQpI .

[iamdamosuzuki]

I used a Sony SVO-5800, which has a built in TBC. Other tapes transferred on that deck have been fine, I'm totally confounded haha!

[dericed]

Is it the same issue, when you use mpv or ffplay directly with the waveform filter?

[dericed]

So thinking that this is not a qctools error, but just an error that qctools reveals. Ping to @richardpl as this reminds of an earlier thread about having a filter report metadata that quantifies disparity of neighboring sample values in a histogram (which would reveal when a video is normalized, badly upscaled, or instances of error concealment in some early digital tapes).

[kieranjol]

Not sure where that concealment filter thread is but +1000.

On 29 Sep 2017 04:35, "Dave Rice" notifications@github.com wrote:

So thinking that this is not a qctools error, but just an error that qctools reveals. Ping to @richardpl https://github.com/richardpl as this reminds of an earlier thread about having a filter report metadata that quantifies disparity of neighboring sample values in a histogram (which would reveal when a video is normalized, badly upscaled, or instances of error concealment in some early digital tapes).

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/bavc/qctools/issues/451#issuecomment-333022888, or mute the thread https://github.com/notifications/unsubscribe-auth/ABEyvi5MUxBVUQufKDSY-RPIF7TacVdxks5snGWBgaJpZM4PmQpI .

[iamdamosuzuki]

Yeah I think it's definitely some sort of digital signal chain error. What's weird is that the 9th and 10th bit have data in them (not just padded zeroes), but that it appears to be bad data.

[kieranjol]

I'm more puzzled by it only happening with one tape..

On 29 Sep 2017 17:27, "Morgan Oscar Morel" notifications@github.com wrote:

Yeah I think it's definitely some sort of digital signal chain error. What's weird is that the 9th and 10th bit have data in them (not just padded zeroes), but that it appears to be bad data.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/bavc/qctools/issues/451#issuecomment-333173577, or mute the thread https://github.com/notifications/unsubscribe-auth/ABEyvh20y01bjqghyk_VFE03CA4RyjY2ks5snRpxgaJpZM4PmQpI .

[iamdamosuzuki]

I found the issue on my end. It was caused by using the S-Video input on a DPS 475. The Component and CAV inputs work just fine however. Wild!

[kieranjol]

That is good to know! Morgan,I might get in touch with you about the blog post on this when I've a bit more free time.

On 29 Sep 2017 20:27, "Morgan Oscar Morel" notifications@github.com wrote:

Closed #451 https://github.com/bavc/qctools/issues/451.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/bavc/qctools/issues/451#event-1272238047, or mute the thread https://github.com/notifications/unsubscribe-auth/ABEyvhltoLDjAXM6IHdppgPBWvQm8mObks5snUSegaJpZM4PmQpI .

[CSchloss385]

We use SDI here at CUNY TV and I actually just checked a few of our files and I saw a similar pattern in the waveform. I also see weird values represented in the histogram. So I guess there is still an issue related to playing a Betacam SP in a Digibeta deck? I might run some tests where I disable the proc amps to see what the histogram looks with and without the proc amps enabled. Definitely interested in reading your blogpost about his issue @kieranjol

[bturkus]

Me too. Very curious. Is it that some digibeta decks quantize at 8 bits sample, some 10?

I just started reviewing the DVW-A500 manual ( http://www.grsv.com/downloads/manuals/Sony-DVW-A500-Manual.pdf, page 141), but I guess I just wait for Kieran.

On Fri, Sep 29, 2017 at 3:56 PM, CSchloss385 notifications@github.com wrote:

We use SDI here at CUNY TV and I actually just checked a few of our files and I saw a similar pattern in the waveform. I also see weird values represented in the histogram. So I guess there is still an issue related to playing a Betacam SP in a Digibeta deck? I might run some tests where I disable the proc amps to see what the histogram looks with and without the proc amps enabled. Definitely interested in reading your blogpost about his issue @kieranjol https://github.com/kieranjol

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bavc/qctools/issues/451#issuecomment-333223690, or mute the thread https://github.com/notifications/unsubscribe-auth/AIkiXmeFSrZzByrMRe4Yk5ct4b_sH38Xks5snUtqgaJpZM4PmQpI .

--

Benjamin Turkus | The New York Public Library Assistant Manager for Audio and Moving Image Preservation

Barbara Goldsmith Preservation Division 40 Lincoln Center Plaza, New York, NY 10023 T. 212.870.1609 <(212)870-1609> | benjaminturkus@nypl.org http://www.nypl.org/preservation https://twitter.com/NYPLPreserve

Lifelong Learning | Advancing Knowledge | Strengthening Our Communities

[kieranjol]

Hey, sorry for the bump - I might write something about all this finally, but my colleague @wendriftwood and I were looking into this again today and figured that a way to check for this in an automated way is to analyse the xml file and check if certain Y/U/V values are divisible by 4, which should (though I'd imagine not always) give a sense of padding. It's been working well so far anyhow.

Here's just the test sketch we knocked up - we're not really XML parsing, just looking through the XML as if it's a textfile for the moment.

#!/usr/bin/env/ python
import sys

def main():
    source = sys.argv[1]
    if source.endswith('.xml'):
        print(' - OK, we will use %s as the source' % source)
        with open(source, 'r') as xml_file:
            lines = xml_file.readlines()
        for line in lines:
            if 'ULOW' in line or 'YLOW' in line or 'YMAX' in line:
                value = int(line.split()[2].split('.')[0].split('=')[1].replace('"', ''))
                if not value % 4 == 0:
                    print value    
    else:
        print(' - NOT XML - THIS SHOULD BE AN XML FILEEEEEE')
        sys.exit()    

if __name__ == '__main__':
    main()

[iamdamosuzuki]

Kieran this is such a great idea! thank you!