input buffer overruns - ffmpegdecklink 4.0 / vrecord 2018-06-11 / decklink studio 4k

[asfinn]

Just upgraded vrecord and ffmpeg and getting buffer underruns regularly after around 35 mins on 10bit uncompressed captures... Our system had been working reliably on ffmpeg 4.0 and vrecord v2018-05-14..

decklink drivers are on 10.9.7 - can't go higher as we're on El Capitan..

Having trouble switching to an earlier vrecord as I must have uninstalled it --force previously!

any suggestions appreciated!

ffmpeg version 4.0 Copyright (c) 2000-2018 the FFmpeg developers built with Apple LLVM version 8.0.0 (clang-800.0.42.1) configuration: --prefix=/usr/local/Cellar/ffmpegdecklink/4.0 --disable-shared --enable-pthreads --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-gpl --enable-ffplay --enable-libfreetype --enable-libmp3lame --enable-libx264 --enable-libxvid --enable-opencl --enable-videotoolbox --disable-lzma --enable-libopenjpeg --disable-decoder=jpeg2000 --extra-cflags=-I/usr/local/Cellar/openjpeg/2.3.0/include/openjpeg-2.3 --enable-nonfree --enable-decklink --extra-cflags=-I/usr/local/include --extra-ldflags=-L/usr/local/include libavutil 56. 14.100 / 56. 14.100 libavcodec 58. 18.100 / 58. 18.100 libavformat 58. 12.100 / 58. 12.100 libavdevice 58. 3.100 / 58. 3.100 libavfilter 7. 16.100 / 7. 16.100 libavresample 4. 0. 0 / 4. 0. 0 libswscale 5. 1.100 / 5. 1.100 libswresample 3. 1.100 / 3. 1.100 libpostproc 55. 1.100 / 55. 1.100 [decklink @ 0x7fe56b002a00] Found Decklink mode 720 x 576 with rate 25.00(i) Guessed Channel Layout for Input Stream #0.0 : 7.1 Input #0, decklink, from 'DeckLink Studio 4K': Duration: N/A, start: 0.000000, bitrate: 233472 kb/s Stream #0:0: Audio: pcm_s32le, 48000 Hz, 7.1, s32, 12288 kb/s Stream #0:1: Video: v210 (V210 / 0x30313256), yuv422p10le(top first), 720x576, 221184 kb/s, 25 fps, 25 tbr, 1000k tbn, 1000k tbc Output #1, nut, to 'pipe:': Metadata: encoder : Lavf58.12.100 Stream #1:0: Audio: pcm_s32le (PSD / 0x20445350), 48000 Hz, 7.1, s32, 12288 kb/s Stream #1:1: Video: v210 (V210 / 0x30313256), yuv422p10le(top first), 720x576, q=2-31, 221184 kb/s, 25 fps, 25 tbr, 1000k tbn, 1000k tbc Stream mapping: Stream #0:0 (pcm_s32le) -> pan Stream #0:1 (v210) -> setfield setdar -> Stream #0:0 (v210) pan -> Stream #0:1 (pcm_s24le) Stream #0:0 -> #1:0 (copy) Stream #0:1 -> #1:1 (copy) [Parsed_pan_3 @ 0x7fe56a416000] Pure channel mapping detected: 0 1 Output #0, mov, to '/Volumes/video-ingest-d-4tb/01_1765.mov': Metadata: creation_time : now encoder : Lavf58.12.100 Stream #0:0: Video: v210 (v210 / 0x30313276), yuv422p10le(tv, bt470bg/bt470bg/bt709, top coded first (swapped)), 720x576 [SAR 16:15 DAR 4:3], q=2-31, 221184 kb/s, 25 fps, 12800 tbn, 25 tbc (default) Metadata: encoder : Uncompressed 10-bit 4:2:2 Stream #0:1: Audio: pcm_s24le (in24 / 0x34326E69), 48000 Hz, stereo, s32, 2304 kb/s (default) Metadata: encoder : Lavc58.18.100 pcm_s24le [decklink @ 0x7fe56b002a00] Decklink input buffer overrun! Last message repeated 163 times av_interleaved_write_frame(): Broken pipe [decklink @ 0x7fe56b002a00] Decklink input buffer overrun! Last message repeated 1 times Error writing trailer of pipe:: Broken pipe frame=76208 fps= 25 q=-0.0 Lq=-1.0 size=83163161kB time=00:50:48.28 bitrate=223494.1kbits/s speed=0.988x
video:164609280kB audio:5429809kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown Conversion failed!

[dericed]

You should be able to workaround this by selecting to run qcli after rather than during the recording. I have two computers locally where one behaves as you mention and another doesn't so am trying to distinguish them to find the issue.

[asfinn]

Ah yes, thanks.. It's set to No at the moment and I assumed it wasn't running - I may have changed this - I'll set it to 'Yes, after recording' to see if this makes a difference..

[dericed]

btw you can force install an earlier version by using the earlier vrecord formula. For instance if you remove vrecord, you could run brew install https://raw.githubusercontent.com/amiaopensource/homebrew-amiaos/2a230467e5eddc50c4943eba2535b43962af318e/vrecord.rb.

This doesn't resolve the issue but could help in testing. I'm surprised to hear that you have buffer overruns in this case as your recording scenario (uncompressed with no concurrent qctools reporting) seems about as simple as feasible. Also note that vrecord uses the ffmpegdecklink install and not ffmpeg and that ffmpegdecklink is still at 4.0.

[retokromer]

Could you possibly indicate which Mac model you are using? Thank you!

[asfinn]

It's an older Mac Pro 4,1 (Early 2009)

[retokromer]

It's an older Mac Pro 4,1 (Early 2009)

Thanks a lot! I’ll check if we have one of those. On iMac from the same period (Early 2008), by replacing the HDD with a SSD, we could work fine with vrecord 2018-04-23.

Update: I’ll test again on legacy devices as soon as the PR ffmpegdecklink 4.0.1 in merged.

[dericed]

can you send your vrecord config file, like cat ~/.vrecord.conf

[asfinn]

I'm surprised to hear that you have buffer overruns in this case as your recording scenario (uncompressed with no concurrent qctools reporting) seems about as simple as feasible.

Yes me too! Hundreds of hours of video have been digitised on this machine with vrecord in uncompressed, FFV1 and ProRes with no previous buffer overruns - It's been very useful and stable...

Here's the vrecord.conf

#vrecord config file
# Set each value to empty quotes (like "") to prompt during run, or set to a provided option.
# Set VIDEO_INPUT_CHOICE to one of these valid options or leave blank to request each run:  "Composite" "SDI" "Component" "S-Video" "quit" 
VIDEO_INPUT_CHOICE="Component"

#Set AUDIO_INPUT_CHOICE to one of these valid options or leave blank to request each run:  "Analog" "SDI Embedded Audio" "Digital Audio (AES/EBU)" "quit" 
AUDIO_INPUT_CHOICE="Analog"

#Set CONTAINER_CHOICE to one of these valid options or leave blank to request each run:  "QuickTime" "Matroska" "AVI" "MXF" "quit" 
CONTAINER_CHOICE="QuickTime"

#Set VIDEO_CODEC_CHOICE to one of these valid options or leave blank to request each run: "Uncompressed Video" "FFV1 version 3" "JPEG2000" "ProRes" "quit" 
VIDEO_CODEC_CHOICE="Uncompressed Video"

#Set VIDEO_BIT_DEPTH_CHOICE to one of these valid options or leave blank to request each run:  "10 bit" "8 bit" "quit" 
VIDEO_BIT_DEPTH_CHOICE="10 bit"

#Set AUDIO_MAPPING_CHOICE to one of these valid options or leave blank to request each run:  "2 Stereo Tracks (Channels 1 & 2 -> 1st Track Stereo, Channels 3 & 4 -> 2nd Track Stereo)" "1 Stereo Track (From Channels 1 & 2)" "1 Stereo Track (From Channels 3 & 4)" "Channel 1 -> 1st Track Mono, Channel 2 -> 2nd Track Mono" "Channel 2 -> 1st Track Mono, Channel 1 -> 2nd Track Mono" "Channel 1 -> Single Track Mono" "Channel 2 -> Single Track Mono" "quit" 
AUDIO_MAPPING_CHOICE="1 Stereo Track (From Channels 1 & 2)"

#Set STANDARD_CHOICE to one of these valid options or leave blank to request each run:  "NTSC" "PAL" "quit" 
STANDARD_CHOICE="PAL"

#Set QCTOOLSXML_CHOICE to one of these valid options or leave blank to request each run:  "Yes, after recording" "Yes, concurrent with recording" "No" "quit" 
QCTOOLSXML_CHOICE="Yes, after recording"

#Set FRAMEMD5_CHOICE to one of these valid options or leave blank to request each run:  "Yes" "No" "quit" 
FRAMEMD5_CHOICE="No"

#Set embed_choice to one of these valid options or leave blank to request each run:  "Yes" "No" "quit" 
EMBED_LOGS_CHOICE="No"

#Set PLAYBACKVIEW_CHOICE to one of these valid options or leave blank to request each run:  "Quality Control View (mpv)" "Broadcast Range Visual" "Full Range Visual" "Visual + Numerical" "Color Matrix" "Bit Planes" "quit" 
PLAYBACKVIEW_CHOICE="Visual + Numerical"

#Set dir to the preferred recording directory or leave blank to request each run:
DIR="/Volumes/video-ingest-d-4tb"
LOGDIR="/Volumes/video-ingest-d-4tb"

INVERT_PHASE="0"

#Set the recording time as a number (integer or decimal) in minutes.
DURATION="70"

#Enter the name of the person digitizing this tape.
TECHNICIAN=""

[asfinn]

mmm.. I've also noticed there's a gradual but significant delay in the vrecord display and this increases until the buffer underruns occur. As a test at the moment, 53 minutes into a capture according to vtr time, vrecord displays 38 minutes....

[retokromer]

As a test at the moment, 53 minutes into a capture according to vtr time, vrecord displays 38 minutes....

Thank you for this additional information: I guess it’s clear, but I‘ll be able to check only next week. Possibly @dericed can do it quicker, as he is living in an earlier timezone.

[retokromer]

@asfinn: Could you possibly post the results of of Disk Speed Test as well? Thank you!

[asfinn]

Disk Speed Test is what I'd expect on one of my injest drives:

[retokromer]

Thank you!

[asfinn]

Upgraded to vrecord 2018-07-03 and still getting buffer overruns after around 30 mins of recording... Going to revert to an earlier version to test...

[retokromer]

Thank you for the update! Do you still notice the same behaviour with 2018–07–03?

I've also noticed there's a gradual but significant delay in the vrecord display and this increases until the buffer underruns occur.

[asfinn]

Yes, same behaviour with 2018-07-03

I've just reverted to 2018-05-14 and this resolved the issue - I managed to get through 60 minutes of uncompressed recording with no buffer overruns..

[asfinn]

Just realised I may have confused this by earlier using buffer underrun accidently!

The reported issues are buffer overruns!

[retokromer]

Yes, of course, overrun (I did read it that way!). I'll fire-up again an older machine for testing (sorry, not before Wednesday). The program seems to be too slow on your computer. I've seen that @dericed did some additional refactoring, but sadly I could not check this so far.

[dericed]

Hi @asfinn, Was the source tape of your tests a digital tape or do you have some method to compare the outputs? I'm curious to know if the error also occurred in the earlier version but only didn't provide an error. Also for the 2018-05-14 test were you still using ffmpeg 4.0?

[dericed]

Actually I was thinking of silent buffer overruns when remembering this comment https://www.mail-archive.com/ffmpeg-devel@ffmpeg.org/msg70840.html but this is after 4.0. At any rate @asfinn I'll review the history between those releases, if you have consistency in your results between 2018-05-14 and 2018-07-03 with the same ffmpeg, are you willing to help testing by bisecting and testing with the intermediate patches?

[asfinn]

Hi @dericed - no all the issues have been when using component inputs and analogue audio, although I haven't tested to check the digital inputs yet.. Using ffmpeg 4.0.1 at the moment and ffmpegdecklink 4.0.1 with vrecord 2018-07-03 and 2018-05-14...

Happy to help test and also given some time later in the month I could replicate the setup in a newer MacPro 5,1 mid 2012 incase it's hardware related..

[libbyhopfauf]

The computers at MIPoPS are also experiencing this same issue. About 45-50 minutes into a capture we start having buffer overruns which result in audio dropped frames. This has been happening consistently for every capture over 45 minutes, regardless of format. I have attached images of the error message, vrecord.conf and disk speed test.

[retokromer]

On an iMac (Early 2008, HDD replaced by SSD, running OS X 10.11.6), I can reproduce.

[asfinn]

Yes, I'm still experiencing this too with vrecord 2018-07-31 and 2018-08-12 on the same system.. I'm going to swap the Studio 4K card into a more recent 6 core 5,1 machine to see if this makes much of a difference..

@dericed happy to help with #356 but not sure how to do this!

[dericed]

I should close https://github.com/amiaopensource/vrecord/pull/356. I had guessed that perhaps splitting the audio and video handling into separate filter chains might make a difference but didn't seem to help.

[libbyhopfauf]

@asfinn did swapping the cards make a difference? We are trying to decide if this is something we need to do as well. For tapes longer than 35 minutes, we were using Blackmagic Media Express and then vrecord for anything under 35 minutes. However, for the last week, we are still experiencing the same buffer overrun issues (resulting in dropped audio frames and sync issues) after about 35 min as we are in vrecord, so I am wondering if we need to try to replace/update any of our equipment.

[asfinn]

@libbyhopfauf haven't done this yet - holiday season has got in the way and we're using a similar workaround to you. I'll hopefully try a card swap in September..

[libbyhopfauf]

MIPoPS recently upgraded our Mac Minis and did a fresh set-up/installs of everything, but we are still encountering the same issues with buffer overruns and long lists of PTS Discontinuity (resulting in dropped audio frame and audio sync issues) for anything over 35-40 minutes. I've run multiple tests with various configurations, but still cannot complete a successful capture past that 35-40 minute mark. This has all been done with the 2018-09-12 release of vrecord.

I have included the logs and results of a Disk Speed Test (I have kept all the other logs/files from the tests and can pass them along if needed): Test_04_capture_options.log Test_04_ffmpeg_decklink_input.log

Also, here are a screenshot of the audio filters (marking the spot where the buffer overrun error occurs and the PTS discontinuities begin) and link to the full report from QCTools: Test_04.mkv.qctools.xml.gz

We have also noticed that even for successful captures (under 35 minutes) we still receive the same conclusion at the end of the ffmpeg deckling input logs:

av_interleaved_write_frame(): Broken pipe
Error writing trailer of pipe:: Broken pipe
frame=10106 fps= 30 q=-0.0 Lq=-0.0 q=-1.0 size= 1898882kB time=00:05:37.20 bitrate=46131.3kbits/s speed=   1x    
video:24826195kB audio:600634kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Conversion failed!

I wasn’t sure if they were related or not.

MIPoPS would LOVE to participating in any testing to resolve this!!

[libbyhopfauf]

As another note, on our old computers, we were experiencing buffer overruns and audio sync/dropped frames issues on both vrecord (since the 2018-07-03 release forward) and on Media Express (starting in mid-August). But on our new system, running 2018-09-12 we continue to experience all of these issues. However, Media Express has consistent successful captures without any of the noted issues. Here's the QCTools report:

MOV Captured on Media Express

[retokromer]

Could you possibly show the output of cat ~/.vrecord.conf as well? I guess, 2018-09-12 includes some relevant debugging information. Thank you!

[libbyhopfauf]

@retokromer sure! Here it is:

Last login: Mon Sep 17 15:16:57 on ttys001
MIPoPSs-Mac-mini:~ mipopsmac2$ cat ~/.vrecord.conf
# Set these variables to a valid option or leave as empty quotes (like "") to request each run.
VIDEO_INPUT_CHOICE="SDI"
AUDIO_INPUT_CHOICE="Analog"
CONTAINER_CHOICE="Matroska"
VIDEO_CODEC_CHOICE="FFV1 version 3"
VIDEO_BIT_DEPTH_CHOICE="10 bit"
AUDIO_MAPPING_CHOICE="1 Stereo Track (From Channels 1 & 2)"
STANDARD_CHOICE="NTSC"
QCTOOLSXML_CHOICE="No"
FRAMEMD5_CHOICE="Yes"
EMBED_LOGS_CHOICE="No"
PLAYBACKVIEW_CHOICE="Visual + Numerical"
DIR="/Users/mipopsmac2/Desktop/ParticipatingInstitutions"
LOGDIR="/Users/mipopsmac2/Desktop/ParticipatingInstitutions"
INVERT_PHASE="0"
DURATION=""
TECHNICIAN=""
MIPoPSs-Mac-mini:~ mipopsmac2$ 

[retokromer]

Sorry, @libbyhopfauf, me wrong! This has finally been put into the log. Yet you have already posted it elsewhere.

[libbyhopfauf]

@retokromer yes, I also posted the same capture log in my original post above ^

We are having two issues, both of which occur with every capture which is why I posted in both places (here and issue #379): the buffer overruns and the PTS discontinuities. I apologize for the confusion. Please let me know if you need me to pass along anymore information!

[retokromer]

Thank you, @libbyhopfauf, I should have all the relevant information to delve into this. I hope I can do it before leaving for Paris (not Texas).

[asfinn]

I'm happy to say our buffer under run issues now seem to have resolved having updated the MacOS on this ingest MacPro to 10.12.6 (Sierra), Desktop Video to 10.11.2 and using the latest vrecord. Interestingly the hardware hasn't changed, it's still a quad core Zeon with 8GB of RAM and the Decklink Studio 4K card.

[retokromer]

Thank you for the update, @asfinn. Can this issue be closed?

[asfinn]

Yes, I think so @retokromer !

[dericed]

@libbyhopfauf can you confirm as well?

[dericed]

Noting that @michi-gato did a test with Desktop Video 10.11.2 and Mac OS 10.14.2 (Mojave) and it produced overruns as well with the latest vrecord. This was with an imac with 4 cores.

computer_model_name: iMac
computer_model_id: iMac17,1
computer_processor_name: Intel Core i5
computer_processor_speed: 3.2 GHz
computer_processor_count: 1
computer_cores: 4
computer_memory: 16 GB

[retokromer]

Interesting indeed! Thank you @michi-gato @dericed, for reporting. Are there other informations about macOS 10.14? (FYI: On our Mac mini Server (Late 2012) under macOS 10.12 it works just fine in production.)

[asfinn]

Ok, we'll stress test more fully next week...so far we've successfully digitised over 60 minutes so far.. On the previous MacOS (10.11.xx and Desktop Video 10.9.7 we'd see underruns consistently after 35 - 40 minutes..

[dericed]

🤔, can you confirm that you're generating framemd5 and qctools at the same time. Also what playback view are you using? ffv1/mkv or something else?

[retokromer]

• Could @michi-gato possibly check if network, WiFi, Bluetooth, anti-virus et similia are enabled or disabled?
• @dericed: FFV1 has the encoding limitations I already mentioned, yet of course yes by using it.
• We’ll run today a longer test digitisation than our usual production items (a 90–120 min films on VHS in long play mode).
• After my teaching in Donostia, I’ll try to find the time to go through the numerous changes which were committed so quickly that I could not review during the last months.

[retokromer]

@dericed I can confirm that we could digitise successfully a 95 min film from a VHS in SD. I suspect that some «other shit» is running on macOS, consuming resources and memory. On my end, I always switch off all what is not needed… Possibly we’ve to go through the system_profiler’s output, a tedious task.

[asfinn]

@dericed @retokromer We've just digitised a 90 minute SD tape to FFV1 using the hardware and software versions listed https://github.com/amiaopensource/vrecord/issues/343#issuecomment-455304035 - no problems still which is reassuring..

[dericed]

happy MLK day

[NMAAHC]

Hi,

Having a buffer overrun issue, as well.

[decklink @ 0x7f8dc60ab800] Decklink input buffer overrun! 0B f=0/0
Last message repeated 1 times [decklink @ 0x7f8dc60ab800] Decklink input buffer overrun! 0B f=0/0
Last message repeated 1 times [decklink @ 0x7f8dc60ab800] Decklink input buffer overrun! 0B f=0/0
/usr/local/bin/vrecord: line 553: 78384 Segmentation fault: 11 "${FFPLAY_DECKLINK}" "${FFPLAY_OPTIONS[@]}" 2> >(tee /tmp/vrecord_input.log 1>&2)

We are on new mac mini, would not think it a hardware issue.

Model Name: Mac mini Model Identifier: Macmini8,1 Processor Name: Intel Core i7 Processor Speed: 3.2 GHz Number of Processors: 1 Total Number of Cores: 6 L2 Cache (per Core): 256 KB L3 Cache: 12 MB Hyper-Threading Technology: Enabled Memory: 16.01 GB

The problem starts about an hour into a tape.

/Users/medialab ≈:≈ cat ./.vrecord.conf Set these variables to a valid option or leave as empty quotes (like "") to request each run. VIDEO_INPUT_CHOICE="SDI" AUDIO_INPUT_CHOICE="SDI Embedded Audio" CONTAINER_CHOICE="Matroska" VIDEO_CODEC_CHOICE="FFV1 version 3" FFV1_SLICE_CHOICE="24" AUDIO_CODEC_CHOICE="24-bit FLAC" VIDEO_BIT_DEPTH_CHOICE="10 bit" AUDIO_MAPPING_CHOICE="2 Stereo Tracks (Channels 1 & 2 -> 1st Track Stereo, Channels 3 & 4 -> 2nd Track Stereo)" TIMECODE_CHOICE="none" STANDARD_CHOICE="NTSC" QCTOOLSXML_CHOICE="Yes, after recording" FRAMEMD5_CHOICE="Yes" EMBED_LOGS_CHOICE="Yes" PLAYBACKVIEW_CHOICE="Visual + Numerical" PLAYBACKVIEW_CHOICE_PASS="Visual + Numerical" DIR="/Users/medialab/Desktop/CCP_Chicago/02_Professional Curation/01_DUSABLE/MV4188" LOGDIR="/Users/medialab/Desktop/CCP_Chicago/02_Professional Curation/01_DUSABLE/MV4188" INVERT_PHASE="false" DURATION="" PREFIX="RFS_20190909_DUSABLE_VHS_MV" USER_SUFFIX="" NO_SUFFIX="true" TECHNICIAN="NFS_MEDIA_DIGITIZATION_LAB"

[NMAAHC]

Also getting the following about broken pipes and conversion errors at the end.

[decklink @ 0x7fd027001200] Frame received (#23) - Input returned - Frames dropped 2 [decklink @ 0x7fd027001200] Frame received (#10940) - No input signal detected - Frames dropped 3 [decklink @ 0x7fd027001200] Frame received (#10986) - Input returned - Frames dropped 4 [matroska,webm @ 0x7ffece88e600] Read error4556KB sq= 0B f=0/0

av_interleaved_write_frame(): Broken pipe Error writing trailer of pipe:: Broken pipe frame=10998 fps= 30 q=-0.0 Lq=-0.0 q=-1.0 size= 3958872kB time=00:06:06.96 bitrate=88375.8kbits/s speed= 1x
video:28949741kB audio:614696kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown Conversion failed! qcli dev build, Nov 3 2018 11:50:36 filters selected: signalstats aphasemeter astats ssim

analyzing input file... /Users/medialab/Desktop/Chicago-RFS/02_Professional Curation/01_DUSABLE/MV4191/RFS_20190909_DUSABLE_VHS_MV4191.mkv .................................................. 100 of 100 % analyzing completed

generating QCTools report... .................................................. 100 of 100 % generating QCTools report... done Vrecord is analyzing your video file. Please be patient. Vrecord is generating graphs from the QCTools data. One moment please

[retokromer]

Can this be closed? Over the past five years, many changes have been made …

[privatezero]

I'd say so? I assume none of the original people reporting the issue are still having this exact problem - I'll close. If anyone is having this issue in the current release of vrecord, feel free to reopen this issue!