search

sourcey / libsourcey

C++14 evented IO libraries for high performance networking and media based applications
https://sourcey.com/libsourcey
GNU Lesser General Public License v2.1
1.31k stars 347 forks source link

Invalid pts on trying to record webcam in devicerecorder sample from av module. #241

Open traw opened 6 years ago

traw commented 6 years ago 
17:21:38 [info] [devicerecorder.cpp(40)] Using video device: HD WebCam: HD WebCam
17:21:38 [debug] [videocapture.cpp(65)] Opening camera: /dev/video0, width=640, height=480, framerate=0, pixelFmt=yuv420p
Input #0, video4linux2,v4l2, from '/dev/video0':
  Duration: N/A, start: 11823.639138, bitrate: 147456 kb/s
    Stream #0:0: Video: rawvideo (YUY2 / 0x32595559), yuyv422, 640x480, 147456 kb/s, 30 fps, 30 tbr, 1000k tbn, 1000k tbc
17:21:38 [debug] [videocontext.cpp(57)] Open: 
    Input: VideoCodec[Unknown:rawvideo:640:480:30:yuyv422:1]
    Output: VideoCodec[Unknown:rawvideo:640:480:30:yuyv422:1]
17:21:38 [info] [recorder.cpp(49)] Using Audio device: Playback/recording through the PulseAudio sound server
Input #0, alsa, from 'default':
  Duration: N/A, start: 1527508299.082254, bitrate: 1411 kb/s
    Stream #0:0: Audio: pcm_s16le, 44100 Hz, 2 channels, s16, 1411 kb/s
17:21:39 [debug] [audiocontext.cpp(52)] Open: 
    Input: AudioCodec[Unknown:pcm_s16le:44100:1411200:2:s16:1]
    Output: AudioCodec[Unknown:pcm_s16le:44100:1411200:2:s16:1]
[libx264 @ 0x242a830] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 AVX2 LZCNT BMI2
[libx264 @ 0x242a830] profile High, level 2.1
[libx264 @ 0x242a830] 264 - core 148 - H.264/MPEG-4 AVC codec - Copyleft 2003-2017 - http://www.videolan.org/x264.html - options: cabac=1 ref=5 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=8 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=2 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=9 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=3 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=50 rc=abr mbtree=1 bitrate=128 ratetol=1.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
17:21:39 [debug] [videocontext.cpp(57)] Open: 
    Input: VideoCodec[Unknown:rawvideo:640:480:30:yuv420p:1]
    Output: VideoCodec[libx264:h264:400:300:25:yuv420p:1]
17:21:39 [debug] [videocontext.cpp(184)] Recreating video conversion context
17:21:39 [debug] [audiocontext.cpp(52)] Open: 
    Input: AudioCodec[Unknown:pcm_s16le:44100:1411200:2:s16:1]
    Output: AudioCodec[aac:aac:44100:64000:2:fltp:1]
17:21:39 [debug] [audiocontext.cpp(151)] Recreating audio resampler context
[mp4 @ 0x240ab70] Using AVStream.codec.time_base as a timebase hint to the muxer is deprecated. Set AVStream.time_base instead.
[mp4 @ 0x240ab70] Using AVStream.codec.time_base as a timebase hint to the muxer is deprecated. Set AVStream.time_base instead.
Output #0, mp4, to 'deveiceoutput.mp4':
  Metadata:
    encoder         : Lavf57.25.100
    Stream #0:0: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv420p, 400x300, q=-1--1, 128 kb/s, 12800 tbn, 25 tbc
    Stream #0:1: Audio: aac (LC) ([64][0][0][0] / 0x0040), 44100 Hz, stereo, fltp, 64 kb/s
17:21:39 [info] [recorder.cpp(64)] Recording video: deveiceoutput.mp4
17:21:39 [debug] [videocontext.cpp(184)] Recreating video conversion context
17:21:39 [warn] [multiplexencoder.cpp(297)] Invalid pts (224) <= last (67363115990158)
17:21:39 [warn] [multiplexencoder.cpp(297)] Invalid pts (20660) <= last (67363115990412)
17:21:39 [warn] [multiplexencoder.cpp(297)] Invalid pts (27529) <= last (67363115990476)
17:21:39 [warn] [multiplexencoder.cpp(297)] Invalid pts (34032) <= last (67363115990476)
17:21:39 [warn] [multiplexencoder.cpp(297)] Invalid pts (40327) <= last (67363115990476)
17:21:39 [warn] [multiplexencoder.cpp(297)] Invalid pts (46639) <= last (67363115990476)
17:21:39 [warn] [multiplexencoder.cpp(297)] Invalid pts (52848) <= last (67363115990476)
17:21:39 [warn] [multiplexencoder.cpp(297)] Invalid pts (59542) <= last (67363115990540)
17:21:39 [warn] [multiplexencoder.cpp(297)] Invalid pts (66402) <= last (67363115990604)
17:21:39 [warn] [multiplexencoder.cpp(297)] Invalid pts (73101) <= last (67363115990604)
17:21:39 [warn] [multiplexencoder.cpp(297)] Invalid pts (627782) <= last (67363115992526)
17:21:39 [warn] [multiplexencoder.cpp(297)] Invalid pts (1497936) <= last (67363115995540)
17:21:39 [warn] [multiplexencoder.cpp(297)] Invalid pts (2317237) <= last (67363115998357)
17:21:39 [warn] [multiplexencoder.cpp(297)] Invalid pts (3187619) <= last (67363116001363)
17:21:39 [warn] [multiplexencoder.cpp(297)] Invalid pts (4006952) <= last (67363116004178)
17:21:39 [warn] [multiplexencoder.cpp(297)] Invalid pts (4877307) <= last (67363116007188)
17:21:39 [warn] [multiplexencoder.cpp(297)] Invalid pts (5696613) <= last (67363116009937)
17:21:39 [warn] [multiplexencoder.cpp(297)] Invalid pts (6566938) <= last (67363116013007)
17:21:39 [warn] [multiplexencoder.cpp(297)] Invalid pts (7437253) <= last (67363116016014)
17:21:39 [warn] [multiplexencoder.cpp(297)] Invalid pts (8256547) <= last (67363116018829)
17:21:39 [warn] [multiplexencoder.cpp(297)] Invalid pts (9126767) <= last (67363116021838)
17:21:39 [warn] [multiplexencoder.cpp(297)] Invalid pts (9946564) <= last (67363116024656)
17:21:39 [warn] [multiplexencoder.cpp(297)] Invalid pts (10816517) <= last (67363116027599)
17:21:40 [warn] [multiplexencoder.cpp(297)] Invalid pts (11635801) <= last (67363116030414)
17:21:40 [warn] [multiplexencoder.cpp(297)] Invalid pts (12506185) <= last (67363116033424)
17:21:40 [warn] [multiplexencoder.cpp(297)] Invalid pts (13376378) <= last (67363116036496)
17:21:40 [warn] [multiplexencoder.cpp(297)] Invalid pts (14195727) <= last (67363116039312)
17:21:40 [warn] [multiplexencoder.cpp(297)] Invalid pts (15066041) <= last (67363116042319)
17:21:40 [warn] [multiplexencoder.cpp(297)] Invalid pts (15885344) <= last (67363116045070)
17:21:40 [warn] [multiplexencoder.cpp(297)] Invalid pts (16755797) <= last (67363116048076)
17:21:40 [warn] [multiplexencoder.cpp(297)] Invalid pts (17576008) <= last (67363116050890)
17:21:40 [warn] [multiplexencoder.cpp(297)] Invalid pts (18445786) <= last (67363116053961)
17:21:40 [warn] [multiplexencoder.cpp(297)] Invalid pts (19265008) <= last (67363116056779)
17:21:40 [warn] [multiplexencoder.cpp(297)] Invalid pts (20134922) <= last (67363116059723)
17:21:40 [warn] [multiplexencoder.cpp(297)] Invalid pts (21005250) <= last (67363116062668)
17:21:40 [warn] [multiplexencoder.cpp(297)] Invalid pts (21824611) <= last (67363116065613)
17:21:40 [warn] [multiplexencoder.cpp(297)] Invalid pts (22695101) <= last (67363116068557)
17:21:40 [warn] [multiplexencoder.cpp(297)] Invalid pts (23514476) <= last (67363116071309)
17:21:41 [warn] [multiplexencoder.cpp(297)] Invalid pts (24384585) <= last (67363116074381)
17:21:41 [warn] [multiplexencoder.cpp(297)] Invalid pts (25204077) <= last (67363116077198)
17:21:41 [warn] [multiplexencoder.cpp(297)] Invalid pts (26074153) <= last (67363116080141)
17:21:41 [warn] [multiplexencoder.cpp(297)] Invalid pts (26944510) <= last (67363116083214)
17:21:41 [warn] [multiplexencoder.cpp(297)] Invalid pts (27763801) <= last (67363116085965)
17:21:41 [warn] [multiplexencoder.cpp(297)] Invalid pts (28634149) <= last (67363116089037)
17:21:41 [warn] [multiplexencoder.cpp(297)] Invalid pts (29453440) <= last (67363116091853)
17:21:41 [warn] [multiplexencoder.cpp(297)] Invalid pts (30323806) <= last (67363116094862)
17:21:41 [warn] [multiplexencoder.cpp(297)] Invalid pts (31143067) <= last (67363116097677)

FFProbe output:

 /mnt/repos/sourcey/workspace/bin/ffprobe /mnt/repos/sourcey/test/deviceoutput.mp4                                                                                                           master    19% ▃  17:27:03 
ffprobe version 3.0-shared Copyright (c) 2007-2016 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.3.0-16ubuntu3~16.04.1)
  configuration: --arch=64 --prefix=/mnt/repos/sourcey/workspace --extra-cflags=-I/mnt/repos/sourcey/workspace/include --extra-ldflags=-L/mnt/repos/sourcey/workspace/lib --extra-version=shared --extra-cflags=--shared --enable-shared --disable-static --enable-pic --disable-debug --disable-ffserver --disable-doc --enable-gpl --enable-version3 --enable-nonfree --enable-runtime-cpudetect --enable-pthreads --enable-libvpx --enable-libmp3lame --enable-libopus --enable-libx264 --enable-libx265 --enable-libfdk-aac --enable-avfilter --enable-filters
  libavutil      55. 17.103 / 55. 17.103
  libavcodec     57. 24.102 / 57. 24.102
  libavformat    57. 25.100 / 57. 25.100
  libavdevice    57.  0.101 / 57.  0.101
  libavfilter     6. 31.100 /  6. 31.100
  libswscale      4.  0.100 /  4.  0.100
  libswresample   2.  0.101 /  2.  0.101
  libpostproc    54.  0.100 / 54.  0.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/mnt/repos/sourcey/test/deviceoutput.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.25.100
  Duration: 00:00:12.91, start: 1527508704.836009, bitrate: 68 kb/s
    Stream #0:0(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 64 kb/s (default)
    Metadata:
      handler_name    : SoundHandler

UPDATE 1: Audio and Video both are enabled.

options.iformat.audio.enabled = true; // enabled if available
options.iformat.video.enabled = true; // enabled if available

UPDATE 2: Audio is working fine and also audio pts is in sync but no video.

I'm using following library versions: LibSourcey-> 1.1.4, WebRTC-native-> branch/66 FFmpeg-> 3.0

traw commented 6 years ago

After some efforts I'm able to capture laptop webcam. That requires to disable setting pts manually, cause av_read_frame sets packet pts value

diff --git a/src/av/src/mediacapture.cpp b/src/av/src/mediacapture.cpp
index 9afe87a..99e9008 100644
--- a/src/av/src/mediacapture.cpp
+++ b/src/av/src/mediacapture.cpp
@@ -201,7 +201,7 @@ void MediaCapture::run()

                 // Realtime PTS calculation in microseconds
                 if (_realtime) {
-                    ipacket.pts = time::hrtime() - startTime;
+                    //ipacket.pts = time::hrtime() + startTime;
                 }
                 else if (_looping) {
                     // Set the PTS offset when looping