search

shaka-project / shaka-packager

A media packaging and development framework for VOD and Live DASH and HLS applications, supporting Common Encryption for Widevine and other DRM Systems.
https://shaka-project.github.io/shaka-packager/
Other
2k stars 510 forks source link

Deadlock in MpdBuilder causing packetization to stop after several hours #45

Closed peyoh closed 9 years ago

peyoh commented 9 years ago

Hello, I'm using v1.2.0. The command line is:

packager --udp_interface_address=172.18.0.10 'input=udp://239.101.2.1:1234,stream=audio,init_segment=/storage/web/dash/bnt1/bnt1_edash-audio.m4a,segment_template=/storage/web/dash/bnt1/bnt1_edash-audio-$Number$.m4a' 'input=udp://239.101.2.1:1234,stream=video,init_segment=/storage/web/dash/bnt1/bnt1_edash-video-1080.m4v,segment_template=/storage/web/dash/bnt1/bnt1_edash-video-1080-$Number$.m4v' 'input=udp://239.102.2.1:1234,stream=video,init_segment=/storage/web/dash/bnt1/bnt1_edash-video-720.m4v,segment_template=/storage/web/dash/bnt1/bnt1_edash-video-720-$Number$.m4v' 'input=udp://239.103.2.1:1234,stream=video,init_segment=/storage/web/dash/bnt1/bnt1_edash-video-576.m4v,segment_template=/storage/web/dash/bnt1/bnt1_edash-video-576-$Number$.m4v' --profile live --mpd_output /storage/web/dash/bnt1/bnt1.mpd -time_shift_buffer_depth 1800 -dump_stream_info

After several hours of operation, edash packager stops to fill the segments directory and .mpd file updates. The three groups are still joined and tcpdump shows correct traffic to the system. The last console output is:

[1026/105614:INFO:demuxer.cc(135)] Demuxer::Run() on file 'udp://239.101.2.1:1234'. [1026/105614:INFO:demuxer.cc(135)] Demuxer::Run() on file 'udp://239.102.2.1:1234'. [1026/105614:INFO:demuxer.cc(135)] Demuxer::Run() on file 'udp://239.103.2.1:1234'. [1026/105614:WARNING:mpd_builder.cc(211)] Video info does not contain frame duration required for calculating framerate. @frameRate is required for DASH IOP. [1026/105614:WARNING:mpd_builder.cc(211)] Video info does not contain frame duration required for calculating framerate. @frameRate is required for DASH IOP. [1026/105615:WARNING:mpd_builder.cc(211)] Video info does not contain frame duration required for calculating framerate. @frameRate is required for DASH IOP.

Any help how I can debug what's wrong? How I can increase the verbosity?

BRS/ Peyo

kqyang commented 9 years ago

I suspect that somehow packager program was no longer able to receive from udp. I created a debug code at https://github.com/kqyang/edash-packager, which includes two changes:

  1. Added a 30 seconds timeout on udp recvfrom
  2. Added verbosity logging flag, which can be enabled through '--vmodule' flag: --vmodule=':'; does wild card matches on the module.

You can try with that code and see what happens. You could increase verbosity level with flag '--vmodule', e.g. --vmodule='*=5', the larger the number, the more loggings.

You could also use vlc to confirm whether the stream is still good: vlc udp://@239.101.2.1:1234 (remember to close packager program first).

peyoh commented 9 years ago

Hello, with this build, only 720 video profile (239.102.2.1:1234)is filled, but without mpd file. I'm using ffmpeg. Please note that with release 1.1, the issue cannot be reproduced. It works fine as expected.

So, with vmodule=1, the last debug lines are: [1027/095148:VERBOSE1:ts_section_psi.cc(103)] Trailing bytes after a PSI section: 16 vs 183 [1027/095148:VERBOSE1:ts_section_psi.cc(103)] Trailing bytes after a PSI section: 32 vs 183 [1027/095148:VERBOSE1:mp2t_media_parser.cc(284)] RegisterPes: pes_pid=256 stream_type=1b [1027/095148:VERBOSE1:mp2t_media_parser.cc(284)] RegisterPes: pes_pid=257 stream_type=f [1027/095148:VERBOSE1:ts_section_psi.cc(103)] Trailing bytes after a PSI section: 16 vs 183 [1027/095148:VERBOSE1:ts_section_psi.cc(103)] Trailing bytes after a PSI section: 21 vs 183 [1027/095148:VERBOSE1:mp2t_media_parser.cc(284)] RegisterPes: pes_pid=256 stream_type=1b

With vmodule=5, the last lines before "hang" are: [1027/095418:VERBOSE3:mp2t_media_parser.cc(381)] OnEmitSample: pid=256 size=51442 dts=942472800 pts=942476400 [1027/095418:VERBOSE5:mp2t_media_parser.cc(216)] Processing PID=256 start_unit=0 [1027/095418:VERBOSE4:h264_parser.cc(433)] NALU type: 8 at: 0x255d9f7 size: 4 ref: 1 [1027/095418:VERBOSE5:mp2t_media_parser.cc(216)] Processing PID=256 start_unit=0 [1027/095418:VERBOSE5:mp2t_media_parser.cc(216)] Processing PID=256 start_unit=0 [1027/095418:VERBOSE3:es_parser_h264.cc(214)] NALU: PPS [1027/095418:VERBOSE5:mp2t_media_parser.cc(216)] Processing PID=256 start_unit=0 [1027/095418:VERBOSE5:mp2t_media_parser.cc(216)] Processing PID=256 start_unit=0 [1027/095418:VERBOSE4:h264_parser.cc(412)] NALU found: size=9 [1027/095418:VERBOSE5:mp2t_media_parser.cc(216)] Processing PID=256 start_unit=0 [1027/095418:VERBOSE5:mp2t_media_parser.cc(216)] Processing PID=256 start_unit=0 [1027/095418:VERBOSE5:mp2t_media_parser.cc(216)] Processing PID=256 start_unit=0 [1027/095418:VERBOSE5:mp2t_media_parser.cc(216)] Processing PID=256 start_unit=0 [1027/095418:VERBOSE4:h264_parser.cc(433)] NALU type: 6 at: 0x255d9ff size: 5 ref: 0 [1027/095418:VERBOSE5:mp2t_media_parser.cc(216)] Processing PID=256 start_unit=0 [1027/095418:VERBOSE5:mp2t_media_parser.cc(216)] Processing PID=256 start_unit=0 [1027/095418:VERBOSE3:es_parser_h264.cc(241)] NALU: 6 [1027/095418:VERBOSE5:mp2t_media_parser.cc(216)] Processing PID=256 start_unit=0 [1027/095418:VERBOSE5:mp2t_media_parser.cc(216)] Processing PID=256 start_unit=0 [1027/095418:VERBOSE5:mp2t_media_parser.cc(216)] Processing PID=256 start_unit=0 [1027/095418:VERBOSE5:mp2t_media_parser.cc(216)] Processing PID=256 start_unit=0 [1027/095418:VERBOSE5:mp2t_media_parser.cc(216)] Processing PID=256 start_unit=0 [1027/095418:VERBOSE5:mp2t_media_parser.cc(216)] Processing PID=256 start_unit=0 [1027/095418:VERBOSE3:mp2t_media_parser.cc(403)] Mp2tMediaParser::EmitRemainingBuffers [1027/095418:VERBOSE5:mp2t_media_parser.cc(216)] Processing PID=256 start_unit=0 [1027/095418:VERBOSE4:h264_parser.cc(412)] NALU found: size=4906 [1027/095418:VERBOSE5:mp2t_media_parser.cc(216)] Processing PID=256 start_unit=0 [1027/095418:VERBOSE4:h264_parser.cc(433)] NALU type: 1 at: 0x255da07 size: 4903 ref: 1 [1027/095418:VERBOSE3:mp2t_media_parser.cc(403)] Mp2tMediaParser::EmitRemainingBuffers [1027/095418:VERBOSE3:es_parser_h264.cc(228)] NALU: slice IDR=0 [1027/095418:VERBOSE4:h264_parser.cc(327)] Could not find start code, end of stream? [1027/095418:VERBOSE4:h264_parser.cc(405)] Could not find next NALU, bytes left in stream: 0 [1027/095418:VERBOSE3:es_parser_h264.cc(275)] Emit frame: stream_pos=6270636 size=4929 [1027/095418:VERBOSE3:mp2t_media_parser.cc(381)] OnEmitSample: pid=256 size=47009 dts=942472800 pts=942476400 [1027/095418:VERBOSE5:mp2t_media_parser.cc(216)] Processing PID=256 start_unit=0 [1027/095418:VERBOSE5:mp2t_media_parser.cc(216)] Processing PID=256 start_unit=0 [1027/095418:VERBOSE5:mp2t_media_parser.cc(216)] Processing PID=256 start_unit=0 [1027/095418:VERBOSE5:mp2t_media_parser.cc(216)] Processing PID=256 start_unit=0 [1027/095418:VERBOSE5:mp2t_media_parser.cc(216)] Processing PID=256 start_unit=0 [1027/095418:VERBOSE5:mp2t_media_parser.cc(216)] Processing PID=256 start_unit=0 [1027/095418:VERBOSE5:mp2t_media_parser.cc(216)] Processing PID=256 start_unit=0 [1027/095418:VERBOSE5:mp2t_media_parser.cc(216)] Processing PID=256 start_unit=0 [1027/095418:VERBOSE5:mp2t_media_parser.cc(216)] Processing PID=256 start_unit=0 [1027/095418:VERBOSE3:mp2t_media_parser.cc(403)] Mp2tMediaParser::EmitRemainingBuffers [1027/095418:VERBOSE3:mpd_builder.cc(1171)] MediaInfo missing field 'bandwidth'. Using estimated from segment size. [1027/095418:VERBOSE2:xmlnode.cc(140)] Passing node. [1027/095418:VERBOSE2:xmlnode.cc(140)] Passing node. [1027/095418:VERBOSE2:xmlnode.cc(140)] Passing node. [1027/095418:VERBOSE2:xmlnode.cc(140)] Passing node.

Please also note, that I'm able to see the incoming traffic with tcpdump on the interface and on parallel computer edash packager release 1.1 is still working.

BRS/ Peyo

kqyang commented 9 years ago

Em... It is good to know that it works with v1.1.

I don't see anything interesting from the log. What happened after "the last debug lines"? Did packager program quit or still running but without any more logging?

Notice that the time difference between the two logs is only 3 minutes. So, you can reproduce the problem in just minutes? You mentioned 'ffmpeg'. Are you using 'ffmpeg' to reproduce it? Can you explain your duplication steps in more detail?

Thanks.

peyoh commented 9 years ago

Hi, after the last logs, the app is still running but without any future output - debug or files. BTW, I've noticed that sometimes it stops after a couple of minutes.

FFmpeg config is quite simple. One MPEG-TS multicast stream is separated in 3 different h264 resolution profiles and re-streamed in 3 different multicast groups. The 3 groups are handles by edash_packager with command line you can see in my first post. The ffmpeg line is: ffmpeg -v verbose -re -i 'udp://@239.100.2.1:1234?localaddr=10.0.4.36&fifo_size=100000&timeout=2&overrun_nonfatal=1' -filter_complex '[0:v]yadif,split=3[out1][out2][out3]' -map '[out1]' -vcodec libx264 -profile main -preset veryfast -s 720x576 -map 0:a -acodec aac -strict -2 -flush_packets 0 -f mpegts 'udp://239.101.2.1:1234?localaddr=172.17.0.10&pkt_size=1316&buffer_size=65536' -map '[out2]' -vcodec libx264 -profile main -preset veryfast -s 642x512 -flush_packets 0 -f mpegts 'udp://239.102.2.1:1234?localaddr=172.17.0.10&pkt_size=1316&buffer_size=65536' -map '[out3]' -vcodec libx264 -profile main -preset veryfast -s 480x384 -flush_packets 0 -f mpegts 'udp://239.103.2.1:1234?localaddr=172.17.0.10&pkt_size=1316&buffer_size=65536'

That's the setup.

BRS/ Peyo

kqyang commented 9 years ago

I tried to duplicate the failure, but still not able to after running for days. Can you use gdb to attach to packager process to find out what it is doing, e.g. extract stack traces?

dviano commented 9 years ago

Hi, I'm seeing this on my setup with a very recent build using docker. In my debugging efforts (and before finding this report) I've managed to generate a full strace (-f) of the packager while failing like this, the whole file is huge (~500M) but here I attach the last lines including the last open()+write()+close() in hope to help pin this down.

packager_strace_after_last_seg.txt (recvfrom data was edited out for size-sake)

The execve was: (sensible data edited out)

21002 execve("/root/edash_packager/Release/packager", ["/root/edash_packager/Release/packager", "input=udp://127.0.0.1:3343,stream=audio,init_segment=a/live-audio-1447177556.mp4,segment_template=a/live-audio-1447177556-$Number$.mp4,bandwidth=140000", "input=udp://127.0.0.1:3343,stream=video,init_segment=0/live-video-1447177556.mp4,segment_template=0/live-video-1447177556-$Number$.mp4,bandwidth=2100000", "input=udp://127.0.0.1:3344,stream=video,init_segment=1/live-video-1447177556.mp4,segment_template=1/live-video-1447177556-$Number$.mp4,bandwidth=990000", "input=udp://127.0.0.1:3345,stream=video,init_segment=2/live-video-1447177556.mp4,segment_template=2/live-video-1447177556-$Number$.mp4,bandwidth=430000", "input=udp://127.0.0.1:3346,stream=video,init_segment=3/live-video-1447177556.mp4,segment_template=3/live-video-1447177556-$Number$.mp4,bandwidth=150000", "--enable_widevine_encryption", "--key_server_url", "xxx", "--content_id", "xxx", "--signer", "xxx", "--aes_signing_key", "xxx", "-aes_signing_iv", "xxx", "--segment_duration", "3", "--min_buffer_time", "30", "--suggested_presentation_delay", "15", "--time_shift_buffer_depth", "300", "--profile", "live", "--mpd_output", "live.mpd"], [/* 13 vars */]) = 0

Let me know if I can help any further.

kqyang commented 9 years ago

Thanks for the report.

Can you try adding a new option "--io_cache_size=0" to see if it resolves the problem?

Also, use debug version which may reveal more information for debugging.

dviano commented 9 years ago

Ok, so I've tried your fork, with and without "--io_cache_size=0", both with "--vmodule='*=5'", both Release version. Both hanged after a while with the following log:

[1112/115416:VERBOSE3:demuxer.cc(177)] Read 1692
[1112/115417:VERBOSE3:demuxer.cc(177)] Read 1316
[1112/115417:VERBOSE3:demuxer.cc(177)] Read 376
[1112/115417:VERBOSE3:demuxer.cc(177)] Read 1692
[1112/115417:VERBOSE3:demuxer.cc(177)] Read 1472
[1112/115417:VERBOSE3:demuxer.cc(177)] Read 13380
[1112/115417:VERBOSE3:demuxer.cc(177)] Read 752
[1112/115417:VERBOSE3:demuxer.cc(177)] Read 564
[1112/115417:VERBOSE3:demuxer.cc(177)] Read 1316
[1112/115417:VERBOSE3:demuxer.cc(177)] Read 940

I'll try the debug version of your fork now, and maybe attach a gdb and try to get a trace.

Let me know if I can help any further.

dviano commented 9 years ago

Hi again, I've tried the debug version, and it fails a lot quickier, but looks like the same error as it just hangs there, attached a gdb while it was hanging there:

(gdb) info threads 
  Id   Target Id         Frame 
  6    Thread 0x7f58825bb700 (LWP 1063) "KeyProductionTh" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  5    Thread 0x7f5880f8d700 (LWP 2421) "RemuxJob/2421" __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
  4    Thread 0x7f587bfff700 (LWP 2422) "RemuxJob/2422" __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
  3    Thread 0x7f587b7fe700 (LWP 2423) "RemuxJob/2423" __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
  2    Thread 0x7f587affd700 (LWP 2424) "RemuxJob/2424" __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
* 1    Thread 0x7f58839bb740 (LWP 1062) "packager" 0x00007f588296e4db in pthread_join (threadid=140018097641216, thread_return=0x0) at pthread_join.c:92

also bt of a RemuxJob:

(gdb) thread 5
[Switching to thread 5 (Thread 0x7f5880f8d700 (LWP 2421))]
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
135     ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: No such file or directory.
(gdb) bt
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f588296f4b9 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f588296f2e0 in __GI___pthread_mutex_lock (mutex=0x1b10378) at ../nptl/pthread_mutex_lock.c:79
#3  0x00000000006b27fb in base::internal::LockImpl::Lock (this=0x1b10378) at ../../packager/base/synchronization/lock_impl_posix.cc:45
#4  0x000000000040bd43 in base::Lock::Acquire (this=0x1b10370) at ../../packager/base/synchronization/lock.h:42
#5  0x000000000042e5d3 in base::AutoLock::AutoLock (this=0x7f5880f89268, lock=...) at ../../packager/base/synchronization/lock.h:99
#6  0x00000000004d47cd in edash_packager::SimpleMpdNotifier::NotifySampleDuration (this=0x1b10350, container_id=2, sample_duration=3600)
    at ../../packager/mpd/base/simple_mpd_notifier.cc:76
#7  0x000000000041df43 in edash_packager::media::MpdNotifyMuxerListener::OnSampleDurationReady (this=0x1b21560, sample_duration=3600)
    at ../../packager/media/event/mpd_notify_muxer_listener.cc:91
#8  0x000000000043fa8f in edash_packager::media::mp4::MultiSegmentSegmenter::WriteSegment (this=0x7f58740853e0)
    at ../../packager/media/formats/mp4/multi_segment_segmenter.cc:188
#9  0x000000000043edcc in edash_packager::media::mp4::MultiSegmentSegmenter::DoFinalizeSegment (this=0x7f58740853e0)
    at ../../packager/media/formats/mp4/multi_segment_segmenter.cc:90
#10 0x0000000000445a5c in edash_packager::media::mp4::Segmenter::FinalizeSegment (this=0x7f58740853e0) at ../../packager/media/formats/mp4/segmenter.cc:355
#11 0x0000000000444e08 in edash_packager::media::mp4::Segmenter::FinalizeFragment (this=0x7f58740853e0, finalize_segment=true, fragmenter=0x7f5874085220)
    at ../../packager/media/formats/mp4/segmenter.cc:422
#12 0x0000000000445503 in edash_packager::media::mp4::Segmenter::AddSample (this=0x7f58740853e0, stream=0x1b1efb0, sample=...)
    at ../../packager/media/formats/mp4/segmenter.cc:303
#13 0x000000000043096d in edash_packager::media::mp4::MP4Muxer::DoAddSample (this=0x1b21480, stream=0x1b1efb0, sample=...)
    at ../../packager/media/formats/mp4/mp4_muxer.cc:133
#14 0x0000000000647217 in edash_packager::media::Muxer::AddSample (this=0x1b21480, stream=0x1b1efb0, sample=...) at ../../packager/media/base/muxer.cc:110
#15 0x00000000006434b7 in edash_packager::media::MediaStream::PushSample (this=0x1b1efb0, sample=...) at ../../packager/media/base/media_stream.cc:48
#16 0x00000000006389e6 in edash_packager::media::Demuxer::NewSampleEvent (this=0x1b116c0, track_id=256, sample=...) at ../../packager/media/base/demuxer.cc:126
#17 0x000000000063b7b6 in base::internal::RunnableAdapter<bool (edash_packager::media::Demuxer::*)(unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>::Run (this=0x7f5880f8b500, object=0x1b116c0, args=..., args=...) at ../../packager/base/bind_internal.h:176
#18 0x000000000063b6e9 in base::internal::InvokeHelper<false, bool, base::internal::RunnableAdapter<bool (edash_packager::media::Demuxer::*)(unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>, base::internal::TypeList<edash_packager::media::Demuxer*, unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&> >::MakeItSo (runnable=..., args=..., args=..., args=...) at ../../packager/base/bind_internal.h:286
#19 0x000000000063b682 in base::internal::Invoker<base::IndexSequence<0ul>, base::internal::BindState<base::internal::RunnableAdapter<bool (edash_packager::media::Demuxer::*)(unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>, bool (edash_packager::media::Demuxer*, unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&), base::internal::TypeList<base::internal::UnretainedWrapper<edash_packager::media::Demuxer> > >, base::internal::TypeList<base::internal::UnwrapTraits<base::internal::UnretainedWrapper<edash_packager::media::Demuxer> > >, base::internal::InvokeHelper<false, bool, base::internal::RunnableAdapter<bool (edash_packager::media::Demuxer::*)(unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>, base::internal::TypeList<edash_packager::media::Demuxer*, unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&> >, bool (unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&)>::Run(base::internal::BindStateBase*, unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&) (base=0x1b0e640, unbound_args=..., unbound_args=...) at ../../packager/base/bind_internal.h:343
#20 0x00000000009d113b in base::Callback<bool (unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>::Run(unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&) const (this=0x1b11198, args=..., args=...) at ../../packager/base/callback.h:396
#21 0x00000000009ccf37 in edash_packager::media::mp2t::Mp2tMediaParser::EmitRemainingSamples (this=0x1b11180) at ../../packager/media/formats/mp2t/mp2t_media_parser.cc:416
#22 0x00000000009cd9e5 in edash_packager::media::mp2t::Mp2tMediaParser::Parse (this=0x1b11180, buf=0x7f5881591010 "GA\001=\001@", size=1472)
    at ../../packager/media/formats/mp2t/mp2t_media_parser.cc:250
#23 0x0000000000638f5e in edash_packager::media::Demuxer::Parse (this=0x1b116c0) at ../../packager/media/base/demuxer.cc:185
#24 0x0000000000639290 in edash_packager::media::Demuxer::Run (this=0x1b116c0) at ../../packager/media/base/demuxer.cc:146
#25 0x000000000040f844 in edash_packager::media::RemuxJob::Run (this=0x1b1b320) at ../../packager/app/packager_main.cc:108
---Type <return> to continue, or q <return> to quit---
#26 0x00000000006c3821 in base::SimpleThread::ThreadMain (this=0x1b1b320) at ../../packager/base/threading/simple_thread.cc:66
#27 0x00000000006c2c3a in base::(anonymous namespace)::ThreadFunc (params=0x7ffe3d367c30) at ../../packager/base/threading/platform_thread_posix.cc:74
#28 0x00007f588296d0a4 in start_thread (arg=0x7f5880f8d700) at pthread_create.c:309
#29 0x00007f58826a204d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

and bt of the KeyProductionThread:

(gdb) thread 6
[Switching to thread 6 (Thread 0x7f58825bb700 (LWP 1063))]
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185     ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or directory.
(gdb) bt
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00000000006b17a5 in base::ConditionVariable::Wait (this=0x7f58825baa98) at ../../packager/base/synchronization/condition_variable_posix.cc:54
#2  0x00000000006b319e in base::WaitableEvent::TimedWait (this=0x1b0edd8, max_time=...) at ../../packager/base/synchronization/waitable_event_posix.cc:214
#3  0x00000000006b2e06 in base::WaitableEvent::Wait (this=0x1b0edd8) at ../../packager/base/synchronization/waitable_event_posix.cc:154
#4  0x000000000064d385 in edash_packager::media::WidevineKeySource::FetchKeysTask (this=0x1b0ecc0) at ../../packager/media/base/widevine_key_source.cc:282
#5  0x0000000000657139 in base::internal::RunnableAdapter<void (edash_packager::media::WidevineKeySource::*)()>::Run (this=0x7f58825bad90, object=0x1b0ecc0)
    at ../../packager/base/bind_internal.h:176
#6  0x000000000065707c in base::internal::InvokeHelper<false, void, base::internal::RunnableAdapter<void (edash_packager::media::WidevineKeySource::*)()>, base::internal::TypeList<edash_packager::media::WidevineKeySource*> >::MakeItSo (runnable=..., args=0x1b0ecc0) at ../../packager/base/bind_internal.h:293
#7  0x0000000000657048 in base::internal::Invoker<base::IndexSequence<0ul>, base::internal::BindState<base::internal::RunnableAdapter<void (edash_packager::media::WidevineKeySource::*)()>, void (edash_packager::media::WidevineKeySource*), base::internal::TypeList<base::internal::UnretainedWrapper<edash_packager::media::WidevineKeySource> > >, base::internal::TypeList<base::internal::UnwrapTraits<base::internal::UnretainedWrapper<edash_packager::media::WidevineKeySource> > >, base::internal::InvokeHelper<false, void, base::internal::RunnableAdapter<void (edash_packager::media::WidevineKeySource::*)()>, base::internal::TypeList<edash_packager::media::WidevineKeySource*> >, void ()>::Run(base::internal::BindStateBase*) (base=0x1b10140) at ../../packager/base/bind_internal.h:343
#8  0x0000000000637c3e in base::Callback<void ()>::Run() const (this=0x1b0ed18) at ../../packager/base/callback.h:396
#9  0x0000000000637bd9 in edash_packager::media::ClosureThread::Run (this=0x1b0ecd0) at ../../packager/media/base/closure_thread.cc:22
#10 0x00000000006c3821 in base::SimpleThread::ThreadMain (this=0x1b0ecd0) at ../../packager/base/threading/simple_thread.cc:66
#11 0x00000000006c2c3a in base::(anonymous namespace)::ThreadFunc (params=0x7ffe3d367870) at ../../packager/base/threading/platform_thread_posix.cc:74
#12 0x00007f588296d0a4 in start_thread (arg=0x7f58825bb700) at pthread_create.c:309
#13 0x00007f58826a204d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Again, let me know if there's anything else I can do to help nail this one.

kqyang commented 9 years ago

Thanks for the traces. Can you get the back trace for all threads?

dviano commented 9 years ago

Sure (kept it running just in case):

(gdb) thread 1
[Switching to thread 1 (Thread 0x7f58839bb740 (LWP 1062))]
#0  0x00007f588296e4db in pthread_join (threadid=140018097641216, thread_return=0x0) at pthread_join.c:92
92      pthread_join.c: No such file or directory.
(gdb) bt
#0  0x00007f588296e4db in pthread_join (threadid=140018097641216, thread_return=0x0) at pthread_join.c:92
#1  0x00000000006c2737 in base::PlatformThread::Join (thread_handle=...) at ../../packager/base/threading/platform_thread_posix.cc:221
#2  0x00000000006c375b in base::SimpleThread::Join (this=0x1b1b320) at ../../packager/base/threading/simple_thread.cc:47
#3  0x000000000040ceae in edash_packager::media::RunRemuxJobs (remux_jobs=std::__debug::vector of length 4, capacity 4 = {...}) at ../../packager/app/packager_main.cc:236
#4  0x000000000040d55f in edash_packager::media::RunPackager (stream_descriptors=std::__debug::multiset with 5 elements = {...}) at ../../packager/app/packager_main.cc:305
#5  0x000000000040d9ea in edash_packager::media::PackagerMain (argc=6, argv=0x7ffe3d3690e0) at ../../packager/app/packager_main.cc:339
#6  0x000000000040da82 in main (argc=31, argv=0x7ffe3d369018) at ../../packager/app/packager_main.cc:346
(gdb) thread 2
[Switching to thread 2 (Thread 0x7f587affd700 (LWP 2424))]
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
135     ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: No such file or directory.
(gdb) bt
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f588296f4b9 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f588296f2e0 in __GI___pthread_mutex_lock (mutex=0x1b10378) at ../nptl/pthread_mutex_lock.c:79
#3  0x00000000006b27fb in base::internal::LockImpl::Lock (this=0x1b10378) at ../../packager/base/synchronization/lock_impl_posix.cc:45
#4  0x000000000040bd43 in base::Lock::Acquire (this=0x1b10370) at ../../packager/base/synchronization/lock.h:42
#5  0x000000000042e5d3 in base::AutoLock::AutoLock (this=0x7f587aff9268, lock=...) at ../../packager/base/synchronization/lock.h:99
#6  0x00000000004d47cd in edash_packager::SimpleMpdNotifier::NotifySampleDuration (this=0x1b10350, container_id=4, sample_duration=3600)
    at ../../packager/mpd/base/simple_mpd_notifier.cc:76
#7  0x000000000041df43 in edash_packager::media::MpdNotifyMuxerListener::OnSampleDurationReady (this=0x1c18a20, sample_duration=3600)
    at ../../packager/media/event/mpd_notify_muxer_listener.cc:91
#8  0x000000000043fa8f in edash_packager::media::mp4::MultiSegmentSegmenter::WriteSegment (this=0x7f586c002660)
    at ../../packager/media/formats/mp4/multi_segment_segmenter.cc:188
#9  0x000000000043edcc in edash_packager::media::mp4::MultiSegmentSegmenter::DoFinalizeSegment (this=0x7f586c002660)
    at ../../packager/media/formats/mp4/multi_segment_segmenter.cc:90
#10 0x0000000000445a5c in edash_packager::media::mp4::Segmenter::FinalizeSegment (this=0x7f586c002660) at ../../packager/media/formats/mp4/segmenter.cc:355
#11 0x0000000000444e08 in edash_packager::media::mp4::Segmenter::FinalizeFragment (this=0x7f586c002660, finalize_segment=true, fragmenter=0x7f586c0430e0)
    at ../../packager/media/formats/mp4/segmenter.cc:422
#12 0x0000000000445503 in edash_packager::media::mp4::Segmenter::AddSample (this=0x7f586c002660, stream=0x1c55e60, sample=...)
    at ../../packager/media/formats/mp4/segmenter.cc:303
#13 0x000000000043096d in edash_packager::media::mp4::MP4Muxer::DoAddSample (this=0x1c2bd80, stream=0x1c55e60, sample=...)
    at ../../packager/media/formats/mp4/mp4_muxer.cc:133
#14 0x0000000000647217 in edash_packager::media::Muxer::AddSample (this=0x1c2bd80, stream=0x1c55e60, sample=...) at ../../packager/media/base/muxer.cc:110
#15 0x00000000006434b7 in edash_packager::media::MediaStream::PushSample (this=0x1c55e60, sample=...) at ../../packager/media/base/media_stream.cc:48
#16 0x00000000006389e6 in edash_packager::media::Demuxer::NewSampleEvent (this=0x1bc6740, track_id=256, sample=...) at ../../packager/media/base/demuxer.cc:126
#17 0x000000000063b7b6 in base::internal::RunnableAdapter<bool (edash_packager::media::Demuxer::*)(unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>::Run (this=0x7f587affb500, object=0x1bc6740, args=..., args=...) at ../../packager/base/bind_internal.h:176
#18 0x000000000063b6e9 in base::internal::InvokeHelper<false, bool, base::internal::RunnableAdapter<bool (edash_packager::media::Demuxer::*)(unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>, base::internal::TypeList<edash_packager::media::Demuxer*, unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&> >::MakeItSo (runnable=..., args=..., args=..., args=...) at ../../packager/base/bind_internal.h:286
#19 0x000000000063b682 in base::internal::Invoker<base::IndexSequence<0ul>, base::internal::BindState<base::internal::RunnableAdapter<bool (edash_packager::media::Demuxer::*)(unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>, bool (edash_packager::media::Demuxer*, unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&), base::internal::TypeList<base::internal::UnretainedWrapper<edash_packager::media::Demuxer> > >, base::internal::TypeList<base::internal::UnwrapTraits<base::internal::UnretainedWrapper<edash_packager::media::Demuxer> > >, base::internal::InvokeHelper<false, bool, base::internal::RunnableAdapter<bool (edash_packager::media::Demuxer::*)(unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>, base::internal::TypeList<edash_packager::media::Demuxer*, unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&> >, bool (unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&)>::Run(base::internal::BindStateBase*, unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&) (base=0x1bc9b00, unbound_args=..., unbound_args=...) at ../../packager/base/bind_internal.h:343
#20 0x00000000009d113b in base::Callback<bool (unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>::Run(unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&) const (this=0x1bc9a68, args=..., args=...) at ../../packager/base/callback.h:396
#21 0x00000000009ccf37 in edash_packager::media::mp2t::Mp2tMediaParser::EmitRemainingSamples (this=0x1bc9a50) at ../../packager/media/formats/mp2t/mp2t_media_parser.cc:416
#22 0x00000000009cd9e5 in edash_packager::media::mp2t::Mp2tMediaParser::Parse (this=0x1bc9a50, buf=0x7f5880f8e010 "GA\001\061\001@", size=1128)
    at ../../packager/media/formats/mp2t/mp2t_media_parser.cc:250
#23 0x0000000000638f5e in edash_packager::media::Demuxer::Parse (this=0x1bc6740) at ../../packager/media/base/demuxer.cc:185
#24 0x0000000000639290 in edash_packager::media::Demuxer::Run (this=0x1bc6740) at ../../packager/media/base/demuxer.cc:146
#25 0x000000000040f844 in edash_packager::media::RemuxJob::Run (this=0x1c60fb0) at ../../packager/app/packager_main.cc:108
---Type <return> to continue, or q <return> to quit---
#26 0x00000000006c3821 in base::SimpleThread::ThreadMain (this=0x1c60fb0) at ../../packager/base/threading/simple_thread.cc:66
#27 0x00000000006c2c3a in base::(anonymous namespace)::ThreadFunc (params=0x7ffe3d367c30) at ../../packager/base/threading/platform_thread_posix.cc:74
#28 0x00007f588296d0a4 in start_thread (arg=0x7f587affd700) at pthread_create.c:309
#29 0x00007f58826a204d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) thread 3
[Switching to thread 3 (Thread 0x7f587b7fe700 (LWP 2423))]
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
135     in ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S
(gdb) bt
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f588296f4b9 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f588296f2e0 in __GI___pthread_mutex_lock (mutex=0x7f58700443c8) at ../nptl/pthread_mutex_lock.c:79
#3  0x00000000006b27fb in base::internal::LockImpl::Lock (this=0x7f58700443c8) at ../../packager/base/synchronization/lock_impl_posix.cc:45
#4  0x000000000040bd43 in base::Lock::Acquire (this=0x7f58700443c0) at ../../packager/base/synchronization/lock.h:42
#5  0x000000000042e5d3 in base::AutoLock::AutoLock (this=0x7f587b7fa140, lock=...) at ../../packager/base/synchronization/lock.h:99
#6  0x00000000004b2003 in edash_packager::AdaptationSet::OnSetFrameRateForRepresentation (this=0x7f5870044360, frame_duration=3600, timescale=90000)
    at ../../packager/mpd/base/mpd_builder.cc:821
#7  0x00000000004b44db in edash_packager::(anonymous namespace)::RepresentationStateChangeListenerImpl::OnSetFrameRateForRepresentation (this=0x7f5868043c30, 
    frame_duration=3600, timescale=90000) at ../../packager/mpd/base/mpd_builder.cc:367
#8  0x00000000004b3934 in edash_packager::Representation::SetSampleDuration (this=0x7f5868043c50, sample_duration=3600) at ../../packager/mpd/base/mpd_builder.cc:1083
#9  0x00000000004d492e in edash_packager::SimpleMpdNotifier::NotifySampleDuration (this=0x1b10350, container_id=3, sample_duration=3600)
    at ../../packager/mpd/base/simple_mpd_notifier.cc:82
#10 0x000000000041df43 in edash_packager::media::MpdNotifyMuxerListener::OnSampleDurationReady (this=0x1bc7630, sample_duration=3600)
    at ../../packager/media/event/mpd_notify_muxer_listener.cc:91
#11 0x000000000043fa8f in edash_packager::media::mp4::MultiSegmentSegmenter::WriteSegment (this=0x7f5868002240)
    at ../../packager/media/formats/mp4/multi_segment_segmenter.cc:188
#12 0x000000000043edcc in edash_packager::media::mp4::MultiSegmentSegmenter::DoFinalizeSegment (this=0x7f5868002240)
    at ../../packager/media/formats/mp4/multi_segment_segmenter.cc:90
#13 0x0000000000445a5c in edash_packager::media::mp4::Segmenter::FinalizeSegment (this=0x7f5868002240) at ../../packager/media/formats/mp4/segmenter.cc:355
#14 0x0000000000444e08 in edash_packager::media::mp4::Segmenter::FinalizeFragment (this=0x7f5868002240, finalize_segment=true, fragmenter=0x7f5868043610)
    at ../../packager/media/formats/mp4/segmenter.cc:422
#15 0x0000000000445503 in edash_packager::media::mp4::Segmenter::AddSample (this=0x7f5868002240, stream=0x1c11dc0, sample=...)
    at ../../packager/media/formats/mp4/segmenter.cc:303
#16 0x000000000043096d in edash_packager::media::mp4::MP4Muxer::DoAddSample (this=0x1bc9970, stream=0x1c11dc0, sample=...)
    at ../../packager/media/formats/mp4/mp4_muxer.cc:133
#17 0x0000000000647217 in edash_packager::media::Muxer::AddSample (this=0x1bc9970, stream=0x1c11dc0, sample=...) at ../../packager/media/base/muxer.cc:110
#18 0x00000000006434b7 in edash_packager::media::MediaStream::PushSample (this=0x1c11dc0, sample=...) at ../../packager/media/base/media_stream.cc:48
#19 0x00000000006389e6 in edash_packager::media::Demuxer::NewSampleEvent (this=0x1bcf2e0, track_id=256, sample=...) at ../../packager/media/base/demuxer.cc:126
#20 0x000000000063b7b6 in base::internal::RunnableAdapter<bool (edash_packager::media::Demuxer::*)(unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>::Run (this=0x7f587b7fc500, object=0x1bcf2e0, args=..., args=...) at ../../packager/base/bind_internal.h:176
#21 0x000000000063b6e9 in base::internal::InvokeHelper<false, bool, base::internal::RunnableAdapter<bool (edash_packager::media::Demuxer::*)(unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>, base::internal::TypeList<edash_packager::media::Demuxer*, unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&> >::MakeItSo (runnable=..., args=..., args=..., args=...) at ../../packager/base/bind_internal.h:286
#22 0x000000000063b682 in base::internal::Invoker<base::IndexSequence<0ul>, base::internal::BindState<base::internal::RunnableAdapter<bool (edash_packager::media::Demuxer::*)(unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>, bool (edash_packager::media::Demuxer*, unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&), base::internal::TypeList<base::internal::UnretainedWrapper<edash_packager::media::Demuxer> > >, base::internal::TypeList<base::internal::UnwrapTraits<base::internal::UnretainedWrapper<edash_packager::media::Demuxer> > >, base::internal::InvokeHelper<false, bool, base::internal::RunnableAdapter<bool (edash_packager::media::Demuxer::*)(unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>, base::internal::TypeList<edash_packager::media::Demuxer*, unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&> >, bool (unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&)>::Run(base::internal::BindStateBase*, unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&) (base=0x1bc3820, unbound_args=..., unbound_args=...) at ../../packager/base/bind_internal.h:343
#23 0x00000000009d113b in base::Callback<bool (unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>::Run(unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&) const (this=0x1bc3788, args=..., args=...) at ../../packager/base/callback.h:396
#24 0x00000000009ccf37 in edash_packager::media::mp2t::Mp2tMediaParser::EmitRemainingSamples (this=0x1bc3770) at ../../packager/media/formats/mp2t/mp2t_media_parser.cc:416
---Type <return> to continue, or q <return> to quit---
#25 0x00000000009cd9e5 in edash_packager::media::mp2t::Mp2tMediaParser::Parse (this=0x1bc3770, buf=0x7f588118f010 "GA\001=\001@", size=1316)
    at ../../packager/media/formats/mp2t/mp2t_media_parser.cc:250
#26 0x0000000000638f5e in edash_packager::media::Demuxer::Parse (this=0x1bcf2e0) at ../../packager/media/base/demuxer.cc:185
#27 0x0000000000639290 in edash_packager::media::Demuxer::Run (this=0x1bcf2e0) at ../../packager/media/base/demuxer.cc:146
#28 0x000000000040f844 in edash_packager::media::RemuxJob::Run (this=0x1bc9d30) at ../../packager/app/packager_main.cc:108
#29 0x00000000006c3821 in base::SimpleThread::ThreadMain (this=0x1bc9d30) at ../../packager/base/threading/simple_thread.cc:66
#30 0x00000000006c2c3a in base::(anonymous namespace)::ThreadFunc (params=0x7ffe3d367c30) at ../../packager/base/threading/platform_thread_posix.cc:74
#31 0x00007f588296d0a4 in start_thread (arg=0x7f587b7fe700) at pthread_create.c:309
#32 0x00007f58826a204d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) thread 4
[Switching to thread 4 (Thread 0x7f587bfff700 (LWP 2422))]
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
135     in ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S
(gdb) bt
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f588296f4b9 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f588296f2e0 in __GI___pthread_mutex_lock (mutex=0x7f5868043d40) at ../nptl/pthread_mutex_lock.c:79
#3  0x00000000006b27fb in base::internal::LockImpl::Lock (this=0x7f5868043d40) at ../../packager/base/synchronization/lock_impl_posix.cc:45
#4  0x000000000040bd43 in base::Lock::Acquire (this=0x7f5868043d38) at ../../packager/base/synchronization/lock.h:42
#5  0x000000000042e5d3 in base::AutoLock::AutoLock (this=0x7f587bff9b78, lock=...) at ../../packager/base/synchronization/lock.h:99
#6  0x00000000004b0ed7 in edash_packager::Representation::GetXml (this=0x7f5868043c50) at ../../packager/mpd/base/mpd_builder.cc:1096
#7  0x00000000004acf5f in edash_packager::AdaptationSet::GetXml (this=0x7f5870044360) at ../../packager/mpd/base/mpd_builder.cc:731
#8  0x00000000004ac718 in edash_packager::MpdBuilder::GenerateMpd (this=0x1b10e80) at ../../packager/mpd/base/mpd_builder.cc:448
#9  0x00000000004b5225 in edash_packager::MpdBuilder::WriteMpdToOutput<std::string> (this=0x1b10e80, output=0x7f587bffaf18) at ../../packager/mpd/base/mpd_builder.cc:419
#10 0x00000000004ac5d9 in edash_packager::MpdBuilder::ToString (this=0x1b10e80, output=0x7f587bffaf18) at ../../packager/mpd/base/mpd_builder.cc:413
#11 0x00000000004cb4e1 in edash_packager::WriteMpdToFile (output_path="live.mpd", mpd_builder=0x1b10e80) at ../../packager/mpd/base/mpd_notifier_util.cc:24
#12 0x00000000004d5090 in edash_packager::SimpleMpdNotifier::Flush (this=0x1b10350) at ../../packager/mpd/base/simple_mpd_notifier.cc:130
#13 0x000000000041e6db in edash_packager::media::MpdNotifyMuxerListener::OnNewSegment (this=0x1bcf0d0, start_time=3403087200, duration=270000, segment_file_size=363131)
    at ../../packager/media/event/mpd_notify_muxer_listener.cc:157
#14 0x000000000043fae1 in edash_packager::media::mp4::MultiSegmentSegmenter::WriteSegment (this=0x7f58700024a0)
    at ../../packager/media/formats/mp4/multi_segment_segmenter.cc:189
#15 0x000000000043edcc in edash_packager::media::mp4::MultiSegmentSegmenter::DoFinalizeSegment (this=0x7f58700024a0)
    at ../../packager/media/formats/mp4/multi_segment_segmenter.cc:90
#16 0x0000000000445a5c in edash_packager::media::mp4::Segmenter::FinalizeSegment (this=0x7f58700024a0) at ../../packager/media/formats/mp4/segmenter.cc:355
#17 0x0000000000444e08 in edash_packager::media::mp4::Segmenter::FinalizeFragment (this=0x7f58700024a0, finalize_segment=true, fragmenter=0x7f5870043f50)
    at ../../packager/media/formats/mp4/segmenter.cc:422
#18 0x0000000000445503 in edash_packager::media::mp4::Segmenter::AddSample (this=0x7f58700024a0, stream=0x1bc2e70, sample=...)
    at ../../packager/media/formats/mp4/segmenter.cc:303
#19 0x000000000043096d in edash_packager::media::mp4::MP4Muxer::DoAddSample (this=0x1bd0740, stream=0x1bc2e70, sample=...)
    at ../../packager/media/formats/mp4/mp4_muxer.cc:133
#20 0x0000000000647217 in edash_packager::media::Muxer::AddSample (this=0x1bd0740, stream=0x1bc2e70, sample=...) at ../../packager/media/base/muxer.cc:110
#21 0x00000000006434b7 in edash_packager::media::MediaStream::PushSample (this=0x1bc2e70, sample=...) at ../../packager/media/base/media_stream.cc:48
#22 0x00000000006389e6 in edash_packager::media::Demuxer::NewSampleEvent (this=0x1b1d080, track_id=256, sample=...) at ../../packager/media/base/demuxer.cc:126
#23 0x000000000063b7b6 in base::internal::RunnableAdapter<bool (edash_packager::media::Demuxer::*)(unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>::Run (this=0x7f587bffd500, object=0x1b1d080, args=..., args=...) at ../../packager/base/bind_internal.h:176
#24 0x000000000063b6e9 in base::internal::InvokeHelper<false, bool, base::internal::RunnableAdapter<bool (edash_packager::media::Demuxer::*)(unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>, base::internal::TypeList<edash_packager::media::Demuxer*, unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&> >::MakeItSo (runnable=..., args=..., args=..., args=...) at ../../packager/base/bind_internal.h:286
#25 0x000000000063b682 in base::internal::Invoker<base::IndexSequence<0ul>, base::internal::BindState<base::internal::RunnableAdapter<bool (edash_packager::media::Demuxer::*)(unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>, bool (edash_packager::media::Demuxer*, unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&), base::internal::TypeList<base::internal::UnretainedWrapper<edash_packager::media::Demuxer> > >, base::internal::TypeList<base::internal::UnwrapTraits<base::internal::UnretainedWrapper<edash_packager::media::Demuxer> > >, base::internal::InvokeHelper<false, bool, base::internal::RunnableAdapter<bool (edash_packager::media::Demuxer::*)(unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>, base::internal::TypeList<edash_packager::media::Demuxer*, unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&> >, bool (unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&)>::Run(base::internal::BindStateBase*, unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&) (base=0x1b190f0, unbound_args=..., unbound_args=...) at ../../packager/base/bind_internal.h:343
#26 0x00000000009d113b in base::Callback<bool (unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>::Run(unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&) const (this=0x1b1b3d8, args=..., args=...) at ../../packager/base/callback.h:396
#27 0x00000000009ccf37 in edash_packager::media::mp2t::Mp2tMediaParser::EmitRemainingSamples (this=0x1b1b3c0) at ../../packager/media/formats/mp2t/mp2t_media_parser.cc:416
---Type <return> to continue, or q <return> to quit---
#28 0x00000000009cd9e5 in edash_packager::media::mp2t::Mp2tMediaParser::Parse (this=0x1b1b3c0, buf=0x7f5881390010 "GA\001=\001@", size=1472)
    at ../../packager/media/formats/mp2t/mp2t_media_parser.cc:250
#29 0x0000000000638f5e in edash_packager::media::Demuxer::Parse (this=0x1b1d080) at ../../packager/media/base/demuxer.cc:185
#30 0x0000000000639290 in edash_packager::media::Demuxer::Run (this=0x1b1d080) at ../../packager/media/base/demuxer.cc:146
#31 0x000000000040f844 in edash_packager::media::RemuxJob::Run (this=0x1bd5ad0) at ../../packager/app/packager_main.cc:108
#32 0x00000000006c3821 in base::SimpleThread::ThreadMain (this=0x1bd5ad0) at ../../packager/base/threading/simple_thread.cc:66
#33 0x00000000006c2c3a in base::(anonymous namespace)::ThreadFunc (params=0x7ffe3d367c30) at ../../packager/base/threading/platform_thread_posix.cc:74
#34 0x00007f588296d0a4 in start_thread (arg=0x7f587bfff700) at pthread_create.c:309
#35 0x00007f58826a204d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
kqyang commented 9 years ago

Cool, thanks for the detailed traces. The traces show that there is a dead lock in mpd generation.

Thread 2: waiting for SimpleMpdNotifier::Lock in NotifySampleDuration Thread 3: hold lock SimpleMpdNotifier::Lock and Representation::Lock waiting for AdaptationSet::Lock inside OnSetFrameRateForRepresentation Thread 4: hold lock AdaptationSet::Lock waiting for Representation::Lock inside GetXml

@rkuroiwa , can you take a look? Thanks.

peyoh commented 9 years ago

Hello, sorry for my late reply. Here also my traces. 

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x00007fc7fa7d0f27 in pthread_join () from /lib64/libpthread.so.0
Missing separate debuginfos, use: debuginfo-install glibc-2.17-78.el7.x86_64 libgcc-4.8.3-9.el7.x86_64 libstdc++-4.8.3-9.el7.x86_64
(gdb) where
#0  0x00007fc7fa7d0f27 in pthread_join () from /lib64/libpthread.so.0
#1  0x00000000006c2737 in base::PlatformThread::Join (thread_handle=...) at ../../packager/base/threading/platform_thread_posix.cc:221
#2  0x00000000006c375b in base::SimpleThread::Join (this=0x2f4f550) at ../../packager/base/threading/simple_thread.cc:47
#3  0x000000000040ceae in edash_packager::media::RunRemuxJobs (remux_jobs=std::__debug::vector of length 3, capacity 4 = {...}) at ../../packager/app/packager_main.cc:236
Python Exception <type 'exceptions.IndexError'> list index out of range: 
#4  0x000000000040d55f in edash_packager::media::RunPackager (stream_descriptors=std::__debug::multiset with 4 elements) at ../../packager/app/packager_main.cc:305
#5  0x000000000040d9ea in edash_packager::media::PackagerMain (argc=5, argv=0x7fff131ee2d8) at ../../packager/app/packager_main.cc:339
#6  0x000000000040da82 in main (argc=13, argv=0x7fff131ee298) at ../../packager/app/packager_main.cc:346
(gdb) thread apply all bt

Thread 7 (Thread 0x7fc7f8003700 (LWP 6437)):
#0  0x00007fc7fa7d3705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000006b17a5 in base::ConditionVariable::Wait (this=0x7fc7f8002068) at ../../packager/base/synchronization/condition_variable_posix.cc:54
#2  0x00000000006b319e in base::WaitableEvent::TimedWait (this=0x2f3cd68, max_time=...) at ../../packager/base/synchronization/waitable_event_posix.cc:214
#3  0x00000000006b2e06 in base::WaitableEvent::Wait (this=0x2f3cd68) at ../../packager/base/synchronization/waitable_event_posix.cc:154
#4  0x000000000042de4e in edash_packager::media::IoCache::Write (this=0x2f3cd30, buffer=0x7fc7f8004010, size=1316) at ../../packager/media/file/io_cache.cc:76
#5  0x0000000000429333 in edash_packager::media::ThreadedIoFile::RunInInputMode (this=0x2f3cd10) at ../../packager/media/file/threaded_io_file.cc:126
#6  0x000000000042b5f9 in base::internal::RunnableAdapter<void (edash_packager::media::ThreadedIoFile::*)()>::Run (this=0x7fc7f8002d90, object=0x2f3cd10) at ../../packager/base/bind_internal.h:176
#7  0x000000000042b53c in base::internal::InvokeHelper<false, void, base::internal::RunnableAdapter<void (edash_packager::media::ThreadedIoFile::*)()>, base::internal::TypeList<edash_packager::media::ThreadedIoFile*> >::MakeItSo (runnable=..., args=0x2f3cd10) at ../../packager/base/bind_internal.h:293
#8  0x000000000042b508 in base::internal::Invoker<base::IndexSequence<0ul>, base::internal::BindState<base::internal::RunnableAdapter<void (edash_packager::media::ThreadedIoFile::*)()>, void (edash_packager::media::ThreadedIoFile*), base::internal::TypeList<base::internal::UnretainedWrapper<edash_packager::media::ThreadedIoFile> > >, base::internal::TypeList<base::internal::UnwrapTraits<base::internal::UnretainedWrapper<edash_packager::media::ThreadedIoFile> > >, base::internal::InvokeHelper<false, void, base::internal::RunnableAdapter<void (edash_packager::media::ThreadedIoFile::*)()>, base::internal::TypeList<edash_packager::media::ThreadedIoFile*> >, void ()>::Run(base::internal::BindStateBase*) (base=0x2f3fe50) at ../../packager/base/bind_internal.h:343
#9  0x0000000000637c3e in base::Callback<void ()>::Run() const (this=0x2f3ca98) at ../../packager/base/callback.h:396
#10 0x0000000000637bd9 in edash_packager::media::ClosureThread::Run (this=0x2f3ca50) at ../../packager/media/base/closure_thread.cc:22
#11 0x00000000006c3821 in base::SimpleThread::ThreadMain (this=0x2f3ca50) at ../../packager/base/threading/simple_thread.cc:66
#12 0x00000000006c2c3a in base::(anonymous namespace)::ThreadFunc (params=0x7fff131ebb60) at ../../packager/base/threading/platform_thread_posix.cc:74
#13 0x00007fc7fa7cfdf5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007fc7fa4fd1ad in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fc7f53ff700 (LWP 6459)):
#0  0x00007fc7fa7d3705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000006b17a5 in base::ConditionVariable::Wait (this=0x7fc7f53fe068) at ../../packager/base/synchronization/condition_variable_posix.cc:54
#2  0x00000000006b319e in base::WaitableEvent::TimedWait (this=0x2f7dcf8, max_time=...) at ../../packager/base/synchronization/waitable_event_posix.cc:214
#3  0x00000000006b2e06 in base::WaitableEvent::Wait (this=0x2f7dcf8) at ../../packager/base/synchronization/waitable_event_posix.cc:154
#4  0x000000000042de4e in edash_packager::media::IoCache::Write (this=0x2f7dcc0, buffer=0x7fc7f5400010, size=1316) at ../../packager/media/file/io_cache.cc:76
#5  0x0000000000429333 in edash_packager::media::ThreadedIoFile::RunInInputMode (this=0x2f7dca0) at ../../packager/media/file/threaded_io_file.cc:126
#6  0x000000000042b5f9 in base::internal::RunnableAdapter<void (edash_packager::media::ThreadedIoFile::*)()>::Run (this=0x7fc7f53fed90, object=0x2f7dca0) at ../../packager/base/bind_internal.h:176
#7  0x000000000042b53c in base::internal::InvokeHelper<false, void, base::internal::RunnableAdapter<void (edash_packager::media::ThreadedIoFile::*)()>, base::internal::TypeList<edash_packager::media::ThreadedIoFile*> >::MakeItSo (runnable=..., args=0x2f7dca0) at ../../packager/base/bind_internal.h:293
#8  0x000000000042b508 in base::internal::Invoker<base::IndexSequence<0ul>, base::internal::BindState<base::internal::RunnableAdapter<void (edash_packager::media::ThreadedIoFile::*)()>, void (edash_packager::media::ThreadedIoFile*), base::internal::TypeList<base::internal::UnretainedWrapper<edash_packager::media::ThreadedIoFile> > >, base::internal::TypeList<base::internal::UnwrapTraits<base::internal::UnretainedWrapper<edash_packager::media::ThreadedIoFile> > >, base::internal::InvokeHelper<false, void, base::internal::RunnableAdapter<void (edash_packager::media::ThreadedIoFile::*)()>, base::internal::TypeList<edash_packager::media::ThreadedIoFile*> >, void ()>::Run(base::internal::BindStateBase*) (base=0x2f3de80) at ../../packager/base/bind_internal.h:343
#9  0x0000000000637c3e in base::Callback<void ()>::Run() const (this=0x2f48028) at ../../packager/base/callback.h:396
#10 0x0000000000637bd9 in edash_packager::media::ClosureThread::Run (this=0x2f47fe0) at ../../packager/media/base/closure_thread.cc:22
#11 0x00000000006c3821 in base::SimpleThread::ThreadMain (this=0x2f47fe0) at ../../packager/base/threading/simple_thread.cc:66
#12 0x00000000006c2c3a in base::(anonymous namespace)::ThreadFunc (params=0x7fff131ebb60) at ../../packager/base/threading/platform_thread_posix.cc:74
#13 0x00007fc7fa7cfdf5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007fc7fa4fd1ad in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fc7edffe700 (LWP 6465)):
#0  0x00007fc7fa7d3705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000006b17a5 in base::ConditionVariable::Wait (this=0x7fc7edffd068) at ../../packager/base/synchronization/condition_variable_posix.cc:54
#2  0x00000000006b319e in base::WaitableEvent::TimedWait (this=0x2fd46f8, max_time=...) at ../../packager/base/synchronization/waitable_event_posix.cc:214
#3  0x00000000006b2e06 in base::WaitableEvent::Wait (this=0x2fd46f8) at ../../packager/base/synchronization/waitable_event_posix.cc:154
#4  0x000000000042de4e in edash_packager::media::IoCache::Write (this=0x2fd46c0, buffer=0x7fc7f47fd010, size=1316) at ../../packager/media/file/io_cache.cc:76
#5  0x0000000000429333 in edash_packager::media::ThreadedIoFile::RunInInputMode (this=0x2fd46a0) at ../../packager/media/file/threaded_io_file.cc:126
#6  0x000000000042b5f9 in base::internal::RunnableAdapter<void (edash_packager::media::ThreadedIoFile::*)()>::Run (this=0x7fc7edffdd90, object=0x2fd46a0) at ../../packager/base/bind_internal.h:176
#7  0x000000000042b53c in base::internal::InvokeHelper<false, void, base::internal::RunnableAdapter<void (edash_packager::media::ThreadedIoFile::*)()>, base::internal::TypeList<edash_packager::media::ThreadedIoFile*> >::MakeItSo (runnable=..., args=0x2fd46a0) at ../../packager/base/bind_internal.h:293
#8  0x000000000042b508 in base::internal::Invoker<base::IndexSequence<0ul>, base::internal::BindState<base::internal::RunnableAdapter<void (edash_packager::media::ThreadedIoFile::*)()>, void (edash_packager::media::ThreadedIoFile*), base::internal::TypeList<base::internal::UnretainedWrapper<edash_packager::media::ThreadedIoFile> > >, base::internal::TypeList<base::internal::UnwrapTraits<base::internal::UnretainedWrapper<edash_packager::media::ThreadedIoFile> > >, base::internal::InvokeHelper<false, void, base::internal::RunnableAdapter<void (edash_packager::media::ThreadedIoFile::*)()>, base::internal::TypeList<edash_packager::m---Type <return> to continue, or q <return> to quit--- 
edia::ThreadedIoFile*> >, void ()>::Run(base::internal::BindStateBase*) (base=0x2fd4910) at ../../packager/base/bind_internal.h:343
#9  0x0000000000637c3e in base::Callback<void ()>::Run() const (this=0x2fd48f8) at ../../packager/base/callback.h:396
#10 0x0000000000637bd9 in edash_packager::media::ClosureThread::Run (this=0x2fd48b0) at ../../packager/media/base/closure_thread.cc:22
#11 0x00000000006c3821 in base::SimpleThread::ThreadMain (this=0x2fd48b0) at ../../packager/base/threading/simple_thread.cc:66
#12 0x00000000006c2c3a in base::(anonymous namespace)::ThreadFunc (params=0x7fff131ebb60) at ../../packager/base/threading/platform_thread_posix.cc:74
#13 0x00007fc7fa7cfdf5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007fc7fa4fd1ad in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fc7ed7fd700 (LWP 6497)):
#0  0x00007fc7fa7d5f7d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007fc7fa7d1d32 in _L_lock_791 () from /lib64/libpthread.so.0
#2  0x00007fc7fa7d1c38 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00000000006b27fb in base::internal::LockImpl::Lock (this=0x2f3cbb8) at ../../packager/base/synchronization/lock_impl_posix.cc:45
#4  0x000000000040bd43 in base::Lock::Acquire (this=0x2f3cbb0) at ../../packager/base/synchronization/lock.h:42
#5  0x000000000042e5d3 in base::AutoLock::AutoLock (this=0x7fc7ed7f9268, lock=...) at ../../packager/base/synchronization/lock.h:99
#6  0x00000000004d47cd in edash_packager::SimpleMpdNotifier::NotifySampleDuration (this=0x2f3cb90, container_id=3, sample_duration=3600) at ../../packager/mpd/base/simple_mpd_notifier.cc:76
#7  0x000000000041df43 in edash_packager::media::MpdNotifyMuxerListener::OnSampleDurationReady (this=0x2f4e350, sample_duration=3600) at ../../packager/media/event/mpd_notify_muxer_listener.cc:91
#8  0x000000000043fa8f in edash_packager::media::mp4::MultiSegmentSegmenter::WriteSegment (this=0x7fc7dc434130) at ../../packager/media/formats/mp4/multi_segment_segmenter.cc:188
#9  0x000000000043edcc in edash_packager::media::mp4::MultiSegmentSegmenter::DoFinalizeSegment (this=0x7fc7dc434130) at ../../packager/media/formats/mp4/multi_segment_segmenter.cc:90
#10 0x0000000000445a5c in edash_packager::media::mp4::Segmenter::FinalizeSegment (this=0x7fc7dc434130) at ../../packager/media/formats/mp4/segmenter.cc:355
#11 0x0000000000444e08 in edash_packager::media::mp4::Segmenter::FinalizeFragment (this=0x7fc7dc434130, finalize_segment=true, fragmenter=0x7fc7dc2adfc0) at ../../packager/media/formats/mp4/segmenter.cc:422
#12 0x0000000000445503 in edash_packager::media::mp4::Segmenter::AddSample (this=0x7fc7dc434130, stream=0x2f54b80, sample=...) at ../../packager/media/formats/mp4/segmenter.cc:303
#13 0x000000000043096d in edash_packager::media::mp4::MP4Muxer::DoAddSample (this=0x2f4b0b0, stream=0x2f54b80, sample=...) at ../../packager/media/formats/mp4/mp4_muxer.cc:133
#14 0x0000000000647217 in edash_packager::media::Muxer::AddSample (this=0x2f4b0b0, stream=0x2f54b80, sample=...) at ../../packager/media/base/muxer.cc:110
#15 0x00000000006434b7 in edash_packager::media::MediaStream::PushSample (this=0x2f54b80, sample=...) at ../../packager/media/base/media_stream.cc:48
#16 0x00000000006389e6 in edash_packager::media::Demuxer::NewSampleEvent (this=0x2f3cc80, track_id=256, sample=...) at ../../packager/media/base/demuxer.cc:126
#17 0x000000000063b7b6 in base::internal::RunnableAdapter<bool (edash_packager::media::Demuxer::*)(unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>::Run (this=0x7fc7ed7fb500, 
    object=0x2f3cc80, args=..., args=...) at ../../packager/base/bind_internal.h:176
#18 0x000000000063b6e9 in base::internal::InvokeHelper<false, bool, base::internal::RunnableAdapter<bool (edash_packager::media::Demuxer::*)(unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>, base::internal::TypeList<edash_packager::media::Demuxer*, unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&> >::MakeItSo (runnable=..., args=..., args=..., args=...)
    at ../../packager/base/bind_internal.h:286
#19 0x000000000063b682 in base::internal::Invoker<base::IndexSequence<0ul>, base::internal::BindState<base::internal::RunnableAdapter<bool (edash_packager::media::Demuxer::*)(unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>, bool (edash_packager::media::Demuxer*, unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&), base::internal::TypeList<base::internal::UnretainedWrapper<edash_packager::media::Demuxer> > >, base::internal::TypeList<base::internal::UnwrapTraits<base::internal::UnretainedWrapper<edash_packager::media::Demuxer> > >, base::internal::InvokeHelper<false, bool, base::internal::RunnableAdapter<bool (edash_packager::media::Demuxer::*)(unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>, base::internal::TypeList<edash_packager::media::Demuxer*, unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&> >, bool (unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&)>::Run(base::internal::BindStateBase*, unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&) (base=0x2f3ceb0, unbound_args=..., unbound_args=...) at ../../packager/base/bind_internal.h:343
#20 0x00000000009d113b in base::Callback<bool (unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>::Run(unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&) const (
    this=0x2f3e758, args=..., args=...) at ../../packager/base/callback.h:396
#21 0x00000000009ccf37 in edash_packager::media::mp2t::Mp2tMediaParser::EmitRemainingSamples (this=0x2f3e740) at ../../packager/media/formats/mp2t/mp2t_media_parser.cc:416
#22 0x00000000009cd9e5 in edash_packager::media::mp2t::Mp2tMediaParser::Parse (this=0x2f3e740, buf=0x7fc7fa206010 "GA", size=13160) at ../../packager/media/formats/mp2t/mp2t_media_parser.cc:250
#23 0x0000000000638f5e in edash_packager::media::Demuxer::Parse (this=0x2f3cc80) at ../../packager/media/base/demuxer.cc:185
#24 0x0000000000639290 in edash_packager::media::Demuxer::Run (this=0x2f3cc80) at ../../packager/media/base/demuxer.cc:146
#25 0x000000000040f844 in edash_packager::media::RemuxJob::Run (this=0x2f4f550) at ../../packager/app/packager_main.cc:108
#26 0x00000000006c3821 in base::SimpleThread::ThreadMain (this=0x2f4f550) at ../../packager/base/threading/simple_thread.cc:66
#27 0x00000000006c2c3a in base::(anonymous namespace)::ThreadFunc (params=0x7fff131eceb0) at ../../packager/base/threading/platform_thread_posix.cc:74
#28 0x00007fc7fa7cfdf5 in start_thread () from /lib64/libpthread.so.0
#29 0x00007fc7fa4fd1ad in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fc7ecffc700 (LWP 6498)):
#0  0x00007fc7fa7d5f7d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007fc7fa7d1d32 in _L_lock_791 () from /lib64/libpthread.so.0
#2  0x00007fc7fa7d1c38 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00000000006b27fb in base::internal::LockImpl::Lock (this=0x7fc7d4003820) at ../../packager/base/synchronization/lock_impl_posix.cc:45
#4  0x000000000040bd43 in base::Lock::Acquire (this=0x7fc7d4003818) at ../../packager/base/synchronization/lock.h:42
#5  0x000000000042e5d3 in base::AutoLock::AutoLock (this=0x7fc7ecff6b78, lock=...) at ../../packager/base/synchronization/lock.h:99
---Type <return> to continue, or q <return> to quit---
#6  0x00000000004b0ed7 in edash_packager::Representation::GetXml (this=0x7fc7d4003730) at ../../packager/mpd/base/mpd_builder.cc:1096
#7  0x00000000004acf5f in edash_packager::AdaptationSet::GetXml (this=0x7fc7d40031a0) at ../../packager/mpd/base/mpd_builder.cc:731
#8  0x00000000004ac718 in edash_packager::MpdBuilder::GenerateMpd (this=0x2f3c740) at ../../packager/mpd/base/mpd_builder.cc:448
#9  0x00000000004b5225 in edash_packager::MpdBuilder::WriteMpdToOutput<std::string> (this=0x2f3c740, output=0x7fc7ecff7f18) at ../../packager/mpd/base/mpd_builder.cc:419
#10 0x00000000004ac5d9 in edash_packager::MpdBuilder::ToString (this=0x2f3c740, output=0x7fc7ecff7f18) at ../../packager/mpd/base/mpd_builder.cc:413
#11 0x00000000004cb4e1 in edash_packager::WriteMpdToFile (output_path="/storage/web/dash/bnt1/bnt1.mpd", mpd_builder=0x2f3c740) at ../../packager/mpd/base/mpd_notifier_util.cc:24
#12 0x00000000004d5090 in edash_packager::SimpleMpdNotifier::Flush (this=0x2f3cb90) at ../../packager/mpd/base/simple_mpd_notifier.cc:130
#13 0x000000000041e6db in edash_packager::media::MpdNotifyMuxerListener::OnNewSegment (this=0x2f705c0, start_time=2409012000, duration=900000, segment_file_size=1248189)
    at ../../packager/media/event/mpd_notify_muxer_listener.cc:157
#14 0x000000000043fae1 in edash_packager::media::mp4::MultiSegmentSegmenter::WriteSegment (this=0x7fc7e01407b0) at ../../packager/media/formats/mp4/multi_segment_segmenter.cc:189
#15 0x000000000043edcc in edash_packager::media::mp4::MultiSegmentSegmenter::DoFinalizeSegment (this=0x7fc7e01407b0) at ../../packager/media/formats/mp4/multi_segment_segmenter.cc:90
#16 0x0000000000445a5c in edash_packager::media::mp4::Segmenter::FinalizeSegment (this=0x7fc7e01407b0) at ../../packager/media/formats/mp4/segmenter.cc:355
#17 0x0000000000444e08 in edash_packager::media::mp4::Segmenter::FinalizeFragment (this=0x7fc7e01407b0, finalize_segment=true, fragmenter=0x7fc7e0031620) at ../../packager/media/formats/mp4/segmenter.cc:422
#18 0x0000000000445503 in edash_packager::media::mp4::Segmenter::AddSample (this=0x7fc7e01407b0, stream=0x2f7c3f0, sample=...) at ../../packager/media/formats/mp4/segmenter.cc:303
#19 0x000000000043096d in edash_packager::media::mp4::MP4Muxer::DoAddSample (this=0x2f77fa0, stream=0x2f7c3f0, sample=...) at ../../packager/media/formats/mp4/mp4_muxer.cc:133
#20 0x0000000000647217 in edash_packager::media::Muxer::AddSample (this=0x2f77fa0, stream=0x2f7c3f0, sample=...) at ../../packager/media/base/muxer.cc:110
#21 0x00000000006434b7 in edash_packager::media::MediaStream::PushSample (this=0x2f7c3f0, sample=...) at ../../packager/media/base/media_stream.cc:48
#22 0x00000000006389e6 in edash_packager::media::Demuxer::NewSampleEvent (this=0x2f4c1b0, track_id=256, sample=...) at ../../packager/media/base/demuxer.cc:126
#23 0x000000000063b7b6 in base::internal::RunnableAdapter<bool (edash_packager::media::Demuxer::*)(unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>::Run (this=0x7fc7ecffa500, 
    object=0x2f4c1b0, args=..., args=...) at ../../packager/base/bind_internal.h:176
#24 0x000000000063b6e9 in base::internal::InvokeHelper<false, bool, base::internal::RunnableAdapter<bool (edash_packager::media::Demuxer::*)(unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>, base::internal::TypeList<edash_packager::media::Demuxer*, unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&> >::MakeItSo (runnable=..., args=..., args=..., args=...)
    at ../../packager/base/bind_internal.h:286
#25 0x000000000063b682 in base::internal::Invoker<base::IndexSequence<0ul>, base::internal::BindState<base::internal::RunnableAdapter<bool (edash_packager::media::Demuxer::*)(unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>, bool (edash_packager::media::Demuxer*, unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&), base::internal::TypeList<base::internal::UnretainedWrapper<edash_packager::media::Demuxer> > >, base::internal::TypeList<base::internal::UnwrapTraits<base::internal::UnretainedWrapper<edash_packager::media::Demuxer> > >, base::internal::InvokeHelper<false, bool, base::internal::RunnableAdapter<bool (edash_packager::media::Demuxer::*)(unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>, base::internal::TypeList<edash_packager::media::Demuxer*, unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&> >, bool (unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&)>::Run(base::internal::BindStateBase*, unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&) (base=0x2f7d570, unbound_args=..., unbound_args=...) at ../../packager/base/bind_internal.h:343
#26 0x00000000009d113b in base::Callback<bool (unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>::Run(unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&) const (
    this=0x2f4e728, args=..., args=...) at ../../packager/base/callback.h:396
#27 0x00000000009ccf37 in edash_packager::media::mp2t::Mp2tMediaParser::EmitRemainingSamples (this=0x2f4e710) at ../../packager/media/formats/mp2t/mp2t_media_parser.cc:416
#28 0x00000000009cd9e5 in edash_packager::media::mp2t::Mp2tMediaParser::Parse (this=0x2f4e710, buf=0x7fc7f7602010 "G\001", size=2632) at ../../packager/media/formats/mp2t/mp2t_media_parser.cc:250
#29 0x0000000000638f5e in edash_packager::media::Demuxer::Parse (this=0x2f4c1b0) at ../../packager/media/base/demuxer.cc:185
#30 0x0000000000639290 in edash_packager::media::Demuxer::Run (this=0x2f4c1b0) at ../../packager/media/base/demuxer.cc:146
#31 0x000000000040f844 in edash_packager::media::RemuxJob::Run (this=0x2f74bf0) at ../../packager/app/packager_main.cc:108
#32 0x00000000006c3821 in base::SimpleThread::ThreadMain (this=0x2f74bf0) at ../../packager/base/threading/simple_thread.cc:66
#33 0x00000000006c2c3a in base::(anonymous namespace)::ThreadFunc (params=0x7fff131eceb0) at ../../packager/base/threading/platform_thread_posix.cc:74
#34 0x00007fc7fa7cfdf5 in start_thread () from /lib64/libpthread.so.0
#35 0x00007fc7fa4fd1ad in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fc7dbfff700 (LWP 6499)):
#0  0x00007fc7fa7d5f7d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007fc7fa7d1d32 in _L_lock_791 () from /lib64/libpthread.so.0
#2  0x00007fc7fa7d1c38 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00000000006b27fb in base::internal::LockImpl::Lock (this=0x7fc7d4003208) at ../../packager/base/synchronization/lock_impl_posix.cc:45
#4  0x000000000040bd43 in base::Lock::Acquire (this=0x7fc7d4003200) at ../../packager/base/synchronization/lock.h:42
#5  0x000000000042e5d3 in base::AutoLock::AutoLock (this=0x7fc7dbffaf18, lock=...) at ../../packager/base/synchronization/lock.h:99
#6  0x00000000004b1b35 in edash_packager::AdaptationSet::OnNewSegmentForRepresentation (this=0x7fc7d40031a0, representation_id=1, start_time=2409012000, duration=900000)
    at ../../packager/mpd/base/mpd_builder.cc:807
#7  0x00000000004b449c in edash_packager::(anonymous namespace)::RepresentationStateChangeListenerImpl::OnNewSegmentForRepresentation (this=0x7fc7d4003690, start_time=2409012000, duration=900000)
    at ../../packager/mpd/base/mpd_builder.cc:361
#8  0x00000000004b2be1 in edash_packager::Representation::AddNewSegment (this=0x7fc7d4003730, start_time=2409012000, duration=900000, size=1238756) at ../../packager/mpd/base/mpd_builder.cc:1062
#9  0x00000000004d4b61 in edash_packager::SimpleMpdNotifier::NotifyNewSegment (this=0x2f3cb90, container_id=1, start_time=2409012000, duration=900000, size=1238756)
    at ../../packager/mpd/base/simple_mpd_notifier.cc:96
#10 0x000000000041e6bf in edash_packager::media::MpdNotifyMuxerListener::OnNewSegment (this=0x2f63fb0, start_time=2409012000, duration=900000, segment_file_size=1238756)
---Type <return> to continue, or q <return> to quit---
    at ../../packager/media/event/mpd_notify_muxer_listener.cc:155
#11 0x000000000043fae1 in edash_packager::media::mp4::MultiSegmentSegmenter::WriteSegment (this=0x7fc7d40029f0) at ../../packager/media/formats/mp4/multi_segment_segmenter.cc:189
#12 0x000000000043edcc in edash_packager::media::mp4::MultiSegmentSegmenter::DoFinalizeSegment (this=0x7fc7d40029f0) at ../../packager/media/formats/mp4/multi_segment_segmenter.cc:90
#13 0x0000000000445a5c in edash_packager::media::mp4::Segmenter::FinalizeSegment (this=0x7fc7d40029f0) at ../../packager/media/formats/mp4/segmenter.cc:355
#14 0x0000000000444e08 in edash_packager::media::mp4::Segmenter::FinalizeFragment (this=0x7fc7d40029f0, finalize_segment=true, fragmenter=0x7fc7d4002f10) at ../../packager/media/formats/mp4/segmenter.cc:422
#15 0x0000000000445503 in edash_packager::media::mp4::Segmenter::AddSample (this=0x7fc7d40029f0, stream=0x2f63df0, sample=...) at ../../packager/media/formats/mp4/segmenter.cc:303
#16 0x000000000043096d in edash_packager::media::mp4::MP4Muxer::DoAddSample (this=0x2f63ed0, stream=0x2f63df0, sample=...) at ../../packager/media/formats/mp4/mp4_muxer.cc:133
#17 0x0000000000647217 in edash_packager::media::Muxer::AddSample (this=0x2f63ed0, stream=0x2f63df0, sample=...) at ../../packager/media/base/muxer.cc:110
#18 0x00000000006434b7 in edash_packager::media::MediaStream::PushSample (this=0x2f63df0, sample=...) at ../../packager/media/base/media_stream.cc:48
#19 0x00000000006389e6 in edash_packager::media::Demuxer::NewSampleEvent (this=0x2fd4610, track_id=256, sample=...) at ../../packager/media/base/demuxer.cc:126
#20 0x000000000063b7b6 in base::internal::RunnableAdapter<bool (edash_packager::media::Demuxer::*)(unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>::Run (this=0x7fc7dbffd500, 
    object=0x2fd4610, args=..., args=...) at ../../packager/base/bind_internal.h:176
#21 0x000000000063b6e9 in base::internal::InvokeHelper<false, bool, base::internal::RunnableAdapter<bool (edash_packager::media::Demuxer::*)(unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>, base::internal::TypeList<edash_packager::media::Demuxer*, unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&> >::MakeItSo (runnable=..., args=..., args=..., args=...)
    at ../../packager/base/bind_internal.h:286
#22 0x000000000063b682 in base::internal::Invoker<base::IndexSequence<0ul>, base::internal::BindState<base::internal::RunnableAdapter<bool (edash_packager::media::Demuxer::*)(unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>, bool (edash_packager::media::Demuxer*, unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&), base::internal::TypeList<base::internal::UnretainedWrapper<edash_packager::media::Demuxer> > >, base::internal::TypeList<base::internal::UnwrapTraits<base::internal::UnretainedWrapper<edash_packager::media::Demuxer> > >, base::internal::InvokeHelper<false, bool, base::internal::RunnableAdapter<bool (edash_packager::media::Demuxer::*)(unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>, base::internal::TypeList<edash_packager::media::Demuxer*, unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&> >, bool (unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&)>::Run(base::internal::BindStateBase*, unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&) (base=0x2fd49c0, unbound_args=..., unbound_args=...) at ../../packager/base/bind_internal.h:343
#23 0x00000000009d113b in base::Callback<bool (unsigned int, scoped_refptr<edash_packager::media::MediaSample> const&)>::Run(unsigned int const&, scoped_refptr<edash_packager::media::MediaSample> const&) const (
    this=0x2fd4b98, args=..., args=...) at ../../packager/base/callback.h:396
#24 0x00000000009ccf37 in edash_packager::media::mp2t::Mp2tMediaParser::EmitRemainingSamples (this=0x2fd4b80) at ../../packager/media/formats/mp2t/mp2t_media_parser.cc:416
#25 0x00000000009cd9e5 in edash_packager::media::mp2t::Mp2tMediaParser::Parse (this=0x2fd4b80, buf=0x7fc7f49fe010 "G\001", size=6580) at ../../packager/media/formats/mp2t/mp2t_media_parser.cc:250
#26 0x0000000000638f5e in edash_packager::media::Demuxer::Parse (this=0x2fd4610) at ../../packager/media/base/demuxer.cc:185
#27 0x0000000000639290 in edash_packager::media::Demuxer::Run (this=0x2fd4610) at ../../packager/media/base/demuxer.cc:146
#28 0x000000000040f844 in edash_packager::media::RemuxJob::Run (this=0x2f644f0) at ../../packager/app/packager_main.cc:108
#29 0x00000000006c3821 in base::SimpleThread::ThreadMain (this=0x2f644f0) at ../../packager/base/threading/simple_thread.cc:66
#30 0x00000000006c2c3a in base::(anonymous namespace)::ThreadFunc (params=0x7fff131eceb0) at ../../packager/base/threading/platform_thread_posix.cc:74
#31 0x00007fc7fa7cfdf5 in start_thread () from /lib64/libpthread.so.0
#32 0x00007fc7fa4fd1ad in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fc7fb81f740 (LWP 6436)):
#0  0x00007fc7fa7d0f27 in pthread_join () from /lib64/libpthread.so.0
#1  0x00000000006c2737 in base::PlatformThread::Join (thread_handle=...) at ../../packager/base/threading/platform_thread_posix.cc:221
#2  0x00000000006c375b in base::SimpleThread::Join (this=0x2f4f550) at ../../packager/base/threading/simple_thread.cc:47
#3  0x000000000040ceae in edash_packager::media::RunRemuxJobs (remux_jobs=std::__debug::vector of length 3, capacity 4 = {...}) at ../../packager/app/packager_main.cc:236
Python Exception <type 'exceptions.IndexError'> list index out of range: 
#4  0x000000000040d55f in edash_packager::media::RunPackager (stream_descriptors=std::__debug::multiset with 4 elements) at ../../packager/app/packager_main.cc:305
#5  0x000000000040d9ea in edash_packager::media::PackagerMain (argc=5, argv=0x7fff131ee2d8) at ../../packager/app/packager_main.cc:339
#6  0x000000000040da82 in main (argc=13, argv=0x7fff131ee298) at ../../packager/app/packager_main.cc:346
kqyang commented 9 years ago

This bug should have been fixed.

@peyoh @dviano Can you try and verify with latest cl submitted by Rintaro? Let us know the results. Thanks again for reporting the issue and helps on debugging.

dviano commented 9 years ago

It's been running fine for 3 hours now (longer than the average before), I'll leave it running for 24hs and confirm again tomorrow.

dviano commented 9 years ago

I can confirm now, that after 24hs it keeps running ok.

kqyang commented 9 years ago

Cool. Thanks!

peyoh commented 9 years ago

Hi, it's ok now.

Thank you.

kqyang commented 9 years ago

Cool. Thanks for the update.