search

flavioribeiro / nginx-audio-track-for-hls-module

:sound: Nginx module that generates audio track for HTTP Live Streaming (HLS) streams on the fly.
GNU General Public License v3.0
136 stars 17 forks source link

nginx crash with many requests #25

Open jalonsoa opened 9 years ago

jalonsoa commented 9 years ago

Hi the nginx crash when many clients request audio files. I think that this is a problem about thread safe.

Are av libraries (avformat, avcodec and avutil) thread safe ?

the bt on nginx crash:

(gdb) bt
#0  0x00007fdd8379cad0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fdd8379d839 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007fdd854bb25c in av_freep () from /usr/lib/x86_64-linux-gnu/libavutil.so.51
#3  0x00007fdd8659f606 in avformat_free_context () from /usr/lib/x86_64-linux-gnu/libavformat.so.53
#4  0x00007fdd8659f811 in avformat_close_input () from /usr/lib/x86_64-linux-gnu/libavformat.so.53
#5  0x00000000004dd128 in ngx_http_aac_extract_audio (destination=0x1f53da8, outputfmt=..., log=0x1e2c820, pool=<optimized out>, source=...)
    at src/modules/nginx-audio-track-for-hls-module/ngx_http_aac_module.c:236
#6  ngx_http_aac_handler (r=0x1f52e20) at src/modules/nginx-audio-track-for-hls-module/ngx_http_aac_module.c:31
#7  0x000000000044db52 in ngx_http_core_content_phase (r=0x1f52e20, ph=0x1e5b010) at src/http/ngx_http_core_module.c:1407
#8  0x0000000000448325 in ngx_http_core_run_phases (r=0x1f52e20) at src/http/ngx_http_core_module.c:888
#9  0x0000000000454451 in ngx_http_process_request (r=0x1f52e20) at src/http/ngx_http_request.c:1902
#10 0x00000000004553a6 in ngx_http_process_request_line (rev=0x1e86250) at src/http/ngx_http_request.c:1012
#11 0x000000000043d731 in ngx_epoll_process_events (cycle=0x1e1e280, timer=<optimized out>, flags=<optimized out>) at src/event/modules/ngx_epoll_module.c:691
#12 0x00000000004340d7 in ngx_process_events_and_timers (cycle=0x1e1e280) at src/event/ngx_event.c:248
#13 0x000000000043b5c9 in ngx_worker_process_cycle (cycle=0x1e1e280, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:822
#14 0x0000000000439f35 in ngx_spawn_process (cycle=0x1e1e280, proc=0x43b510 <ngx_worker_process_cycle>, data=0x0, name=0x516bde "worker process", respawn=0) at src/os/unix/ngx_process.c:198
#15 0x000000000043c9c2 in ngx_reap_children (cycle=0x1e1e280) at src/os/unix/ngx_process_cycle.c:631
#16 ngx_master_process_cycle (cycle=0x1e1e280) at src/os/unix/ngx_process_cycle.c:184
#17 0x0000000000419c0d in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:407
flavioribeiro commented 9 years ago

did you saw these errors before the pull request?

flavioribeiro commented 9 years ago

other question, are you caching the results of this location? what do you mean by many requests?

jalonsoa commented 9 years ago

Hi Flavio, yes, I'm using the module with the new features.

I have this nginx server behind varnish. But we have 5 varnish farms (diferent cdn providers). When we have more of 3 simultaneous request to the same audio file, the nginx crash.

I think that my installation of av libs don't have thread support (seems Ubuntu forgot add the --threads flag when compiled).

I'm trying with mutex before and after free context ... I'll tell you how works.

If this not work, I will try with av_lockmgr_register (http://ffmpeg.org/doxygen/trunk/group__lavc__misc.html#gae120ec60cdf49cf0f793eff3ec731bcc) on module init

j

jalonsoa commented 9 years ago

Hi Flavio, all work fine after put mutex around avformat_free_context and avformat_close_input.

But, I've been thinking about this, and I think that the best solution is create one init function for the module and move the av_register_all and av_log_set_level. And, in this module init function add the av_lockmgr_register.

What think about this ?

thanks j

flavioribeiro commented 9 years ago

hi @jalonsoa, sounds good to me. I'm still surprised that I hadn't face this problem here. Have you tried compiling your libav with threads?

Just curious, can you tell me how many RPM nginx can serve with this mutex?

jalonsoa commented 9 years ago

Hi Flavio, sounds like my libav have threads ... (at least, is linked with the library). Maybe the mpegts output format don't work fine, I don't known.

On this server, all request = 1751 RPM, only audio.ts related, 115 RPM

j

On Fri, Jan 16, 2015 at 1:26 PM, Flávio Ribeiro notifications@github.com wrote:

hi @jalonsoa https://github.com/jalonsoa, sounds good to me. I'm still surprised that I hadn't face this problem here. Have you tried compiling your libav with threads?

Just curious, can you tell me how many RPM nginx can serve with this mutex?

— Reply to this email directly or view it on GitHub https://github.com/flavioribeiro/nginx-audio-track-for-hls-module/issues/25#issuecomment-70247050 .

flavioribeiro commented 9 years ago

It's ok for me @jalonsoa, can you make another pull request?

flavioribeiro commented 9 years ago

add yourself on AUTHORS file also :beers:

jalonsoa commented 9 years ago

Hi Flavio, I'm trying the changes that I tell you, but don't work. I created the init function put the initialitations function into it, but the nginx crash into libav function (different functions - av_write_trailer, avformat_close_input, av_interleaved_write_frame).

I'm trying to update my libav libs to ensure support of threads or solve bugs in libav libs. I will tell you about progress on this way.

j

flavioribeiro commented 9 years ago

great, thank you for making me updated.

flavioribeiro commented 9 years ago

hey @jalonsoa any updates on this?

jalonsoa commented 9 years ago

Hi Flavio, sorry. I'm bussy this week .. ;-( .. yesterday compile new libav for my machine (this time I'm sure that have threads support), and recompile nginx against the new lib. Bad news ... nginx crash again.

Any idea ?

(gdb) bt

0 0x00007ffc914b83c4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6

1 0x00007ffc9379d650 in av_parser_close () from

/usr/lib/x86_64-linux-gnu/libavcodec.so.56

2 0x00007ffc944a0d24 in ?? () from

/usr/lib/x86_64-linux-gnu/libavformat.so.56

3 0x00007ffc944a1137 in ?? () from

/usr/lib/x86_64-linux-gnu/libavformat.so.56

4 0x00000000004dd0c4 in ngx_http_aac_extract_audio

(destination=0x1916b48, outputfmt=..., log=0x18fdc80, pool=, source=...) at /html/src/nginx/nginx-1.7.4/debian/modules/nginx-audio-track-for-hls-module/ngx_http_aac_module.c:245

5 ngx_http_aac_handler (r=0x1915bd0) at

/html/src/nginx/nginx-1.7.4/debian/modules/nginx-audio-track-for-hls-module/ngx_http_aac_module.c:31

6 0x000000000044db52 in ngx_http_core_content_phase (r=0x1915bd0,

ph=0x189bdb0) at src/http/ngx_http_core_module.c:1407

7 0x0000000000448325 in ngx_http_core_run_phases (r=0x1915bd0) at

src/http/ngx_http_core_module.c:888

8 0x0000000000454451 in ngx_http_process_request (r=0x1915bd0) at

src/http/ngx_http_request.c:1902

9 0x00000000004553a6 in ngx_http_process_request_line (rev=0x18c8d18) at

src/http/ngx_http_request.c:1012

10 0x000000000043d731 in ngx_epoll_process_events (cycle=0x185f060,

timer=, flags=) at src/event/modules/ngx_epoll_module.c:691

11 0x00000000004340d7 in ngx_process_events_and_timers (cycle=0x185f060)

at src/event/ngx_event.c:248

12 0x000000000043b5c9 in ngx_worker_process_cycle (cycle=0x185f060,

data=) at src/os/unix/ngx_process_cycle.c:822

13 0x0000000000439f35 in ngx_spawn_process (cycle=0x185f060, proc=0x43b510

, data=0x0, name=0x516b1e "worker process", respawn=-3) at src/os/unix/ngx_process.c:198 #14 0x000000000043b804 in ngx_start_worker_processes (cycle=0x185f060, n=1, type=-3) at src/os/unix/ngx_process_cycle.c:368 #15 0x000000000043c486 in ngx_master_process_cycle (cycle=0x185f060) at src/os/unix/ngx_process_cycle.c:140 #16 0x0000000000419c0d in main (argc=, argv=) at src/core/nginx.c:407 On Tue, Jan 27, 2015 at 5:50 PM, Flávio Ribeiro notifications@github.com wrote: > hey @jalonsoa https://github.com/jalonsoa any updates on this? > > — > Reply to this email directly or view it on GitHub > https://github.com/flavioribeiro/nginx-audio-track-for-hls-module/issues/25#issuecomment-71682107 > .
jalonsoa commented 9 years ago

I try this week without the patchs about lock manager

j

On Wed, Jan 28, 2015 at 1:44 PM, Justo Alonso justo.alonso@gmail.com wrote:

Hi Flavio, sorry. I'm bussy this week .. ;-( .. yesterday compile new libav for my machine (this time I'm sure that have threads support), and recompile nginx against the new lib. Bad news ... nginx crash again.

Any idea ?

(gdb) bt

0 0x00007ffc914b83c4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6

1 0x00007ffc9379d650 in av_parser_close () from

/usr/lib/x86_64-linux-gnu/libavcodec.so.56

2 0x00007ffc944a0d24 in ?? () from

/usr/lib/x86_64-linux-gnu/libavformat.so.56

3 0x00007ffc944a1137 in ?? () from

/usr/lib/x86_64-linux-gnu/libavformat.so.56

4 0x00000000004dd0c4 in ngx_http_aac_extract_audio

(destination=0x1916b48, outputfmt=..., log=0x18fdc80, pool=, source=...) at /html/src/nginx/nginx-1.7.4/debian/modules/nginx-audio-track-for-hls-module/ngx_http_aac_module.c:245

5 ngx_http_aac_handler (r=0x1915bd0) at

/html/src/nginx/nginx-1.7.4/debian/modules/nginx-audio-track-for-hls-module/ngx_http_aac_module.c:31

6 0x000000000044db52 in ngx_http_core_content_phase (r=0x1915bd0,

ph=0x189bdb0) at src/http/ngx_http_core_module.c:1407

7 0x0000000000448325 in ngx_http_core_run_phases (r=0x1915bd0) at

src/http/ngx_http_core_module.c:888

8 0x0000000000454451 in ngx_http_process_request (r=0x1915bd0) at

src/http/ngx_http_request.c:1902

9 0x00000000004553a6 in ngx_http_process_request_line (rev=0x18c8d18) at

src/http/ngx_http_request.c:1012

10 0x000000000043d731 in ngx_epoll_process_events (cycle=0x185f060,

timer=, flags=) at src/event/modules/ngx_epoll_module.c:691

11 0x00000000004340d7 in ngx_process_events_and_timers (cycle=0x185f060)

at src/event/ngx_event.c:248

12 0x000000000043b5c9 in ngx_worker_process_cycle (cycle=0x185f060,

data=) at src/os/unix/ngx_process_cycle.c:822

13 0x0000000000439f35 in ngx_spawn_process (cycle=0x185f060,

proc=0x43b510 , data=0x0, name=0x516b1e "worker process", respawn=-3) at src/os/unix/ngx_process.c:198

14 0x000000000043b804 in ngx_start_worker_processes (cycle=0x185f060,

n=1, type=-3) at src/os/unix/ngx_process_cycle.c:368

15 0x000000000043c486 in ngx_master_process_cycle (cycle=0x185f060) at

src/os/unix/ngx_process_cycle.c:140

16 0x0000000000419c0d in main (argc=, argv=<optimized

out>) at src/core/nginx.c:407

On Tue, Jan 27, 2015 at 5:50 PM, Flávio Ribeiro notifications@github.com wrote:

hey @jalonsoa https://github.com/jalonsoa any updates on this?

— Reply to this email directly or view it on GitHub https://github.com/flavioribeiro/nginx-audio-track-for-hls-module/issues/25#issuecomment-71682107 .

flavioribeiro commented 9 years ago

I think we can go ahead with lock manager.

jalonsoa commented 9 years ago

Hi Flavio, I try without lock manager, and all work fine now (from this morning without crashes).

I'm preparing a patch to send you a patch ...

thanks ! j

On Wed, Jan 28, 2015 at 4:57 PM, Flávio Ribeiro notifications@github.com wrote:

I think we can go ahead with lock manager.

— Reply to this email directly or view it on GitHub https://github.com/flavioribeiro/nginx-audio-track-for-hls-module/issues/25#issuecomment-71858937 .

flavioribeiro commented 9 years ago

Awesome, thanks :beers:

jalonsoa commented 9 years ago

Hi Flavio, sorry, but the nginx crash again. I install debug symbols for libc and the new libav libs ... and this is the backtrace:

(gdb) bt
#0  0x00007f3321ebead0 in malloc_consolidate (av=0x7f33221f8720) at malloc.c:4286
#1  0x00007f3321ebf839 in malloc_consolidate (av=0x7f33221f8720) at malloc.c:4255
#2  _int_free (av=0x7f33221f8720, p=<optimized out>, have_lock=0) at malloc.c:4186
#3  0x00007f3323be335c in av_freep (arg=0x372e438) at /usr/src/libav/libav-11/libavutil/mem.c:201
#4  0x00007f3324eaa8ad in avformat_free_context (s=0x372e420) at /usr/src/libav/libav-11/libavformat/utils.c:2484
#5  0x00007f3324eaaa7d in avformat_close_input (ps=0x7fffac2509f8) at /usr/src/libav/libav-11/libavformat/utils.c:2511
#6  0x0000000000543c62 in ngx_http_aac_extract_audio (pool=0x37362c0, log=0x360f9c0, source=..., outputfmt=..., destination=0x3737288)
    at /html/src/nginx/nginx-1.7.4/debian/modules/nginx-audio-track-for-hls-module/ngx_http_aac_module.c:241
#7  0x000000000054333d in ngx_http_aac_handler (r=0x3736310) at /html/src/nginx/nginx-1.7.4/debian/modules/nginx-audio-track-for-hls-module/ngx_http_aac_module.c:31
#8  0x0000000000463d0f in ngx_http_core_content_phase (r=0x3736310, ph=0x363e4c8) at src/http/ngx_http_core_module.c:1407
#9  0x0000000000462845 in ngx_http_core_run_phases (r=0x3736310) at src/http/ngx_http_core_module.c:888
#10 0x00000000004627bc in ngx_http_handler (r=0x3736310) at src/http/ngx_http_core_module.c:871
#11 0x00000000004724fc in ngx_http_process_request (r=0x3736310) at src/http/ngx_http_request.c:1902
#12 0x0000000000470e15 in ngx_http_process_request_headers (rev=0x366cc20) at src/http/ngx_http_request.c:1333
#13 0x000000000047018e in ngx_http_process_request_line (rev=0x366cc20) at src/http/ngx_http_request.c:1012
#14 0x0000000000475040 in ngx_http_keepalive_handler (rev=0x366cc20) at src/http/ngx_http_request.c:3181
#15 0x000000000045284f in ngx_epoll_process_events (cycle=0x3601ff0, timer=10070, flags=1) at src/event/modules/ngx_epoll_module.c:691
#16 0x000000000044296c in ngx_process_events_and_timers (cycle=0x3601ff0) at src/event/ngx_event.c:248
#17 0x000000000045071a in ngx_worker_process_cycle (cycle=0x3601ff0, data=0x0) at src/os/unix/ngx_process_cycle.c:822
#18 0x000000000044ce54 in ngx_spawn_process (cycle=0x3601ff0, proc=0x450519 <ngx_worker_process_cycle>, data=0x0, name=0x592773 "worker process", respawn=-3)
    at src/os/unix/ngx_process.c:198
#19 0x000000000044f432 in ngx_start_worker_processes (cycle=0x3601ff0, n=1, type=-3) at src/os/unix/ngx_process_cycle.c:368
#20 0x000000000044ea6c in ngx_master_process_cycle (cycle=0x3601ff0) at src/os/unix/ngx_process_cycle.c:140
#21 0x000000000041968b in main (argc=1, argv=0x7fffac251338) at src/core/nginx.c:407

It crash freeing pointer from s->priv_data (s is the input_format_context). Reading the docs from libav, the priv_data holds information about options, only if codecs need it. I don't known about my input ts files (generates by nginx-rtmp module) need thes structs and your files don't need (or don't crash here).

This is the info from AVFormatContext when crash

(gdb) print *(AVFormatContext *)s
$13 = {av_class = 0x7f33250ee120, iformat = 0x7f33250f8d20, oformat = 0x0, priv_data = 0x3749360, pb = 0x3729ae0, ctx_flags = 1, nb_streams = 2, streams = 0x3651990, 
  filename = "/html/nginx/hls/0_zkamo5t3_1/1200756780.ts", '\000' <repeats 981 times>, start_time = 13342281333, duration = 9985000, bit_rate = 83748, packet_size = 0, 
  max_delay = -1, flags = 512, probesize = 5000000, max_analyze_duration = 5000000, key = 0x0, keylen = 0, nb_programs = 1, programs = 0x0, video_codec_id = AV_CODEC_ID_NONE, 
  audio_codec_id = AV_CODEC_ID_NONE, subtitle_codec_id = AV_CODEC_ID_NONE, max_index_size = 1048576, max_picture_buffer = 3041280, nb_chapters = 0, chapters = 0x0, 
  metadata = 0x0, start_time_realtime = 0, fps_probe_size = -1, error_recognition = 1, interrupt_callback = {callback = 0, opaque = 0x0}, debug = 0, 
  max_interleave_delta = 10000000, strict_std_compliance = 0, event_flags = 0, packet_buffer = 0x0, packet_buffer_end = 0x0, data_offset = 0, raw_packet_buffer = 0x0, 
  raw_packet_buffer_end = 0x0, parse_queue = 0x0, parse_queue_end = 0x0, raw_packet_buffer_remaining_size = 2500000, offset = 0, offset_timebase = {num = 0, den = 0}, 
  internal = 0x360af80}
(gdb)

and the priv_data before free it

(gdb) print *(PESContext *)ptr
$14 = {pid = 57971552, pcr_pid = 0, stream_type = 57842400, ts = 0x200000001, stream = 0x3651990, st = 0x676e2f6c6d74682f, sub_st = 0x2f736c682f786e69, state = 1803181872, 
  data_index = 896494945, flags = 828322676, total_size = 808595759, pes_header_size = 909457200, extended_stream_id = 774912055, pts = 29556, dts = 0, ts_packet_pos = 0, 
  header = '\000' <repeats 263 times>, buffer = 0x0, sl = {use_au_start = 0, use_au_end = 0, use_rand_acc_pt = 0, use_padding = 0, use_timestamps = 0, use_idle = 0, 
    timestamp_res = 0, timestamp_len = 0, ocr_len = 0, au_len = 0, inst_bitrate_len = 0, degr_prior_len = 0, au_seq_num_len = 0, packet_seq_num_len = 0}}
(gdb)

What do you think about this ?