search

kaltura / nginx-vod-module

NGINX-based MP4 Repackager
GNU Affero General Public License v3.0
2.01k stars 440 forks source link

Nginx vod is using to much memory #942

Open momkin opened 5 years ago

momkin commented 5 years ago

Hello , i have dedicated server with 24GB of ram , Nginx vod is using the full 24GB memory in 3 or 4 days !! which causes the system to crush

the config is this : mp4_buffer_size 1M; mp4_max_buffer_size 10M; open_file_cache max=1000 inactive=5m; open_file_cache_valid 2m; open_file_cache_min_uses 1; open_file_cache_errors on; access_log off; server_tokens off; keepalive_timeout 0; types_hash_max_size 2048; error_log /dev/null crit; resolver 8.8.4.4 8.8.8.8 valid=300s ipv6=off; resolver_timeout 10s; client_max_body_size 80000M;

gzip

gzip off;

vod_mapping_cache mapping_cache 10m; # few MB vod_response_cache response_cache 128m; # 8-128MB vod_metadata_cache metadata_cache 6000m; # large cache

vod hls; vod_mode local; vod_secret_key "2BL52NF2V4kdpFdiwNsa$vod_filepath"; vod_hls_encryption_method aes-128; vod_segment_duration 5000; vod_last_modified 'Sun, 19 Nov 1999 09:09:09 GMT'; vod_last_modified_types *; vod_hls_absolute_master_urls off; vod_hls_absolute_index_urls off;

nginx version: nginx/1.12.1 built by gcc 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) built with OpenSSL 1.0.2k-fips 26 Jan 2017 TLS SNI support enabled configure arguments: --with-http_stub_status_module --with-http_mp4_module --with-http_secure_link_module --add-module=../nginx-secure-token-module-master --add-module=../nginx-vod-module-master --with-file-aio --with-threads --with-http_ssl_module --with-http_v2_module --with-ld-opt=-Wl,-E

Any ideas ??

erankor commented 5 years ago

Maybe you have requests that do not close for some reason, I suggest you look at nginx status and check the active requests value. If you see this value keeps growing, that is most likely the problem. We've seen this happen at some point due to aio requests that never complete. Increasing worker_aio_requests solved the issue for us.

Two more things -

  1. Please attach the full nginx.conf
  2. From the conf part you pasted, it seems you are ignoring the error log, maybe there is something there... I suggest you connect it to a real file + use the default 'error' severity.
momkin commented 5 years ago

Hi , You mean when aio is active ? because i didn't activate it ! or i should rebuild the nginx without aio ??

erankor commented 5 years ago

If you don't use aio, then it's not the same as the problem we had, but I still suggest you check the active requests in nginx status.

momkin commented 5 years ago

I have enabled the nginx status which one of those should i look into ??

Active connections: 597 server accepts handled requests 1273 1273 5476 Reading: 0 Writing: 118 Waiting: 485

also i have enabled the error log and i'm getting this

2019/01/07 10:49:44 [error] 3495#3495: *321 hls_muxer_process: no data was handled, probably a truncated file while processing frames, client: 5.41.30.9, server: localh

2019/01/07 10:49:47 [alert] 3495#3495: *321 header already sent while processing frames, client: 5.41.30.9, server: localhost, request: "GET /stream/hls/6/b9a96bdafd40e8eaef8a651623ea985e/,720,480,360,p.mp4.urlset/seg-727-f1-v1-a1.ts?token=oXdFve_ceWz6lOAdvgfnnw&expire=1546958685 HTTP/2.0", host: "cy8dpqdu.stream.com", referrer: "https://www.website.co/video_player?nv2=true&uid=0&vid=b9a96bdafd40e8eaef8a651623ea985e&img=https://i2.wp.com/www.website.co/wp-content/uploads/2018/11/MV5BMjQ1MzYzMDU5NV5BMl5BanBnXkFtZTgwMDU2MjU2NjM@._V1_SY1000_SX676_AL_.jpg?resize=440%2C630"

erankor commented 5 years ago

Active Connections, if it keeps growing / has a value significantly higher than the established conns on the server, you have stuck requests. The first error you pasted is legit - it means the mp4 file is corrupt, the second error is not, it means some setup problem/bug. If you curl this URL, does it reproduce? do you have many of these?

momkin commented 5 years ago

Hi , Yes i have at least 2 line every 10 seconds

erankor commented 5 years ago

Ok, but does it reproduce if you curl it manually? This may be related to the fact you use HTTP/2, I never tested it... Please provide the full nginx.conf