search

AirenSoft / OvenMediaEngine

OvenMediaEngine (OME) is a Sub-Second Latency Live Streaming Server with Large-Scale and High-Definition. #WebRTC #LLHLS
https://airensoft.com/ome.html
GNU Affero General Public License v3.0
2.61k stars 1.06k forks source link

APP crush if transcoding more than 1 variants #1713

Closed GeghamSimonyan closed 1 month ago

GeghamSimonyan commented 1 month ago

Describe the bug OvenMedia engine hang when transcoding more than 1 variants over nv module

To Reproduce Steps to reproduce the behavior:

  1. Set Server.xml as follows 

    
<HWAccels>
<Decoder>
            <Enable>true</Enable>

            <Modules>nv</Modules>
</Decoder>
<Encoder>
            <Enable>true</Enable>
            <Modules>nv</Modules>
</Encoder>
</HWAccels>

On encoders section 


3. With Encoder 'FFMPEG, OBS, all the same' 
4. See error
Inconsistency detected by ld.so: dl-lookup.c: 106: check_match: Assertion `version->filename == NULL || ! _dl_name_match_p (version->filename, map)' failed!
**Expected behavior**
1 stream should be transcoded to 2 streams

**Logs**
[2024-10-07 13:28:57.984] I [Decaac:129839] Transcoder | filter_resampler.cpp:159  | Resampler parameters. track(#1 -> #4). desc(src: time_base=1/1000:sample_rate=48000:sample_fmt=fltp:channel_layout=stereo -> output: asettb=1/48000,aresample=async=1000,aresample=48000,aformat=sample_fmts=s16:channel_layouts=stereo,asetnsamples=n=960)
[2024-10-07 13:28:58.213] I [Dech264NV:129838] Transcoder | decoder_avc_nv.cpp:259  | [#default#app/obs(100)] input stream information: [video] h264 (Constrained Baseline 3.1), cuda, 1280x720 [SAR 1:1 DAR 16:9], 40 fps, 195 kbps, timebase: 1/40, 
[2024-10-07 13:28:58.238] I [Dech264NV:129838] Transcoder | transcoder_encoder.cpp:282  | The encoder has been created successfully. track(#2), codec(H264), module(nvenc:0)
[2024-10-07 13:28:58.260] I [Dech264NV:129838] Transcoder | transcoder_encoder.cpp:282  | The encoder has been created successfully. track(#1), codec(H264), module(nvenc:0)
[2024-10-07 13:28:58.260] I [Dech264NV:129838] Transcoder | filter_rescaler.cpp:322  | Rescaler parameters. track(#0 -> #2), module(nvenc:0 -> nvenc:0). desc(src:video_size=1280x720:pix_fmt=cuda:time_base=1/1000:pixel_aspect=1/1 -> output:settb=1/90000,scale_npp=1280:720,format=cuda)
[2024-10-07 13:28:58.261] I [Dech264NV:129838] Transcoder | filter_rescaler.cpp:322  | Rescaler parameters. track(#0 -> #1), module(nvenc:0 -> nvenc:0). desc(src:video_size=1280x720:pix_fmt=cuda:time_base=1/1000:pixel_aspect=1/1 -> output:settb=1/90000,scale_npp=1920:1280,format=cuda)
Inconsistency detected by ld.so: dl-lookup.c: 106: check_match: Assertion `version->filename == NULL || ! _dl_name_match_p (version->filename, map)' failed!

**Server (please complete the following information):**
 - OS: ubuntu 22
 - OvenMediaEngine Version: [e.g. v0.16.8]
 - Branch: [e.g. release]

**Player (please complete the following information):**
not relvant
**Additional context**
nvidia-smi
Mon Oct  7 13:33:06 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03              Driver Version: 560.35.03      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX 4000 SFF Ada ...    Off |   00000000:01:00.0 Off |                  Off |
| 30%   41C    P2             23W /   70W |     168MiB /  20475MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A    129915      C   /usr/bin/OvenMediaEngine                      158MiB |
+-----------------------------------------------------------------------------------------+
Keukhan commented 1 month ago

@GeghamSimonyan

This is my first time seeing this error. But I think the issue with ld might be because OvenMediaEngine is using the wrong version or path for a library. Can you share the details of the libraries it depends on using the ldd command like this?

$ ldd ./OvenMediaEngine 
        linux-vdso.so.1 (0x00007fff899df000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f8f4bdb5000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f8f4bd92000)
        libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f8f4bd89000)
        libsrt.so.1.5 => /opt/ovenmediaengine/lib/libsrt.so.1.5 (0x00007f8f4bc9c000)
        libavformat.so.59 => /opt/ovenmediaengine/lib/libavformat.so.59 (0x00007f8f4bb9f000)
        libavfilter.so.8 => /opt/ovenmediaengine/lib/libavfilter.so.8 (0x00007f8f4bb4f000)
        libavcodec.so.59 => /opt/ovenmediaengine/lib/libavcodec.so.59 (0x00007f8f4b5af000)
        libavutil.so.57 => /opt/ovenmediaengine/lib/libavutil.so.57 (0x00007f8f4b3f1000)
        libssl.so.3 => /opt/ovenmediaengine/lib/libssl.so.3 (0x00007f8f4b344000)
        libcrypto.so.3 => /opt/ovenmediaengine/lib/libcrypto.so.3 (0x00007f8f4aed6000)
        libsrtp2.so.1 => /opt/ovenmediaengine/lib/libsrtp2.so.1 (0x00007f8f4aebf000)
        libpcre2-8.so.0 => /opt/ovenmediaengine/lib/libpcre2-8.so.0 (0x00007f8f4ae2b000)
        libhiredis.so.1.0.0 => /opt/ovenmediaengine/lib/libhiredis.so.1.0.0 (0x00007f8f4ae14000)
        libcuda.so.1 => /lib/x86_64-linux-gnu/libcuda.so.1 (0x00007f8f496ec000)
        libnvidia-ml.so.1 => /lib/x86_64-linux-gnu/libnvidia-ml.so.1 (0x00007f8f49054000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f8f48e72000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f8f48d23000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f8f48d08000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8f48b14000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f8f4da4a000)
        libswscale.so.6 => /opt/ovenmediaengine/lib/libswscale.so.6 (0x00007f8f48a7c000)
        libswresample.so.4 => /opt/ovenmediaengine/lib/libswresample.so.4 (0x00007f8f48a5c000)
        libnppig.so.10 => /lib/x86_64-linux-gnu/libnppig.so.10 (0x00007f8f46f8a000)
        libnppicc.so.10 => /lib/x86_64-linux-gnu/libnppicc.so.10 (0x00007f8f4694d000)
        libvpx.so.7 => /opt/ovenmediaengine/lib/libvpx.so.7 (0x00007f8f46873000)
        libfdk-aac.so.2 => /opt/ovenmediaengine/lib/libfdk-aac.so.2 (0x00007f8f46727000)
        libopus.so.0 => /opt/ovenmediaengine/lib/libopus.so.0 (0x00007f8f466c8000)
        libopenh264.so.7 => /opt/ovenmediaengine/lib/libopenh264.so.7 (0x00007f8f46550000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f8f4654a000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f8f4653e000)
        libnppc.so.10 => /lib/x86_64-linux-gnu/libnppc.so.10 (0x00007f8f462c3000)
GeghamSimonyan commented 1 month ago

idk why lld didn't work lld --version lld is a generic driver. Invoke ld.lld (Unix), ld64.lld (macOS), lld-link (Windows), wasm-ld (WebAssembly) instead and 

ld.lld OvenMediaEngine 
ld.lld: error: corrupt input file: version definition index 4 for symbol _ZTVSt9basic_iosIcSt11char_traitsIcEE is out of bounds
>>> defined in OvenMediaEngine

ld.lld: error: corrupt input file: version definition index 20 for symbol _ZTISt12bad_weak_ptr is out of bounds
>>> defined in OvenMediaEngine

ld.lld: error: corrupt input file: version definition index 13 for symbol _ZTIPi is out of bounds
>>> defined in OvenMediaEngine

objdump -x ./OvenMediaEngine

./OvenMediaEngine: file format elf64-x86-64 ./OvenMediaEngine architecture: i386:x86-64, flags 0x00000150: HAS_SYMS, DYNAMIC, D_PAGED start address 0x00000000000c6dd0

Program Header: PHDR off 0x0000000000000040 vaddr 0x0000000000000040 paddr 0x0000000000000040 align 23 filesz 0x0000000000000310 memsz 0x0000000000000310 flags r-- INTERP off 0x0000000000000350 vaddr 0x0000000000000350 paddr 0x0000000000000350 align 20 filesz 0x000000000000001c memsz 0x000000000000001c flags r-- LOAD off 0x0000000000000000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 212 filesz 0x00000000000633c0 memsz 0x00000000000633c0 flags r-- LOAD off 0x0000000000064000 vaddr 0x0000000000064000 paddr 0x0000000000064000 align 212 filesz 0x000000000050ecb5 memsz 0x000000000050ecb5 flags r-x LOAD off 0x0000000000573000 vaddr 0x0000000000573000 paddr 0x0000000000573000 align 212 filesz 0x00000000001d3ee4 memsz 0x00000000001d3ee4 flags r-- LOAD off 0x0000000000747bb0 vaddr 0x0000000000748bb0 paddr 0x0000000000748bb0 align 212 filesz 0x0000000000028960 memsz 0x000000000002f9f8 flags rw- DYNAMIC off 0x000000000076e930 vaddr 0x000000000076f930 paddr 0x000000000076f930 align 23 filesz 0x0000000000000320 memsz 0x0000000000000320 flags rw- NOTE off 0x0000000000000370 vaddr 0x0000000000000370 paddr 0x0000000000000370 align 23 filesz 0x0000000000000030 memsz 0x0000000000000030 flags r-- NOTE off 0x00000000000003a0 vaddr 0x00000000000003a0 paddr 0x00000000000003a0 align 22 filesz 0x0000000000000044 memsz 0x0000000000000044 flags r-- TLS off 0x0000000000747bb0 vaddr 0x0000000000748bb0 paddr 0x0000000000748bb0 align 20 filesz 0x0000000000000000 memsz 0x0000000000000010 flags r-- 0x6474e553 off 0x0000000000000370 vaddr 0x0000000000000370 paddr 0x0000000000000370 align 23 filesz 0x0000000000000030 memsz 0x0000000000000030 flags r-- EH_FRAME off 0x00000000005c9170 vaddr 0x00000000005c9170 paddr 0x00000000005c9170 align 22 filesz 0x000000000001d3b4 memsz 0x000000000001d3b4 flags r-- STACK off 0x0000000000000000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 24 filesz 0x0000000000000000 memsz 0x0000000000000000 flags rw- RELRO off 0x0000000000747bb0 vaddr 0x0000000000748bb0 paddr 0x0000000000748bb0 align 20 filesz 0x0000000000028450 memsz 0x0000000000028450 flags r--

Dynamic Section: NEEDED libz.so.1 NEEDED libuuid.so.1 NEEDED libsrt.so.1.5 NEEDED libavformat.so.59 NEEDED libavfilter.so.8 NEEDED libavcodec.so.59 NEEDED libavutil.so.57 NEEDED libssl.so.3 NEEDED libcrypto.so.3 NEEDED libsrtp2.so.1 NEEDED libpcre2-8.so.0 NEEDED libhiredis.so.1.0.0 NEEDED libcuda.so.1 NEEDED libnvidia-ml.so.1 NEEDED libjemalloc.so.2 NEEDED libstdc++.so.6 NEEDED libm.so.6 NEEDED libgcc_s.so.1 NEEDED libc.so.6 RUNPATH /opt/ovenmediaengine/lib:/opt/ovenmediaengine/lib64 INIT 0x0000000000064000 FINI 0x0000000000572ca8 INIT_ARRAY 0x0000000000748bb0 INIT_ARRAYSZ 0x0000000000000be8 FINI_ARRAY 0x0000000000749798 FINI_ARRAYSZ 0x0000000000000008 GNU_HASH 0x00000000000003e8 STRTAB 0x00000000000046d0 SYMTAB 0x00000000000005c0 STRSZ 0x000000000000411b SYMENT 0x0000000000000018 DEBUG 0x0000000000000000 PLTGOT 0x000000000076fc50 PLTRELSZ 0x0000000000003828 PLTREL 0x0000000000000007 JMPREL 0x000000000005fb98 RELA 0x0000000000009078 RELASZ 0x0000000000056b20 RELAENT 0x0000000000000018 FLAGS 0x0000000000000008 FLAGS_1 0x0000000008000001 VERNEED 0x0000000000008d58 VERNEEDNUM 0x000000000000000b VERSYM 0x00000000000087ec RELACOUNT 0x00000000000030bc

Version References: required from libgcc_s.so.1: 0x0b792650 0x00 35 GCC_3.0 required from libuuid.so.1: 0x09da27b0 0x00 22 UUID_1.0 required from libm.so.6: 0x09691a75 0x00 32 GLIBC_2.2.5 0x06969189 0x00 18 GLIBC_2.29 required from libavutil.so.57: 0x0f818437 0x00 17 LIBAVUTIL_57 required from libssl.so.3: 0x06702b20 0x00 10 OPENSSL_3.0.0 required from libc.so.6: 0x069691b2 0x00 38 GLIBC_2.32 0x0d696913 0x00 34 GLIBC_2.3 0x0d696919 0x00 31 GLIBC_2.9 0x06969197 0x00 26 GLIBC_2.17 0x0d696914 0x00 25 GLIBC_2.4 0x069691b4 0x00 23 GLIBC_2.34 0x09691974 0x00 21 GLIBC_2.3.4 0x069691b3 0x00 16 GLIBC_2.33 0x09691973 0x00 15 GLIBC_2.3.3 0x06969194 0x00 14 GLIBC_2.14 0x09691972 0x00 12 GLIBC_2.3.2 0x09691a75 0x00 08 GLIBC_2.2.5 required from libavfilter.so.8: 0x0ececed8 0x00 07 LIBAVFILTER_8 required from libavformat.so.59: 0x0c7032c9 0x00 06 LIBAVFORMAT_59 required from libstdc++.so.6: 0x0297f864 0x00 40 GLIBCXX_3.4.14 0x0297f876 0x00 39 GLIBCXX_3.4.26 0x02297f85 0x00 37 GLIBCXX_3.4.5 0x0297f869 0x00 36 GLIBCXX_3.4.19 0x0297f840 0x00 33 GLIBCXX_3.4.30 0x0bafd175 0x00 30 CXXABI_1.3.5 0x0297f879 0x00 29 GLIBCXX_3.4.29 0x0297f872 0x00 28 GLIBCXX_3.4.22 0x02297f89 0x00 27 GLIBCXX_3.4.9 0x0297f861 0x00 24 GLIBCXX_3.4.11 0x0297f865 0x00 20 GLIBCXX_3.4.15 0x0297f868 0x00 19 GLIBCXX_3.4.18 0x056bafd3 0x00 13 CXXABI_1.3 0x0297f870 0x00 11 GLIBCXX_3.4.20 0x0bafd172 0x00 09 CXXABI_1.3.2 0x0297f871 0x00 05 GLIBCXX_3.4.21 0x08922974 0x00 04 GLIBCXX_3.4 required from libavcodec.so.59: 0x035ff8b9 0x00 03 LIBAVCODEC_59 required from libcrypto.so.3: 0x06702b20 0x00 02 OPENSSL_3.0.0

Sections: Idx Name Size VMA LMA File off Algn 0 .interp 0000001c 0000000000000350 0000000000000350 00000350 20 CONTENTS, ALLOC, LOAD, READONLY, DATA 1 .note.gnu.property 00000030 0000000000000370 0000000000000370 00000370 23 CONTENTS, ALLOC, LOAD, READONLY, DATA 2 .note.gnu.build-id 00000024 00000000000003a0 00000000000003a0 000003a0 22 CONTENTS, ALLOC, LOAD, READONLY, DATA 3 .note.ABI-tag 00000020 00000000000003c4 00000000000003c4 000003c4 22 CONTENTS, ALLOC, LOAD, READONLY, DATA 4 .gnu.hash 000001d4 00000000000003e8 00000000000003e8 000003e8 23 CONTENTS, ALLOC, LOAD, READONLY, DATA 5 .dynsym 00004110 00000000000005c0 00000000000005c0 000005c0 23 CONTENTS, ALLOC, LOAD, READONLY, DATA 6 .dynstr 0000411b 00000000000046d0 00000000000046d0 000046d0 20 CONTENTS, ALLOC, LOAD, READONLY, DATA 7 .gnu.version 0000056c 00000000000087ec 00000000000087ec 000087ec 21 CONTENTS, ALLOC, LOAD, READONLY, DATA 8 .gnu.version_r 00000320 0000000000008d58 0000000000008d58 00008d58 23 CONTENTS, ALLOC, LOAD, READONLY, DATA 9 .rela.dyn 00056b20 0000000000009078 0000000000009078 00009078 23 CONTENTS, ALLOC, LOAD, READONLY, DATA 10 .rela.plt 00003828 000000000005fb98 000000000005fb98 0005fb98 23 CONTENTS, ALLOC, LOAD, READONLY, DATA 11 .init 0000001b 0000000000064000 0000000000064000 00064000 22 CONTENTS, ALLOC, LOAD, READONLY, CODE 12 .plt 00002580 0000000000064020 0000000000064020 00064020 24 CONTENTS, ALLOC, LOAD, READONLY, CODE 13 .plt.got 00000050 00000000000665a0 00000000000665a0 000665a0 24 CONTENTS, ALLOC, LOAD, READONLY, CODE 14 .plt.sec 00002570 00000000000665f0 00000000000665f0 000665f0 24 CONTENTS, ALLOC, LOAD, READONLY, CODE 15 .text 0050a145 0000000000068b60 0000000000068b60 00068b60 24 CONTENTS, ALLOC, LOAD, READONLY, CODE 16 .fini 0000000d 0000000000572ca8 0000000000572ca8 00572ca8 22 CONTENTS, ALLOC, LOAD, READONLY, CODE 17 .rodata 00056170 0000000000573000 0000000000573000 00573000 25 CONTENTS, ALLOC, LOAD, READONLY, DATA 18 .eh_frame_hdr 0001d3b4 00000000005c9170 00000000005c9170 005c9170 22 CONTENTS, ALLOC, LOAD, READONLY, DATA 19 .eh_frame 000a78a4 00000000005e6528 00000000005e6528 005e6528 23 CONTENTS, ALLOC, LOAD, READONLY, DATA 20 .gcc_except_table 000b9118 000000000068ddcc 000000000068ddcc 0068ddcc 22 CONTENTS, ALLOC, LOAD, READONLY, DATA 21 .tbss 00000010 0000000000748bb0 0000000000748bb0 00747bb0 20 ALLOC, THREAD_LOCAL 22 .init_array 00000be8 0000000000748bb0 0000000000748bb0 00747bb0 23 CONTENTS, ALLOC, LOAD, DATA 23 .fini_array 00000008 0000000000749798 0000000000749798 00748798 23 CONTENTS, ALLOC, LOAD, DATA 24 .data.rel.ro 00026190 00000000007497a0 00000000007497a0 007487a0 25 CONTENTS, ALLOC, LOAD, DATA 25 .dynamic 00000320 000000000076f930 000000000076f930 0076e930 23 CONTENTS, ALLOC, LOAD, DATA 26 .got 00001398 000000000076fc50 000000000076fc50 0076ec50 23 CONTENTS, ALLOC, LOAD, DATA 27 .data 00000510 0000000000771000 0000000000771000 00770000 25 CONTENTS, ALLOC, LOAD, DATA 28 .bss 00007088 0000000000771520 0000000000771520 00770510 25 ALLOC 29 .comment 0000002b 0000000000000000 0000000000000000 00770510 20 CONTENTS, READONLY SYMBOL TABLE: no symbols

bchah commented 1 month ago

idk why lld didn't work

Try ldd, they are different commands.

GeghamSimonyan commented 1 month ago

I just moved to docker build to avoid this issue , i think this can be closed for now