search

OpenVisualCloud / SVT-VP9

SVT VP9 encoder. Scalable Video Technology (SVT) is a software-based video coding technology that is highly optimized for Intel® Xeon® processors. Using the open source SVT-VP9 encoder, it is possible to spread video encoding processing across multiple Intel® Xeon® processors to achieve a real advantage of processing efficiency.
Other
215 stars 46 forks source link

Multiple instances of libsvt-vp9 #160

Closed lp24db closed 1 year ago

lp24db commented 1 year ago

Hi there. I am testing the svt-vp9 in a self developed, ffmpeg-based encoder (written in c) and i'm very happy with it. My code is nearly identical to the transcoding-example of ffmpeg (no os ffmpeg call) Actually i am using libvpx for VP9 encodes and because of the bad multithreading i am forking serveral threads with libvpx instances in order to get a good performance. I recently switched to libsvt and did not reduce the threading algo. In this case (multiple instances of libsvt-vp9) i experienced a deadlock while transcoding files which cannot be resolved by myself. My dev-system runs on a Dual-Socket E5-2690v4 Xeon. I know that the svt-lib uses both sockets on all cores for the transcoding, so there would be no need for multiple instances but a did a test with libsvt-hevc and did not see any deadlock there - even with four or more multiple instances. So even there is no expectation in an improvement of overall speed using multiple instances, i'm not sure if this behaviour is correct. I was able to catch a stack trace and i see that the lib got stuck while acquiring a lock. If i only use a single libsvt-vp9 instance on one system, the issue does not occur. Maybe this helps to improve the stability. If it is not designed to run multiple, it is fine for me. I just wanted to clear, whether it could be a bug or not.

regards andreas

Thread 12 (Thread 0x7f5117ff7700 (LWP 102163) "worker"):

0 futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f50b07e34b0) at ../sysdeps/nptl/futex-internal.h:323

1 do_futex_wait (sem=sem@entry=0x7f50b07e34b0, abstime=0x0, clockid=0) at sem_waitcommon.c:112

2 0x00007f53a1c30278 in __new_sem_wait_slow (sem=0x7f50b07e34b0, abstime=0x0, clockid=0) at sem_waitcommon.c:184

3 0x00007f539fded759 in eb_vp9_block_on_semaphore () from /usr/local/lib/libSvtVp9Enc.so.1

4 0x00007f539fded37b in eb_vp9_get_empty_object () from /usr/local/lib/libSvtVp9Enc.so.1

5 0x00007f539fd95169 in eb_vp9_svt_enc_send_picture () from /usr/local/lib/libSvtVp9Enc.so.1

6 0x00007f53a37132be in ?? () from /usr/local/lib/ffmpeg/libavcodec.so.59

7 0x00007f53a34ce5bf in ?? () from /usr/local/lib/ffmpeg/libavcodec.so.59

8 0x00007f53a34ce821 in avcodec_send_frame () from /usr/local/lib/ffmpeg/libavcodec.so.59

9 0x000055d91d58d4ad in CWEBMEncoder::EncodeVideo (this=0x7f511c000720) at src/modules/WEBMEncoder.cpp:1704

10 0x000055d91d586a2a in CWEBMEncoder::DoJob (this=0x7f511c000720) at src/modules/WEBMEncoder.cpp:115

11 0x000055d91d5e8d12 in workerThread (context=0x7f538c007620) at src/core/EncodingThread.cpp:149

12 0x00007f53a1c26ea7 in start_thread (arg=) at pthread_create.c:477

13 0x00007f53a1b46aef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

1480c1 commented 1 year ago

Hi, for all svt encoders, it is not recommended to have multiple instances of the library running inside one application, as it is not tested and as far as I remember, all 3 use some global variables here and there which could lead to issues when trying to run them all. Additionally, all encoders are tunes on the assumption that there will be no other encoder within the same application.

lp24db commented 1 year ago

It would be nice if you were able to use all SVTs on the same machine at same time. But if it is caused by design, it looks like it is not possible at all. thank you.