search

OpenVisualCloud / SVT-HEVC

SVT HEVC encoder. Scalable Video Technology (SVT) is a software-based video coding technology that is highly optimized for Intel® Xeon® processors. Using the open source SVT-HEVC encoder, it is possible to spread video encoding processing across multiple Intel® Xeon® processors to achieve a real advantage of processing efficiency.
Other
518 stars 170 forks source link

Random crash after running for a long time #613

Open gaetanaxs opened 2 years ago

gaetanaxs commented 2 years ago

We found a random crash issue after running encoder for a long time (e.g. 40 hours). The crash happens randomly when we run command like below:

ffmpeg -y -stream_loop -1 -c:v h264 -i 1.mp4 -an -c:v libsvt_hevc -rc 1 -preset 9 -r 25 -flags +cgop -g 8 -keyint_min 8 -forced-idr 1 -hielevel 0 -pred_struct 0 -bf 0 -la_depth -1 -sc_detection 0 -vf scale=w=1280:h=720 -b:v 500000 -maxrate:v 1000000 -bufsize:v 1500000 -f segment -segmenttime 0.32 ./seq1%d.mp4

With gdb, we captured two crash core dumps as follows: 1st time crash -- image 2nd time crash -- image

Both of them point to the function of EncodeQuantizedCoefficients_SSE2 in https://github.com/OpenVisualCloud/SVT-HEVC/blob/master/Source/Lib/Codec/EbEntropyCoding.c

It seems that sometimes the while(1) loop cannot be ended up successfully, leading to the final crash.

Could you help us have a look on it? Thanks a lot!

tianjunwork commented 2 years ago

Hi @gaetanaxs, what is the hw setup? Could you use svt-hevc debug library and attach the information it prints before starting encode? Thanks.

gaetanaxs commented 2 years ago

Thanks! @tianjunwork CPU: AMD EPYC Milan memory: 128G OS: Linux 4.14.105-19-0012 x86_64 GCC: gcc version 4.8.5 SVT CommitID: b09984ce1b88ec424b11a62fb6d09108cbeb634d WXWorkCapture_16421375808165(1)

tianjunwork commented 2 years ago

Hi @gaetanaxs, Thanks providing the snapshot. I will try to reproduce on my side. How often did you get random crash? Every time app crashes it is around 40 hours? Your GCC is a very old version. For better performance, newer GCC is recommended. You can switch GCC versions: https://askubuntu.com/a/26502.

gaetanaxs commented 2 years ago

Hi @tianjunwork, thanks for your suggestions for switching newer GCC. This random crash appears after running for a quite long time, sometimes it is around 40 hours and sometimes 110 hours. You can launch multiple commands (e.g., 16) on the machine to increase the probability of getting this random crash.

gaetanaxs commented 2 years ago

@tianjunwork In the captured two crashes, we found that in the function of EncodeQuantizedCoefficients_SSE2, the while(1) loop cannot be smoothly ended up by triggering "break" sentence (as shown in the figure below), leading to the continuous increasing of "subSetIndex". Hope this may help. Thank you! image