search

bilibili / ijkplayer

Android/iOS video player based on FFmpeg n3.4, with MediaCodec, VideoToolbox support.
GNU General Public License v2.0
32.46k stars 8.12k forks source link

0.8.8 ios硬解码h264/h265丢帧或失败 #4141

Open cupidfantasy opened 6 years ago

cupidfantasy commented 6 years ago

经查发现是可能的原因是： IJKVideoToolBoxSync.m和IJKVideoToolBoxAsync.m文件中vtbformat_init函数注释掉以下代码

//                if (!validate_avcC_spc(extradata, extrasize, &fmt_desc->max_ref_frames, &sps_level, &sps_profile)) {
            //goto failed;
//                }

导致fmt_desc->max_ref_frames没有正确赋值。经过fmt_desc->max_ref_frames = FFMAX(fmt_desc->max_ref_frames, 2);之后变为2，但仍可能小于视频实际的max_ref_frames。

错误的max_ref_frames导致帧很快被送走，ctx->m_sort_queue的depth不超过2。

        if ((ctx->m_queue_depth > ctx->fmt_desc.max_ref_frames)) {
            QueuePicture(ctx);
        }

然后在接下来的检查时，新来帧的pts小于队列里最小pts（因为本组的帧数都已经超过2，上一组已经全部被送走，如果队列中至少有一个上一组的帧，新来的帧肯定会小于队列最小值，则不会被抛弃），导致此帧被抛弃

        if (ctx->m_sort_queue && newFrame->pic.pts < ctx->m_sort_queue->pic.pts) {
            goto failed;
        }

以上仅是个人推测，请作者帮忙指正一下。

cupidfantasy commented 6 years ago

今天发现ios硬解h265 NALU失败，看了代码发现在vtbformat_init中对265从annexB=>mp4转换时需要使用ff_isom_write_hvcc，而不能重用264的ff_isom_write_avcc。

        if ((extradata[0] == 0 && extradata[1] == 0 && extradata[2] == 0 && extradata[3] == 1) ||
            (extradata[0] == 0 && extradata[1] == 0 && extradata[2] == 1)) {
            AVIOContext *pb;
            if (avio_open_dyn_buf(&pb) < 0) {
                goto fail;
            }

            fmt_desc->convert_bytestream = true;

            if (codec == AV_CODEC_ID_HEVC) {
                ff_isom_write_hvcc(pb, extradata, extrasize, 1); //hvcc isom is different from avcc
            } else {
                ff_isom_write_avcc(pb, extradata, extrasize);
            }

            extradata = NULL;
            extrasize = avio_close_dyn_buf(pb, &extradata);

另外发现喂H265数据时，如果带有HEVC_NAL_AUD帧会导致videotoolbox解码失败。

raymond1012 commented 6 years ago

麻烦提供下demo

Zeaa commented 5 years ago

@raymond1012 测试 h265 时也遇到了同样的问题，开启使用硬解码失败，转成了软解 用的最新的版本，0.8.8，iPhone 8 plus，iOS 12.0。视频地址：http://demo.polyv.net/h5/265/01.m3u8 AVDRegister - AppleAVD HEVC codec registered could not init video tool box decoder !!!vtb fail!!! switch to ffmpeg decode!!!! 日志

ijkmediaplayer version : ===== custom modules begin =====
register demuxer : ijklivehook
===== custom modules end =====
av_version_info: ff3.4--ijk0.8.7--20180103--001
ijk_version_info: 
ijkmp_set_inject_opaque(0x280120ba0)
ijkmp_set_inject_opaque()=void
ijkmp_set_ijkio_inject_opaque(0x280120ba0)
ijkmp_set_ijkio_inject_opaque()=void

2018-10-18 19:18:19.308433+0800 IJKMediaDemo[4023:1410916] IJKSDLGLView: setupDisplay not ready
ijkmp_set_data_source(url="http://demo.polyv.net/h5/265/01.m3u8")
ijkmp_set_data_source(url="http://demo.polyv.net/h5/265/01.m3u8")=0
ijkmp_prepare_async()
===== versions =====
ijkplayer    : 
FFmpeg       : ff3.4--ijk0.8.7--20180103--001
libavutil    : 55.78.100
libavcodec   : 57.107.100
libavformat  : 57.83.100
libswscale   : 4.8.100
libswresample: 2.9.100
===== options =====
player-opts : video-pictq-size             = 3
player-opts : overlay-format               = fcc-_es2
player-opts : max-fps                      = 30
player-opts : framedrop                    = 0
player-opts : videotoolbox-max-frame-width = 960
player-opts : videotoolbox                 = 1
player-opts : start-on-prepared            = 1
format-opts : ijkapplication               = 4401983488
format-opts : ijkiomanager                 = 4402001648
format-opts : user-agent                   = ijkplayer
format-opts : auto_convert                 = 0
format-opts : timeout                      = 30000000
format-opts : reconnect                    = 1
format-opts : safe                         = 0
===================
ijkmp_prepare_async()=0
Opening 'http://demo.polyv.net/h5/265/01.m3u8' for reading
Setting default whitelist 'http,https,tls,rtp,tcp,udp,crypto,httpproxy'
Add dns cache hostname = demo.polyv.net, ip = 183.61.83.211
the user-agent option is deprecated, please use user_agent option
request: GET /h5/265/01.m3u8 HTTP/1.1

User-Agent: ijkplayer
Accept: */*
Range: bytes=0-
Connection: close
Host: demo.polyv.net

Icy-MetaData: 1

2018-10-18 19:18:19.348616+0800 IJKMediaDemo[4023:1410760] invalidateRenderBuffer
2018-10-18 19:18:19.348817+0800 IJKMediaDemo[4023:1410936] IJKSDLGLView: setupDisplay not ready
Format hls,applehttp probed with size=2048 and score=100
HLS request for url 'http://demo.polyv.net/h5/265/out000.ts', offset 0, playlist 0
Opening 'http://demo.polyv.net/h5/265/out000.ts' for reading
Hit DNS cache hostname = demo.polyv.net
the user-agent option is deprecated, please use user_agent option
request: GET /h5/265/out000.ts HTTP/1.1

User-Agent: ijkplayer
Accept: */*
Connection: close
Host: demo.polyv.net
Icy-MetaData: 1

Format mpegts probed with size=2048 and score=50
stream=0 stream_type=24 pid=100 prog_reg_desc=
stream=1 stream_type=f pid=101 prog_reg_desc=
Option safe not found.
Before avformat_find_stream_info() pos: 372 bytes read:372 seeks:0 nb_streams:2
2018-10-18 19:18:19.384978+0800 IJKMediaDemo[4023:1410760] FFP_MSG_OPEN_INPUT:
All programs have pmt, headers found
nal_unit_type: 35(AUD), nuh_layer_id: 0, temporal_id: 0
nal_unit_type: 32(VPS), nuh_layer_id: 0, temporal_id: 0
nal_unit_type: 33(SPS), nuh_layer_id: 0, temporal_id: 0
nal_unit_type: 34(PPS), nuh_layer_id: 0, temporal_id: 0
nal_unit_type: 39(SEI_PREFIX), nuh_layer_id: 0, temporal_id: 0
nal_unit_type: 19(IDR_W_RADL), nuh_layer_id: 0, temporal_id: 0
Decoding VPS
Main profile bitstream
Decoding SPS
Main profile bitstream
Decoding VUI
Decoding PPS
Decoding SEI
Skipped PREFIX SEI 5
nal_unit_type: 35(AUD), nuh_layer_id: 0, temporal_id: 0
nal_unit_type: 32(VPS), nuh_layer_id: 0, temporal_id: 0
nal_unit_type: 33(SPS), nuh_layer_id: 0, temporal_id: 0
nal_unit_type: 34(PPS), nuh_layer_id: 0, temporal_id: 0
nal_unit_type: 39(SEI_PREFIX), nuh_layer_id: 0, temporal_id: 0
nal_unit_type: 19(IDR_W_RADL), nuh_layer_id: 0, temporal_id: 0
Invalid NAL unit 0, skipping.
nal_unit_type: 35(AUD), nuh_layer_id: 0, temporal_id: 0
nal_unit_type: 32(VPS), nuh_layer_id: 0, temporal_id: 0
nal_unit_type: 33(SPS), nuh_layer_id: 0, temporal_id: 0
nal_unit_type: 34(PPS), nuh_layer_id: 0, temporal_id: 0
nal_unit_type: 39(SEI_PREFIX), nuh_layer_id: 0, temporal_id: 0
nal_unit_type: 19(IDR_W_RADL), nuh_layer_id: 0, temporal_id: 0
Decoding VPS
Main profile bitstream
Decoding SPS
Main profile bitstream
Decoding VUI
Decoding PPS
Decoding SEI
Skipped PREFIX SEI 5
Decoded frame with POC 0.
nal_unit_type: 35(AUD), nuh_layer_id: 0, temporal_id: 0
...
All info found
After avformat_find_stream_info() pos: 372 bytes read:372 seeks:0 frames:44
max_frame_duration: 10.000
Input #0, hls,applehttp, from 'http://demo.polyv.net/h5/265/01.m3u8':
  Duration: 00:01:30.96, start: 1.436889, bitrate: 0 kb/s
  Program 0 
    Metadata:
      variant_bitrate : 0
    Stream #0:0, 21, 1/90000: Video: hevc, 1 reference frame ([36][0][0][0] / 0x0024), yuv420p(tv), 960x540 (960x544), 0/1, 24 fps, 24 tbr, 90k tbn, 24 tbc2018-10-18 19:18:19.407661+0800 IJKMediaDemo[4023:1410760] FFP_MSG_FIND_STREAM_INFO:

    Metadata:
      variant_bitrate : 0
    Stream #0:1, 23, 1/90000: Audio: aac ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp
    Metadata:
      variant_bitrate : 0
aout_open_audio()
AudioCodec: avcodec, aac
aout_pause_audio(0)
2018-10-18 19:18:19.487403+0800 IJKMediaDemo[4023:1410760] FFP_MSG_AUDIO_RENDERING_START:
detected 6 logical cores
nal_unit_type: 32(VPS), nuh_layer_id: 0, temporal_id: 0
nal_unit_type: 33(SPS), nuh_layer_id: 0, temporal_id: 0
nal_unit_type: 34(PPS), nuh_layer_id: 0, temporal_id: 0
Decoding VPS
Main profile bitstream
Decoding SPS
Main profile bitstream
Decoding VUI
Decoding PPS
2018-10-18 19:18:19.796378+0800 IJKMediaDemo[4023:1411082] AVDRegister - AppleAVD HEVC codec registered
could not init video tool box decoder !!!vtb fail!!! switch to ffmpeg decode!!!! 
VideoCodec: avcodec, hevc
fps: 24.000000 (normal)
fps: 24.000000 (normal)
ijkmp_get_msg: FFP_MSG_PREPARED
2018-10-18 19:18:19.847164+0800 IJKMediaDemo[4023:1410760] FFP_MSG_VIDEO_DECODER_OPEN: false
2018-10-18 19:18:19.847315+0800 IJKMediaDemo[4023:1410760] FFP_MSG_COMPONENT_OPEN:
2018-10-18 19:18:19.847452+0800 IJKMediaDemo[4023:1410760] FFP_MSG_VIDEO_SIZE_CHANGED: 960, 540
nal_unit_type: 35(AUD), nuh_layer_id: 0, temporal_id: 0
nal_unit_type: 0(TRAIL_N), nuh_layer_id: 0, temporal_id: 0
2018-10-18 19:18:19.847651+0800 IJKMediaDemo[4023:1410760] FFP_MSG_SAR_CHANGED: 0, 1
2018-10-18 19:18:19.847758+0800 IJKMediaDemo[4023:1410760] FFP_MSG_PREPARED:
ijkmp_get_meta_l
ijkmp_get_meta_l()=void
2018-10-18 19:18:19.847983+0800 IJKMediaDemo[4023:1410760] fps in meta 24.000000
Sunny-Pro commented 5 years ago

@cupidfantasy 发现即便走了validate_avcC_spc，也有可能获取不正确的ref_frame,导致丢帧，请问解决了吗，是否需要单独去获取HEVC编码的ref_frame,264的sps解析函数可能不适用265

cupidfantasy commented 5 years ago

@JerryIsAgoodBoy 是不通用，要重写解析部分了，后来简单处理给了个稍微大一点的max_ref_frames，至今工作正常。这个值最大也不会超过5吧。

Sunny-Pro commented 5 years ago

@cupidfantasy 好的，谢谢

hu5980 commented 4 years ago

@cupidfantasy 你好 请问你的有没有出现 “libavformat/hevc.h” 文件找不到

cupidfantasy commented 4 years ago

@hu5980 修改libavformat的Makefile，把hevc.h加到HEADERS里

heberthe commented 4 years ago

今天发现ios硬解h265 NALU失败，看了代码发现在vtbformat_init中对265从annexB=>mp4转换时需要使用ff_isom_write_hvcc，而不能重用264的ff_isom_write_avcc。

        if ((extradata[0] == 0 && extradata[1] == 0 && extradata[2] == 0 && extradata[3] == 1) ||
            (extradata[0] == 0 && extradata[1] == 0 && extradata[2] == 1)) {
            AVIOContext *pb;
            if (avio_open_dyn_buf(&pb) < 0) {
                goto fail;
            }

            fmt_desc->convert_bytestream = true;

            if (codec == AV_CODEC_ID_HEVC) {
                ff_isom_write_hvcc(pb, extradata, extrasize, 1); //hvcc isom is different from avcc
            } else {
                ff_isom_write_avcc(pb, extradata, extrasize);
            }

            extradata = NULL;
            extrasize = avio_close_dyn_buf(pb, &extradata);

另外发现喂H265数据时，如果带有HEVC_NAL_AUD帧会导致videotoolbox解码失败。

@cupidfantasy 你好，请问你是如何扩展ff_isom_write_hvcc 方法的呢？

cupidfantasy commented 4 years ago

@heberthe ff_isom_write_hvcc是ffmpeg内置函数，把hevc.h开放出来就可以了

heberthe commented 4 years ago

@cupidfantasy 好的，谢谢！已经找到了，但是发现还是会有 H265 的硬解失败，给max_ref_frames 大点的值也不行。

cupidfantasy commented 4 years ago

@heberthe AUD帧过滤掉了吗，之前发现AUD送给硬解码器之后就报错，过滤掉就好了，至于为什么也没继续研究

359796875 commented 4 years ago

@cupidfantasy 您好，请教一下，修改libavformat的Makefile，把hevc.h加到HEADERS里，ff_isom_write_hvcc这个方法编译的的时候报错，Undefined symbols for architecture arm64:"_ff_isom_write_hvcc"，referenced from: _vtbformat_init in IJKMediaFramework(IJKVideoToolBoxSync.o)，有另外需要修改的地方么？谢谢