关于不能直播FLV流的理解

xuanjinliang commented 7 years ago

研究了FLVJS的原码一段时间，发现在360安全浏览器等其它chrome内核浏览器，对FLV直播的视频只能播放第一次加载的数据，后续都是卡死，但chrome的最新版却能播放，原因分析：

1、通过fetch获取视频流，作者通过把FLV视频流转换为浏览器识别的流后，用MediaSource创建视频流并且用appendBuffer()，不断的添加视频流；

2、理解后其原理，进行分析，为什么旧版的chrome只能播放第一次加载的数据但新版本的chrome却能播放，解码出问题了？我觉得不对，因为两个浏览器都是用Uint8Array处理二进制数据，所以数据处理方式是一样的，所以猜测appendBuffer出问题；

3、跟据chrome的官方文档，有说明，只有chrome50以上的版本才有sourceBuffer.mode的方法，可以进行 addSourceBuffer 不断的添加，低版本浏览器只能播放第一次加载的数据；

4、而为什么却youtube可以实现了？个人理解youtube用的是点播方式，不断的轮循获取二进制的视频流进行播放（因为我看不到有用websocket或者SSE使用，有同学看到请指正），每次取一点点数据，不断的添加新的MediaSource二进制流进行播放，但这种方式会对视频出现延迟，因为服务器要对视频进行切片，所以对实时直播要求高的视频是不适用的；

以上都是个人的理解，如有误，请个位大神指正，谢谢；

xqq commented 7 years ago

频繁执行的是 appendBuffer 而不是 addSourceBuffer addSourceBuffer() 一般只会执行两次，分别对应音视频轨道

只能播放第一次加载的数据是啥？

xuanjinliang commented 7 years ago

@xqq 对的，是appendBuffer，我贴错代码，发现当fetch获取回来的数据后，只能播放第一次获取的视频流，一但数据更新，用appendBuffer再次添加流，视频卡住不动，如果视频是有大小的（不是直播，回放之类），在chrome 50以下也能正常播放；

xqq commented 7 years ago

第一次获取的是啥？数据更新是啥？

xuanjinliang commented 7 years ago

好吧，我详细点说，当fetch链接成功后，会首次返回二进制的流，后续就不断的等待流服务器的推送（其实就是建立一个长链接），当首次获取到流数据进行解码，转为Uint8Array通用流，然后分块push(appendBuffer)到this._sourceBuffers对象中的video与audio中，此时为第一次获取到流，后续当服务器再推送流过来时，还是一样的操作push到this._sourceBuffers对象中，等浏览器自动加载播放这些流，可惜后续添加进来的流在chrome 50以下的都不能播放；

以上是我的理解，有误请指正，谢谢；

xqq commented 7 years ago

你的理解有误。

不能播放的具体表现是？提供 chrome://media-internals 下的日志？

xuanjinliang commented 7 years ago

Player Properties

    render_id: 387
    player_id: 0
    pipeline_state: kStopped
    event: PLAY
    url: blob:http%3A//127.0.0.1%3A8000/721f2b2b-0578-47c0-a582-a029a3b7f4de
    info: Audio codec: mp4a.40.2
    found_video_stream: true
    video_codec_name: h264
    duration: unknown
    found_audio_stream: true
    audio_codec_name: aac
    audio_dds: false
    audio_decoder: FFmpegAudioDecoder
    video_dds: false
    video_decoder: FFmpegVideoDecoder
    seek_target: 0.05999999865889549
    error: video decode error
    pipeline_error: pipeline: decode error

log

    00:00:00 00 pipeline_state  kCreated
    00:00:00 00 event   WEBMEDIAPLAYER_CREATED
    00:00:00 00 url blob:http%3A//127.0.0.1%3A8000/721f2b2b-0578-47c0-a582-a029a3b7f4de
    00:00:00 01 pipeline_state  kInitDemuxer
    00:00:01 39 info    Video codec: avc1.64016
    00:00:01 39 found_video_stream  true
    00:00:01 39 video_codec_name    h264
    00:00:01 39 duration    unknown
    00:00:01 39 info    Audio codec: mp4a.40.2
    00:00:01 39 found_audio_stream  true
    00:00:01 39 audio_codec_name    aac
    00:00:01 39 pipeline_state  kInitRenderer
    00:00:01 41 audio_dds   false
    00:00:01 41 audio_decoder   FFmpegAudioDecoder
    00:00:01 43 video_dds   false
    00:00:01 43 video_decoder   FFmpegVideoDecoder
    00:00:01 43 pipeline_state  kPlaying
    00:00:01 49 seek_target 0.05999999865889549
    00:00:01 50 pipeline_state  kSeeking
    00:00:01 50 pipeline_state  kPlaying
    00:00:01 58 event   PLAY
    00:00:05 288    error   video decode error
    00:00:05 338    pipeline_error  pipeline: decode error
    00:00:05 338    pipeline_state  kStopping
    00:00:05 339    pipeline_state  kStopped

xqq commented 7 years ago

decode error 解码错误

尝试其它视频流/更换推流端？

xuanjinliang commented 7 years ago

这就是问题所在，当一个视频是有大小的（从头到尾都可以播放），一个视频是直播的（却只能播放几秒钟），两个视频流一样的......

xqq commented 7 years ago

直播流的 duration 天然为 0 ，并且浏览器也会因此判断该流为实时流。

能否提供用于测试的样本或在线流

songguangyu commented 7 years ago

我也同样遇到了在点播一个flv视频时候，只能加载第一段当buffer 满了之后后续文件便不会加载，seek 一下就会出现bug，代码还没研究透，不是很了解具体是咋回事，文件是1.5G的flv文件。

xqq commented 7 years ago

@songguangyu console报错信息？服务端是否支持range请求？是否为跨域请求，是否包含301/302跳转？

songguangyu commented 7 years ago

console.log 没报错，服务端支持range 我是在本地127.0.0.1调试的第一个fetch 请求发出去之后 seek 一下就没有后续的请求继续发出了

Moran349 commented 7 years ago

@xqq 在flv.js对接直播流时，我也遇到了和 @xuanjinliang 相同的问题，即chrome只能加载第一个I帧图像，然后就没有反映了。在mse-controller中添加日志分析得知，appendMediaSegment和_doAppendSegments被正常调用。 chrome://media-internals的日志如下：还有您之前提到直播流的duration天然为0，而在日志中duration为unknown，是否是这个原因？该从什么方面着手解决呢？而且日志中出现不断丢audio帧又是什么原因导致的呢？

xqq commented 7 years ago

duration为0的流，chrome内部将其视为live stream，log输出为unknown，API提供Infinity，意义是一样的；其它原因不明，建议扒一段FLV文件用FlvBugger手工分析。

Moran349 commented 7 years ago

@xqq 最近在做直播的时候发现，正常的10分钟视频经过flv.js直播后会多出20秒左右才播完。初步估计是cts、dts以及pts在写入时精度偏差，想请教一下对于audio和video来说cts、dts、pts具体该怎样设置，或者说他们有什么具体的含义（浏览代码时发现audio并没有cts），望不吝赐教？补充一下：音视频流的时间戳是从私有码流中获取的并不是取自flv文件格式中

xqq commented 7 years ago

音频没有类似h264的B帧向后参考，PTS和DTS相等

liekkas commented 5 years ago

@xqq 使用官方的例子，播flv直播流，没报错，但也没图形一直在转圈。帮忙看下！谢谢。

这是自带的console日志。

[MSEController] > MediaSource onSourceOpen
[FLVDemuxer] > Parsed onMetaData
[FLVDemuxer] > Parsed AVCDecoderConfigurationRecord
[MSEController] > Received Initialization Segment, mimeType: video/mp4;codecs=avc1.42c01f

这是chrome://media-internals/的日志。

{
  "8:0": {
    "id": "8:0",
    "properties": {
      "render_id": 8,
      "player_id": 0,
      "origin_url": "chrome-extension://oeopbcgkkoapgobdbedcemjljbihmemj/",
      "frame_url": "chrome-extension://oeopbcgkkoapgobdbedcemjljbihmemj/background.html",
      "frame_title": "Checker Plus for Gmail - Background page",
      "url": "chrome-extension://oeopbcgkkoapgobdbedcemjljbihmemj/sounds/chime.ogg",
      "total_bytes": 12708,
      "streaming": false,
      "single_origin": true,
      "passed_cors_access_check": false,
      "range_header_supported": true,
      "pipeline_state": "kSuspended",
      "info": "Effective playback rate changed from 0 to 1",
      "audio_channels_count": 1,
      "audio_codec_name": "vorbis",
      "audio_sample_format": "Float 32-bit planar",
      "audio_samples_per_second": 44100,
      "bitrate": 162024,
      "found_audio_stream": true,
      "found_video_stream": false,
      "max_duration": 0.62746,
      "start_time": -0.002902,
      "audio_dds": false,
      "audio_decoder": "FFmpegAudioDecoder",
      "is_platform_audio_decoder": false,
      "audio_buffering_state": "BUFFERING_HAVE_ENOUGH",
      "debug": "FFmpegDemuxer: av_read_frame(): End of file",
      "for_suspended_start": false,
      "pipeline_buffering_state": "BUFFERING_HAVE_ENOUGH",
      "event": "PAUSE",
      "duration": 0.62746
    },
    "allEvents": [
      {
        "time": 0,
        "key": "origin_url",
        "value": "chrome-extension://oeopbcgkkoapgobdbedcemjljbihmemj/"
      },
      {
        "time": 0.020999997854232788,
        "key": "frame_url",
        "value": "chrome-extension://oeopbcgkkoapgobdbedcemjljbihmemj/background.html"
      },
      {
        "time": 0.027000010013580322,
        "key": "frame_title",
        "value": "Checker Plus for Gmail - Background page"
      },
      {
        "time": 0.25300000607967377,
        "key": "url",
        "value": "chrome-extension://oeopbcgkkoapgobdbedcemjljbihmemj/sounds/chime.ogg"
      },
      {
        "time": 350.90000000596046,
        "key": "total_bytes",
        "value": 12708
      },
      {
        "time": 350.91400000452995,
        "key": "streaming",
        "value": false
      },
      {
        "time": 350.9350000023842,
        "key": "single_origin",
        "value": true
      },
      {
        "time": 350.93800000846386,
        "key": "passed_cors_access_check",
        "value": false
      },
      {
        "time": 350.9399999976158,
        "key": "range_header_supported",
        "value": true
      },
      {
        "time": 367.6330000013113,
        "key": "pipeline_state",
        "value": "kStarting"
      },
      {
        "time": 369.12900000810623,
        "key": "info",
        "value": "FFmpegDemuxer: created audio stream, config codec: vorbis bytes_per_channel: 4 channel_layout: 2 channels: 1 samples_per_second: 44100 sample_format: 6 bytes_per_frame: 4 seek_preroll: 0ms codec_delay: 0 has extra data? true encryption scheme: Unencrypted discard decoder delay? true"
      },
      {
        "time": 369.27800001204014,
        "key": "audio_channels_count",
        "value": 1
      },
      {
        "time": 369.27800001204014,
        "key": "audio_codec_name",
        "value": "vorbis"
      },
      {
        "time": 369.27800001204014,
        "key": "audio_sample_format",
        "value": "Float 32-bit planar"
      },
      {
        "time": 369.27800001204014,
        "key": "audio_samples_per_second",
        "value": 44100
      },
      {
        "time": 369.27800001204014,
        "key": "bitrate",
        "value": 162024
      },
      {
        "time": 369.27800001204014,
        "key": "found_audio_stream",
        "value": true
      },
      {
        "time": 369.27800001204014,
        "key": "found_video_stream",
        "value": false
      },
      {
        "time": 369.27800001204014,
        "key": "max_duration",
        "value": 0.62746
      },
      {
        "time": 369.27800001204014,
        "key": "start_time",
        "value": -0.002902
      },
      {
        "time": 516.6520000100136,
        "key": "audio_dds",
        "value": false
      },
      {
        "time": 516.6590000092983,
        "key": "audio_decoder",
        "value": "FFmpegAudioDecoder"
      },
      {
        "time": 516.6609999984503,
        "key": "is_platform_audio_decoder",
        "value": false
      },
      {
        "time": 516.6930000036955,
        "key": "info",
        "value": "Selected FFmpegAudioDecoder for audio decoding, config: codec: vorbis bytes_per_channel: 4 channel_layout: 2 channels: 1 samples_per_second: 44100 sample_format: 6 bytes_per_frame: 4 seek_preroll: 0ms codec_delay: 0 has extra data? true encryption scheme: Unencrypted discard decoder delay? true"
      },
      {
        "time": 516.832000002265,
        "key": "pipeline_state",
        "value": "kPlaying"
      },
      {
        "time": 519.6239999979734,
        "key": "audio_buffering_state",
        "value": "BUFFERING_HAVE_ENOUGH"
      },
      {
        "time": 520.5729999989271,
        "key": "debug",
        "value": "FFmpegDemuxer: av_read_frame(): End of file"
      },
      {
        "time": 527.1979999989271,
        "key": "for_suspended_start",
        "value": false
      },
      {
        "time": 527.1979999989271,
        "key": "pipeline_buffering_state",
        "value": "BUFFERING_HAVE_ENOUGH"
      },
      {
        "time": 527.3210000097752,
        "key": "info",
        "value": "Effective playback rate changed from 0 to 1"
      },
      {
        "time": 527.6420000046492,
        "key": "event",
        "value": "PLAY"
      },
      {
        "time": 369.24800001084805,
        "key": "duration",
        "value": 0.62746
      },
      {
        "time": 1257.2530000060797,
        "key": "event",
        "value": "ENDED"
      },
      {
        "time": 1257.6520000100136,
        "key": "event",
        "value": "PAUSE"
      },
      {
        "time": 16267.068000003695,
        "key": "pipeline_state",
        "value": "kSuspending"
      },
      {
        "time": 16267.689999997616,
        "key": "pipeline_state",
        "value": "kSuspended"
      }
    ],
    "lastRendered": 0,
    "firstTimestamp_": 86763292.983
  },
  "21:0": {
    "id": "21:0",
    "properties": {
      "render_id": 21,
      "player_id": 0,
      "pipeline_state": "kStopped",
      "event": "WEBMEDIAPLAYER_DESTROYED"
    },
    "allEvents": [
      {
        "time": 0,
        "key": "pipeline_state",
        "value": "kStopping"
      },
      {
        "time": 1.63400000333786,
        "key": "pipeline_state",
        "value": "kStopped"
      },
      {
        "time": 2.6019999980926514,
        "key": "event",
        "value": "WEBMEDIAPLAYER_DESTROYED"
      }
    ],
    "lastRendered": 0,
    "firstTimestamp_": 90921773.33,
    "destructed": true
  },
  "190:19": {
    "id": "190:19",
    "properties": {
      "render_id": 190,
      "player_id": 19,
      "pipeline_state": "kStopped",
      "event": "WEBMEDIAPLAYER_DESTROYED"
    },
    "allEvents": [
      {
        "time": 0,
        "key": "pipeline_state",
        "value": "kStopping"
      },
      {
        "time": 0.031999990344047546,
        "key": "pipeline_state",
        "value": "kStopped"
      },
      {
        "time": 0.07199999690055847,
        "key": "event",
        "value": "WEBMEDIAPLAYER_DESTROYED"
      }
    ],
    "lastRendered": 0,
    "firstTimestamp_": 114644273.812,
    "destructed": true
  },
  "205:1": {
    "id": "205:1",
    "properties": {
      "render_id": 205,
      "player_id": 1,
      "origin_url": "http://localhost:8080/",
      "frame_url": "http://localhost:8080/lpgs/test.html",
      "frame_title": "flv.js demo (v1.4.2)",
      "url": "blob:http://localhost:8080/0486f4e7-3f98-470b-a216-f2dd7b1de473",
      "info": "ChunkDemuxer: buffering by DTS",
      "pipeline_state": "kStopped",
      "event": "WEBMEDIAPLAYER_DESTROYED"
    },
    "allEvents": [
      {
        "time": 0,
        "key": "origin_url",
        "value": "http://localhost:8080/"
      },
      {
        "time": 0.012999996542930603,
        "key": "frame_url",
        "value": "http://localhost:8080/lpgs/test.html"
      },
      {
        "time": 0.017000004649162292,
        "key": "frame_title",
        "value": "flv.js demo (v1.4.2)"
      },
      {
        "time": 0.2290000021457672,
        "key": "url",
        "value": "blob:http://localhost:8080/0486f4e7-3f98-470b-a216-f2dd7b1de473"
      },
      {
        "time": 0.2979999929666519,
        "key": "info",
        "value": "ChunkDemuxer: buffering by DTS"
      },
      {
        "time": 2.741999998688698,
        "key": "pipeline_state",
        "value": "kStarting"
      },
      {
        "time": 629821.1069999933,
        "key": "pipeline_state",
        "value": "kStopping"
      },
      {
        "time": 629821.1369999945,
        "key": "pipeline_state",
        "value": "kStopped"
      },
      {
        "time": 629821.2290000021,
        "key": "event",
        "value": "WEBMEDIAPLAYER_DESTROYED"
      }
    ],
    "lastRendered": 0,
    "firstTimestamp_": 114648806.603,
    "destructed": true
  },
  "205:4": {
    "id": "205:4",
    "properties": {
      "render_id": 205,
      "player_id": 4,
      "origin_url": "http://localhost:8080/",
      "frame_url": "http://localhost:8080/lpgs/test.html",
      "frame_title": "flv.js demo (v1.4.2)",
      "url": "blob:http://localhost:8080/c24ae213-cc84-4d24-86e9-6349a52bd0e3",
      "info": "ChunkDemuxer: buffering by DTS",
      "pipeline_state": "kStarting"
    },
    "allEvents": [
      {
        "time": 0,
        "key": "origin_url",
        "value": "http://localhost:8080/"
      },
      {
        "time": 0.009000003337860107,
        "key": "frame_url",
        "value": "http://localhost:8080/lpgs/test.html"
      },
      {
        "time": 0.013999998569488525,
        "key": "frame_title",
        "value": "flv.js demo (v1.4.2)"
      },
      {
        "time": 0.08699999749660492,
        "key": "url",
        "value": "blob:http://localhost:8080/c24ae213-cc84-4d24-86e9-6349a52bd0e3"
      },
      {
        "time": 0.12700000405311584,
        "key": "info",
        "value": "ChunkDemuxer: buffering by DTS"
      },
      {
        "time": 0.15600000321865082,
        "key": "pipeline_state",
        "value": "kStarting"
      }
    ],
    "lastRendered": 0,
    "firstTimestamp_": 115278776.7
  }
}

xqq commented 5 years ago

没音频

liekkas commented 5 years ago

@xqq 谢谢了

bilibili / flv.js

关于不能直播FLV流的理解 #84

Player Properties

log