search

kingslay / KSPlayer

A video player for iOS、macOS、tvOS、visionOS , based on AVPlayer and FFmpeg, support the horizontal, vertical screen. support adjust volume, brightness and seek by slide, SwiftUI, support subtitles.
https://apps.apple.com/app/tracyplayer/id6450770064
GNU General Public License v3.0
955 stars 193 forks source link

Dropped Frames #369

Closed UnknownCoder807 closed 1 year ago

UnknownCoder807 commented 1 year ago

@kingslay I've emailed you about this and provided some test stream URLs.

It has been brought to my attention by many of my customers that they are experiencing continuous dropped frames. They are getting stuttering, lag, speed up and slow downs. As a result the audio goes out of sync.

To rule out my application I have tested with your TracyPlayer demo in both SwiftUI and UKit versions and get this issue. I also tested with a basic SwiftUI ContentView with a simple NavigationLink to KSVideoPlayerView() and get this issue.

This happens on iPhone or tvOS.

The worst part is, a stream might show these dropped frames, then a few mins later play fine.

I have tried 5 different stream providers and they all have the same problems. (different channels) and my customers are unlikely to be using the same providers as me.

Please look into this as I believe it's a real issue that will affect others.

KSPlayer: KSPlayerLayer.swift:149 state | playerStateDidChange - bufferFinished
KSPlayer: MEPlayerItem.swift:701 getVideoOutputRender(force:) | video delay time: 0.4011555555553059 dropped video frame frameCount: 15 frameMaxCount: 16
KSPlayer: MEPlayerItem.swift:701 getVideoOutputRender(force:) | video delay time: 0.40209999999933643 dropped video frame frameCount: 15 frameMaxCount: 16
KSPlayer: MEPlayerItem.swift:701 getVideoOutputRender(force:) | video delay time: 0.40516666666735546 dropped video frame frameCount: 10 frameMaxCount: 16
KSPlayer: MEPlayerItem.swift:701 getVideoOutputRender(force:) | video delay time: 0.4030222222208977 dropped video frame frameCount: 15 frameMaxCount: 16
UnknownCoder807 commented 1 year ago

With debug logs enabled I get

KSPlayer: MEPlayerItem.swift:127 init(url:options:) | nal_unit_type: 6(SEI), nal_ref_idc: 0

KSPlayer: MEPlayerItem.swift:127 init(url:options:) | nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 0

KSPlayer: MEPlayerItem.swift:127 init(url:options:) | ct_type:0 pic_struct:0

KSPlayer: MEPlayerItem.swift:701 getVideoOutputRender(force:) | video delay time: 0.4059222222222161 dropped video frame frameCount: 12 frameMaxCount: 16
KSPlayer: MEPlayerItem.swift:701 getVideoOutputRender(force:) | video delay time: 0.4037777777775773 dropped video frame frameCount: 8 frameMaxCount: 16
KSPlayer: MEPlayerItem.swift:127 init(url:options:) | nal_unit_type: 9(AUD), nal_ref_idc: 0
UnknownCoder807 commented 1 year ago

@kingslay I sent you a screen recording of my app + xcode showing the dropped frames in a related issue to this one. It shows 2 videos. first video has the frame drops. the second video doesn't have the issue. I provided both video links. maybe the encoding of the video is the issue.

JeremTM93 commented 1 year ago

I have these problems on all streams except 50fps. VOD in 23.976fps has "jerks". Channels in 50i (25fps) are not smooth.

Sometimes the stream speeds up, slows down and finally stabilises, other times you have to restart the stream.

I also confirm a random shift between sound and image.

kingslay commented 1 year ago

@UnknownCoder807 你可以试下 options.dropVideoFrame = false 看下效果怎么样

UnknownCoder807 commented 1 year ago

@UnknownCoder807 你可以试下 options.dropVideoFrame = false 看下效果怎么样

Sadly makes no difference.

UnknownCoder807 commented 1 year ago

Sorry, not fixed. sent you an email with the video stream link.

kingslay commented 1 year ago

Sorry, not fixed. sent you an email with the video stream link.

我在tvOS上试了。不会有delay。你那边delay的时间是多少呢？还是0.4s吗？

UnknownCoder807 commented 1 year ago

Sorry, not fixed. sent you an email with the video stream link.

我在tvOS上试了。不会有delay。你那边delay的时间是多少呢？还是0.4s吗？

i don't think this is translating properly for me. I don't understand what you mean.

kingslay commented 1 year ago

你把日志发给我看下

UnknownCoder807 commented 1 year ago 
KSPlayer: KSMEPlayer.swift:331 prepareToPlay() | prepareToPlay <KSPlayer.KSMEPlayer: 0x60000351a7c0>
KSPlayer: KSPlayerLayer.swift:137 state | playerStateDidChange - buffering
KSPlayer: MEPlayerItem.swift:127 init(url:options:) | non-existing SPS 0 referenced in buffering period

KSPlayer: MEPlayerItem.swift:127 init(url:options:) | non-existing SPS 0 referenced in buffering period

2023-01-30 14:16:56.409089+0000 xxxxxxxxx[42720:3945460] [plugin] AddInstanceForFactory: No factory registered for id <CFUUID 0x6000000b3220> F8BB1C28-BAE8-11D6-9C31-00039315CD46
KSPlayer: AVFFmpegExtension.swift:396 layoutTag | FFmepg channelLayout: stereo order: AVChannelOrder(rawValue: 1)
KSPlayer: AudioEnginePlayer.swift:150 prepare(options:audioDescriptor:) | outputFormat channelLayout AudioFormat: <AVAudioFormat 0x6000023dee90:  2 ch,  44100 Hz, Float32, deinterleaved>
KSPlayer: AudioEnginePlayer.swift:152 prepare(options:audioDescriptor:) | outputFormat channelLayout tag: 6619138
KSPlayer: AudioEnginePlayer.swift:153 prepare(options:audioDescriptor:) | outputFormat channelLayout channelDescriptions: []
KSPlayer: AVFoundationExtension.swift:135 channelLayout() | KSOptions channelLayout: AudioChannelLayout(mChannelLayoutTag: 6619138, mChannelBitmap: __C.AudioChannelBitmap(rawValue: 0), mNumberChannelDescriptions: 0, mChannelDescriptions: __C.AudioChannelDescription(mChannelLabel: 0, mChannelFlags: __C.AudioChannelFlags(rawValue: 1), mCoordinates: (0.0, 1.9783765e-05, 1e-45)))
KSPlayer: AVFoundationExtension.swift:161 channelLayout() | out channelLayout mask: 3 nb_channels: 2
KSPlayer: Resample.swift:193 init(audioDescriptor:audioFormat:) | out channelLayout: stereo
KSPlayer: KSPlayerLayer.swift:137 state | playerStateDidChange - readyToPlay
KSPlayer: MEPlayerItem.swift:127 init(url:options:) | non-existing SPS 0 referenced in buffering period

KSPlayer: MEPlayerItem.swift:127 init(url:options:) | Failed setup for format videotoolbox_vld: hwaccel initialisation returned error.

KSPlayer: KSMEPlayer.swift:338 play() | play <KSPlayer.KSMEPlayer: 0x60000351a7c0>
KSPlayer: KSPlayerLayer.swift:137 state | playerStateDidChange - buffering
KSPlayer: KSMEPlayer.swift:338 play() | play <KSPlayer.KSMEPlayer: 0x60000351a7c0>
KSPlayer: KSPlayerLayer.swift:358 changeLoadState(player:) | ["openTime": 1.8353903333481867, "dnsTime": 2.17916676774621e-05, "tcpTime": 0.1407597083307337, "readyTime": 0.003968750010244548, "readVideoTime": 0.166235583339585, "readAudioTime": 0.16627637500641868, "firstTime": 3.426528000010876, "initTime": 0.004433249996509403, "decodeVideoTime": 0.07624020832008682, "findTime": 0.8704921249882318, "decodeAudioTime": 0.00235966665786691]
KSPlayer: KSPlayerLayer.swift:137 state | playerStateDidChange - bufferFinished
KSPlayer: MEPlayerItem.swift:701 getVideoOutputRender(force:) | video delay time: 1.575077777777551 dropped video frame frameCount: 31 frameMaxCount: 32
KSPlayer: MEPlayerItem.swift:701 getVideoOutputRender(force:) | video delay time: 2.9701111111116916 dropped video frame frameCount: 29 frameMaxCount: 32
KSPlayer: MEPlayerItem.swift:701 getVideoOutputRender(force:) | video delay time: 2.970944444444285 dropped video frame frameCount: 27 frameMaxCount: 32
KSPlayer: MEPlayerItem.swift:701 getVideoOutputRender(force:) | video delay time: 2.972777777777992 dropped video frame frameCount: 25 frameMaxCount: 32
KSPlayer: MEPlayerItem.swift:701 getVideoOutputRender(force:) | video delay time: 2.9687999999996464 dropped video frame frameCount: 23 frameMaxCount: 32
KSPlayer: MEPlayerItem.swift:701 getVideoOutputRender(force:) | video delay time: 2.9754333333339673 dropped video frame frameCount: 21 frameMaxCount: 32
KSPlayer: MEPlayerItem.swift:701 getVideoOutputRender(force:) | video delay time: 2.9772666666667646 dropped video frame frameCount: 19 frameMaxCount: 32
KSPlayer: MEPlayerItem.swift:701 getVideoOutputRender(force:) | video delay time: 2.9723000000003594 dropped video frame frameCount: 17 frameMaxCount: 32
KSPlayer: MEPlayerItem.swift:701 getVideoOutputRender(force:) | video delay time: 2.962511111110871 dropped video frame frameCount: 31 frameMaxCount: 32
KSPlayer: MEPlayerItem.swift:701 getVideoOutputRender(force:) | video delay time: 2.941111111110331 dropped video frame frameCount: 29 frameMaxCount: 32
KSPlayer: MEPlayerItem.swift:701 getVideoOutputRender(force:) | video delay time: 2.918733333332966 dropped video frame frameCount: 27 frameMaxCount: 32
KSPlayer: MEPlayerItem.swift:701 getVideoOutputRender(force:) | video delay time: 2.9031444444435692 dropped video frame frameCount: 25 frameMaxCount: 32

Repeated …
UnknownCoder807 commented 1 year ago 
2023-01-30 14:21:09.788588+0000 test ksplayer playback ui[42921:3949270] Metal GPU Frame Capture Enabled
2023-01-30 14:21:09.788694+0000 test ksplayer playback ui[42921:3949270] Metal API Validation Enabled
KSPlayer: KSMEPlayer.swift:331 prepareToPlay() | prepareToPlay <KSPlayer.KSMEPlayer: 0x600001f58f00>
KSPlayer: KSPlayerLayer.swift:137 state | playerStateDidChange - buffering
KSPlayer: MEPlayerItem.swift:127 init(url:options:) | non-existing SPS 0 referenced in buffering period

KSPlayer: MEPlayerItem.swift:127 init(url:options:) | non-existing SPS 0 referenced in buffering period

2023-01-30 14:21:13.548071+0000 test ksplayer playback ui[42921:3949428] [plugin] AddInstanceForFactory: No factory registered for id <CFUUID 0x600002ac3b60> F8BB1C28-BAE8-11D6-9C31-00039315CD46
KSPlayer: AVFFmpegExtension.swift:396 layoutTag | FFmepg channelLayout: stereo order: AVChannelOrder(rawValue: 1)
KSPlayer: AudioEnginePlayer.swift:150 prepare(options:audioDescriptor:) | outputFormat channelLayout AudioFormat: <AVAudioFormat 0x60000090eb20:  2 ch,  44100 Hz, Float32, deinterleaved>
KSPlayer: AudioEnginePlayer.swift:152 prepare(options:audioDescriptor:) | outputFormat channelLayout tag: 6619138
KSPlayer: AudioEnginePlayer.swift:153 prepare(options:audioDescriptor:) | outputFormat channelLayout channelDescriptions: []
KSPlayer: AVFoundationExtension.swift:135 channelLayout() | KSOptions channelLayout: AudioChannelLayout(mChannelLayoutTag: 6619138, mChannelBitmap: __C.AudioChannelBitmap(rawValue: 0), mNumberChannelDescriptions: 0, mChannelDescriptions: __C.AudioChannelDescription(mChannelLabel: 0, mChannelFlags: __C.AudioChannelFlags(rawValue: 9500656), mCoordinates: (3.4438e-41, 0.0, 0.0)))
KSPlayer: AVFoundationExtension.swift:161 channelLayout() | out channelLayout mask: 3 nb_channels: 2
KSPlayer: Resample.swift:193 init(audioDescriptor:audioFormat:) | out channelLayout: stereo
KSPlayer: KSPlayerLayer.swift:137 state | playerStateDidChange - readyToPlay
KSPlayer: MEPlayerItem.swift:127 init(url:options:) | non-existing SPS 0 referenced in buffering period

KSPlayer: MEPlayerItem.swift:127 init(url:options:) | Failed setup for format videotoolbox_vld: hwaccel initialisation returned error.

KSPlayer: KSMEPlayer.swift:338 play() | play <KSPlayer.KSMEPlayer: 0x600001f58f00>
KSPlayer: KSPlayerLayer.swift:137 state | playerStateDidChange - buffering
KSPlayer: KSMEPlayer.swift:338 play() | play <KSPlayer.KSMEPlayer: 0x600001f58f00>
KSPlayer: KSPlayerLayer.swift:358 changeLoadState(player:) | ["firstTime": 4.224358458333882, "tcpTime": 0.14004233333980665, "findTime": 0.8823624583310448, "initTime": 0.019388999993680045, "readyTime": 0.0037846250052098185, "decodeAudioTime": 0.0023315833532251418, "readVideoTime": 0.16894470833358355, "openTime": 2.6069333749765065, "decodeVideoTime": 0.09067195834359154, "readAudioTime": 0.16897895833244547, "dnsTime": 1.320833689533174e-05]
KSPlayer: KSPlayerLayer.swift:137 state | playerStateDidChange - bufferFinished
KSPlayer: MEPlayerItem.swift:127 init(url:options:) | Stream ends prematurely at 30376460, should be 18446744073709551615

KSPlayer: MEPlayerItem.swift:127 init(url:options:) | Will reconnect at 30376460 in 0 second(s), error=Input/output error.

KSPlayer: MEPlayerItem.swift:127 init(url:options:) | Packet corrupt (stream = 0, dts = 538532521)
KSPlayer: MEPlayerItem.swift:127 init(url:options:) | .

KSPlayer: MEPlayerItem.swift:127 init(url:options:) | Stream ends prematurely at 15217296, should be 18446744073709551615

KSPlayer: MEPlayerItem.swift:127 init(url:options:) | Will reconnect at 15217296 in 0 second(s), error=Input/output error.

KSPlayer: MEPlayerItem.swift:127 init(url:options:) | Packet corrupt (stream = 0, dts = 539340361)
KSPlayer: MEPlayerItem.swift:127 init(url:options:) | .

KSPlayer: MEPlayerItem.swift:127 init(url:options:) | error while decoding MB 59 41, bytestream -8

KSPlayer: MEPlayerItem.swift:701 getVideoOutputRender(force:) | video delay time: 11.371355555555056 dropped video frame frameCount: 31 frameMaxCount: 32
KSPlayer: MEPlayerItem.swift:701 getVideoOutputRender(force:) | video delay time: 11.34897777777769 dropped video frame frameCount: 29 frameMaxCount: 32
KSPlayer: MEPlayerItem.swift:701 getVideoOutputRender(force:) | video delay time: 11.333388888888294 dropped video frame frameCount: 27 frameMaxCount: 32
KSPlayer: MEPlayerItem.swift:701 getVideoOutputRender(force:) | video delay time: 11.323611111111859 dropped video frame frameCount: 25 frameMaxCount: 32
KSPlayer: MEPlayerItem.swift:701 getVideoOutputRender(force:) | video delay time: 11.301211111111115 dropped video frame frameCount: 23 frameMaxCount: 32
KSPlayer: MEPlayerItem.swift:701 getVideoOutputRender(force:) | video delay time: 11.401311111111681 dropped video frame frameCount: 26 frameMaxCount: 32

Repeated...
Kimentanm commented 1 year ago

332 I have a same issue

UnknownCoder807 commented 1 year ago

The fix you did is slightly better but still have the issue

KSPlayer: KSPlayerLayer.swift:137 state | playerStateDidChange - bufferFinished
KSPlayer: MEPlayerItem.swift:127 init(url:options:) | IO error: Operation timed out

KSPlayer: MEPlayerItem.swift:127 init(url:options:) | Will reconnect at 9903432 in 0 second(s), error=Operation timed out.

KSPlayer: KSPlayerLayer.swift:137 state | playerStateDidChange - buffering
KSPlayer: MEPlayerItem.swift:127 init(url:options:) | Packet corrupt (stream = 0, dts = 2759776470)
KSPlayer: MEPlayerItem.swift:127 init(url:options:) | .

KSPlayer: MEPlayerItem.swift:127 init(url:options:) | error while decoding MB 56 17, bytestream -6

KSPlayer: KSPlayerLayer.swift:137 state | playerStateDidChange - bufferFinished
KSPlayer: MEPlayerItem.swift:701 getVideoOutputRender(force:) | video delay time: 13.482211111113429 frameCount: 31 frameMaxCount: 32
KSPlayer: MEPlayerItem.swift:704 getVideoOutputRender(force:) | dropped video frame
KSPlayer: MEPlayerItem.swift:708 getVideoOutputRender(force:) | video track seek to 30664.118211111112
KSPlayer: MEPlayerItem.swift:701 getVideoOutputRender(force:) | video delay time: 13.459811111111776 frameCount: 29 frameMaxCount: 32
KSPlayer: MEPlayerItem.swift:704 getVideoOutputRender(force:) | dropped video frame
KSPlayer: MEPlayerItem.swift:708 getVideoOutputRender(force:) | video track seek to 30664.12981111111
KSPlayer: MEPlayerItem.swift:701 getVideoOutputRender(force:) | video delay time: 13.44422222222056 frameCount: 27 frameMaxCount: 32
UnknownCoder807 commented 1 year ago

Getting the same error in your TracyPlayer demo too. (swiftui, tvos)

kingslay commented 1 year ago

这个错误是无法避免的。因为音频和视频的时间戳不同步。所以会有这个报错。但是报错会减少很多了吧。不会一直报错了吧。

UnknownCoder807 commented 1 year ago

Still not working properly. There are less dropped frame messages, but the frames are still dropped and the audio becomes out of sync.

I played the same stream using VLC and got 0 dropped frames over 10 minutes.

I don't understand why you don't see this issue with the url I sent to you yesterday. Sometimes you need to wait for it to happen, other times it happens from the very start. using tvOS and SwiftUI.

kingslay commented 1 year ago

Still not working properly. There are less dropped frame messages, but the frames are still dropped and the audio becomes out of sync.

I played the same stream using VLC and got 0 dropped frames over 10 minutes.

I don't understand why you don't see this issue with the url I sent to you yesterday. Sometimes you need to wait for it to happen, other times it happens from the very start. using tvOS and SwiftUI.

vlc没有drop frame，那它会loading吗？

UnknownCoder807 commented 1 year ago

vlc没有drop frame，那它会loading吗？

VLC loads and plays the stream perfectly fine. No dropped frames and no out of sync sound because of it.

I just tried the stream again and get dropped frames within seconds of playback, using TracyPlayer.

Kimentanm commented 1 year ago

vlc没有drop frame，那它会loading吗？

VLC loads and plays the stream perfectly fine. No dropped frames and no out of sync sound because of it.

I just tried the stream again and get dropped frames within seconds of playback, using TracyPlayer.

When VLC and TracyPlayer play videos at the same time, whether VLC starts loading at the location where TracyPlayer drops frames

JeremTM93 commented 1 year ago

Dear @kingslay your fix is almost done; I have frame drops when I use interlaced live streams (AppleTV). If you want I can provide you a few interlaced streams.

Maybe create a new ticket for interlaced streams? The lack of fluidity of a flow is a critical issue.

kingslay commented 1 year ago

Dear @kingslay your fix is almost done; I have frame drops when I use interlaced live streams (AppleTV). If you want I can provide you a few interlaced streams.

Maybe create a new ticket for interlaced streams? The lack of fluidity of a flow is a critical issue.

恩 创建一个新的吧

UnknownCoder807 commented 1 year ago

So far, the stream I had problems with seems to play ok now. I've had 0 dropped frames in the 40 minutes or so of testing I have done. Thanks for the great work @kingslay

If the problem resurfaces I will post here.

kingslay commented 1 year ago

Dear @kingslay your fix is almost done; I have frame drops when I use interlaced live streams (AppleTV). If you want I can provide you a few interlaced streams.

@fedalcode 麻烦你提供下测试url给我下，谢谢。

UnknownCoder807 commented 1 year ago

Still have this issue. Same logs as before.