My list for Santa Claus

[LauraKami]

Hello, it's me again. So my last request for 2022 is an improvement of your library for 2023 (not for Christmas ;o)

I made a video to show you and especially to listen, the bug to be fixed. And as long as you do, to greatly improve and become the best library for reading and listening to video.

Sorry the video is a bit long 14 minutes, so I used k-pop, a young band, it's more enjoyable.

As I use a less powerful PC (Intel Pentium Silver N5030 CPU @ 1.10-2.5GHz), I may detect problems more easily, which can go unnoticed on a fast PC.

---- EDIT ---- I removed the video, I showed that on a low-end PC, the performance of FlyLeaf is excellent on the video part, superior to that of Microsoft, and as good or better than VLC. On the other hand, on the sound side, some improvements are possible, especially if you change the speed to 0.5X or 2X. Another older software FFME, also has difficulties on a less powerful PC but the use of the SoundTouch library through ffmpeg, offers possibilities that interest me a lot on slowed down or accelerated sound and image. ---- EDIT ----

Thanks again for your great work on this library, have a great holiday and see you next year.

PS: indicate as soon as you have watched and listened to the video, I will remove it.

[SuRGeoNix]

Hey @LauraKami,

I've watched your cool video - 14 minutes of k-pop (first time), it was OK but I would prefer to listen Flyleaf - I'm so sick 😄

There are two different audio crackling / gaps issues and I will try to explain them for now (and to fix them later on):

1. Audio Crackling on Frame Drops

This is happening because Flyleaf re-buffers (MediaBuffer) on a single frame drop and it will not allow audio to play properly until it will be stable to play video as well. As you correctly mention, this is easier to reproduce with a not powerful CPU/GPU. VLC it also drops frames I guess (you can double check it by going to Menu -> Tools -> Media Information -> Statistics and check Lost frames).

Flyleaf works this way mainly for network streams and tries to respect the Min Buffer Duration so if it doesn't have enough frames demuxed/decoded will restart buffering. I will review this in 2023 and I will try to handle different when the issue happens because of CPU/GPU instead of network, however I'm not sure if it will worth the risk losing audio/video accurate sync (see last paragraph).

2. Audio Crackling on Speed != 1

This is happening because when you change the Speed of video you will need to apply an audio filter to change also the audio speed properly. Now this filter requires a lot of research, I think FFmpeg includes this in avfilters so it could be used directly from there but that would make avfilters dll as required which is not really a good idea. Might worth also checking soundtouch.dll but I would prefer to not use other libraries. Probably, I will let the user optionally add also avfilter and I will enable those filters only if the dll exists in the ffmpeg folder.

Flyleaf currently just drops audio samples that's why you have those gaps. I will review this as well in 2023!

---

Generally, in your comparison you should ensure that all players are using video / hardware / GPU acceleration (you can just check CPU/GPU usage, if they mainly use CPU then there is no hardware acceleration. For example, FFME it does not support it that's why it struggles more than the other players, even if it supports it for decoding it will 'download' the hardware frame as a software frame to present it.

By adding those features to Flyleaf there is a risk that it will not have accurate sync between video / audio frames which currently is more important rather than not having those gaps for now. It's really important that any changes will not affect this and it will not brake other things such as timestamp change randomly (that can happen also on network streams and flyleaf handles this by restarting with a new starting point of the new timestamps)

Thanks for you feedback and wish you happy holidays and happy new year!

[SuRGeoNix]

And something you can try that it could fix this:

Go from Settings -> Player -> Video Frames Max. (set this to 30-50) and Buffer Duration Min. (set this to 60000ms this is temporary as if it works I could wait for decoded packets to be full)

[LauraKami]

Thanks for watching and listening, and inferring some improvements for a future release. I hope to level up and one day be able to help you, but for now I'm going to stick with using your library, which from a video standpoint works perfectly well for my project on my low-end pc. (and therefore the PCs that I target for my project because it is aimed at young teenagers without a budget...) Depending on the style of video you are watching, the priority may be different, for example sync for a movie/series so that the sound channel is in lip sync. For the animations (or video clip) on the songs, the sound is the most important, no matter that there is a little delay on the image (except when we make a close-up on the face...).

I note for Hardware acceleration, I did not know that it was necessary to touch it... (not autodetect?)

Happy holidays and New Year and rest helps to work well afterwards...

[SuRGeoNix]

It's true that Flyleaf is focusing on Video rather than Audio so far. There are a lot of reasons for that (e.g. weird formats, network bottlenecks, bad rips, wrong timestamps, etc.) that could cause a lot issues during the playback and the application should be able to continue playing if that happens. If it is not possible to handle those in a single playback method might Flyleaf should provide two different ones (one for Video priority and one for Audio). Another good example is when you have a audio starting earlier without video (or a video gap in the middle of the media), currently Flyleaf will consider this as a network issue and it will skip until the first video frame, so you will not be able to play those audio samples. It does currently provide an audio only player or you can just disable the video stream to avoid having the gaps as well, but of course that's not a proper solution.

To be able to play with hardware acceleration, it should be supported by the library that you use (e.g. FFME does not support it). Secondly, your hardware (GPU) should be able to support hardware decoding for the specific video codec (eg. x264). Finally, it should be supported also from your operating system (DirectX, eg. Win7 requires DirectX 9 and Flyleaf does not supported). Generally, it should be enabled by default (autodetect as you said).

[LauraKami]

As I'm curious, I read your Flyleaf source code and that of FFME to better understand the difficulties. (I made mistakes in my interpretation, it is indeed the C code of ffmpeg with your improvements (hls.c and mov.c)

It seems to me that you rewrote and improved a few pieces of ffmpeg, in particular to better follow network streams and re-synchronize. The decoding part is too complex, but if you used the ffmpeg library, it is normally possible to use the filters, including the slowing down and speeding up audio filters. (if the dll is in the ffmpeg folder, it takes it into account). But I'm sure it seriously slows down sound decoding, which would explain ffme's difficulties. Another major element of improvement is that you redirect the image stream to a directX texture, which allows you to apply scale, move and rotate effects to it, and also, through the use of a shader, color corrections . (elements that I appreciate very much and that I use). I also have a doubt, if the decoding is done by hardware, do we still have this possibility of scale, move, rotate?

In any case, I'm convinced that I couldn't help you, I'm too old, to master these technologies it requires a complete investment, especially since the novelties keep appearing (like av1). But also, opensource is good, but openAnalyse could be better ;o)

So thank you again for this beautiful library, and I hope for 2023, I'll cut the sound in the meantime ;o)

[SuRGeoNix]

The C code has not been written by me, those are the FFmpeg's code but patched for an HLS issue that it crashes while you seek (near to the begin or end, or when you switch between video streams). The demuxing and decoding uses FFmpeg's API directly (through FFmpeg.AutoGen bindings).

The most important thing for Flyleaf is the performance, everything that it currently supports it happens directly in your GPU with pixel shaders that's why it doesn't use any AVFilters so far as they don't support DirectX hardware acceleration (for now at least). That's why you can Pan Move/Zoom/Rotate smoothly ;)