JetBrains / compose-multiplatform

Compose Multiplatform, a modern UI framework for Kotlin that makes building performant and beautiful user interfaces easy and enjoyable.
https://jetbrains.com/lp/compose-multiplatform
Apache License 2.0
15.9k stars 1.15k forks source link

VideoPlayer issues on macOs #1089

Open igordmn opened 3 years ago

igordmn commented 3 years ago

(from https://github.com/JetBrains/compose-jb/pull/1088#issuecomment-903090784)

  1. Fullscreen is too slow on macOs on my machine (MacBook Pro 2019 16" i9), but not slow on the theapache64 machine.
  2. I noticed a frame tearing (as happened when vsync is disabled)
  3. Encountered:
    java.io.IOException: Native library (darwin/libvlc.dylib) not found in resource path

    until I installed VLC:

    brew install vlc

    I think we need to bundle VLC as a part of VideoPlayer. The separate issue

mahozad commented 1 year ago

I embedded the VLC in my Compose Multiplatform app (only for Windows). Wanted to share it here. In the end it just became about 10 MB of VLC files that needed to be embedded in the app (still a lot but tolerable). With this, there will be no need for the VLC to have been installed on the system.

First, downloaded and installed latest version of the VLC media player (v3.0.18) on the system (made sure to select the x64 version from the dropdown): https://www.videolan.org/vlc/download-windows.html

My case is that I need to play only MPEG-TS files with these abilities:

So, the following is the minimum required .dll files from VLC for the above use case (added ⚠️ comments for some of them):

📂 <VLC_INSTALLTION_DIRCTORY>
  ├─── libvlc.dll
  ├─── libvlccore.dll
  └─── 📂 plugins
      ├─── 📂 access
      │   └─── libfilesystem_plugin.dll
      ├─── 📂 audio_filter
      │   ├─── libnormvol_plugin.dll (⚠️ Along with audio_output/libmmdevice_plugin.dll, normalizes audio loudness)
      │   ├─── libscaletempo_pitch_plugin.dll (⚠️ For human voice to not distort when changing the speed)
      │   └─── libscaletempo_plugin.dll (⚠️ For audio when changing the speed)
      ├─── 📂 audio_output
      │   └─── libdirectsound_plugin.dll
      │   └─── libmmdevice_plugin.dll (⚠️ Along with audio_filter/libnormvol_plugin.dll, normalizes audio loudness)
      ├─── 📂 codec
      │   └─── libavcodec_plugin.dll (⚠️ When almost all other DLLs were available (not deleted) I deleted this file and the player still worked but the video flickered (when paused and sometimes during playback))
      ├─── 📂 demux
      │   └─── libts_plugin.dll (⚠️ For MPEG-TS files)
      ├─── 📂 packetizer
      │   ├─── libpacketizer_mpeg4audio_plugin.dll
      │   └─── libpacketizer_mpeg4video_plugin.dll
      ├─── 📂 stream_filter
      │   └─── libcache_read_plugin.dll (⚠️ For speedup of live-stream video to work correctly and smoothly; speedup of finished videos works without this)
      ├─── 📂 text_renderer (⚠️ Can delete this to hide the text overlay (save path) when taking a screenshot)
      │   └─── libfreetype_plugin.dll
      ├─── 📂 video_chroma
      │   └─── libswscale_plugin.dll
      ├─── 📂 video_filter
      │   └─── libdeinterlace_plugin.dll (⚠️ To de-interlace the video playback; otherwise, not needed)
      └─── 📂 video_output
          ├─── libdirect3d9_plugin.dll (⚠️ On my system, video was black without this file)
          ├─── libdirect3d11_plugin.dll (⚠️ On my system, video was black without this file)
          ├─── libdirectdraw_plugin.dll
          └─── libdrawable_plugin.dll

Copied the contents of <VLC_INSTALLTION_DIRCTORY> (preserving the directory structure and names) into <compose_project_directory>/assets/windows/vlc/.

Then downloaded the UPX program to reduce the size of VLC .dll files (to about 50% of their original size!):

cd <compose_project_directory>/assets/windows/vlc/
find -type f -name *.dll | xargs -n1 path/to/upx.exe

Then modified the app root build.gradle.kts file to add the assets/ as the compose multiplatform custom resources directory:

compose.desktop {
    application {
        mainClass = // ...
        nativeDistributions {
            // ...
            appResourcesRootDir = (project.rootDir.toPath() / "assets").toFile()
        }
    }
}

Finally, updated the video player:

// ...
val assetsPath = System
    .getProperty("compose.application.resources.dir")
    ?.let(::Path)
    ?: error("The compose.application.resources.dir is not set. Make sure to execute the app using Gradle tasks named `run...`")
val vlcPath = (assetsPath / "vlc").absolutePathString()

NativeDiscovery(object : NativeDiscoveryStrategy {
    override fun discover() = vlcPath
    override fun supported() = true
    override fun onFound(path: String) = true
    override fun onSetPluginPath(path: String) = true
}).discover()

if (isMacOS()) {
    CallbackMediaPlayerComponent()
} else {
    EmbeddedMediaPlayerComponent()
}
// ...

I tried to also do this in Ubuntu with VLC .so files but the app started with the error failed to get a new native library instance probably because, as the author of vlcj library said in https://stackoverflow.com/a/23897920 , the .so files are not relocatable.

These are probably related issues:

avently commented 12 months ago

I tried to also do this in Ubuntu with VLC .so files but the app started with the error failed to get a new native library instance probably because, as the author of vlcj library said in https://stackoverflow.com/a/23897920 , the .so files are not relocatable.

It's possible but you need to load more plugins and their helper libs too. Also you need to use patchelf in order to change rpath in libs to $ORIGIN. So in Linux I can setup it using some libs from VLC packages but I found easier to use libs from packed AppImage: https://github.com/cmatomic/VLCplayer-AppImage/releases

Here is some commits I made which make help you to understand what to do. Its still in progress so can't say that it works perfectly (it works bad on arm64 mac but on Linux it's ok).

https://github.com/simplex-chat/simplex-chat/commit/116df5c428cac5ccd8f5c1fc9e4689663e41f4c4 (you don't really need it but in case...) https://github.com/simplex-chat/simplex-chat/commit/46bfae111290a47cd9a00a9e4566681c1392ec6e https://github.com/simplex-chat/simplex-chat/commit/440035facf1bd8881ca6704d2e6317becc85d12d https://github.com/simplex-chat/simplex-chat/commit/87190cb60da56d54631c3140464120cc2b690a75 https://github.com/simplex-chat/simplex-chat/commit/cf144be391a859f050fc72e7c9020820af758a8a

Will send a link to final commit with all the code when I finish.

All packaging things for Linux is here: https://github.com/simplex-chat/simplex-chat/blob/av/desktop-playing-video/scripts/desktop/prepare-vlc-linux.sh

mahozad commented 12 months ago

@avently

That would be great.

What more plugins are needed? And what are those helper libs?

Could you please provide a little bit more detail on how to do this with newest version of VLC in Linux?


Sidenote: Wanted to mention that it is possible to directly show the video in Compose desktop without the need for SwingPanel, which gives more power and also makes it possible to easily overlay other things on video (for example, using Box {): https://github.com/JetBrains/compose-multiplatform/pull/3336

avently commented 11 months ago

@mahozad, now have time to answer with details. I successfully use VLC on Linux, Windows and mac. As a downside I noticed is that at least on mac videos from iPhone (mov) have no image. It can probably fixed by including one more plugin. Also when my code at some point initialize the player once per run of the app, the thread freezes for 0.5-1-2 seconds. After that every function that use VLC player starts without this delay. This happens on all OSes. Basically, you can see these scripts I use to prepare the VLC libs: https://github.com/simplex-chat/simplex-chat/blob/master/scripts/desktop/prepare-vlc-linux.sh https://github.com/simplex-chat/simplex-chat/blob/master/scripts/desktop/prepare-vlc-windows.sh https://github.com/simplex-chat/simplex-chat/blob/master/scripts/desktop/prepare-vlc-mac.sh

Script for the Linux uses AppImage I found on GitHub that works pretty good so I made my work a little easier by just unpacking the AppImage and taking all libs I need from it. In the script I take more libs than needed but you can strip almost all of them if you need just most common formats.

In code after line with exit 0 you can see the code that I wrote before I found AppImage. Using that code you can have a basic things working but in order to see the picture on videos you have to add a ffmpeg libs to your libs as well. Also if you want to use that method you have to load this lib manually, otherwise, there will be no image too (who knows why): val libXcb = "libvlc_xcb_events.so.0.0.0" System.load(File(File(vlcDir, "vlc"), libXcb).absolutePath)

Maybe everything will work fine without loading this lib, I'm not sure, because I have a lot of changes in logic of loading libs after I discovered inability to have a picture working without loading this lib manually.

So, moving to the next step. The idea is that you have to prepare the libs that comes with VLC and it's plugins to be able to play different formats. Scripts for different OSes prepare the libs.

When you have pointed the VLC libs to search in their relative directories (by patching the libs) you need to point JNA to search libs in this directories:

System.setProperty("jna.library.path", vlcDir.absolutePath) // https://github.com/simplex-chat/simplex-chat/blob/master/apps/multiplatform/desktop/src/jvmMain/kotlin/chat/simplex/desktop/Main.kt#L31C3-L31C62

Before I packed all libs in resources directory to jar and extracted them in runtime but it takes sooooo long time after start to copy them to tmp directory. So I started to pack them differently using compose packager that moves libs to resources directory close to jar files so I don't have to unpack the libs in runtime.

appResourcesRootDir.set(project.file("../build/links")) // https://github.com/simplex-chat/simplex-chat/blob/master/apps/multiplatform/desktop/build.gradle.kts#L55

Here I just point compose to directory with symlinks that point to real directory with all libs per $OS-$ARCH.

Also I should use the latest jna lib version in order to make something (I don't remember what) working:

implementation("net.java.dev.jna:jna:5.13.0") // https://github.com/simplex-chat/simplex-chat/blob/master/apps/multiplatform/desktop/build.gradle.kts#L24

Please, ask questions if you have problems in implementing all this in your case.

And big-big thank you for giving me a link to non-Swing implementation of video renderer. It saved ton hours of my time and made possible to make inline videos (multiple videos in lazy list) and removed all glitches in videos I had. Sooooo good!

mahozad commented 11 months ago

@avently Thanks for the detailed answer.

I also struggled with showing the media picture (it only had audio). With the below procedure, the problem was resolved and there was also no need for any additional libraries (at least for video formats and codecs that I tested).

Fortunately, VLC is available in Snap format too. So, I used its Snap package instead of the AppImage package (I think both of them, or at least, the Snap version, contain all the required dependencies and libraries).

Downloads of various VLC versions for various OSes are also available in https://get.videolan.org/vlc/.

Here is what I did for Linux (Ubuntu):

  1. Removed the default installed VLC (if any): sudo apt autoremove sudo apt remove vlc-nox sudo apt remove vlc

  2. Made sure no VLC is installed: which vlc whereis vlc

  3. Downloaded the snap package of VLC (instead of installing it) it will be downloaded in the current working directory sudo snap download vlc --channel=latest/stable

  4. Mounted (extracted) the downloaded VLC snap file sudo mount -t squashfs -o ro /path/to/my-vlc.snap /path/to/<mount-folder-name>

  5. Copied the mounted directory to another directory (because it was read-only): sudo cp -r vlc-mount/ vlc-mount-copy/

  6. Unmounted and removed the original mounted folder: sudo umount vlc-mount/ && rm -r vlc-mount/

  7. Installed chrpath tool (it was easier for me compared to patchelf): sudo apt install chrpath

  8. (Optional) Viewed all .so libraries that have rpath= or runpath= in them find . -name "*.so*" | xargs -n1 chrpath | grep "="

  9. Did these steps in order:

    1. cd vlc-mount-copy/usr/lib/
    2. sudo chrpath -r '$ORIGIN' libvlc.so Changes rptah or runpath of libvlc.so
    3. cd vlc/ subdirectory of vlc-mount-copy/usr/lib/
    4. sudo chrpath -r '$ORIGIN/..' libvlc_pulse.so.0.0.0
    5. sudo chrpath -r '$ORIGIN/..' libvlc_xcb_events.so.0.0.0
    6. cd plugins/ subdirectory of vlc-mount-copy/usr/lib/vlc/
    7. find . -name "*.so*" | sudo xargs -n1 chrpath -r '$ORIGIN/../../..'
    8. rm -r vlc-mount-copy/usr/lib/ssl/ Removes the ssl/ directory to prevent some errors
    9. rm -r vlc-mount-copy/usr/lib/jvm/ Removes the jvm/ directory to prevent some errors
    10. rm -r vlc-mount-copy/usr/lib/debug/ Removes the debug/ directory to prevent some errors
    11. cp -r vlc-mount-copy/usr/lib/x86_64-linux-gnu/* vlc-mount-copy/usr/lib/ copies everything in lib/x86_64-linux-gnu/ directory to the lib/ directory
    12. rm -r vlc-mount-copy/usr/lib/x86_64-linux-gnu/

Finally, in the app code, there was no need to manually load libraries. Just used a custom vlcj discovery as before (https://github.com/JetBrains/compose-multiplatform/issues/1089#issuecomment-1617782890).

And big-big thank you for ...

Glad it was helpful. Thanks goes to DrewCarlson.

okushnikov commented 2 weeks ago

Please check the following ticket on YouTrack for follow-ups to this issue. GitHub issues will be closed in the coming weeks.