search

intel / openvino-plugins-for-obs-studio

Apache License 2.0
53 stars 7 forks source link

Plugin build compatible with the official OBS flatkpak release #1

Closed cleaton closed 1 year ago

cleaton commented 1 year ago

Hi, I am trying to build & package the plugins for use in the official flatpak release of OBS. Because the flatpak does not have opencv, openvinio etc I adding them all as static dependencies. The problem I am currently facing is onetbbbind which I am not able to add statically.

I've tried disabling onetbb from openvinio -DENABLE_SYSTEM_TBB=OFF -DENABLE_TBBBIND_2_5=OFF -DTHREADING=SEQ but seems the output still has symbols that requires TBB.

Anyone successfully made these plugins work on the official flatpak?

RyanMetcalfeInt8 commented 1 year ago

Hi @cleaton, I'm not too knowledgeable on flatpak, but happy to help where I can. As far as I understand, you are trying to disable TBB from OpenVINO build using those cmake options, right? Where do you see that the output still requires those symbols.. at OpenVINO build (link) time, when OBS Studio is trying to dlopen the plugins, or someplace else?

cleaton commented 1 year ago

Hi @RyanMetcalfeInt8, The missing symbols is when OBS is trying to dlopen the pluging. Some background on my setup, I got a docker file for the build environment:

FROM ubuntu:22.04
WORKDIR /opt/build
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update
RUN apt-get install -y wget build-essential gcc g++ cmake bash libopencv-dev git
RUN wget https://github.com/obsproject/obs-studio/releases/download/29.0.2/obs-studio_29.0.2-0obsproject1_jammy_amd64.deb && \
    apt-get install -y ./obs-studio_29.0.2-0obsproject1_jammy_amd64.deb
RUN git clone https://github.com/openvinotoolkit/openvino.git && cd openvino && git checkout 2022.3.0 && git submodule update --init --recursive
RUN git clone https://github.com/oneapi-src/oneTBB.git && cd oneTBB && git checkout v2021.8.0 && git submodule update --init --recursive
RUN export TBBROOT=/opt/build/oneTBB && cd openvino && mkdir build && cd build && cmake -DCMAKE_INSTALL_PREFIX=/opt/openvino -DCMAKE_BUILD_TYPE=Release -DENABLE_INTEL_VPU=OFF -DENABLE_INTEL_MYRIAD=OFF -DENABLE_INTEL_MYRIAD_COMMON=OFF -DENABLE_HDDL=OFF -DENABLE_SYSTEM_TBB=OFF -DENABLE_TBBBIND_2_5=ON  -DTHREADING=SEQ -DENABLE_DOCS=OFF -DENABLE_INTEL_GNA=OFF -DENABLE_PYTHON=OFF -DENABLE_TESTS=OFF -DBUILD_SHARED_LIBS=OFF -DENABLE_INTEL_GPU=OFF  .. && make -j$(nproc) && make install
RUN cd oneTBB && mkdir build && cd build && cmake -DBUILD_SHARED_LIBS=OFF -DTBB_TEST=OFF -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/opt/onetbb .. && make -j$(nproc) && make install
RUN chmod 777 -R /opt/build

and then to build the plugin (I also have a tweaked cmake to static link opencv):

docker build -t openvino-plugins-build . && docker run --rm -it \
     --user $(id -u):$(id -g) \
      -v $(pwd):/opt/src \
      openvino-plugins-build \
      bash -c "source /opt/openvino/setupvars.sh && cmake /opt/src && make -j$(nproc) && cp -r intel64/Release/* /opt/src/build-out-linux/"

In OBS logs I can see log messages about tbb task symbol missing, I've tried to change the openvino build configuration or manually link static tbb to the plugin. I belive the symbol comes from tbbbind which is not included in a staic build of onetbb

11:15:42.460: os_dlopen(/home/jesper/.var/app/com.obsproject.Studio/config/obs-studio/plugins/obs-backgroundconcealment-ov/bin/64bit/obs-backgroundconcealment-ov.so->/home/jesper/.var/app/com.obsproject.Studio/config/obs-studio/plugins/obs-backgroundconcealment-ov/bin/64bit/obs-backgroundconcealment-ov.so): /home/jesper/.var/app/com.obsproject.Studio/config/obs-studio/plugins/obs-backgroundconcealment-ov/bin/64bit/obs-backgroundconcealment-ov.so: undefined symbol: _ZTIN3tbb4taskE
11:15:42.460: 
11:15:42.463: os_dlopen(/home/jesper/.var/app/com.obsproject.Studio/config/obs-studio/plugins/obs-backgroundconcealment-ov/bin/64bit/obs-backgroundconcealment-ov.so->/home/jesper/.var/app/com.obsproject.Studio/config/obs-studio/plugins/obs-backgroundconcealment-ov/bin/64bit/obs-backgroundconcealment-ov.so): /home/jesper/.var/app/com.obsproject.Studio/config/obs-studio/plugins/obs-backgroundconcealment-ov/bin/64bit/obs-backgroundconcealment-ov.so: undefined symbol: _ZTIN3tbb4taskE
RyanMetcalfeInt8 commented 1 year ago

Hi @cleaton,

I'm trying to reproduce this on my side. As my native Ubuntu install is 20.04, I'm using a slightly modified docker than what you shared (FROM ubuntu:20.04, and grabbing focal obs packages instead of jammy).

The docker builds okay, and I'm also able to produce the OBS plugins using the same docker run cmd's as you shared. I can see that they have been statically linked with OpenVINO libs, as expected.

You mention that you 'have a tweaked cmake to static link opencv', can you give some details about that? My build shows dynamic dependencies (via ldd):

libopencv_imgcodecs.so.4.2 => /lib/x86_64-linux-gnu/libopencv_imgcodecs.so.4.2 (0x00007fd979ddf000)
libopencv_imgproc.so.4.2 => /lib/x86_64-linux-gnu/libopencv_imgproc.so.4.2 (0x00007fd979838000)

After copying these plugins to OBS plugin directory, I don't see the dlopen issues that you mention. But, given that I'm using different build / test environment (Ubuntu 20 vs. 22, dynamically linked opencv, etc.) I don't know what to make of it.

One question I have about your testing -- once you build the plugins, what are the steps you're doing to actually test it? As I'm not too familiar (or at all familiar I should say) with flatpak, could you explain how these plugins get installed to the flatpak installation?

cleaton commented 1 year ago

Hi @RyanMetcalfeInt8 Thanks for taking a look at it. For now I just hardcoded the static opencv library location in ubuntu, in the main cmake file I added:

function(convert_to_static_libraries LIBRARIES_LIST STATIC_LIBRARIES_LIST)
  set(LOCAL_STATIC_VARIABLES "")
  foreach(lib ${${LIBRARIES_LIST}})
    list(APPEND LOCAL_STATIC_VARIABLES "/usr/lib/x86_64-linux-gnu/lib${lib}.a")
  endforeach()
  SET(${STATIC_LIBRARIES_LIST} ${LOCAL_STATIC_VARIABLES} PARENT_SCOPE)
endfunction()
convert_to_static_libraries(OpenCV_LIBRARIES OpenCV_STATIC_LIBRARIES)

and then in target_link_libraries I use OpenCV_STATIC_LIBRARIES instead of OpenCV_LIBRARIES

then for my full build script (which also does install to the obs flatpak plugin directory ~/.var/app/com.obsproject.Studio/config/obs-studio/plugins/<plugin-name>):

#!/bin/sh
while test $# != 0
do
    case "$1" in
    -fi) flatpak_install=true ;;
    esac
    shift
done
mkdir build-out-linux || true
docker build -t openvino-plugins-build . && docker run --rm -it \
     --user $(id -u):$(id -g) \
      -v $(pwd):/opt/src \
      openvino-plugins-build \
      bash -c "source /opt/openvino/setupvars.sh && cmake /opt/src && make -j$(nproc) && cp -r intel64/Release/* /opt/src/build-out-linux/"

if [ "$flatpak_install" = true ] ; then
    for i in obs-smart-framing-ov obs-face-mesh-ov obs-backgroundconcealment-ov
    do 
       echo "installing flatpak ~/.var/app/com.obsproject.Studio/config/obs-studio/plugins/$i"
       mkdir -p ~/.var/app/com.obsproject.Studio/config/obs-studio/plugins/$i/bin/64bit/ || true
       mkdir -p ~/.var/app/com.obsproject.Studio/config/obs-studio/plugins/$i/data || true
       cp -r build-out-linux/lib/lib$i.so ~/.var/app/com.obsproject.Studio/config/obs-studio/plugins/$i/bin/64bit/$i.so
       cp -r build-out-linux/lib/libovmediapipe.a ~/.var/app/com.obsproject.Studio/config/obs-studio/plugins/$i/bin/64bit/
    done
    cp -r openvino-models ~/.var/app/com.obsproject.Studio/config/obs-studio/plugins/
fi

to test it I just launch flatpak OBS, view logs from OBS menu: Help -> Log Files -> View Current Log. I suspect the model path also need to be adjusted for flatpak but I'll look at that after I can make the plugin to load :-)

RyanMetcalfeInt8 commented 1 year ago

Hi @cleaton,

Thanks for the details on your method used to statically link OpenCV libs, as well as the flatpak installation instructions.

I was able to reproduce the TBB symbol error, so I've started to dig deeper into it. I am suspecting that the symbol error is coming from the way OpenCV is getting linked, rather than an OpenVINO issue.. but I'm not sure of it yet.

I say this because, in the docker image, this symbol doesn't appear to be found within the OpenVINO runtime libraries, but shows up in the system opencv libs:

I have no name!@361b67dffe23:/opt/build$ grep -R _ZTIN3tbb4taskE /opt/openvino/runtime/lib/*
I have no name!@361b67dffe23:/opt/build$ grep -R _ZTIN3tbb4taskE /usr/lib/x86_64-linux-gnu/libopencv_*
grep: /usr/lib/x86_64-linux-gnu/libopencv_core.a: binary file matches
grep: /usr/lib/x86_64-linux-gnu/libopencv_core.so: binary file matches
grep: /usr/lib/x86_64-linux-gnu/libopencv_core.so.4.5.4d: binary file matches
grep: /usr/lib/x86_64-linux-gnu/libopencv_core.so.4.5d: binary file matches

What I started experimenting with is, incorporating a version of opencv into the docker build to statically link against. Here's my updated Dockerfile (I think it only differs from yours with the opencv build stuff, second to last line):

FROM ubuntu:22.04
WORKDIR /opt/build
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update
RUN apt-get install -y wget build-essential gcc g++ cmake bash libopencv-dev git
RUN wget https://github.com/obsproject/obs-studio/releases/download/29.0.2/obs-studio_29.0.2-0obsproject1_jammy_amd64.deb && \
    apt-get install -y ./obs-studio_29.0.2-0obsproject1_jammy_amd64.deb
RUN git clone https://github.com/openvinotoolkit/openvino.git && cd openvino && git checkout 2022.3.0 && git submodule update --init --recursive
RUN git clone https://github.com/oneapi-src/oneTBB.git && cd oneTBB && git checkout v2021.8.0 && git submodule update --init --recursive
RUN export TBBROOT=/opt/build/oneTBB && cd openvino && mkdir build && cd build && cmake -DCMAKE_INSTALL_PREFIX=/opt/openvino -DCMAKE_BUILD_TYPE=Release -DENABLE_INTEL_VPU=OFF -DENABLE_INTEL_MYRIAD=OFF -DENABLE_INTEL_MYRIAD_COMMON=OFF -DENABLE_HDDL=OFF -DENABLE_SYSTEM_TBB=OFF -DENABLE_TBBBIND_2_5=ON  -DTHREADING=SEQ -DENABLE_DOCS=OFF -DENABLE_INTEL_GNA=OFF -DENABLE_PYTHON=OFF -DENABLE_TESTS=OFF -DBUILD_SHARED_LIBS=OFF -DENABLE_INTEL_GPU=OFF  .. && make -j$(nproc) && make install
RUN cd oneTBB && mkdir build && cd build && cmake -DBUILD_SHARED_LIBS=OFF -DTBB_TEST=OFF -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/opt/onetbb .. && make -j$(nproc) && make install
RUN git clone https://github.com/opencv/opencv.git && cd opencv && git checkout 4.7.0 && mkdir build && cd build && cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX=/opt/opencv -DBUILD_TESTS=OFF -DBUILD_PERF_TESTS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_opencv_apps=OFF .. && make -j$(nproc) && make install
RUN chmod 777 -R /opt/build

And updated plugin build cmd (setting OpenCV_DIR env.):

docker run --rm -it --user $(id -u):$(id -g)  -v $(pwd):/opt/src   openvino-plugins-build-ocv-custom  bash -c "source /opt/openvino/setupvars.sh && export OpenCV_DIR=/opt/opencv/lib/cmake/opencv4/ && cmake /opt/src && make -j$(nproc) VERBOSE=1 && cp -r intel64/Release/* /opt/src/build-out-linux/"

The above method isn't quite working yet. While it appears that static opencv libs are used for the build, it's still dynamically linking against things like libjpeg.so, etc. and so there are still dlopen issues from OBS due to those missing libraries.

So anyway.. still looking at this.

cleaton commented 1 year ago

Thanks! It didn't occur to me that opencv might use TBB as well... I tweaked your opencv line in docker file to force build static 3rd party libraries

RUN git clone https://github.com/opencv/opencv.git && cd opencv && git checkout 4.7.0 && mkdir build && cd build && export BUILD_SHARED_LIBS=OFF && cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF -DOPENCV_FORCE_3RDPARTY_BUILD=ON -DCMAKE_INSTALL_PREFIX=/opt/opencv -DBUILD_TESTS=OFF -DBUILD_PERF_TESTS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_opencv_apps=OFF .. && make -j$(nproc) && make install

I can see the library loading in logs

17:57:26.551: Failed to load 'en-US' text for module: 'obs-backgroundconcealment-ov.so'
17:57:26.551: [obs-backgroundconcealment-ov] plugin loaded successfully (version 0.4.0)
17:57:26.650: Failed to load 'en-US' text for module: 'obs-face-mesh-ov.so'
17:57:26.650: [obs-face-mesh-ov] plugin loaded successfully (version 0.4.0)
17:57:26.744: Failed to load 'en-US' text for module: 'obs-smart-framing-ov.so'
17:57:26.744: [obs-smart-framing-ov] plugin loaded successfully (version 0.4.0)
17:57:26.744: ---------------------------------
17:57:26.744:   Loaded Modules:
17:57:26.744:     obs-smart-framing-ov.so
17:57:26.744:     obs-face-mesh-ov.so
17:57:26.744:     obs-backgroundconcealment-ov.so

and I can also the the new filter options in UI. Next i'll try and make the model work as well :-) thank you for the help!

on a side note, seems there is a way to package plugins for flatpak including dynamic libraries using flatpak application extensions: https://blog.tingping.se/2018/03/18/flatpaking-plugins.html https://github.com/flathub/io.github.Hexchat.Plugin.Perl/tree/master and obs does add extension point for plugins: https://github.com/obsproject/obs-studio/blob/master/CI/flatpak/com.obsproject.Studio.json

though I've not seen anyone do it this way yet. I've only seen people manually copy files to the folder for now...

RyanMetcalfeInt8 commented 1 year ago

Hi @cleaton,

Ah, thanks for figuring out the -DOPENCV_FORCE_3RDPARTY_BUILD=ON option for cmake. I am now also able to successfully load the build plugins.

After playing around a bit, I figured out that for the flatpak installation, it considers this the plugin 'module' directory (e.g. for background concealment): .var/app/com.obsproject.Studio/config/obs-studio/plugins/obs-backgroundconcealment-ov/data/

So, it's going to look for 'openvino-models' directory one directory up from there (i.e. '../openvino-models'). So, an update to your build script that should put the models in the right spot:

if [ "$flatpak_install" = true ] ; then
    # Store a copy of the openvino-models directory here, and then create links in each
    #  plugin directory.
    cp -r openvino-models ~/.var/app/com.obsproject.Studio/config/obs-studio/plugins/
    for i in obs-smart-framing-ov obs-face-mesh-ov obs-backgroundconcealment-ov
    do
       echo "installing flatpak ~/.var/app/com.obsproject.Studio/config/obs-studio/plugins/$i"
       mkdir -p ~/.var/app/com.obsproject.Studio/config/obs-studio/plugins/$i/bin/64bit/ || true
       mkdir -p ~/.var/app/com.obsproject.Studio/config/obs-studio/plugins/$i/data || true
       ln -s ../openvino-models ~/.var/app/com.obsproject.Studio/config/obs-studio/plugins/$i/
       cp -r build-out-linux/lib/lib$i.so ~/.var/app/com.obsproject.Studio/config/obs-studio/plugins/$i/bin/64bit/$i.so
    done
fi

Anyway, give that a try!

on a side note, seems there is a way to package plugins for flatpak including dynamic libraries using flatpak application extensions: https://blog.tingping.se/2018/03/18/flatpaking-plugins.html https://github.com/flathub/io.github.Hexchat.Plugin.Perl/tree/master and obs does add extension point for plugins: https://github.com/obsproject/obs-studio/blob/master/CI/flatpak/com.obsproject.Studio.json

though I've not seen anyone do it this way yet. I've only seen people manually copy files to the folder for now...

Thanks for the heads up, I'll take a look at this.

RyanMetcalfeInt8 commented 1 year ago

Hi @cleaton, did you get a chance to verify if this worked for you? Thanks!

cleaton commented 1 year ago

@RyanMetcalfeInt8 sorry, got caught up in other projects. Yes I've verified this works for me :) including on 29.1.0 beta 3 flatpak. I can't use it yet because it turns out most sources I use, screen recording and browser source (vdo.ninja video) does not support video filter, only effects & audio filters. I was only able to confirm the plugin is working using a virtual camera and V4L2 loopback source. I'll close this issue, as the build works great now. Thanks for all the help!