search

software-mansion / live-compositor

Media server for real-time, low latency, programmable video and audio mixing.
https://compositor.live
Other
267 stars 18 forks source link

Info about minimum features on targeted GPU #527

Closed ludolpif closed 6 months ago

ludolpif commented 6 months ago

Hi, I hit a wall trying to run video_compositor examples from membrane.

ludolpif@lud-5490:~/git/membrane_video_compositor_plugin/examples$ mix run lib/layout_with_shader.exs
[...]
thread 'main' panicked at src/server.rs:25:9:
Failed to start event loop.
Failed to initialize a wgpu context.
Failed to get a wgpu device.
Unsupported features were requested: Features(UNIFORM_BUFFER_AND_STORAGE_TEXTURE_ARRAY_NON_UNIFORM_INDEXING)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I tried this on a Dell 5490 Laptop with Debian 12 tweaked a bit to get ffmpeg 6, vulkaninfo says : 

$ vulkaninfo | less
GPU0:
VkPhysicalDeviceProperties:
    apiVersion        = 1.3.230 (4206822)
    driverVersion     = 22.3.6 (92286982)
    vendorID          = 0x8086
    deviceID          = 0x5917
    deviceType        = PHYSICAL_DEVICE_TYPE_INTEGRATED_GPU
    deviceName        = Intel(R) UHD Graphics 620 (KBL GT2)

$ vulkaninfo | egrep '(fragmentStoresAndAtomics|independentBlending|maxDrawInd' | sort | uniq
        fragmentStoresAndAtomics                = true
        maxDrawIndexedIndexValue                        = 4294967295
        VK_KHR_storage_buffer_storage_class          : extension revision 1
        VK_KHR_storage_buffer_storage_class         : extension revision 1

It seems because there is :

 required_features: wgpu::Features::TEXTURE_BINDING_ARRAY
                    | wgpu::Features::PUSH_CONSTANTS
                    | wgpu::Features::SAMPLED_TEXTURE_AND_STORAGE_BUFFER_ARRAY_NON_UNIFORM_INDEXING
                    | wgpu::Features::UNIFORM_BUFFER_AND_STORAGE_TEXTURE_ARRAY_NON_UNIFORM_INDEXING,
            },

in https://github.com/membraneframework/video_compositor/blob/04f1e0576e10977ecb7cfa83e42fab90d86098b5/compositor_render/src/wgpu/ctx.rs#L83

I can't find any info about what the project aim in term of GPUs, did I missed it somewere ? May can help to write some "hardware requirements" closer to where newcomers put their eyes ?

wkozyra95 commented 6 months ago

Hi

There are no hardware requirements. In particular, the compositor supports software-based rendering on the CPU only.

The missing features are likely software side, and unfortunately, we don't have that documented on most platforms. Every platform and even most Linux distributions differ in how you install vulkan/mesa/gpu drivers.

Currently, the only maintained configuration we are verifying is our Docker image https://github.com/membraneframework/video_compositor/blob/master/build_tools/docker/slim.Dockerfile. It, of course, works on other systems, we use macOS, Arch, NixOS and PopOS for development, it just documenting those configurations is problematic.

Also, CPU-based rendering has some additional requirements that GPU rendering does not (e.g. requires newer mesa).

For example, If you use ubuntu:

I'm not sure what exactly is needed on Debian, but I would check the following stuff first:

ludolpif commented 6 months ago

Thanks for the extensive reply ! About mesa-vulkan-drivers on debian 12 setup :

ludolpif@lud-5490:~$ LANG=C apt policy mesa-vulkan-drivers
mesa-vulkan-drivers:
  Installed: 22.3.6-1+deb12u1
  Candidate: 22.3.6-1+deb12u1
  Version table:
 *** 22.3.6-1+deb12u1 990
        990 http://ftp.fr.debian.org/debian bookworm/main amd64 Packages
        100 /var/lib/dpkg/status

About packages needed for CPU render, I didn't have installed 2 of them when initially tried and when captured the above log : 

ludolpif@lud-5490:~$ dpkg -l | egrep 'libegl1-mesa-dev|libgl1-mesa-dri|libxcb-xfixes0-dev|mesa-vulkan-drivers'
ii  libgl1-mesa-dri:amd64                                       22.3.6-1+deb12u1                    amd64        free implementation of the OpenGL API -- DRI modules
ii  mesa-vulkan-drivers:amd64                                   22.3.6-1+deb12u1                    amd64        Mesa Vulkan graphics drivers
ludolpif@lud-5490:~$

I am unsure about how to clean run the shipped video_compositor out of membrane and/or getting the log for humans. My first guess is this way :

ludolpif@lud-5490:~$ ./git/membrane_video_compositor_plugin/examples/_build/dev/lib/membrane_video_compositor_plugin/priv/v0.2.0-rc.4/linux_x86_64/video_compositor/video_compositor

A got a unformatted JSON, and if I pipe it to "jq" it seems get lines you refer with some metadata. I see some LoggerConfig.format: json in the way, I didn't learnt how to configure it yet. It seems to use vulkan backend and the Intel GPU right before the wgpu init failure messages show in the original message of ths issue.

{
  "timestamp": "2024-04-22T09:33:08.699290Z",
  "level": "INFO",
  "fields": {
    "message": "Starting LiveCompositor with config:\nConfig {\n    instance_id: \"live_compositor_604953949\",\n    api_port: 8081,\n    logger: LoggerConfig {\n        ffmpeg_logger_level: Warn,\n        format: Json,\n        level: \"info,wgpu_hal=warn,wgpu_core=warn\",\n    },\n    stream_fallback_timeout: 500ms,\n    web_renderer: WebRendererInitOptions {\n        enable: false,\n        enable_gpu: true,\n    },\n    force_gpu: false,\n    download_root: \"/tmp\",\n    queue_options: QueueOptions {\n        ahead_of_time_processing: false,\n        output_framerate: Framerate {\n            num: 30,\n            den: 1,\n        },\n        run_late_scheduled_events: false,\n        never_drop_output_frames: false,\n    },\n    output_sample_rate: 48000,\n}",                                                                                                                
    "log.target": "video_compositor::server",
    "log.module_path": "video_compositor::server",
    "log.file": "src/server.rs",
    "log.line": 23
  },
  "target": "video_compositor::server"
}
{
  "timestamp": "2024-04-22T09:33:08.857136Z",
  "level": "WARN",
  "fields": {
    "message": "Unable to find extension: VK_EXT_swapchain_colorspace",
    "log.target": "wgpu_hal::vulkan::instance",
    "log.module_path": "wgpu_hal::vulkan::instance",
    "log.file": "/home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wgpu-hal-0.19.1/src/vulkan/instance.rs",
    "log.line": 291
  },
  "target": "wgpu_hal::vulkan::instance"
}
{
  "timestamp": "2024-04-22T09:33:08.891347Z",
  "level": "WARN",
  "fields": {
    "message": "Detected skylake derivative running on mesa i915. Clears to srgb textures will use manual shader clears.",
    "log.target": "wgpu_hal::gles::adapter",
    "log.module_path": "wgpu_hal::gles::adapter",
    "log.file": "/home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wgpu-hal-0.19.1/src/gles/adapter.rs",
    "log.line": 774
  },
  "target": "wgpu_hal::gles::adapter"
}
{
  "timestamp": "2024-04-22T09:33:08.894883Z",
  "level": "INFO",
  "fields": {
    "message": "Available adapters: \n - AdapterInfo { name: \"Intel(R) UHD Graphics 620 (KBL GT2)\", vendor: 32902, device: 22807, device_type: IntegratedGpu, driver: \"Intel open-source Mesa driver\", driver_info: \"Mesa 22.3.6\", backend: Vulkan }\n - AdapterInfo { name: \"llvmpipe (LLVM 15.0.6, 256 bits)\", vendor: 65541, device: 0, device_type: Cpu, driver: \"llvmpipe\", driver_info: \"Mesa 22.3.6 (LLVM 15.0.6)\", backend: Vulkan }\n - AdapterInfo { name: \"Mesa Intel(R) UHD Graphics 620 (KBL GT2)\", vendor: 32902, device: 0, device_type: IntegratedGpu, driver: \"\", driver_info: \"\", backend: Gl }",                                                                                                                               
    "log.target": "compositor_render::wgpu::ctx",
    "log.module_path": "compositor_render::wgpu::ctx",
    "log.file": "compositor_render/src/wgpu/ctx.rs",
    "log.line": 144
  },
  "target": "compositor_render::wgpu::ctx"
}
{
  "timestamp": "2024-04-22T09:33:08.897989Z",
  "level": "WARN",
  "fields": {
    "message": "Detected skylake derivative running on mesa i915. Clears to srgb textures will use manual shader clears.",
    "log.target": "wgpu_hal::gles::adapter",
    "log.module_path": "wgpu_hal::gles::adapter",
    "log.file": "/home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wgpu-hal-0.19.1/src/gles/adapter.rs",
    "log.line": 774
  },
  "target": "wgpu_hal::gles::adapter"
}
{
  "timestamp": "2024-04-22T09:33:08.899573Z",
  "level": "INFO",
  "fields": {
    "message": "Using Intel(R) UHD Graphics 620 (KBL GT2) adapter with Vulkan backend",
    "log.target": "compositor_render::wgpu::ctx",
    "log.module_path": "compositor_render::wgpu::ctx",
    "log.file": "compositor_render/src/wgpu/ctx.rs",
    "log.line": 64
  },
  "target": "compositor_render::wgpu::ctx"
}
ludolpif commented 6 months ago

I have doubts about going in the docker environment because I hope using physical devices (cams, mics) , network NDI or SRT streams in a not-so-static fashion (and the GPU accel). Am I getting it wrong thinking "this will be pain with docker" ? I am considering include membrane and video_compositor in a Live Streaming setup to got less shaders things in OBS or not using OBS at all. I know that is early and geeky but explorative.

wkozyra95 commented 6 months ago

Mesa 22.3.6 is to old for CPU based rendering, but I'm not sure if it is too old for GPU based. It's possible that mesa version is ok but GPU driver does not support it. I'll try to look into that more latter this week.

As for the log formatting, if you want to have human-readable logs from compositor you can set LIVE_COMPOSITOR_LOGGER_FORMAT=compact env (https://compositor.live/docs/deployment/configuration#live_compositor_logger_format)

Am I getting it wrong thinking "this will be pain with docker"

For your use case probably better to avoid it. Passing dedicated GPU to docker is not that hard, but I'm not sure how it would work with other devices or integrated GPU. This approach is more intended for cloud deployments.

ludolpif commented 6 months ago

On #wgpu-users:matrix.org chan were I've asked before understanding there is feature requested by the live compositor, there were a reply for my particular GPU and UNIFORM_BUFFER_AND_STORAGE_TEXTURE_ARRAY_NON_UNIFORM_INDEXING feature : cwfitzgerald (wgpu, rend3) said : That gpu doesn't support that particular feature flag, unfortunately.

ludolpif commented 6 months ago

As a hacky try : I managed to compile and run video_compositor on the aforementioned graphics card by commenting out the requested feature. simple.rs example with shader works, some others too, some didn't display any window nor errors. I am unsure where this requested feature is used in the compositor right now.

As reasonable try : I tried to run the compositor on a desktop computer with a dedicated Nvidia GTX 1060 and Ubuntu 23.10, I got frame drops (avg FPS below 25 it seems) with shipped examples, even tweaking a few to stick with "ultrafast" profile on output (instead of fast). Is there some tips to know to try to figure out where the bottleneck(s) is/are ?

wkozyra95 commented 6 months ago

I am unsure where this requested feature is used in the compositor right now.

Currently, it is not used in our shaders, but it can be used in user-defined shaders.

I tried to run the compositor on a desktop computer with a dedicated Nvidia GTX 1060 and Ubuntu 23.10. I got frame drops (avg FPS below 25 it seems)

Sounds like a missing driver issue, find the log like 

"Using Intel(R) UHD Graphics 620 (KBL GT2) adapter with Vulkan backend"

And make sure that is using Vulkan and that nvidia card. If you see llvmpipe there that means CPU rendering.

Is there some tips to know to try to figure out where the bottleneck(s) is/are ?

If switching to ultrafast did not help then most likely rendering is a bottleneck. Any graphics card should be enough to handle rendering, so I suspect you are rendering on CPU.

ludolpif commented 6 months ago

After trying again on the desktop computer, I see NVIDIA / Vulkan and frame drops are gone if I switch simple.rs from fast to ultrafast today. I must messed up something or got a ressource-eating process left in background when I first tried. Sorry for the noise.

About minimum features and the abort I got : I will suggest two PR to (1) don't panic (but errror!() and exit) on missing feature by asking wgpu adapter supported features, (2) let start in a non-nominal mode for some not-so-critical Features instead of aborting. (2) is for my poor's man/hw use case and may not be the current goals of the project.

May we add a note in a README or so about targeted GPU / known non working for now situations and close this issue ? If it can help : say me where it best fit from your point of view and I may try write some concise lines for future newcomers.

wkozyra95 commented 6 months ago

May we add a note in a README or so about targeted GPU / known non working for now situations and close this issue ?

I'm not sure what you would like to write there. For now, the only non-working situation is if you have to old drivers. Identifying a minimal version of drivers for each GPU vendor would require a ton of research and testing. Even if did that just the version of the driver is not really helpful to the user.

We plan to improve the documentation about what the compositor requires, but it will still be up to the user to translate those requirements to their specific operating system, hardware, and driver versions.