redroid nVidia GPU support

iervn6341-ovo commented 1 year ago

looks like my docekr not using my nvidia gpu

OS:Ubuntu 22.04 GPU:GeForce 1050ti(notebook)

nvidia-smi output:

start container as: sudo docker run -itd --privileged \ -v ~/data11:/data \ -p 5555:5555 \ redroid-11-libndk:latest \ ro.product.cpu.abilist=x86_64,arm64-v8a,x86,armeabi-v7a,armeabi \ ro.product.cpu.abilist64=x86_64,arm64-v8a \ ro.product.cpu.abilist32=x86,armeabi-v7a,armeabi \ ro.dalvik.vm.isa.arm=x86 \ ro.dalvik.vm.isa.arm64=x86_64 \ ro.enable.native.bridge.exec=1 \ ro.dalvik.vm.native.bridge=libndk_translation.so \ ro.ndk_translation.version=0.2.2 \ androidboot.redroid_width=1080 \ androidboot.redroid_height=2340 \ androidboot.redroid_dpi=440 i'm already installed nvidia-docker2 and checked gpu support for docker.

zhouziyang commented 1 year ago

For Nvidia GPUs, several options:

setup VM with virtio-gpu (supported by redroid currently)
use reversed driver nouveau (possible buggy, supported by redroid currently)
use GLES_emulation / gfxstream (just like Android Emulator, not available in redroid yet)
use Nvidia proprietary Android drivers (Limited Android versions, and need proprietary GPU kernel driver)

he110jian commented 9 months ago

For NVidia GPUs, several options:

setup VM, use virtio-gpu (redroid supported currently)

use reversed driver nouveau (possible buggy, redroid supported currently)

use GLES_emulation (like Android Emulator, not available in redroid yet)

waiting for open source NVK driver (zink on top)

use NVidia proprietary Android drivers

anything new? so when i select the virtio-gpu, this situtaion will be like : ubuntu(quem(ubuntu(docker(redroid))), is there any simple way? BTW Will there be any performance loss?

zhouziyang commented 9 months ago

For NVidia GPUs, several options:

setup VM, use virtio-gpu (redroid supported currently)

use reversed driver nouveau (possible buggy, redroid supported currently)

use GLES_emulation (like Android Emulator, not available in redroid yet)

waiting for open source NVK driver (zink on top)

use NVidia proprietary Android drivers

anything new? so when i select the virtio-gpu, this situtaion will be like : ubuntu(quem(ubuntu(docker(redroid))), is there any simple way? BTW Will there be any performance loss?

It's a correct setup for virtio-gpu; And performance / compatibility is not good as native Nvidia proprietary Android drivers of course.

tzmtl commented 8 months ago

For NVidia GPUs, several options:

setup VM, use virtio-gpu (redroid supported currently)

use reversed driver nouveau (possible buggy, redroid supported currently)

use GLES_emulation (like Android Emulator, not available in redroid yet)

waiting for open source NVK driver (zink on top)

use NVidia proprietary Android drivers

For the last option, the Nvidia proprietary Android driver, can you please explain a little bit more about the procedure? After the Nvidia proprietary GPU driver is installed to the host, do we have to modify anything inside redroid, or configure anything in the host?

Thanks in advance!

zhouziyang commented 8 months ago

@tzmtl AFAIK, this is a Android specialized driver (including kernel space and user space); Try contact Nvidia to get this licensed product.

tzmtl commented 8 months ago

I'm trying to launch redroid 14 in a host with kernel 6.6 and Nvidia GPU. There are some changes with nouveau driver in kernel 6.6. So, I hope it can provide better support for Nvidia GPU.

However, it's unable to launch redroid properly. I got some errors like below,

03-21 17:39:26.004  3464  3464 F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
03-21 17:39:26.004  3464  3464 F DEBUG   : Build fingerprint: 'redroid/redroid_x86_64/redroid_x86_64:14/UP1A.231005.007.A1/eng.frank.20231013.082327:userdebug/test-keys'
03-21 17:39:26.004  3464  3464 F DEBUG   : Revision: '0'
03-21 17:39:26.004  3464  3464 F DEBUG   : ABI: 'x86_64'
03-21 17:39:26.004  3464  3464 F DEBUG   : Timestamp: 2024-03-21 17:39:25.962637638+0000
03-21 17:39:26.004  3464  3464 F DEBUG   : Process uptime: 1s
03-21 17:39:26.004  3464  3464 F DEBUG   : Cmdline: /system/bin/surfaceflinger
03-21 17:39:26.004  3464  3464 F DEBUG   : pid: 3450, tid: 3450, name: surfaceflinger  >>> /system/bin/surfaceflinger <<<
03-21 17:39:26.004  3464  3464 F DEBUG   : uid: 1000
03-21 17:39:26.004  3464  3464 F DEBUG   : signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
03-21 17:39:26.004  3464  3464 F DEBUG   : Abort message: 'FORTIFY: pthread_mutex_destroy called on a destroyed mutex (0x725d80244880)'
03-21 17:39:26.004  3464  3464 F DEBUG   :     rax 0000000000000000  rbx 00007ffd5bcffaf8  rcx 0000725f2a270610  rdx 0000000000000006
03-21 17:39:26.004  3464  3464 F DEBUG   :     r8  0000000000000000  r9  0000000000000000  r10 00007ffd5bcffb00  r11 0000000000000207
03-21 17:39:26.004  3464  3464 F DEBUG   :     r12 0000725c83f2e960  r13 0000000000000000  r14 0000000000000d7a  r15 0000000000000d7a
03-21 17:39:26.004  3464  3464 F DEBUG   :     rdi 0000000000000d7a  rsi 0000000000000d7a
03-21 17:39:26.004  3464  3464 F DEBUG   :     rbp 0000725d80244638  rsp 00007ffd5bcffaf0  rip 0000725f2a270610
03-21 17:39:26.004  3464  3464 F DEBUG   : 17 total frames
03-21 17:39:26.004  3464  3464 F DEBUG   : backtrace:
03-21 17:39:26.004  3464  3464 F DEBUG   :       #00 pc 0000000000061610  /apex/com.android.runtime/lib64/bionic/libc.so (abort+192) (BuildId: fa337969c798946280caa45e2d71a2e7)
03-21 17:39:26.004  3464  3464 F DEBUG   :       #01 pc 00000000000633a0  /apex/com.android.runtime/lib64/bionic/libc.so (__fortify_fatal(char const*, ...)+160) (BuildId: fa337969c798946280caa45e2d71a2e7)
03-21 17:39:26.004  3464  3464 F DEBUG   :       #02 pc 00000000000cdf4f  /apex/com.android.runtime/lib64/bionic/libc.so (HandleUsingDestroyedMutex(pthread_mutex_t*, char const*)+47) (BuildId: fa337969c798946280caa45e2d71a2e7)
03-21 17:39:26.004  3464  3464 F DEBUG   :       #03 pc 00000000000ce6ca  /apex/com.android.runtime/lib64/bionic/libc.so (pthread_mutex_destroy+90) (BuildId: fa337969c798946280caa45e2d71a2e7)
03-21 17:39:26.004  3464  3464 F DEBUG   :       #04 pc 0000000000b2d24e  /vendor/lib64/dri/libgallium_dri.so (BuildId: dc62e08fc216108ca3cf815b69d2d7a807a9941f)
03-21 17:39:26.004  3464  3464 F DEBUG   :       #05 pc 0000000000b2f27e  /vendor/lib64/dri/libgallium_dri.so (BuildId: dc62e08fc216108ca3cf815b69d2d7a807a9941f)
03-21 17:39:26.004  3464  3464 F DEBUG   :       #06 pc 000000000002a853  /vendor/lib64/egl/libEGL_mesa.so
03-21 17:39:26.004  3464  3464 F DEBUG   :       #07 pc 000000000002f24f  /vendor/lib64/egl/libEGL_mesa.so
03-21 17:39:26.004  3464  3464 F DEBUG   :       #08 pc 000000000002ec2d  /vendor/lib64/egl/libEGL_mesa.so
03-21 17:39:26.004  3464  3464 F DEBUG   :       #09 pc 000000000002b7e8  /vendor/lib64/egl/libEGL_mesa.so
03-21 17:39:26.004  3464  3464 F DEBUG   :       #10 pc 000000000001c1d4  /vendor/lib64/egl/libEGL_mesa.so (eglInitialize+228)
03-21 17:39:26.004  3464  3464 F DEBUG   :       #11 pc 0000000000018c83  /system/lib64/libEGL.so (android::egl_display_t::initialize(int*, int*)+339) (BuildId: f2922d3b37a9eb91881db71e18b968ce)
03-21 17:39:26.004  3464  3464 F DEBUG   :       #12 pc 00000000002dc22f  /system/bin/surfaceflinger (android::renderengine::gl::GLESRenderEngine::create(android::renderengine::RenderEngineCreationArgs const&)+63) (BuildId: 6d7d6aad628562199b2107f057fc932c)
03-21 17:39:26.004  3464  3464 F DEBUG   :       #13 pc 00000000002dbc98  /system/bin/surfaceflinger (android::renderengine::RenderEngine::create(android::renderengine::RenderEngineCreationArgs const&)+632) (BuildId: 6d7d6aad628562199b2107f057fc932c)
03-21 17:39:26.004  3464  3464 F DEBUG   :       #14 pc 0000000000217b92  /system/bin/surfaceflinger (android::SurfaceFlinger::init()+818) (BuildId: 6d7d6aad628562199b2107f057fc932c)
03-21 17:39:26.004  3464  3464 F DEBUG   :       #15 pc 0000000000274146  /system/bin/surfaceflinger (main+1190) (BuildId: 6d7d6aad628562199b2107f057fc932c)
03-21 17:39:26.004  3464  3464 F DEBUG   :       #16 pc 00000000000529ef  /apex/com.android.runtime/lib64/bionic/libc.so (__libc_init+95) (BuildId: fa337969c798946280caa45e2d71a2e7)
03-21 17:39:26.125  3307  3307 F libc    : /buildbot/src/android/ndk-release-r23/toolchain/llvm-project/libcxx/../../../toolchain/llvm-project/libcxxabi/src/abort_message.cpp:72: abort_message: assertion "terminating" failed
03-21 17:39:26.125  3307  3307 F libc    : Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 3307 (binder:3307_2), pid 3307 (binder:3307_2)

Does redroid support kernel 6.6? Can is it possible to use the new nouveau driver for hardware acceleration?

zhouziyang commented 8 months ago

This is a known issue of mesa3d prebuilt. check #568

firemeteorxx commented 1 month ago

use GLES_emulation / gfxstream (just like Android Emulator, not available in redroid yet)

Hi @zhouziyang, I have some questions related to this option:

Does this option allows lightweight GPU acceleration (i.e. more container styled instead of VM or emulation)?
What's the prerequisite to make this option happen?
- I noticed a gfxstream related MR hass recently been accepted into MESA as an experimental Vulkan driver.
- Does this change makes it easier to implement this option in redroid?
- On the other hand, Mesa already had Venus before this change. Could Venus serve a similar purpose if gfxstream did not come?

zhouziyang commented 1 month ago

Quoted from gfxstream:

From a virtual machine guest to host for virtualized graphics From one process to another for IPC graphics From one computer to another via network sockets

Well, virtio-gpu is preferred which needs setup a VM.

firemeteorxx commented 1 month ago

Why would you say the VM based virtio-gpu is the preferred option? It's it a judgement from a technical perspective, or is it simply because it just works right now?

Forget about the efforts behind each option for a moment, isn't the following looks more appealing than a VM wrapper?

From one process to another for IPC graphics

Here appears to be a demonstration of such IPC graphics work without VM: https://crosvm.dev/book/appendix/rutabaga_gfx.html#kumquat-media-server

zhouziyang commented 1 month ago

Just much more efficient than IPC, I think.

firemeteorxx commented 1 month ago

Just much more efficient than IPC, I think.

This is really a statement that I never anticipated. Could you explain the rationale behind it? Was not able to find apple-to-apple comparison on the net directly.

From my naive understanding, the two sides of the communication should roughly stay the same between the VM and IPC setup. VM appears to be a heavier IPC tunnel, which leads me to believe IPC should be slightly better, unless the VM layer provides more features than tunneling?

tzmtl commented 1 month ago

Hi @zhouziyang,

Recently Maurossi implemented the android component of NVK driver.

I've built the driver and integrated with redroid. But I see different errors when SurfaceFlinger is starting. Do you have any interest to have a look and test it?

zhouziyang commented 1 month ago

NVK already added (check libvulkan_nouveau.so).

tzmtl commented 1 month ago

I see the lib libvulkan_nouveau.so was updated in commit "mesa3d 234.3.0 -> 24.0.8", about 5 months ago. I tested that version already, it doesn't work. Android HAL was not implemented in mesa3d nvk driver. It's implemented in the commit I pasted above about 2 months ago.

The script gpu_config.sh is not updated for nvk too.

NVK already added (check libvulkan_nouveau.so).

zhouziyang commented 1 month ago

Well, not test this reverse engineered driver. BTW, try addro.hardware.vulkan=nouveau to load NVK when start redroid container.

remote-android / redroid-doc

redroid nVidia GPU support #282