RK3588移植问题请教

VincentHQL commented 2 years ago

您好，我尝试将RK3588原厂提供的sdk中的 libGLES_mali.so ， android.hardware.graphics.allocator@4.0-impl-bifrost.so 和android.hardware.graphics.mapper@4.0-impl-bifrost.so移植到redroid中。容器运行成功并且能够识别出GPU。识别信息如下：

但是我必须在宿主机上运行glmark2-es2-wayland 才能正常运行，否则会收到内核错误，如下：

[  121.267421] mali fb000000.gpu: Unhandled Page fault in AS1 at VA 0x0000000000000020
[  121.267421] Reason: Memory is not mapped on the GPU
[  121.267421] raw fault status: 0x246002C1
[  121.267421] exception type 0xC1: UNKNOWN
[  121.267421] access type 0x2: READ
[  121.267421] source id 0x2460
[  121.267421] pid: 3548
[  125.444803] mali fb000000.gpu: Unhandled Page fault in AS1 at VA 0x000080002EC00000
[  125.444803] Reason: Memory is not mapped on the GPU
[  125.444803] raw fault status: 0x201001C2
[  125.444803] exception type 0xC2: UNKNOWN
[  125.444803] access type 0x1: EXECUTE
[  125.444803] source id 0x2010
[  125.444803] pid: 3268

似乎是内核中的内存映射错误。请问是不是容器内 aloocator的内存和宿主机中有冲突？

zhouziyang commented 2 years ago

道理上不会冲突，关注下render相关devices的permission

VincentHQL commented 2 years ago

问题已经解决，感谢回答。原因是设备节点/dev/mali0，在宿主机中 ls -i /dev/mali0 inode id 和容器中的 inode id不一致导致的。

运行时添加 --mount=type=bind,source=/dev/mali0,destination=/dev/mali0

容器中
ccfcc44bb96f:/ # ls -i /dev/mali0
83 /dev/mali0

宿主机：
~/docker$ ls -i /dev/mali0
203 /dev/mali0

在启动容器时会报错：

[38000.966756] **mali fb000000.gpu: mali_file_inode: 203  f_inode: 83**
[38000.966806] ------------[ cut here ]------------
[38000.966826] WARNING: CPU: 1 PID: 15606 at drivers/gpu/arm/bifrost/csf/mali_kbase_csf.c:1727 kbase_csf_ctx_init+0xa8/0x180
[38000.966830] Modules linked in: ahci_platform libahci_platform libahci libata scsi_mod rtk_btusb 8821cu btusb btrtl btbcm btintel
[38000.966884] CPU: 1 PID: 15606 Comm: RenderThread Tainted: G        W         5.10.66 #54
[38000.966888] Hardware name: Firefly ROC-RK3588S-PC HDMI(Linux) (DT)
[38000.966896] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO BTYPE=--)
[38000.966901] pc : kbase_csf_ctx_init+0xa8/0x180
[38000.966907] lr : kbase_csf_ctx_init+0x94/0x180
[38000.966910] sp : ffffffc0176f3bb0
[38000.966914] x29: ffffffc0176f3bb0 x28: ffffff815b943c00
[38000.966923] x27: 0000000000000000 x26: 0000000000000000
[38000.966931] x25: 0000000000000000 x24: 0000000000000018
[38000.966940] x23: 0000000000000009 x22: ffffffc0112ccba0
[38000.966949] x21: ffffff8106198118 x20: ffffff8106198000
[38000.966958] x19: ffffffc012d77000 x18: 0000000000000000
[38000.966966] x17: 0000000000000000 x16: 0000000000000000
[38000.966975] x15: 000000000000000a x14: 00000000000ec064
[38000.966983] x13: 0000000000000006 x12: ffffffffffffffff
[38000.966992] x11: 0000000000000010 x10: ffffffc0976f37f7
[38000.967000] x9 : ffffffc0100a52b0 x8 : 65646f6e695f6620
[38000.967009] x7 : 20333032203a6564 x6 : ffffffc0129943f0
[38000.967017] x5 : c0000001000987b9 x4 : ffffff83fd5ecab0
[38000.967026] x3 : 0000000000000000 x2 : 0000000000000000
[38000.967034] x1 : ffffff810566aee8 x0 : ffffff8185f0d6a8
[38000.967044] Call trace:
[38000.967051] kbase_csf_ctx_init+0xa8/0x180
[38000.967059] kbase_create_context+0x74/0x144
[38000.967067] kbase_file_create_kctx+0x7c/0x1e0
[38000.967073] kbase_ioctl+0x3a8/0x2bfc
[38000.967082] vfs_ioctl+0x34/0x54
[38000.967087] __arm64_sys_ioctl+0x84/0xb8
[38000.967096] el0_svc_common.constprop.0+0x154/0x204
[38000.967101] do_el0_svc+0x8c/0x98
[38000.967108] el0_svc+0x20/0x30
[38000.967113] el0_sync_handler+0xd8/0x184
[38000.967120] el0_sync+0x1a0/0x1c0
[38000.967124] \x0aPC: 0xffffffc010935928:
[38000.967127] 5928  3940a345 aa1903e2 b9408a64 aa1503e0 b957a263 f94012e1 97f59d03 aa1a03e0
[38000.967154] 5948  97fffd43 a9400281 f9000420 f9000001 f9000294 f9000694 17ffffd6 a94153f3
[38000.967180] 5968  a9425bf5 a94363f7 a9446bf9 a8c77bfd d50323bf d65f03c0 d503245f aa1e03e9
[38000.967206] 5988  d503201f d503233f a9bd7bfd 912b6001 910003fd a90153f3 aa0003f3 f90013f5
[38000.967231] 59a8  f9056c01 f9057001 d2828c01 8b010001 f90a3001 f90a3401 f9400414 94005164
[38000.967257] 59c8  f90a3a7f 91046295 aa1503e0 9420c3f0 f9400260 b4000280 f9587e81 f9401000
[38000.967283] 59e8  b5000041 f9187e80 f9400260 f9401001 f9587e80 f9402023 b00069e1 f9402002
[38000.967309] 5a08  91297821 f9401280 94201007 f9400260 f9587e81 f9401000 eb00003f 54000040

mizcara commented 2 years ago

Hi.

I am also attempting to run redroid on a rk3588 board too. I manage to run redroid using the BSP kernel with custom kernel config, but am unable to tap into the host GPU, e.g.

redroid_gpu_mode unset (=auto) - high CPU usage;
redroid_gpu_mode=host - surfaceflinger crashed on "eglQueryStringImplementationANDROID(EGL_VERSION) failed"

How should I go about connecting things together so the Mali GPU can be used inside the redroid container?

zhouziyang commented 2 years ago

Should customize redroid images just like @VincentHQL mentioned above (GLES, allocator, mapper etc.). There are reversed Mali GPU drivers (Panfrost). But not sure whether worked or stable.

mizcara commented 2 years ago

I tried to shove the following files on top of redroid/redroid:12.0.0-latest via a Dockerfile:

/vendor/lib64/hw/android.hardware.graphics.mapper@4.0-impl-bifrost.so
/vendor/lib64/hw/android.hardware.graphics.allocator@4.0-impl-bifrost.so
/vendor/lib64/egl/libGLES_mali.so
/vendor/lib64/libdrm.so (reported dependency)
/vendor/lib64/arm.graphics-V1-ndk_platform.so (reported dependency)

libGLES_mali.so seems to get loaded but failed with errors:

10-28 11:22:06.660 374 374 D libEGL : loaded /vendor/lib64/egl/libGLES_mali.so 10-28 11:22:06.669 374 374 E mali : Failed creating base context during opening of kernel driver. 10-28 11:22:06.669 374 374 E mali : Kernel module may not have been loaded

Am I heading in the right direction with the Dockerfile approach? Any clues/hints on the "Failed creating base context" error?

Thanks.

zhouziyang commented 2 years ago

Possible incorrect permission (0x666) of the render node (/dev/mali...)?

Tiananmen8964 commented 1 year ago

Could anyone guide me how to customize redroid image and get it work, thanks. I have these files from SDK already.

tangzhi1996 commented 1 year ago

您好，我尝试将RK3588原厂提供的sdk中的 libGLES_mali.so ， android.hardware.graphics.allocator@4.0-impl-bifrost.so 和android.hardware.graphics.mapper@4.0-impl-bifrost.so移植到redroid中。容器运行成功并且能够识别出GPU。识别信息如下：

但是我必须在宿主机上运行glmark2-es2-wayland 才能正常运行，否则会收到内核错误，如下：
[  121.267421] mali fb000000.gpu: Unhandled Page fault in AS1 at VA 0x0000000000000020
[  121.267421] Reason: Memory is not mapped on the GPU
[  121.267421] raw fault status: 0x246002C1
[  121.267421] exception type 0xC1: UNKNOWN
[  121.267421] access type 0x2: READ
[  121.267421] source id 0x2460
[  121.267421] pid: 3548
[  125.444803] mali fb000000.gpu: Unhandled Page fault in AS1 at VA 0x000080002EC00000
[  125.444803] Reason: Memory is not mapped on the GPU
[  125.444803] raw fault status: 0x201001C2
[  125.444803] exception type 0xC2: UNKNOWN
[  125.444803] access type 0x1: EXECUTE
[  125.444803] source id 0x2010
[  125.444803] pid: 3268
似乎是内核中的内存映射错误。请问是不是容器内 aloocator的内存和宿主机中有冲突？

兄弟,最后你解决了没有呀,我直接是连Android的容器都无法正常连接上,

VincentHQL commented 1 year ago

您好，我尝试将RK3588原厂提供的sdk中的 libGLES_mali.so ， android.hardware.graphics.allocator@4.0-impl-bifrost.so 和android.hardware.graphics.mapper@4.0-impl-bifrost.so移植到redroid中。容器运行成功并且能够识别出GPU。识别信息如下：但是我必须在宿主机上运行glmark2-es2-wayland 才能正常运行，否则会收到内核错误，如下：
[  121.267421] mali fb000000.gpu: Unhandled Page fault in AS1 at VA 0x0000000000000020
[  121.267421] Reason: Memory is not mapped on the GPU
[  121.267421] raw fault status: 0x246002C1
[  121.267421] exception type 0xC1: UNKNOWN
[  121.267421] access type 0x2: READ
[  121.267421] source id 0x2460
[  121.267421] pid: 3548
[  125.444803] mali fb000000.gpu: Unhandled Page fault in AS1 at VA 0x000080002EC00000
[  125.444803] Reason: Memory is not mapped on the GPU
[  125.444803] raw fault status: 0x201001C2
[  125.444803] exception type 0xC2: UNKNOWN
[  125.444803] access type 0x1: EXECUTE
[  125.444803] source id 0x2010
[  125.444803] pid: 3268
似乎是内核中的内存映射错误。请问是不是容器内 aloocator的内存和宿主机中有冲突？
兄弟,最后你解决了没有呀,我直接是连Android的容器都无法正常连接上,

解决了

tangzhi1996 commented 1 year ago

您好，我尝试将RK3588原厂提供的sdk中的 libGLES_mali.so ， android.hardware.graphics.allocator@4.0-impl-bifrost.so 和android.hardware.graphics.mapper@4.0-impl-bifrost.so移植到redroid中。容器运行成功并且能够识别出GPU。识别信息如下：但是我必须在宿主机上运行glmark2-es2-wayland 才能正常运行，否则会收到内核错误，如下：
[  121.267421] mali fb000000.gpu: Unhandled Page fault in AS1 at VA 0x0000000000000020
[  121.267421] Reason: Memory is not mapped on the GPU
[  121.267421] raw fault status: 0x246002C1
[  121.267421] exception type 0xC1: UNKNOWN
[  121.267421] access type 0x2: READ
[  121.267421] source id 0x2460
[  121.267421] pid: 3548
[  125.444803] mali fb000000.gpu: Unhandled Page fault in AS1 at VA 0x000080002EC00000
[  125.444803] Reason: Memory is not mapped on the GPU
[  125.444803] raw fault status: 0x201001C2
[  125.444803] exception type 0xC2: UNKNOWN
[  125.444803] access type 0x1: EXECUTE
[  125.444803] source id 0x2010
[  125.444803] pid: 3268
似乎是内核中的内存映射错误。请问是不是容器内 aloocator的内存和宿主机中有冲突？
兄弟,最后你解决了没有呀,我直接是连Android的容器都无法正常连接上,
解决了

出个教程,让我也用上全力使用GPU的快乐,来自不懂C,C++的人

VincentHQL commented 1 year ago

您好，我尝试将RK3588原厂提供的sdk中的 libGLES_mali.so ， android.hardware.graphics.allocator@4.0-impl-bifrost.so 和android.hardware.graphics.mapper@4.0-impl-bifrost.so移植到redroid中。容器运行成功并且能够识别出GPU。识别信息如下：但是我必须在宿主机上运行glmark2-es2-wayland 才能正常运行，否则会收到内核错误，如下：
[  121.267421] mali fb000000.gpu: Unhandled Page fault in AS1 at VA 0x0000000000000020
[  121.267421] Reason: Memory is not mapped on the GPU
[  121.267421] raw fault status: 0x246002C1
[  121.267421] exception type 0xC1: UNKNOWN
[  121.267421] access type 0x2: READ
[  121.267421] source id 0x2460
[  121.267421] pid: 3548
[  125.444803] mali fb000000.gpu: Unhandled Page fault in AS1 at VA 0x000080002EC00000
[  125.444803] Reason: Memory is not mapped on the GPU
[  125.444803] raw fault status: 0x201001C2
[  125.444803] exception type 0xC2: UNKNOWN
[  125.444803] access type 0x1: EXECUTE
[  125.444803] source id 0x2010
[  125.444803] pid: 3268
似乎是内核中的内存映射错误。请问是不是容器内 aloocator的内存和宿主机中有冲突？
兄弟,最后你解决了没有呀,我直接是连Android的容器都无法正常连接上,
解决了
出个教程,让我也用上全力使用GPU的快乐,来自不懂C,C++的人

抱歉了，这是公司项目，不好发出来

tangzhi1996 commented 1 year ago

那就算了

------------------ 原始邮件 ------------------ 发件人: @.>; 发送时间: 2023年3月7日(星期二) 上午9:18 收件人: @.>; 抄送: @.>; @.>; 主题: Re: [remote-android/redroid-doc] RK3588移植问题请教 (Issue #228)

您好，我尝试将RK3588原厂提供的sdk中的 libGLES_mali.so ， @. @.移植到redroid中。容器运行成功并且能够识别出GPU。识别信息如下：但是我必须在宿主机上运行glmark2-es2-wayland 才能正常运行，否则会收到内核错误，如下： [ 121.267421] mali fb000000.gpu: Unhandled Page fault in AS1 at VA 0x0000000000000020 [ 121.267421] Reason: Memory is not mapped on the GPU [ 121.267421] raw fault status: 0x246002C1 [ 121.267421] exception type 0xC1: UNKNOWN [ 121.267421] access type 0x2: READ [ 121.267421] source id 0x2460 [ 121.267421] pid: 3548 [ 125.444803] mali fb000000.gpu: Unhandled Page fault in AS1 at VA 0x000080002EC00000 [ 125.444803] Reason: Memory is not mapped on the GPU [ 125.444803] raw fault status: 0x201001C2 [ 125.444803] exception type 0xC2: UNKNOWN [ 125.444803] access type 0x1: EXECUTE [ 125.444803] source id 0x2010 [ 125.444803] pid: 3268
似乎是内核中的内存映射错误。请问是不是容器内 aloocator的内存和宿主机中有冲突？

兄弟,最后你解决了没有呀,我直接是连Android的容器都无法正常连接上,

解决了

出个教程,让我也用上全力使用GPU的快乐,来自不懂C,C++的人

抱歉了，这是公司项目，不好发出来

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

web3-coder-hub commented 1 year ago

@VincentHQL @tangzhi1996 你好，请问这几个so文件你们是在哪里获取的？

VincentHQL commented 1 year ago

@VincentHQL @tangzhi1996 你好，请问这几个so文件你们是在哪里获取的？

原厂给的sdk里面有

netPokey commented 1 year ago

大佬可以考虑一下给出一部分教程吗，或者加个联系方式一下你有个粉色的rock5a 想折腾折腾，我已经折腾好几天了，根据大佬的给出的几个项目，我现在有做到几个地方 1.在gpu_config 中添加对应代码，2将rk_gpu 代码根据rock5b 安卓12的代码放到vendor 下然后make ，我最后编译是成功了不加mali 也可以正常启动，但是加了mali 启动就sh能进去，进去后没有网络，求大佬帮助解答。谢谢联系方式：qq群:681972981 欢迎大家一起来交流讨论

chaixiang2002 commented 10 months ago

大佬可以考虑一下给出一部分教程吗，或者加个联系方式一下你有个粉色的rock5a 想折腾折腾，我已经折腾好几天了，根据大佬的给出的几个项目，我现在有做到几个地方 1.在gpu_config 中添加对应代码，2将rk_gpu 代码根据rock5b 安卓12的代码放到vendor 下然后make ，我最后编译是成功了不加mali 也可以正常启动，但是加了mali 启动就sh能进去，进去后没有网络，求大佬帮助解答。谢谢联系方式：qq群:681972981 欢迎大家一起来交流讨论

你好，我如今也遇到这个问题，我现在有一个rk3558的版，我应该如何在redroid项目中使用maliG610,QQ群能不能通过一下

ice-black-tea commented 9 months ago

按照各位大佬思路，我集成了一个rk3588的版本，代码和编译方式都在redroid-rockchip了

CuiZhaoHui commented 7 months ago

I tried to shove the following files on top of redroid/redroid:12.0.0-latest via a Dockerfile:

/vendor/lib64/hw/android.hardware.graphics.mapper@4.0-impl-bifrost.so

/vendor/lib64/hw/android.hardware.graphics.allocator@4.0-impl-bifrost.so

/vendor/lib64/egl/libGLES_mali.so

/vendor/lib64/libdrm.so (reported dependency)

/vendor/lib64/arm.graphics-V1-ndk_platform.so (reported dependency)

libGLES_mali.so seems to get loaded but failed with errors:

10-28 11:22:06.660 374 374 D libEGL : loaded /vendor/lib64/egl/libGLES_mali.so 10-28 11:22:06.669 374 374 E mali : Failed creating base context during opening of kernel driver. 10-28 11:22:06.669 374 374 E mali : Kernel module may not have been loaded

Am I heading in the right direction with the Dockerfile approach? Any clues/hints on the "Failed creating base context" error?

Thanks.

Did you resolve this? Can you share your plan? @mizcara

netPokey commented 7 months ago

I tried to shove the following files on top of redroid/redroid:12.0.0-latest via a Dockerfile:

/vendor/lib64/hw/android.hardware.graphics.mapper@4.0-impl-bifrost.so

/vendor/lib64/hw/android.hardware.graphics.allocator@4.0-impl-bifrost.so

/vendor/lib64/egl/libGLES_mali.so

/vendor/lib64/libdrm.so (reported dependency)

/vendor/lib64/arm.graphics-V1-ndk_platform.so (reported dependency)

libGLES_mali.so seems to get loaded but failed with errors:

10-28 11:22:06.660 374 374 D libEGL : loaded /vendor/lib64/egl/libGLES_mali.so 10-28 11:22:06.669 374 374 E mali : Failed creating base context during opening of kernel driver. 10-28 11:22:06.669 374 374 E mali : Kernel module may not have been loaded

Am I heading in the right direction with the Dockerfile approach? Any clues/hints on the "Failed creating base context" error? Thanks.

Did you resolve this? Can you share your plan? @mizcara

你可以看下https://github.com/remote-android/redroid-doc/issues/228#issuecomment-1951096612 这个大佬的

remote-android / redroid-doc

RK3588移植问题请教 #228