jcdutton
commented
2 years ago @Coreforge That is good progress. Where does Xorg stop? What errors ?
Open geerlingguy opened 2 years ago
jcdutton
commented
2 years ago @Coreforge That is good progress. Where does Xorg stop? What errors ?
Coreforge
commented
2 years ago Xorg shuts down without any real error that I can see. dmesg is full of messages like this though:
[ 102.911975] [drm:drm_ioctl [drm]] comm="Xorg" pid=742, dev=0xe202, auth=1, RADEON_GEM_WAIT_IDLE
[ 102.916670] [drm:drm_ioctl [drm]] comm="Xorg", pid=742, ret=-512The GPU then gets reset due to a ring 0 lockup, and Xorg shuts down. The last thing that's happening before the ring stall is an indirect buffer getting executed. IBs usually work though, so either data gets corrupted when it gets written into this one, or it contains some code that doesn't work.
jcdutton
commented
2 years ago @Coreforge For memmove, one does not need to allocate any extra buffers. If the address of src > dest then copy the data from start to end. If the address of src < dest then copy the data from the end to the start. If the address of src == dest, don't do anything.
Coreforge
commented
2 years ago I'll change it to that then. This was just a quick way to do it to see if it would work, which is does.
geerlingguy
commented
2 years ago /me just wanted to comment from the sidelines that this is a fascinating debugging session to tail via email 🍿
geerlingguy
commented
2 years ago Also—Coreforge, if you'd like me to send you my SOQuartz, I'd be more than willing to loan it out for a while and pay any shipping costs for return. Right now it's in a box in my office as I don't have the time to mess with it, and I have a Radxa CM3 as well that I will probably get to first with the same SoC.
jcdutton
commented
2 years ago @geerlingguy We just about have the SOQuartz working. Its somewhat easier to work with than the Raspberry Pi as it has fewer hardware bugs. Just need to fix / implement aligned writes in libc6 aarch64 assembly for memcpy and memmove.
Coreforge
commented
2 years ago For now it looks like the issue is mostly the same on the cm4, so I can just use that for debugging. If some issue turns up though that is only on the SOQuartz (which is doesn't really look like right now), having one could be useful. For now I'll continue on the cm4 though.
adminy
commented
2 years ago SoQuartz is incredibly good compared to the Pi, but only if you have a good io board ... that supports sd card, since you can only boot of that or an emmc module that you buy separately ...
pgwipeout
commented
2 years ago I managed to fix the problem at the source by borrowing the alignment concept from the kernel and u-boot's write commands. With this I got glmark2-drm to run to completion. Unfortunately it isn't enough for startx to work.
Patch:
diff --git a/src/gallium/drivers/r600/r600_buffer_common.c b/src/gallium/drivers/r600/r600_buffer_common.c
index eda8554b8d5f..a262cad70f47 100644
--- a/src/gallium/drivers/r600/r600_buffer_common.c
+++ b/src/gallium/drivers/r600/r600_buffer_common.c
@@ -31,6 +31,44 @@
#include <inttypes.h>
#include <stdio.h>
+#define IS_ALIGNED(x, a) (((x) & ((__typeof__(x))(a) - 1)) == 0)
+
+#define __arch_putb(v,a) (*(volatile unsigned char *)(a) = (v))
+#define __arch_putw(v,a) (*(volatile unsigned short *)(a) = (v))
+#define __arch_putl(v,a) (*(volatile unsigned int *)(a) = (v))
+#define __arch_putq(v,a) (*(volatile unsigned long long *)(a) = (v))
+
+#define __raw_writeb(v,a) __arch_putb(v,a)
+#define __raw_writew(v,a) __arch_putw(v,a)
+#define __raw_writel(v,a) __arch_putl(v,a)
+#define __raw_writeq(v,a) __arch_putq(v,a)
+
+void memcpy_toio(volatile void *to, const void *from, size_t count);
+
+void memcpy_toio(volatile void *to, const void *from, size_t count)
+{
+ while (count && !IS_ALIGNED((unsigned long)to, 8)) {
+ __raw_writeb(*(uint8_t *)from, to);
+ from++;
+ to++;
+ count--;
+ }
+
+ while (count >= 8) {
+ __raw_writeq(*(uint64_t *)from, to);
+ from += 8;
+ to += 8;
+ count -= 8;
+ }
+
+ while (count) {
+ __raw_writeb(*(uint8_t *)from, to);
+ from++;
+ to++;
+ count--;
+ }
+}
+
bool r600_rings_is_buffer_referenced(struct r600_common_context *ctx,
struct pb_buffer *buf,
unsigned usage)
@@ -567,7 +605,12 @@ void r600_buffer_subdata(struct pipe_context *ctx,
if (!map)
return;
- memcpy(map, data, size);
+ if (!IS_ALIGNED((unsigned long)map, 16)) {
+// fprintf(stderr, "UNALIGNED MEMCPY\n");
+ memcpy_toio(map, data, size);
+ } else {
+ memcpy(map, data, size);
+ }
r600_buffer_transfer_unmap(ctx, transfer);
}Results:
=======================================================
glmark2 2021.12
=======================================================
OpenGL Information
GL_VENDOR: X.Org
GL_RENDERER: AMD TURKS (DRM 2.50.0 / 5.17.0-rc5-00097-gccb1df4cf6b5-dirty, LLVM 13.0.0)
GL_VERSION: 3.1 Mesa 22.0.0 (git-a82920af6e)
Surface Config: buf=32 r=8 g=8 b=8 a=8 depth=24 stencil=0
Surface Size: 1920x1080 fullscreen
=======================================================
[build] use-vbo=false: FPS: 60 FrameTime: 16.667 ms
[build] use-vbo=true: FPS: 60 FrameTime: 16.667 ms
[texture] texture-filter=nearest: FPS: 60 FrameTime: 16.667 ms
[texture] texture-filter=linear: FPS: 60 FrameTime: 16.667 ms
[texture] texture-filter=mipmap: FPS: 60 FrameTime: 16.667 ms
[shading] shading=gouraud: FPS: 60 FrameTime: 16.667 ms
[shading] shading=blinn-phong-inf: FPS: 59 FrameTime: 16.949 ms
[shading] shading=phong: FPS: 59 FrameTime: 16.949 ms
[shading] shading=cel: FPS: 59 FrameTime: 16.949 ms
[bump] bump-render=high-poly: FPS: 60 FrameTime: 16.667 ms
[bump] bump-render=normals: FPS: 60 FrameTime: 16.667 ms
[bump] bump-render=height: FPS: 60 FrameTime: 16.667 ms
[effect2d] kernel=0,1,0;1,-4,1;0,1,0;: FPS: 60 FrameTime: 16.667 ms
[effect2d] kernel=1,1,1,1,1;1,1,1,1,1;1,1,1,1,1;: FPS: 59 FrameTime: 16.949 ms
[pulsar] light=false:quads=5:texture=false: FPS: 60 FrameTime: 16.667 ms
[desktop] blur-radius=5:effect=blur:passes=1:separable=true:windows=4: FPS: 59 FrameTime: 16.949 ms
[desktop] effect=shadow:windows=4: FPS: 59 FrameTime: 16.949 ms
[buffer] columns=200:interleave=false:update-dispersion=0.9:update-fraction=0.5:update-method=map: FPS: 30 FrameTime: 33.333 ms
[buffer] columns=200:interleave=false:update-dispersion=0.9:update-fraction=0.5:update-method=subdata: FPS: 29 FrameTime: 34.483 ms
[buffer] columns=200:interleave=true:update-dispersion=0.9:update-fraction=0.5:update-method=map: FPS: 59 FrameTime: 16.949 ms
[ideas] speed=duration: FPS: 49 FrameTime: 20.408 ms
[jellyfish] <default>: FPS: 59 FrameTime: 16.949 ms
[terrain] <default>: FPS: 27 FrameTime: 37.037 ms
[shadow] <default>: FPS: 59 FrameTime: 16.949 ms
[refract] <default>: FPS: 49 FrameTime: 20.408 ms
[conditionals] fragment-steps=0:vertex-steps=0: FPS: 60 FrameTime: 16.667 ms
[conditionals] fragment-steps=5:vertex-steps=0: FPS: 60 FrameTime: 16.667 ms
[conditionals] fragment-steps=0:vertex-steps=5: FPS: 59 FrameTime: 16.949 ms
[function] fragment-complexity=low:fragment-steps=5: FPS: 60 FrameTime: 16.667 ms
[function] fragment-complexity=medium:fragment-steps=5: FPS: 60 FrameTime: 16.667 ms
[loop] fragment-loop=false:fragment-steps=5:vertex-steps=5: FPS: 60 FrameTime: 16.667 ms
[loop] fragment-steps=5:fragment-uniform=false:vertex-steps=5: FPS: 60 FrameTime: 16.667 ms
[loop] fragment-steps=5:fragment-uniform=true:vertex-steps=5: FPS: 60 FrameTime: 16.667 ms
=======================================================
glmark2 Score: 56
=======================================================
miguemely
commented
2 years ago Also—Coreforge, if you'd like me to send you my SOQuartz, I'd be more than willing to loan it out for a while and pay any shipping costs for return. Right now it's in a box in my office as I don't have the time to mess with it, and I have a Radxa CM3 as well that I will probably get to first with the same SoC.
Ditto here. I have a few I can send out as well (I think 3 8GBs w/o EMMC)
pgwipeout
commented
2 years ago Found the next memcpy error, this time when running gnome-shell --wayland --display-server
This one is a little less trivial to fix.
Thread 1 "gnome-shell" received signal SIGBUS, Bus error.
__memcpy_generic () at ../sysdeps/aarch64/multiarch/../memcpy.S:139
139 ../sysdeps/aarch64/multiarch/../memcpy.S: No such file or directory.
(gdb) where
#0 __memcpy_generic () at ../sysdeps/aarch64/multiarch/../memcpy.S:139
#1 0x0000007f6599fb74 in st_TexSubImage (ctx=0x55558110d0, dims=2,
texImage=0x55565a4eb0, xoffset=1, yoffset=65, zoffset=0, width=9,
height=9, depth=1, format=32993, type=5121, pixels=0x555660f7c0,
unpack=0x5555843408) at ../src/mesa/state_tracker/st_cb_texture.c:2108
#2 0x0000007f659272b4 in texture_sub_image (ctx=0x55558110d0, dims=2,
texObj=0x55565a4a50, texImage=0x55565a4eb0, target=3553, level=0,
xoffset=1, yoffset=65, zoffset=0, width=9, height=9, depth=1,
format=32993, type=5121, pixels=0x555660f7c0)
at ../src/mesa/main/teximage.c:3563
#3 0x0000007f659274e0 in texsubimage_err (ctx=0x55558110d0, dims=2,
target=3553, level=0, xoffset=1, yoffset=65, zoffset=0, width=9, height=9,
depth=1, format=32993, type=5121, pixels=0x555660f7c0,
callerName=0x7f667e6858 "glTexSubImage2D")
at ../src/mesa/main/teximage.c:3621
#4 0x0000007f65928200 in _mesa_TexSubImage2D (target=3553, level=0,
xoffset=1, yoffset=65, width=9, height=9, format=32993, type=5121,
pixels=0x555660f7c0) at ../src/mesa/main/teximage.c:3843
#5 0x0000007ff4c484fc in glTexSubImage2D (target=3553, level=0, xoffset=1,
yoffset=65, width=9, height=9, format=32993, type=5121,
pixels=0x555660f7c0)
at /home/master/build/mesa/build/src/mapi/glapi/gen/glapi_mapi_tmp.h:3724
#6 0x0000007ff67230c8 in ?? ()Progress
jcdutton
commented
2 years ago Also—Coreforge, if you'd like me to send you my SOQuartz, I'd be more than willing to loan it out for a while and pay any shipping costs for return. Right now it's in a box in my office as I don't have the time to mess with it, and I have a Radxa CM3 as well that I will probably get to first with the same SoC.
Ditto here. I have a few I can send out as well (I think 3 8GBs w/o EMMC)
If you have any spare, I would like one.
jcdutton
commented
2 years ago Found the next memcpy error, this time when running
gnome-shell --wayland --display-serverThis one is a little less trivial to fix.Thread 1 "gnome-shell" received signal SIGBUS, Bus error. __memcpy_generic () at ../sysdeps/aarch64/multiarch/../memcpy.S:139 139 ../sysdeps/aarch64/multiarch/../memcpy.S: No such file or directory. (gdb) where #0 __memcpy_generic () at ../sysdeps/aarch64/multiarch/../memcpy.S:139
@pgwipeout
Like I said before, libc6 or glibc has the aarch64 assembly code for memcpy that needs fixing. Are you able to do a "disassemble" and an "i r" so we can check if the problem is with a load or a store ?
pgwipeout
commented
2 years ago @jcdutton There's actually three problems. load and store I've seen both, but the third one is size alignment.
I've rewritten my work to catch all three. Unfortunately mutter + wayland still has hilarious display corruption. But the performance on glmark2 is pretty good.
pgwipeout
commented
2 years ago =======================================================
glmark2 2021.12
=======================================================
OpenGL Information
GL_VENDOR: X.Org
GL_RENDERER: AMD TURKS (DRM 2.50.0 / 5.17.0-rc5-00097-gccb1df4cf6b5-dirty, LLVM 13.0.0)
GL_VERSION: 3.1 Mesa 22.0.0 (git-59144c5be6)
Surface Config: buf=32 r=8 g=8 b=8 a=8 depth=24 stencil=0
Surface Size: 800x600 windowed
=======================================================
[build] use-vbo=false: FPS: 318 FrameTime: 3.145 ms
[build] use-vbo=true: FPS: 556 FrameTime: 1.799 ms
[texture] texture-filter=nearest: FPS: 559 FrameTime: 1.789 ms
[texture] texture-filter=linear: FPS: 556 FrameTime: 1.799 ms
[texture] texture-filter=mipmap: FPS: 574 FrameTime: 1.742 ms
[shading] shading=gouraud: FPS: 492 FrameTime: 2.033 ms
[shading] shading=blinn-phong-inf: FPS: 492 FrameTime: 2.033 ms
[shading] shading=phong: FPS: 473 FrameTime: 2.114 ms
[shading] shading=cel: FPS: 465 FrameTime: 2.151 ms
[bump] bump-render=high-poly: FPS: 323 FrameTime: 3.096 ms
[bump] bump-render=normals: FPS: 571 FrameTime: 1.751 ms
[bump] bump-render=height: FPS: 562 FrameTime: 1.779 ms
[effect2d] kernel=0,1,0;1,-4,1;0,1,0;: FPS: 465 FrameTime: 2.151 ms
[effect2d] kernel=1,1,1,1,1;1,1,1,1,1;1,1,1,1,1;: FPS: 229 FrameTime: 4.367 ms
[pulsar] light=false:quads=5:texture=false: FPS: 532 FrameTime: 1.880 ms
[desktop] blur-radius=5:effect=blur:passes=1:separable=true:windows=4: FPS: 201 FrameTime: 4.975 ms
[desktop] effect=shadow:windows=4: FPS: 327 FrameTime: 3.058 ms
[buffer] columns=200:interleave=false:update-dispersion=0.9:update-fraction=0.5:update-method=map: FPS: 63 FrameTime: 15.873 ms
[buffer] columns=200:interleave=false:update-dispersion=0.9:update-fraction=0.5:update-method=subdata: FPS: 77 FrameTime: 12.987 ms
[buffer] columns=200:interleave=true:update-dispersion=0.9:update-fraction=0.5:update-method=map: FPS: 104 FrameTime: 9.615 ms
[ideas] speed=duration: FPS: 121 FrameTime: 8.264 ms
[jellyfish] <default>: FPS: 353 FrameTime: 2.833 ms
[terrain] <default>: FPS: 36 FrameTime: 27.778 ms
[shadow] <default>: FPS: 143 FrameTime: 6.993 ms
[refract] <default>: FPS: 48 FrameTime: 20.833 ms
[conditionals] fragment-steps=0:vertex-steps=0: FPS: 417 FrameTime: 2.398 ms
[conditionals] fragment-steps=5:vertex-steps=0: FPS: 417 FrameTime: 2.398 ms
[conditionals] fragment-steps=0:vertex-steps=5: FPS: 418 FrameTime: 2.392 ms
[function] fragment-complexity=low:fragment-steps=5: FPS: 417 FrameTime: 2.398 ms
[function] fragment-complexity=medium:fragment-steps=5: FPS: 417 FrameTime: 2.398 ms
[loop] fragment-loop=false:fragment-steps=5:vertex-steps=5: FPS: 417 FrameTime: 2.398 ms
[loop] fragment-steps=5:fragment-uniform=false:vertex-steps=5: FPS: 417 FrameTime: 2.398 ms
[loop] fragment-steps=5:fragment-uniform=true:vertex-steps=5: FPS: 229 FrameTime: 4.367 ms
=======================================================
glmark2 Score: 357
=======================================================Interestingly enough, glmark2 in full screen mode works fine, even with the wayland corruption underneath.
jcdutton
commented
2 years ago @pgwipeout Please can you post the output of "uname -a" I want to know if you are using aarch32 or aarch64
jcdutton
commented
2 years ago There's actually three problems. load and store I've seen both, but the third one is size alignment. I've rewritten my work to catch all three. Unfortunately mutter + wayland still has hilarious display corruption.
What did your "rewritten" involve? I have had a chat with various mesa people, and the general consensus is that the best way to fix this is by fixing the memcpy assembly code on aarch64, rather than attempt to fix anything in mesa. The reason for that, is that any application/game can mmap stuff, and they will probably just try to use memcpy as well, so to catch everything, fixing memcpy is the best choice.
jcdutton
commented
2 years ago Unfortunately mutter + wayland still has hilarious display corruption. But the performance on glmark2 is pretty good.
So, I don't know how we are going to track down those problems with mutter and wayland. Without an associated bus error, or crash, it is difficult to know where to start with it. Do you see any display artifacts on glmark2 ? Does a different window manager help? E.g. XFCE4 ?
pgwipeout
commented
2 years ago So instead of just flailing about, I revived my original investigation into this from a year ago. Some snips from the experts: Christian König christian.koenig@amd.com:
> The CM4 team is convinced this is an issue with memcpy in glibc, but
> I'm not convinced it's that simple.
Yes exactly that.
Both OpenGL and Vulkan allow the application to mmap() device memory and
do any memory access they want with that.
This means that changing memcpy is just a futile effort, it's still
possible for the application to make an unaligned memory access and that
is perfectly valid.Robin Murphy robin.murphy@arm.com:
> Robin:
> My questions for you, since you're the smartest person I know about
> arm64 memory management:
> Could cache snooping permit unaligned accesses to IO to be safe?
No.
> Or
> Is it the lack of an IOMMU that's causing the alignment faults to become fatal?
No.
> Or
> Am I insane here?
No. (probably)
CPU access to PCIe has nothing to do with PCIe's access to memory. From
what you've described, my guess is that a GPU BAR gets put in a
non-prefetchable window, such that it ends up mapped as Device memory
(whereas if it were prefetchable it would be Normal Non-Cacheable).I've already confirmed making a prefetchable window makes a difference, but now I'm running into the issue the prefetch isn't actually happening. TLDR: I'm still pretty sure there's no way to make these non compliant PCIe controllers magically compliant, even with a massive performance hit. But I'll keep running this into the ground because I want an exact answer.
jcdutton
commented
2 years ago @pgwipeout Looking at the lspci -vv that you posted previously, I am not sure it is setting up the bus correctly. On x86, the Bridge addresses are a superset of the device addresses attached to the bridge. From the arm lspci, the bridge addresses are not matching the devices. There are address translations that PCI busses do, so this might be fine, but the address translations are not listed in lspci, so its difficult to know if it is set up correctly. But, it would be good to check the 256MB bar is prefetchable, and that the prefetching is working.
jcdutton
commented
2 years ago @pgwipeout There are bugs in the aarch64 memcpy, so fixing them would help. That will not catch the cases where an app does its own non-aligned read/write to device memory.
pgwipeout
commented
2 years ago Those aren't bugs, they are architectural limitations. Coming from x86, you're going to be spoiled about how x86 handles things. In the RISC world, a lot of the rules you broke in x86 because it just ignored the violations will bite you here.
memcpy and any other direct memory access is inherently unsafe. A lot of work has gone into armv7 and armv8 to make sure normal system ram can make a safe environment, but that doesn't get extended to IO devices, because it literally cannot. The PCIe spec permits devices to be mapped as normal system ram, with all benefits of that. We already know BRCM doesn't care about complying with the spec, and their implementation is severely broken. It seems the rk35xx series also doesn't comply, but not nearly as bad.
I'm trying to narrow down exactly how bad, so that I can document it as we finalize the PCIe support for mainline.
pgwipeout
commented
2 years ago And yes, you're correct, the address mapping in Arm64 is fun compared to x86, because we are setting it up instead of firmware. A lot of the "bugs" you see in PCIe boil down to programmers taking advantage of x86 ignoring certain specifications and rules.
dtischler
commented
2 years ago You all are WAY above my head from a debugging perspective, but thought I'd chime in with some extra data points:
I am running a Quartz64 Model A, @pgwipeout kernel built from 5.16 branch, and an Armbian Station M2 OS. I take the image file that Armbian produces, flash to SD Card, then overwrite the Kernel (file is literally just named image) and the rk3566-quartz64-a.dtb file (also borrowed from Peter). With those two files overwritten on the SD Card, I can boot right up.
My kernel build from Peter's GitLab repo simply adds Radeon driver, but compiles directly in the radeon/oland_ce.bin radeon/oland_mc.bin radeon/oland_me.bin radeon/oland_pfp.bin radeon/oland_rlc.bin radeon/oland_smc.bin radeon/TAHITI_uvd.bin radeon/TAHITI_vce.bin as needed for my GPU.
All said, here are my logs:
pgwipeout
commented
2 years ago @dtischler Due to:
[ 2.472176] [drm:r600_ring_test] *ERROR* radeon: ring 0 test failed (scratch(0x850C)=0xCAFEDEAD)
[ 2.472972] radeon 0000:01:00.0: disabling GPU accelerationthe gpu is just a display output. Graphical acceleration is completely disabled.
dtischler
commented
2 years ago Oh definitely...just wanted to chime in as another user of the amazing work you all are doing. :-)
jcdutton
commented
2 years ago Those aren't bugs, they are architectural limitations. Coming from x86, you're going to be spoiled about how x86 handles things. In the RISC world, a lot of the rules you broke in x86 because it just ignored the violations will bite you here.
memcpy and any other direct memory access is inherently unsafe. A lot of work has gone into armv7 and armv8 to make sure normal system ram can make a safe environment, but that doesn't get extended to IO devices, because it literally cannot. The PCIe spec permits devices to be mapped as normal system ram, with all benefits of that. We already know BRCM doesn't care about complying with the spec, and their implementation is severely broken. It seems the rk35xx series also doesn't comply, but not nearly as bad.
I'm trying to narrow down exactly how bad, so that I can document it as we finalize the PCIe support for mainline.
Here is one for you. The board (not via PCIe) has a HDMI output, and graphics work fine of that. So, aarch64 can work with GPUs. The problem is the implementation PCIe on the aarch64 such that GPUs don't work across it. One of the purposes, (I might be wrong here), of PCIe "prefetchable" BAR is that the BAR is mapped as if it was memory, and not as if it was an IO Device. So, that is why I think it would be a good idea to check that the "prefetchable" bit is actually working correctly.
From your message earlier: "Robin Murphy robin.murphy@arm.com: CPU access to PCIe has nothing to do with PCIe's access to memory. From what you've described, my guess is that a GPU BAR gets put in a non-prefetchable window, such that it ends up mapped as Device memory (whereas if it were prefetchable it would be Normal Non-Cacheable)."
So, we need it to not appear as "Device memory" and instead appear as "Normal Non-Cacheable" We get bus errors from Device memory", but should not from "Normal Non-Cacheable" .
jcdutton
commented
2 years ago So, although lspci -vv is "saying" the 256MB BAR is prefetchable, it might not actually be doing it.
pgwipeout
commented
2 years ago @jcdutton I highly recommend you read up on how Arm handles the integrated GPU.
pgwipeout
commented
2 years ago For that matter, how the AXI interconnect handles things, how the memory controller ties into this, how cache works, and how the PCIe controller expects to handle this. TLDR: If the control lines aren't there, you can't magically make them exist. IRT the integrated GPU, extremely simplified TLDR: It is already attached to system ram.
jcdutton
commented
2 years ago Some ARM SOC documentation: Base Address Register Overview The AXI Memory Mapped to PCI Express core in Endpoint configuration supports up to three 32-bit BARs or three 64-bit BARs. The AXI Memory Mapped to PCI Express in Root Port configuration supports one 32-bit BARs or one 64-bit BAR. All BAR registers share these options: • Checkbox: Click the checkbox to enable BAR. Deselect the checkbox to disable BAR. • Type: BARs can be Memory apertures only. Memory BARs can be either 64-bit or 32-bit. Prefetch is enabled for 64-bit and not enabled for 32-bit. When a BAR is set as 64 bits, it uses the next BAR for the extended address space, making it inaccessible. • Size: The available Size range depends on the PCIe ® Device/Port Type and the Type of BAR selected.
Now, this documentation is for a different arm SOC, so it might not be valid for the SOQuartz soc.
jcdutton
commented
2 years ago @pgwipeout I have read all the docs now. I am still no wiser with regards to why we cannot answer Robin Murphy's question above.
gdumee
commented
2 years ago Hello
I used @pgwipeout image to install debian on the eMMC module. I copied the image directly to the eMMC, and installed from there, so I can boot from eMMC without any SDcard. It's working well (i just needed to delete parts 6 and 7 from the image before installing debian, to free some primary partitions). There is only one problem, when a SDcard is inserted in the slot, the system is ok, but when no SDCard is inserted in the slot, there are a lot of messages about failing to set a clock rate :
[ 110.775314] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 110.791088] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 110.794706] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 110.812916] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 110.830014] dwmmc_rockchip fe2b0000.mmc: failed to set rate 200000Hz [ 110.847289] dwmmc_rockchip fe2b0000.mmc: failed to set rate 200000Hz [ 110.851096] dwmmc_rockchip fe2b0000.mmc: failed to set rate 200000Hz [ 110.869005] dwmmc_rockchip fe2b0000.mmc: failed to set rate 200000Hz [ 110.886078] dwmmc_rockchip fe2b0000.mmc: failed to set rate 100000Hz [ 110.907141] dwmmc_rockchip fe2b0000.mmc: failed to set rate 100000Hz [ 110.911062] dwmmc_rockchip fe2b0000.mmc: failed to set rate 100000Hz [ 112.023268] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 112.040552] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 112.044184] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 112.062191] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 112.079089] dwmmc_rockchip fe2b0000.mmc: failed to set rate 200000Hz [ 112.096365] dwmmc_rockchip fe2b0000.mmc: failed to set rate 200000Hz [ 112.099994] dwmmc_rockchip fe2b0000.mmc: failed to set rate 200000Hz [ 112.135100] dwmmc_rockchip fe2b0000.mmc: failed to set rate 100000Hz [ 112.156195] dwmmc_rockchip fe2b0000.mmc: failed to set rate 100000Hz [ 112.160143] dwmmc_rockchip fe2b0000.mmc: failed to set rate 100000Hz [ 113.271344] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 113.288621] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 113.292339] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 113.310248] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 113.327087] dwmmc_rockchip fe2b0000.mmc: failed to set rate 200000Hz [ 113.344368] dwmmc_rockchip fe2b0000.mmc: failed to set rate 200000Hz [ 113.347988] dwmmc_rockchip fe2b0000.mmc: failed to set rate 200000Hz [ 113.365890] dwmmc_rockchip fe2b0000.mmc: failed to set rate 200000Hz [ 113.383008] dwmmc_rockchip fe2b0000.mmc: failed to set rate 100000Hz [ 113.404063] dwmmc_rockchip fe2b0000.mmc: failed to set rate 100000Hz [ 113.407994] dwmmc_rockchip fe2b0000.mmc: failed to set rate 100000Hz [ 114.519311] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 114.536582] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 114.541787] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 114.561650] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 114.579105] dwmmc_rockchip fe2b0000.mmc: failed to set rate 200000Hz [ 114.595086] dwmmc_rockchip fe2b0000.mmc: failed to set rate 200000Hz [ 114.598707] dwmmc_rockchip fe2b0000.mmc: failed to set rate 200000Hz
As I do not need the SDcard (the mainboard I use has none), I have extracted the dts from dtb, removed the lines "mmc@fe2b0000 { ... }", and recompiled the dtb. Now I do not have message anymore, but I think it would be better to change the configuration of the SD slot and not removing it. Could you guide me please?
pgwipeout
commented
2 years ago See https://patchwork.kernel.org/project/linux-rockchip/list/?series=620648
On Wed, Apr 6, 2022, 09:06 Gilles Dumée @.***> wrote:
Hello
I used @pgwipeout https://github.com/pgwipeout image to install debian on the eMMC module. I copied the image directly to the eMMC, and installed from there, so I can boot from eMMC without any SDcard. It's working well (i just needed to delete parts 6 and 7 from the image before installing debian, to free some primary partitions). There is only one problem, when a SDcard is inserted in the slot, the system is ok, but when no SDCard is inserted in the slot, there are a lot of messages about failing to set a clock rate :
[ 110.775314] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 110.791088] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 110.794706] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 110.812916] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 110.830014] dwmmc_rockchip fe2b0000.mmc: failed to set rate 200000Hz [ 110.847289] dwmmc_rockchip fe2b0000.mmc: failed to set rate 200000Hz [ 110.851096] dwmmc_rockchip fe2b0000.mmc: failed to set rate 200000Hz [ 110.869005] dwmmc_rockchip fe2b0000.mmc: failed to set rate 200000Hz [ 110.886078] dwmmc_rockchip fe2b0000.mmc: failed to set rate 100000Hz [ 110.907141] dwmmc_rockchip fe2b0000.mmc: failed to set rate 100000Hz [ 110.911062] dwmmc_rockchip fe2b0000.mmc: failed to set rate 100000Hz [ 112.023268] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 112.040552] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 112.044184] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 112.062191] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 112.079089] dwmmc_rockchip fe2b0000.mmc: failed to set rate 200000Hz [ 112.096365] dwmmc_rockchip fe2b0000.mmc: failed to set rate 200000Hz [ 112.099994] dwmmc_rockchip fe2b0000.mmc: failed to set rate 200000Hz [ 112.135100] dwmmc_rockchip fe2b0000.mmc: failed to set rate 100000Hz [ 112.156195] dwmmc_rockchip fe2b0000.mmc: failed to set rate 100000Hz [ 112.160143] dwmmc_rockchip fe2b0000.mmc: failed to set rate 100000Hz [ 113.271344] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 113.288621] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 113.292339] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 113.310248] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 113.327087] dwmmc_rockchip fe2b0000.mmc: failed to set rate 200000Hz [ 113.344368] dwmmc_rockchip fe2b0000.mmc: failed to set rate 200000Hz [ 113.347988] dwmmc_rockchip fe2b0000.mmc: failed to set rate 200000Hz [ 113.365890] dwmmc_rockchip fe2b0000.mmc: failed to set rate 200000Hz [ 113.383008] dwmmc_rockchip fe2b0000.mmc: failed to set rate 100000Hz [ 113.404063] dwmmc_rockchip fe2b0000.mmc: failed to set rate 100000Hz [ 113.407994] dwmmc_rockchip fe2b0000.mmc: failed to set rate 100000Hz [ 114.519311] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 114.536582] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 114.541787] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 114.561650] dwmmc_rockchip fe2b0000.mmc: failed to set rate 300000Hz [ 114.579105] dwmmc_rockchip fe2b0000.mmc: failed to set rate 200000Hz [ 114.595086] dwmmc_rockchip fe2b0000.mmc: failed to set rate 200000Hz [ 114.598707] dwmmc_rockchip fe2b0000.mmc: failed to set rate 200000Hz
As I do not need the SDcard (the mainboard I use has none), I have extracted the dts from dtb, removed the lines @.*** { ... }", and recompiled the dtb. Now I do not have message anymore, but I think it would be better to change the configuration of the SD slot and not removing it. Could you guide me please?
— Reply to this email directly, view it on GitHub https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/336#issuecomment-1090246518, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFWB7P4E2Y5Z257UG3YZB3VDWD4JANCNFSM5JNBKOVQ . You are receiving this because you were mentioned.Message ID: @.***>
gdumee
commented
2 years ago Thanks @pgwipeout I saw on your Gitlab that newer artefacts were deleted, the latest available is 4 months old. Do you have a patched kernel ?
pgwipeout
commented
2 years ago The longest public gitlab permits artifacts to stick around is 30 days, unless you specifically save them. I'm currently working on rebasing to 5.18, so that the majority of what I'm holding out of tree can finally be submitted up to mainline.
On Wed, Apr 6, 2022 at 10:43 AM Gilles Dumée @.***> wrote:
Thanks. I saw on your Gitlab that newer artefacts were deleted, the latest available is 4 months old. Do you have a patched kernel ?
— Reply to this email directly, view it on GitHub https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/336#issuecomment-1090354065, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFWB7PQVVJNCUX4WDZJ3YTVDWPJBANCNFSM5JNBKOVQ . You are receiving this because you were mentioned.Message ID: @.***>
hongkongkiwi
commented
2 years ago I would like to use the Pine64 SOQuartz CM4 with the DFRobot router board (which adds an additional ethernet over pcie), at the codes current state will this work?
pgwipeout
commented
2 years ago Unfortunately I can't answer that positively yes. PCIe works on the CM4IO board, but others using custom carriers have experienced issues with PCIe. It seems the CM4IO board has the data lanes polarity swapped, while the custom boards do not. Someone will be sending me one to investigate exactly what's going on there.
On Thu, Apr 7, 2022 at 4:18 AM Andy @.***> wrote:
I would like to use the Pine64 with the DFRobot router board (which adds an additional ethernet over pcie), at the codes current state will this work?
— Reply to this email directly, view it on GitHub https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/336#issuecomment-1091277901, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFWB7I4I7IEUWYUDEBGY5DVD2K3NANCNFSM5JNBKOVQ . You are receiving this because you were mentioned.Message ID: @.***>
loganepicpower
commented
2 years ago Hi, I am trying to install the debian over the SOQUARTZ (on the CM4 board) and an eMMC and I am unable to get it running. I am using the @pgwipeout img and tried installing through SD or directly from eMMC. I have the problem regarding kernel modules on installing that I bypass and when it is installed, the bootloader of the SOQUARTZ does not detect the OS. If I try to boot from the SD Card and then use option 4, then I get a Kernel Panic: and this msg: VFS: Cannot open root device "mmcblk1p7" or unknown-block(179,39): error -6
Do you have any kind of "guide" on how to install the debian on this module?
gdumee
commented
2 years ago Hi @loganepicpower On the SDcard installation, the partition 5 is the boot part. You need to mount it, and modify the file extlinux/extlinux.conf, which is the boot menu conf file. The third entry is the SDCard boot, and it points to the rootfs partition /dev/mmcblk0p7, but when intalling debian on the SDcard, the rootfs is on /dev/mmcblkp0p9.
When you installed debian on eMMC, did you use the SDCard to launch the installer and select destination on the eMMC, or did you copy the pgwipeout image to the eMMC and installed from there without any SDcard ? The first case cannot work without the SDcard, because debian installer do not install a boot manager, and the kernel is not the good one. In the second case, you may have the same config problem I explained, and you need to change 4th entry of the boot menu to point to the /dev/mmcblk1p9 partition.
loganepicpower
commented
2 years ago Thank you @gdumee ! That worked nice. If you need any kind of tests, please just ask.
loganepicpower
commented
2 years ago I have seen that WiFi Module is detected:
`root@debian:~# udevadm info /sys/bus/sdio/devices/* P: /devices/platform/fe2c0000.mmc/mmc_host/mmc2/mmc2:0001/mmc2:0001:1 L: 0 E: DEVPATH=/devices/platform/fe2c0000.mmc/mmc_host/mmc2/mmc2:0001/mmc2:0001:1 E: SDIO_CLASS=00 E: SDIO_ID=02D0:A9BF E: SDIO_REVISION=0.0 E: MODALIAS=sdio:c00v02D0dA9BF E: SUBSYSTEM=sdio E: USEC_INITIALIZED=4700771 E: ID_VENDOR_FROM_DATABASE=Broadcom Corp. E: ID_SDIO_CLASS_FROM_DATABASE=Non-standard SDIO interface
P: /devices/platform/fe2c0000.mmc/mmc_host/mmc2/mmc2:0001/mmc2:0001:2 L: 0 E: DEVPATH=/devices/platform/fe2c0000.mmc/mmc_host/mmc2/mmc2:0001/mmc2:0001:2 E: SDIO_CLASS=00 E: SDIO_ID=02D0:A9BF E: SDIO_REVISION=0.0 E: MODALIAS=sdio:c00v02D0dA9BF E: SUBSYSTEM=sdio E: USEC_INITIALIZED=4695100 E: ID_VENDOR_FROM_DATABASE=Broadcom Corp. E: ID_SDIO_CLASS_FROM_DATABASE=Non-standard SDIO interface
P: /devices/platform/fe2c0000.mmc/mmc_host/mmc2/mmc2:0001/mmc2:0001:3 L: 0 E: DEVPATH=/devices/platform/fe2c0000.mmc/mmc_host/mmc2/mmc2:0001/mmc2:0001:3 E: SDIO_CLASS=02 E: SDIO_ID=02D0:A9BF E: SDIO_REVISION=0.0 E: MODALIAS=sdio:c02v02D0dA9BF E: SUBSYSTEM=sdio E: USEC_INITIALIZED=4696642 E: ID_VENDOR_FROM_DATABASE=Broadcom Corp. E: ID_SDIO_CLASS_FROM_DATABASE=Bluetooth Type-A standard interface ` however, no evidence in dmesg; maybe kernel update should be needed?
FlameKat53
commented
2 years ago Hi. I have purchased a GPi Case 2 from RetroFlag and I am attempting to get it to work on the SOQuartz (2GB model). I need some help getting the display to work. (I did not purchase the dock version) supposedly this uses a DPI GPIO display. I don't know what that is. If anyone can help, my discord is setLillie#5644 and you can mention me here on GitHub.
acjohnson
commented
2 years ago I bought a Mcuzone RPi CM4_GIGA_USB3.0 expansion board (3*1Gbps Ethernet, 2*USB3.0) with the metal case from Mcuzone as well as a Pine64 SOQuartz 8GB. I've got @pgwipeout's rk3566-soquartz-cm4.dtb.img installed on a 32GB eMMC that I purchased with the soquartz. So far I'm happy with it.
I was able to install debian to the eMMC using a micro sdcard (without needing to use a USB writer) by following the steps here https://wiki.pine64.org/wiki/Installing_Debian_on_the_Quartz64 with one minor change to the extlinux.conf file from: fdt /dtbs/rockchip/rk3566-quartz64-a.dtb to: fdt /dtbs/rockchip/rk3566-soquartz-cm4.dtb
Also looking at the unmodified image once flashed to sdcard, I don't understand how some of you are able to get it to boot (past the recovery mode) or install because when I tried using 3:.Boot Root SDMMC I got an error that it could not find /sbin/init and since there are already 7 partitions you would have to manually create an 8th partition outside of the debian-installer using gnu parted or something. Is that what y'all have been doing?
Thanks everyone for this thread I would have been lost without it.
gdumee
commented
2 years ago hello @acjohnson I don't know why, but the debian installer can create all the partitions when installing on a SDCard, but when running from the eMMC it cannot create the needed partitions (it seems to be blocked after 1 GB space).
In the wiki that you mentioned, it is explained to boot to "Build-Root Recovery" and do manual paritionning. In your case, you just need to delete the partitions 6 and 7 and rerun the installer (these partitions are created in the original image, but are not used).
In the installer you can choose to use all free space from eMMC (debian will create 3 partitions, but 1 is a useless new boot part), or manually create 2 partitions (rootfs and swap). At the end, the installer will mention the rootfs partition, and you must return to recovery mode to modify the extlinux.conf from the boot partition 5, and change to the correct rootfs partition.
7ROLI101
commented
2 years ago Is there a guide on how to install debian on the SOQuartz? I tried to use this link:
https://wiki.pine64.org/wiki/Installing_Debian_on_the_Quartz64
and also replaced this line inside of this installation
fdt /dtbs/rockchip/rk3566-quartz64-a.dtb to fdt /dtbs/rockchip/rk3566-soquartz-cm4.dtb
but to no avail.
loganepicpower
commented
2 years ago @7ROLI101 I used a usb to emmc converter, then flashed the image. After that, boot into Debian Installer option and when it finished, modified the extlinux.conf as @gdumee said (and replaced in that file default boot to emmc also)
As with the Radxa CM3, I would also like to test Pine64's SOQuartz with some CM4 boards, since it's supposed to be pin-compatible.
@timonsku mentioned the Wiki (linked above) and this dtb artifact are the two best ways to get started with it. I'd like to write up my experience trying to get the thing to boot, and also seeing if it fits and works in a few popular CM4 boards (starting with the official IO Board).