search

Plebian-Linux / quartz64-images

GitHub Actions Repository for automatically generated images for the Quartz64 family of single board computers
https://plebian.org
GNU General Public License v3.0
41 stars 10 forks source link

soquartz-cm4 not detecting hardware #33

Closed zero-pytagoras closed 1 year ago

zero-pytagoras commented 1 year ago

hello, I am using CM4 with Tofu board, I am booting it with emmc from pine64. I have tried burning images several times, yet non of the peripherials are detected after install.

CounterPillow commented 1 year ago

Hello,

using the CM4 I/O board device tree on a different carrier board will not really work. Someone needs to write a device tree for this board in combination with a SOQuartz and submit it upstream so that it can be included in Plebian.

Wi-Fi not working on the SOQuartz modules in Plebian right now is a known issue though that I'm gonna have to look into.

zero-pytagoras commented 1 year ago

is there any way i can help. my background is in devops but may be i can do something ?

CounterPillow commented 1 year ago

Can you try the plebian-debian-bookworm-soquartz-model-a.img.xz image? This is a bit of an ugly hack but might make peripherals work if the problem is what I think it is.

zero-pytagoras commented 1 year ago

can you send me the links with wifi issues? tried the image but it is still the same - i'll try to update configs and install firmware - will update you in case something will pop up

CounterPillow commented 1 year ago

Regarding the Wi-Fi issues, it's documented here: https://github.com/Plebian-Linux/quartz64-images/blob/main/RUNNING.md#wi-fi (outdated by now, Model B is supported.) It's probably easy to create the right symlink for the Wi-Fi firmware to make it work, I just haven't looked into it yet.

tried the image but it is still the same

So USB and PCIe still don't work?

zero-pytagoras commented 1 year ago

So USB and PCIe still don't work?

for now - still not working

zero-pytagoras commented 1 year ago

I have tried every possible script - nothing works. seems to me that next step would be compiling kernel, yet these boards are not x86 machines, meaning I don't really know what to add to kernel to compile it - is there any documentation/guideline for trouble shooting ?

CounterPillow commented 1 year ago

The output of your dmesg would be handy to see if there are any device errors.

To cross-compile a kernel on an x86 machine, do make ARCH=ARM64 CROSS_COMPILE=aarch64-linux-gnu- bindeb-pkg and then install the resulting linux-image .deb in the parent directory on the board.

I'll try to obtain one of the boards myself to be able to further debug this, maybe they'll send me one for free for this purpose.

zero-pytagoras commented 1 year ago

regarding the dmesg pasted at pastebin. hope it will help. keep in mind that is the systems i have installed heavily on it bunch of firmware and compilations. I try to build kernel and test it later. while at it: this board came in this morning and for some unknown reason wifi and usb work - have not tried yet the vanilla image, I have good feeling about it.

CounterPillow commented 1 year ago

rockchip-naneng-combphy fe840000.phy: failed to create combphy

Well that would certainly throw a spanner in the works, that PHY is used for PCIe. I wonder why devm_phy_create is seemingly failing (repeatedly).

USB is seemingly picking up hubs though, so something is being detected.

zero-pytagoras commented 1 year ago

is there any code we can write/copy/run that might shed a light on the issue ? or it might be chip/board issue ?

CounterPillow commented 1 year ago

Output of sudo cat /sys/kernel/debug/devices_deferred might show something. If it's empty, that's good.

lsusb -v might also be useful for debugging why USB doesn't work.

I've looked at the TOFU schematics and didn't quite see anything that would explain both PCIe and USB not working, as they aren't like controlled by GPIOs or anything. Maybe something sketchy happening with the nEXTRST pin. I've requested a board sample for now so that I can try to reproduce the issue locally and then do some device tree writing.

zero-pytagoras commented 1 year ago

/sys/kernel/debug/devices_deferred file exists, and it is empty lsusb how one learn these stuff ? there is no course on it unless you part of kernel community, but then you need to be C developer

CounterPillow commented 1 year ago

lsusb doesn't show the Microchip USB2514B hub that's in the schematic, so right now I'm musing whether it's FSUSB42 mux not working for some reason. It's enabled by the USBOTG_ID pin, which in turn is seemingly connected to the RK3566 SoC directly judging by the SOQuartz schematics. So I'm looking at the RK356x TRM's GRF definition to see whether there's a way to control this, and whether that's even the issue. If that is the issue, then you'd only get USB on the USB-C connector, rather than the USB hub.

Then there's also the theory that 3.3V goes entirely byebye because the regulator isn't set to be always on, which would take the mux with it and the hub, as you can see from the TOFU schematic.

So if I had a board I'd try to somehow probe the USBOTG line and the 3.3v line if they have testpoints on the board anywhere. I don't see a test point for the USBOTG thing on the TOFU board but it might be possible to probe it at the CM4 connector somehow. Then I'd look up in the FSUSB42 datasheet which output is enabled with low or high signal.

how one learn these stuff ?

It sounds like I know what I'm doing but as you can see from me thinking out loud, it's really just trial and error to eliminate possible reasons for the problem until there's only one answer left.

zero-pytagoras commented 1 year ago

lsusb doesn't show the Microchip USB2514B hub that's in the schematic, so right now I'm musing whether it's FSUSB42 mux not working for some reason. It's enabled by the USBOTG_ID pin, which in turn is seemingly connected to the RK3566 SoC directly judging by the SOQuartz schematics. So I'm looking at the RK356x TRM's GRF definition to see whether there's a way to control this, and whether that's even the issue. If that is the issue, then you'd only get USB on the USB-C connector, rather than the USB hub.

i'll test the usb-c, may be something will show up, either in kernel or in lsus/lspci

Then there's also the theory that 3.3V goes entirely byebye because the regulator isn't set to be always on, which would take the mux with it and the hub, as you can see from the TOFU schematic.

I think I have usb connector, and I can try to login to device with it... haven't done it for very long, because I burned my last device and I am somewhat discouraged to try again.

So if I had a board I'd try to somehow probe the USBOTG line and the 3.3v line if they have testpoints on the board anywhere. I don't see a test point for the USBOTG thing on the TOFU board but it might be possible to probe it at the CM4 connector somehow. Then I'd look up in the FSUSB42 datasheet which output is enabled with low or high signal.

Is there a way to probe from software ? sorry - it's not clear to me.

It sounds like I know what I'm doing but as you can see from me thinking out loud, it's really just trial and error to eliminate possible reasons for the problem until there's only one answer left.

still there are many basic things that I am missing: reading and understanding schematics, writing code based on schematics and so on...

Thanks for your help - I appreciate it a lot !

stevenlafl commented 1 year ago

Two cents here, I have the same problem on the Turing Pi v2 board.

CounterPillow commented 1 year ago

Two cents here, I have the same problem on the Turing Pi v2 board.

Doesn't seem to have schematics available so I won't be looking into this $260 board, though it might get fixed by accident if I can get this one working.

CounterPillow commented 1 year ago

I've received a TOFU board sample from Oratek and got to work right away. I've made a fix for the problem which I will push to Plebian's device tree package and u-boot once I get some reviews on it.

You can read more about the problem in this article I wrote.

stevenlafl commented 1 year ago

You're a legend. I know nothing about the kernel or compiling those packages, but I'd love to test it for my use case. Thanks for your hard work on this very fine distribution.

zero-pytagoras commented 1 year ago

I've received a TOFU board sample from Oratek and got to work right away. I've made a fix for the problem which I will push to Plebian's device tree package and u-boot once I get some reviews on it.

You can read more about the problem in this article I wrote.

Hi. I am traveling and will be with the board next week. I'll test it will upate you.

CounterPillow commented 1 year ago

Fix was merged into kernel for 6.4 fixes (will trickle down to older releases), so I've applied to to Plebian's device tree package. To get the updated device trees, simply run sudo apt update && sudo apt upgrade. No unusual manual intervention is required.

I also am publishing a new release of images just so the updated package is in there in new installations.

I've not yet carried over the dts fix to u-boot. It doesn't seem to need it, and I'd rather make the next u-boot update a bit bigger than just this small change because for users updating u-boot isn't very nice.

Small note on stuff: I've noticed that PCIe can still be a bit squirrely at times with the Transcend M.2 module I've used on the TOFU. Sometimes it just works, sometimes a manual rescan is needed, sometimes not even a rescan makes it show up. Not sure what's to blame here, but that it shows up at all means one of the problems was fixed. USB is always present now. So I'm closing this for now as the main bug that caused this issue is now fixed.

Thank you everyone for your patience and thank you to Oratek for the hardware that allowed me to debug this.