Test PCIe switches and adapters

[geerlingguy]

I need to add a new section to the site for bridges/switches/adapters... because I'm finding that at least one of the three I've purchased doesn't seem to work with the Pi at all, giving me pcie: link is down on any attempt to use it with any PCIe device (including known good ones that work when I plug them in directly or through other adapters.

Right now I'm testing these:

IO Crest 1 to 2 ports PCIe x1 adapter — seems to work fine at least for some cards. Build quality of included floppy to molex power adapter is pretty bad, though:

Exacq Technologies 1 to 2 ports PCIe x1 adapter using the TI XIO3130 PCIe switch chip, which uses a 4-pin molex connector for power. This one seems like it came out of a video server of some sort. Got it on eBay:

PCIe 16x to 1x adapter (generic) - Works with everything so far, as if I plugged it straight in. It should though, since it's just a dumb extension cable:

Rosewill RCRC-17001 Mining Card 1x to 16x - Sometimes works, sometimes causes a kernel panic for some reason. I even bought two revisions (003, with a PCIe power connector, and 888, with a Molex 4-pin power connector), and both of them exhibit this weird behavior:

PCE164P-N06 VER008S PCIe 1x to 16x riser - Haven't tested yet, but I like how it has a plethora of power options!

One common annoyance is they all have different power adapter cables. I should probably just go buy a bulky PC power supply and drop it on my desk. I did by a 600W fully modular PSU.

[geerlingguy]

@theodric - Wow, excellent news! And great sleuthing in that thread. I'm pretty sure all the issues I've seen on the Pi 5's bus have been due to clock problems with signal integrity, but now I'm also wondering if certain weird behaviors on the CM4 with other devices could be related too.

[arqtv]

We have seen similar idiosyncratic PCIe behaviour with the CM4 on our in-house designed carrier board for the Viota Pi-Cube and initially blamed signal integrity. The design employs an ASM1182e two port chip with downstream to a VL805 and an M.2 connector for NMVE SSDs. We found this chip (and the 4 port variant) to be the most reliable of the several bridge chip vendors that we tested on various PCIe adapter boards on the CM4IO board. The problem was a random failure to enumerate one or other or any of the downstream devices or a failure to enumerate the NVME device at 5GTs - not getting beyond 2.5GTs. We burned many hours trying to track this problem down including re-iterating our board design (to improve signal integrity) but then after one of the several CM4 eeprom firmware upgrade releases the problem seemed to disappear and enumeration of both downstream devices and at 5GTs became significantly more reliable. No other changes were made on the test rig other than the firmware upgrade so our conclusion was that whilst good enough timing at the hardware level is important, the enumeration process is also likely relevant, as in what order things get done and when. The code is closed source so it is only a best guess but IMHO worth knowing. It sounds like there may be some similar "timing" problems with the Pi5 and as mine has just arrived this morning it is high on the list to investigate. A Pi5 PCIe connector pinout would be helpful, so if anyone has worked that out it would be a big help to post it here - with the usual caveat emptor !! We plan on making some Pi5 PCIe to M.2 adapter boards and will make them available to the community when done.

[arqtv]

Dr George Smart has posted his reverse engineering hack of the Pi5 PCIe connector pinout on Github and YouTube - and saved me cannibalising my Pi5! Big thank you George - will take it as valuable input for an M.2 HAT and keep you posted.

m1geo/Pi5_PCIe

[albertz]

The PSU is quite girthy, and also has to have a jumper across two pins on the motherboard connector so it will switch on. That was unforeseen, but obvious in hindsight.

I bought this ATX Power Breakout Board 20Pin 24Pin:

This gives you the power on/off switch, and additionally also more easy access to the power output. I use this also together with such USB C to 2-Pin blank cable:

Or this USB-A connector:

[albertz]

Regarding PCIe, there are a couple of Raspberry NVME M.2 bridges, like this Geekworm X1001 PCIe to M.2 NVMe Key-M SSD Shield for Raspberry Pi 5:

And then there are PCIe to M.2 bridges like this CERRXIAN M.2 NGFF zu PCIE X4 Slot Riser Karte, M Key M.2 2260 2280:

So far, I only tried a M.2 to SATA bridge, specifically this Bewinner M.2 to SATA3.0:

And the kernel seems to recognize this SATA controller. I see this in lspci:

0000:01:00.0 SATA controller: ASMedia Technology Inc. ASM1166 Serial ATA Controller (rev 02)

However, when attaching some HDD to it, powering the HDD via my PSU, I just get this:

[    8.120087] ata1: link is slow to respond, please be patient (ready=0)
[    8.591928] ata1: SATA link down (SStatus FFFFFFFF SControl FFFFFFFF)
[    8.855645] ahci 0000:01:00.0: AHCI controller unavailable!
[    9.150697] ahci 0000:01:00.0: AHCI controller unavailable!

(I'm also writing down my experience on this here.)

So next I will try the PCIe to M.2 bridge and then attach a SATA controller to the PCIe. I wonder if that works better.

I see that all the solutions here are PCIe to USB3. I wonder how that compares to solutions which directly use the PCIe from the Pi board? What would be the possible bandwidth? I see this in dmesg regarding the PCIe bandwidth:

pci 0000:01:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x1 link at 0000:00:00.0 (capable of 15.752 Gb/s with 8.0 GT/s PCIe x2 link)

But I don't exactly understand what it means. What is GT/s? So how can I get this capable 15.752 Gb/s? Gb = Gbit? And what bandwidth would I have via USB3? 5 Gbit/s?

The Radxa Penta SATA HAT (#615) seems like another nice solution. But I cannot easily/cheaply get it currently in Germany.