Test PCIe switches and adapters

[geerlingguy]

I need to add a new section to the site for bridges/switches/adapters... because I'm finding that at least one of the three I've purchased doesn't seem to work with the Pi at all, giving me pcie: link is down on any attempt to use it with any PCIe device (including known good ones that work when I plug them in directly or through other adapters.

Right now I'm testing these:

IO Crest 1 to 2 ports PCIe x1 adapter — seems to work fine at least for some cards. Build quality of included floppy to molex power adapter is pretty bad, though:

Exacq Technologies 1 to 2 ports PCIe x1 adapter using the TI XIO3130 PCIe switch chip, which uses a 4-pin molex connector for power. This one seems like it came out of a video server of some sort. Got it on eBay:

PCIe 16x to 1x adapter (generic) - Works with everything so far, as if I plugged it straight in. It should though, since it's just a dumb extension cable:

Rosewill RCRC-17001 Mining Card 1x to 16x - Sometimes works, sometimes causes a kernel panic for some reason. I even bought two revisions (003, with a PCIe power connector, and 888, with a Molex 4-pin power connector), and both of them exhibit this weird behavior:

PCE164P-N06 VER008S PCIe 1x to 16x riser - Haven't tested yet, but I like how it has a plethora of power options!

One common annoyance is they all have different power adapter cables. I should probably just go buy a bulky PC power supply and drop it on my desk. I did by a 600W fully modular PSU.

[geerlingguy]

With the 1 to 2 ports adapter, I'm able to get multiple devices to load, so that's nice:

$ lspci
00:00.0 PCI bridge: Broadcom Limited Device 2711 (rev 20)
01:00.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
02:01.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
02:02.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
03:00.0 SATA controller: Marvell Technology Group Ltd. Device 9215 (rev 11)
04:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981

I'm going to confirm I can only transfer data at up to 3.2 Gbps though. Maybe less with the overhead of the PCI bridge?

[geerlingguy]

To test NVMe + SATA, or anything plus anything requiring extra power... I'm going to need to grab an external PSU. I just don't have enough adapter cables to get from my single 4-pin molex power adapter to both the floppy connector on the PCIe switch plus SATA power for the drives.

Might as well buy a PC case at some point to contain all this crazy mess haha.

Anyways, I have my eye on the Redragon GC-PS003 600W fully modular PSU... mostly because it's close, relatively cheap, and available whenever. But I'm doing plenty of other stuff so it's not yet a priority.

Edit: Just ordered that PSU, and I'll have a little more fun with the switch next week!

[geerlingguy]

The PSU is quite girthy, and also has to have a jumper across two pins on the motherboard connector so it will switch on. That was unforeseen, but obvious in hindsight.

[Chaz6]

I found another one possibly worth testing: https://www.ebay.co.uk/itm/Excellent-PCI-e-Express-1X-to-3-1X-Switch-Multiplier-HUB-Riser-Card-USB-Cable-GN/353247990975

[geerlingguy]

@Chaz6 - I'm actually going to try another switch, which might also be nice on the CM4, though it will likely cause a bit of strain on the connector... https://www.ebay.com/itm/PCIe-1x-to-Dual-PCIe-1x-90-Degree-Riser-Card-with-Power-PCIe-Switch-Expansion/273855896755?hash=item3fc3155cb3:g:fVwAAOSwHu5c4uvd

[TheGuyDanish]

@Chaz6 I have that card (or a variant of it, anyway. Same design). The sticker covers it, but the chip on the board is of the same series as the IOCrest card that Jeff already tested, albeit one with a slightly expanded feature set (adding an extra port). They utilize the same kernel driver, so should ideally function exactly the same.

That said, I'm going to test it when my CM4 turns up one of these days and report back the results.

[markus-k]

I'm currently testing that card from @Chaz6 in my PC (CM4 not here yet), but I'm having trouble getting multi-lane cards to work. Have you been experiencing the same? The only 1x card I have is a Realtek Gigabit card, which seems to work fine. But I haven't gotten any other card to work, I tested a Samsung 970 EVO (with 4x adapter), Mellanox 2 10GbE card and a dual gigabit card pulled from a HP server, none working. I cut open the connector to plug in longer cards since I don't have a riser card right now, but the 1x card still works in that port (so I guess nothing broke).

The cards not working do light up or get warm, but nothing in lspci, and nothing obvious in dmesg.

[geerlingguy]

@markus-k - I can't speak to @Chaz6's card, but the IO Crest seems to so far only work with devices plugged in through slot 2 (maybe slot 1 is dead, maybe something in the Pi's PCIe stack doesn't work with it)... but I haven't done any rigorous testing with it yet.

I'm still focused right now on a number of individual cards and haven't yet done thorough testing with multiple cards.

[markus-k]

Just to give an update, I figured out what the problem was (or is) after some digging into the switch and PCIe documentation. Not really related to the CM4, but probably useful to everyone thinking about cutting up their 1x PCIe switch ports to plug many multi-lane cards into their CM4.

When you just google about cutting open 1x PCIe ports everyone will tell you, it'll work no problem. That is not entirely true though, depending on how presence detection is implemented on the PCIe carrier board. PCIe cards have two pins, PRSNT1# and PRSNT2#, which need to be directly connected on the card. PCIe cards with more than one lane have multiple PRSNT2# pins, one for each lane (i.e. one for 1x, 4x, 8x, 16x), but only the one at the maximum number of supported lanes has to be connected to PRSNT1#.

Now, when you cut open your 1x port, PRSNT1# is connected to a pin that is not even present on the slot connector, and the 1x PRSNT2# is floating since the card supports more lanes. Thus, the PCIe switch does not detect that a card is plugged in and disables the port. When you use a proper 1x to Nx adapter, the correct PRSNTx# pins will usually be connected and the card will be detected.

Presence detection via PRSNTx# can be disabled in the switch, but the EEPROM is not fitted on the board, and you'd need to modify it's contents even if it was there. One other whacky solution, if you don't want to order a proper raiser, I used is to get out the soldering iron and botch a wire between PRSNT1# and PRSNT2# on the board, so the switch thinks there is always a card plugged in. PRSNT1# is connected to ground on this particular board, and PRSNT2# connected to the PCIe switch. I wouldn't really recommend doing this, but if you are too cheap to order yet an other PCIe adapter, it's probably fine.

And see, now cards with more than one lane are detected on the host.

[6by9]

@Chaz6 I have an almost identical 3 way card https://www.ebay.co.uk/itm/Excellent-PCI-e-Express-1X-to-3-1X-Switch-Multiplier-HUB-Riser-Card-USB-Cable-L-/154052637457. Works fine with all the cards I've plugged into it so far (providing they work without the switch of course)

[geerlingguy]

I also wanted to post an update—earlier I do remember having at least the Marvell 9215 (IO Crest) SATA card and Samsung 970 EVO NVMe working at the same time on my 2-slot switch, so it did work at least. I'll have to continue doing more testing. I have another new riser that has two slots and a different chip.

[danielkucera]

This one seems to be working too: https://www.aliexpress.com/item/32960541989.html

[geerlingguy]

@danielkucera - Can you confirm having more than one device plugged in works, identifying all the devices? I noticed that many times it would only show devices plugged into the 2nd slot on my switch.

[darkstar]

Can you try one of these PCIe to PCI(-X) cards too? Like this for example.

It would make it possible to use many old PCI cards on the RPi

[6by9]

@darkstar Tried already - https://www.raspberrypi.org/forums/viewtopic.php?p=1772997#p1772997 Results not positive as most cards seem to rely on mapping as I/O registers as that was the way that life used to work on x86 systems.

[dlednik]

is anyone able to test ASM1184e chipset, I'm planing to build my own riser card with 1 x1 to 2 x4 + 2 M.2

It uses ASMedia chip ASM1184e: https://www.aliexpress.com/item/32868079091.html Used to be popular in mining community.

[PBXForums]

I have just put two cards in the first one pictured in Jeff's post and get this:

root@raspberrypi:~# lspci
00:00.0 PCI bridge: Broadcom Limited Device 2711 (rev 20)
01:00.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
02:01.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
02:02.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
03:00.0 SATA controller: Marvell Technology Group Ltd. Device 9215 (rev 11)
04:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
04:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
root@raspberrypi:~# dmesg | grep igb
[    3.616121] igb: Intel(R) Gigabit Ethernet Network Driver
[    3.616139] igb: Copyright (c) 2007-2014 Intel Corporation.
[    3.616786] igb 0000:04:00.0: enabling device (0000 -> 0002)
[    3.842546] igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection
[    3.842572] igb 0000:04:00.0: eth1: (PCIe:2.5Gb/s:Width x1) 80:61:5f:05:0b:ae
[    3.842676] igb 0000:04:00.0: eth1: PBA No: E43709-006
[    3.842694] igb 0000:04:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[    3.843153] igb 0000:04:00.1: enabling device (0000 -> 0002)
[    4.035061] igb 0000:04:00.1: Intel(R) Gigabit Ethernet Network Connection
[    4.035087] igb 0000:04:00.1: eth2: (PCIe:2.5Gb/s:Width x1) 80:61:5f:05:0b:af
[    4.035192] igb 0000:04:00.1: eth2: PBA No: E43709-006
[    4.035208] igb 0000:04:00.1: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)

What does seem to be the case though is that it will only recognise pciex1 ie no pciex4 on a riser, I cannot get it to recognise a single pciex4 device.

[PBXForums]

I have tried the one dlednik recommended, this is the one for us guys!!

On my Amazon it is listed as a different brand but looks like its the same as is the case with many of these devices:

https://www.amazon.co.uk/gp/product/B07NZ5W8KJ/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1

To test it, I plugged in four devices, all four worked and the cool thing is that this is a mixture of x1 and x4 devices

All four devices working. A 4x Intel 10G network card, a 1x 2 port network card, a marvel 1x 4 port sata card and a 4x nvme card

00:00.0 PCI bridge: Broadcom Limited Device 2711 (rev 20)
01:00.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
02:01.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
02:03.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
02:05.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
02:07.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
03:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
04:00.0 Non-Volatile memory controller: Sandisk Corp Device 5009 (rev 01)
05:00.0 SATA controller: Marvell Technology Group Ltd. Device 9215 (rev 11)
06:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
06:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

pi@raspberrypi:~ $ dmesg | grep ixgbe
[    3.517035] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[    3.517054] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[    3.522232] ixgbe 0000:03:00.0: enabling device (0000 -> 0002)
[    4.738177] ixgbe 0000:03:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[    4.738550] ixgbe 0000:03:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x1 link at 0000:00:00.0 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)
[    4.738912] ixgbe 0000:03:00.0: MAC: 2, PHY: 1, PBA No: G18786-004
[    4.738926] ixgbe 0000:03:00.0: 90:e2:ba:84:5e:f8
[    4.746219] ixgbe 0000:03:00.0: Intel(R) 10 Gigabit Network Connection
[    4.746761] libphy: ixgbe-mdio: probed
[    4.747202] ixgbe 0000:03:00.1: enabling device (0000 -> 0002)
[    5.932951] ixgbe 0000:03:00.1: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[    5.933290] ixgbe 0000:03:00.1: 4.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x1 link at 0000:00:00.0 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)
[    5.933631] ixgbe 0000:03:00.1: MAC: 2, PHY: 1, PBA No: G18786-004
[    5.933637] ixgbe 0000:03:00.1: 90:e2:ba:84:5e:f9
[    5.937171] ixgbe 0000:03:00.1: Intel(R) 10 Gigabit Network Connection
[    5.937435] libphy: ixgbe-mdio: probed

pi@raspberrypi:~ $ dmesg | grep igb
[    3.572119] igb: Intel(R) Gigabit Ethernet Network Driver
[    3.572136] igb: Copyright (c) 2007-2014 Intel Corporation.
[    3.572567] igb 0000:06:00.0: enabling device (0000 -> 0002)
[    3.760881] igb 0000:06:00.0: Intel(R) Gigabit Ethernet Network Connection
[    3.760906] igb 0000:06:00.0: eth1: (PCIe:2.5Gb/s:Width x1) 80:61:5f:05:0b:ae
[    3.760999] igb 0000:06:00.0: eth1: PBA No: E43709-006
[    3.761015] igb 0000:06:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[    3.761513] igb 0000:06:00.1: enabling device (0000 -> 0002)
[    3.946915] igb 0000:06:00.1: Intel(R) Gigabit Ethernet Network Connection
[    3.946940] igb 0000:06:00.1: eth2: (PCIe:2.5Gb/s:Width x1) 80:61:5f:05:0b:af
[    3.947033] igb 0000:06:00.1: eth2: PBA No: E43709-006
[    3.947049] igb 0000:06:00.1: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)

pi@raspberrypi:~ $ dmesg | grep nvme
[    3.463076] nvme nvme0: pci function 0000:04:00.0
[    3.463299] nvme 0000:04:00.0: enabling device (0000 -> 0002)
[    3.497918] nvme nvme0: allocated 32 MiB host memory buffer.
[    3.506935] nvme nvme0: 4/0/0 default/read/poll queues
[    3.516354]  nvme0n1: p1

pi@raspberrypi:~ $ dmesg | grep SATA
[    3.604608] ahci 0000:05:00.0: AHCI 0001.0000 32 slots 4 ports 6 Gbps 0xf impl SATA mode
[    3.650437] ata1: SATA max UDMA/133 abar m2048@0x601540000 port 0x601540100 irq 61
[    3.650455] ata2: SATA max UDMA/133 abar m2048@0x601540000 port 0x601540180 irq 61
[    3.650471] ata3: SATA max UDMA/133 abar m2048@0x601540000 port 0x601540200 irq 61
[    3.650486] ata4: SATA max UDMA/133 abar m2048@0x601540000 port 0x601540280 irq 61
[    4.123960] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    4.128017] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    4.128100] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    4.131946] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)

Happy days :D

[markus-k]

I have just put two cards in the first one pictured in Jeff's post and get this:

[...]

What does seem to be the case though is that it will only recognise pciex1 ie no pciex4 on a riser, I cannot get it to recognise a single pciex4 device.

@PBXForums Have a look at my previous comment: https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/14#issuecomment-735793184. Check if your Riser properly connects both PRSNT# pins.

[PBXForums]

@markus-k They work in the one in my above post, the two port one is now being scrapped in favor of that one which seems to be working perfectly.

[markus-k]

Some devices (like the CM4IO) don't check presence with the PRSNT# pins but whether theres a 50 Ohm load on the PCIe lanes. You can achieve the same effect on the Pericom switches by just shorting the PRSNT# pins (or configuring the switch, which is a lot harder to do).

[PBXForums]

@markus-k Thanks, I want at least three ports anyway.

[geerlingguy]

Testing this again with the IOCrest switch but with my 700W power supply for a NAS build... I finally seem to have it powered stably enough to get two devices working, and the system doesn't completely lock up when I run lspci:

$ lspci
00:00.0 PCI bridge: Broadcom Limited Device 2711 (rev 20)
01:00.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
02:01.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
02:02.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
03:00.0 SATA controller: JMicron Technology Corp. Device 0585
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. Device 8125 (rev 04)

I was trying the same thing with the Exacq switch, and kept either having only one of the two cards work, or the Pi would lock up at various points through the boot process.

Note that I tried powering it both via dedicated 2A molex power supply and via my 700W (100+W to the 5v bus) PSU, and it seemed to work more stably connected through my PSU. It still required one extra reboot before both cards were fully functional though.

[geerlingguy]

Seems to be extremely stable now (the IOCrest switch). I think for any of these switches, a solid power supply capable of handling 3-5A or more is going to be pretty important.

But even then, some of the adapters and switches just... fail sometimes, and cause kernel panic. No idea why that happens randomly.

[inzi]

Sorry if this is a dumb question - with using these that have a USB cable - do these perform as if they were simply USB devices or are they using a USB interface only?

My goal is to connect a drive and a VPU (like the coral).

In doing that, would it be better to just leave the drive on USB?

If this isn't the place to ask, feel free to delete or point me in the right direction to learn more.

[6by9]

Sorry if this is a dumb question - with using these that have a USB cable - do these perform as if they were simply USB devices or are they using a USB interface only?

They're using USB3 cables purely as a convenient physical cable. They do not convert to/from USB at an electrical or protocol level. Connecting to a genuine USB host or slave may well result in damage to either or both ends.

As Jeff noted, there is also no standard wiring with these adapters, so ensure you use the correct pairs of adapters.

[mplinuxgeek]

I've had my eye on these for my mining shinanigans and they look perfect for Raspberry Pi usage given their connectivity is a USB socket instead of a PCIe slot but can't seem to find them anywhere else and the vendor didn't answer my request for quote but thought I'd share anyway incase anyone knows where to get them.

https://www.alibaba.com/product-detail/pcie-splitter-pcie-to-pci-adapter_60572718142.html

[RuhanSA079]

Was anyone able to test the PI7C9X2G404SL chip? I just need a working PCIe switch that works with the CM4, Ubuntu Core 20. Unless I have to look at the ASM1184e PCIe switch variants?

[vnagara]

@geerlingguy Wow dude, that is awesome what you are trying to do.

[csoutreach]

Was anyone able to test the PI7C9X2G404SL chip? I just need a working PCIe switch that works with the CM4, Ubuntu Core 20.

I tried that chip in Ubuntu Core 20 without success -- but I haven't confirmed the adapter card definitely works. Interested to know experience of other people. -- not had chance to look to see if it was a driver issue for ubuntu

[juthi]

A PCIe compliant switch is supported by the native pci driver. Unless the hardware is defective, the PI7C9X2G404SL should be enumerated across all platforms. Full disclosure, our company is a mfr of many PCIe designs and use this component with success.

[csoutreach]

Thanks @juthi that's what I wanted to know. Looks like poor build quality, have another one on order so will be able to confirm for @RuhanSA079 .

[l00nix]

Bought three of the Exacq Technologies 1 to 2 ports PCIe x1 adapter none of them show up with lspci - all broken?

[juthi]

@100nix, are you powering the PCIe adapter using the molex connector? That is required.

[l00nix]

Thanks @juthi - yes, see picture - are there any LEDs on the card that should light up when operational? I wonder if all three cards I bought are duds.

[juthi]

Personally do not have this adapter and as a company, we did not use the TI bridge ages ago due to compatibility issues with the PCIe enumeration. We camped onto the PEX8112 bridge which worked in every brand we tested - we have 1M+ adapters in the field using their bridge. Early on a major brand name of PC had enumeration faults but they corrected the issue via BIOS code within 48 hours of notification.

1) Be sure the adapter is properly seated inside your motherboard.

2) It is possible that the PCIe fanout board REQUIRES a seated PCIe adapter inside one of its PCIe slots to enable. Do you have a PCIe x1 board for testing ?

3) If possible, measure the voltages on the large coil on the adapter to confirm that the onboard DC-DC converter is working.

4) LEDs would have been nice - we use them always for such purposes but do not see it on the adapter from the posted pix.

Forgot to note, this is a standard PCIe bridge (fanout) design. You could also test the same inside of your Windows PC.

Just be sure that the PCIe gold finger keying is correct & apply the PC power supply molex connector then check your Device Manager.

If you are DOS person - you can use one of the freeware PCI utilities to scan & dump your PCIe bus to see if it enumerating on your PC.

[l00nix]

Thanks @juthi - plugged in one of my PCI to NVMe adapters - doesn't see it.

Here is what I see in dmesg:

[ 1.276644] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges: [ 1.279389] brcm-pcie fd500000.pcie: No bus range found for /scb/pcie@7d500000, using [bus 00-ff] [ 1.282156] brcm-pcie fd500000.pcie: MEM 0x0600000000..0x07ffffffff -> 0x0000000000 [ 1.284922] brcm-pcie fd500000.pcie: IB MEM 0x0000000000..0x01ffffffff -> 0x0200000000 [ 1.606276] brcm-pcie fd500000.pcie: link down

[juthi]

Hi. Not a rpi guy...yet but suggest the following:

1) if the Linux distribution offers the following, please try:

lspci -vv ; that is 2 * 'v' as in Victor and not a 'w'

this will display all PCI devices found by the host CPU. Inside this listing, check to see if you can locate the Texas Instrument PCIe switch. From the TI datasheet, you can find the PCI IDs (Vendor & Device ID). If found, then the PCIe switch is being detected by the host CPU. Then could be a s/w issue with the RPI distribution but not sure.

Respectively, you can also seat the same adapter inside a normal PC and check to see if the switch is found. These switches (like others) use the CLASS CODE to be auto-enumerated by every operating system. Respectively, no device driver is required for your Windows / Linux OS running on your x86 / AMD / Intel CPU. This is to only validate that the adapter is working.

2) I did watch the linked video at the start of this thread and the seating inside the keyed PCIe slot looks correct. Assuming that the external molex adapter is supplying enough current, you should be operational. Have you tested both of the x1 black PCIe slots for your NVMe adapter ? Can only assume that the vendor shipped you tested boards.

3) If you remove this adapter and seat in your PCIe to NVMe adapter -> does the rpi see the NVMe adapter natively (without this PCIe switch)? That will be a good test. I am assuming that your NVMe adapter is for the PCIe slot and not the PCI slot otherwise that adds another layer of complication (rpi PCIe slot -> PCIe switch (TI) -> PCIe to PCI bridge (?? brand / make) -> PCI NVMe adapter. It can be done but with care. We have built many designs like this early on for our product line.

[juthi]

Silly but important concern, can you post a clear picture of the molex connector onboard your external power adapter? Also check against the industry standard pinout. A standard PCIe slot offers 3v3 and 12v0 rails. Be sure this molex connector is supplying these voltages to the proper PCIe adapter pins.

[l00nix]

Hey @juthi

I moved on for the moment to play around with the ASHATA PCE4PCE-A01 1 to 4 Port PCIe Switch.

It seems to work, as you can see on the picture I am using the same power adapter with a sata power plug.

I will revert back once I get a chance to try the Exacq Technologies 1 to 2 ports PCIe x1 adapter again.

Thanks

Alex

[Coreforge]

Thanks @juthi - yes, see picture - are there any LEDs on the card that should light up when operational? I wonder if all three cards I bought are duds.

I'm not sure if it's the issue here, but try powering the pi and the card from one power supply, e.g. a PC power supply, as there have been issues with using multiple power supplies (bad ground connection resulting in connection issues. Jeff had those issues with GPUs)

[juthi]

Excellent advice @Coreforge. On a standard PC PCIe slot, the PCIe adapter must be fully powered and awake before the host bios can enumerate the PCIe function. On some protos, we intentionally had some soft start power rails to reduce in-rush currents and found that some branded PCs would miss the detection of our adapter. This may not be the same issue with the rpi but the idea to test with a PC power supply is a good one. Furthermore, the CM4 is powered from a 12V power adapter (coax / barrel jack). You could use a common PC power supply to power the molex connector on the PCIe switch + apply the yellow (+12v) and black (ground) from the same PC power supply onto the coax connector. Be sure to check on the polarity of the coax connector and their through hole location on the CM4 PCB. Do not reverse wire the power supply!!

A few more comments - the TI PCIe switch is a Gen1 (2.5Gbps) switch for the fanout. The CM4 is reported to be a Gen2 PCIe switch (5.0Gbps). If the 5.0 Gbps is confirmed then it will be more practical to install a Gen2 PCIe switch to allow for the downstream adapters to run faster than with the TI switch.

In the meantime, can you share the output from the following (found from Jeff's logs):

lspci -vvnnk ; run this command on your CM4 00:00.0 PCI bridge [0604]: Broadcom Limited Device [14e4:2711] (rev 20) (prog-if 00 [Normal decode]) Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 64 Bus: primary=00, secondary=01, subordinate=02, sec-latency=0 I/O behind bridge: 00000000-00000fff Memory behind bridge: c0000000-c00fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: access denied Kernel driver in use: pcieport

01:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 01) (prog-if 00 [Normal decode]) Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 64 Bus: primary=01, secondary=02, subordinate=02, sec-latency=0 Memory behind bridge: c0000000-c00fffff Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort+ >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: access denied

02:00.0 FireWire (IEEE 1394) [0c00]: VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller [1106:3044] (rev 80) (prog-if 10 [OHCI]) Subsystem: VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller [1106:3044] Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping+ SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 0 Region 0: Memory at 600000000 (32-bit, non-prefetchable) [disabled] [size=2K] Region 1: I/O ports at [disabled] Capabilities: access denied

The above should list all enumerated devices on the CM4 and should also display the PCIe switch if it is being enumerated.

reference: https://gist.github.com/geerlingguy/9d78ea34cab8e18d71ee5954417429df

[Coreforge]

The IO Board also has a floppy connector for power that can just be used with an adapter and an atx power supply, or just an older one that still has a floppy connector. Especially with older ones, but maybe also with newer ones, they need some kind of base load though on either the 5V or 12V rail to have a stable output and some just shut off without one. Either a power resistor or a turn indicator bulb work well as one.

[vukitoso]

Hello. Do you always need to connect additional power to the board? Or only when the device needs more power?

[juthi]

It is best to apply the additional power. Unless the vendor claims otherwise, the additional power supply is mandatory. Then check if the PCIe switch is being enumerated. Next, of course, check if your docked widget is being found through the PCIe switch.

[pamunthe]

I know I'm kind of late to the party, but what version of raspbian did you do your initial testing on ? My freshly installed Rasbian 11 32-bit just just outputs segmentation fault when running lspci as a regular user, and ust freezes when same command run as root.

the board I'm using is the Exacq Technologies 1 to 2 ports PCIe x1 adapter using the TI XIO3130 PCIe switch chip powered through the J20 powerheader.

Update: Running now on rasbian 11 64-bir and I have now been able to get the PCIe switch recognized by the OS. I have tried to plug both a NVMe adapter and a Realtek Nic into it and only the NIC is recognized, The driver for the nic is loaded but no device is present in the OS, just eth0. Could this be a power issue as it is not stable. Sometimes it just freezes when issuing lspci

[geerlingguy]

Testing with the Raspberry Pi 5 and an NVMe drive...

pi@pi5:~ $ lspci
0000:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21)
0000:01:00.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
0000:02:01.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
0000:02:02.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
0001:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21)
0001:01:00.0 Ethernet controller: Device 1de4:0001

For some reason I'm not seeing the NVMe drive. Going to try something else.

[geerlingguy]

My soundblaster card shows up behind the bridge, though:

pi@pi5:~ $ lspci
0000:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21)
0000:01:00.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
0000:02:01.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
0000:02:02.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
0000:04:00.0 Audio device: Creative Labs EMU20k2 [Sound Blaster X-Fi Titanium Series] (rev 03)
0001:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21)
0001:01:00.0 Ethernet controller: Device 1de4:0001
pi@pi5:~ $ lspci -tv
-+-[0000:00]---00.0-[01-04]----00.0-[02-04]--+-01.0-[03]--
 |                                           \-02.0-[04]----00.0  Creative Labs EMU20k2 [Sound Blaster X-Fi Titanium Series]
 \-[0001:00]---00.0-[01]----00.0  Device 1de4:0001

Hmm... maybe it's just one of the two ports on my riser is having an issue, because swapping the NVMe to the other side gets it to show up:

pi@pi5:~ $ lspci
0000:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21)
0000:01:00.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
0000:02:01.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
0000:02:02.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
0000:04:00.0 Non-Volatile memory controller: KIOXIA Corporation Device 0010 (rev 01)
0001:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21)
0001:01:00.0 Ethernet controller: Device 1de4:0001
pi@pi5:~ $ lspci -tv
-+-[0000:00]---00.0-[01-04]----00.0-[02-04]--+-01.0-[03]--
 |                                           \-02.0-[04]----00.0  KIOXIA Corporation Device 0010
 \-[0001:00]---00.0-[01]----00.0  Device 1de4:0001

Edit: Hmm, now it's showing fine in slot 1 too, lol. Maybe just bad luck!

pi@pi5:~ $ lspci -tv
-+-[0000:00]---00.0-[01-04]----00.0-[02-04]--+-01.0-[03]----00.0  KIOXIA Corporation Device 0010
 |                                           \-02.0-[04]--
 \-[0001:00]---00.0-[01]----00.0  Device 1de4:0001

[geerlingguy]

Seems stable now, but like on the CM4, it looks like the firmware only looks for root-level NVMe drives—at least right now. I'll ask RPi about whether they'll support deeper NVMe enumeration for boot.

[theodric]

In case anybody's using one of these Pericom PI7C9X2G304 / PI7C9X2G404 PCIe switches in production and experiencing random kernel panics, there is a fix in the works over here (which I have personally validated on 5.15.92): https://github.com/raspberrypi/linux/issues/5352