Group buy interest? - Githubissues

j4cbo commented 4 months ago

A few people have mentioned interest in buying one of these so I'm starting a thread to discuss. Price will be $50 each plus shipping.

RoxOnix commented 2 months ago

Hi, I'll take 2, check your email.

su-thomas commented 2 months ago

Hi, interested in buying, email sent

neutronstriker commented 2 months ago

Hi @j4cbo , I am interested to buy 1 as well, however my email got blocked by your spam filter, is there an alternative way to communicate with you. I am in India.

kitor commented 2 months ago

My riser arrived too, already installed and working!

Some fun(?) facts i found in my setup.

Let's start with remark about PCIe topology in those devices:

Both NVME slots go to PCH
Factory x8 riser goes to CPU. Factory x4 riser goes to PCH
j4cbo riser has pcie x4 slot on CPU, first (J2) m.2 on CPU, 2nd (J1) m.2 on PCH
(not sure about WiFi slot, I don't use it atm)

I have:

2x NVME for boot devices (proxmox)
2x NVME for TrueNAS VM (pcie passthrough)
1x Aquantia AQC100 based 10G SFP+ NIC - Sonnet Solo 10G.

TL;DR:

If you want to pass through a device to VM, use one of PCH slots to keep system going into low power C states
If you need to pass through a device on bifurcated CPU slot, you likely need to pass through both devices.
If you have Aquantia AQC100 based 10G NIC, there's a way to achieve C7 with very specific ASPM configuration

PCIe passthrough and C states

While still running on standard riser with bifurcation card (and without 10G NIC) I found that system idles at C7, but the moment I boot TrueNAS VM it stops at C3. I had both passed through drivers at bifurcation card. As soon as I stop VM, system idles back at C7.

After some trial and error I found that for some reason passing through from bifurcated slot makes this happen. When I moved one of my storage SSDs to factory slot and passed only that drive - everything went to C6 (and even sometimes C7) just fine.

With new riser installed I confirmed this to be a case. I settled down by having one of boot and storage SSDs in factory slots, 2nd boot SSD in riser slot 1 (CPU) and 2nd storage ssd in riser slot 2 (PCH). This way BIOS still properly recognizes boot device in factory slot 1 and I pass through disks connected only to PCH slots.

AQC100 card and ASPM / C states

When card is plugged in, system reports max C2. Looking at lspci output gives this:

03:00.0 Ethernet controller: Aquantia Corp. Device 80b1 (rev 02)
        LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited
        LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+

This is sus, as "unlimited" latency to my knowledge means once device enters state, it will never return from it.

Let's force ASPM enable on the card anyway:

root@pve:~# /opt/enable-aspm.sh 01:00.0 3
Device: 01:00.0 Ethernet controller: Aquantia Corp. Device 80b1 (rev 02)
    0x50 : 0x42 --> 0x43 ... [SUCCESS] L1 and L0s

Device goes briefly to C7... aaaand hard lock. After some debugging I noticed in my boot scripts previously enabled L1 on PCIe port where this card is connected:

00:01.2 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x4) (rev 0a) (prog-if 00 [Normal decode])
        LnkCap: Port #4, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <256ns, L1 <8us
        LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+

After disabling it (PCIe port ASPM disabled, Card ASPM L1 Enabled)... system works fine and goes to C7 again?! I double checked that on stock M720 with factory riser - same happens. I don't know wh it works but it works. Everything is running stable now.

PCIe passthrough on bifurcated CPU slot (or on PCIe x4 / m.2 J2 on this riser)

After I received the riser, during my initial tests I found something odd about PCIe passthrough of bifurcated slot. On the old riser I had both drives in that slot. With the new riser NIC and one SSD goes to that slot.

If I passed through this particular SSD, as soon as I started VM, NIC went down. I tested pass through for other slots and just this one had that issue.

j4cbo commented 2 months ago

Thanks for all the info! In my own system I'm currently running TrueNAS SCALE on bare metal, but I had been thinking of putting a Proxmox layer underneath. (TrueNAS works well enough as a VM host but Proxmox is much more powerful.) It's interesting that there's an interaction between passthrough and bifurcation, not terribly surprising but it seems like it "should" work.

I'm now sold out of the first batch of 20 boards! I'll ask the manufacturer about pricing for a follow-up batch. Anyone else who's interested in buying one, please let me know.

kitor commented 2 months ago

One more thought: I previously ran I5-8400 (non-T), now I have I9-9900T engineering sample QQC0. Behaviour is exactly the same on both. Move from 8400 to 9900T had no impact on my average power usage. That's with average load just slightly below 1.0

Combination of 2.5G to 10G upgrade (I previously had 2.5G m.2 card) and fixing C states ended in... lowering my average power consumption by roughly 3-4 watts. With a note that's 10G over SFP+ copper DAC which is the low power, low latency solution for 10G.

Go0oSer commented 2 months ago

Interested in 2.

j4cbo commented 2 months ago

Update: I've ordered more boards! :) I'll reach out to everyone who's contacted me once they're ready to ship.

joseal commented 1 month ago

Hi @j4cbo, please count with me for one more board to EU. If there is anyone else interested in EU, I don't mind to join for a group buy.

thank you!

marcojk commented 1 month ago

I wonder where @j4cbo ordered the first batch. Maybe it would be just cheaper to order a second batch directly to someone in EU ... assuming there's enough interest of course.

This is the option i was referring to. Collect the order, pay to j4cbo, order shipped directly to europe and from here we will manage somehow :)

I'd be interested in 2 boards, and I still believe this could me an interesting alternative...

joseal commented 1 month ago

I wonder where @j4cbo ordered the first batch. Maybe it would be just cheaper to order a second batch directly to someone in EU ... assuming there's enough interest of course.

This is the option i was referring to. Collect the order, pay to j4cbo, order shipped directly to europe and from here we will manage somehow :)

I'd be interested in 2 boards, and I still believe this could me an interesting alternative...

if @j4cbo agrees, I'm in and I don't mind to help with the distrubution...

AndrewL82 commented 1 month ago

Hey, I'd be interested in buying 2 if shipping to Australia isn't too much :)

su-thomas commented 1 month ago

Hey, I'd be interested in buying 2 if shipping to Australia isn't too much :)

@AndrewL82 I'm also interested in Australia, perhaps we can bundle shipping to save on costs

randlor commented 3 weeks ago

Hi, in the UK and interested in one. Thanks.

wouterbouvy commented 2 weeks ago

Awesome work @j4cbo! I would be interested in buying 3 or 6 of your risers once your new batch arrives. Shipment to the EU.

j4cbo commented 2 weeks ago

The new batch is here! :) Send me an email if you haven't already.

j4cbo / tiny5-m2-riser

Group buy interest? #1

Some fun(?) facts i found in my setup.

PCIe passthrough and C states

AQC100 card and ASPM / C states

PCIe passthrough on bifurcated CPU slot (or on PCIe x4 / m.2 J2 on this riser)