Open j4cbo opened 4 months ago
Hi, I'll take 2, check your email.
Hi, interested in buying, email sent
Hi @j4cbo , I am interested to buy 1 as well, however my email got blocked by your spam filter, is there an alternative way to communicate with you. I am in India.
My riser arrived too, already installed and working!
Let's start with remark about PCIe topology in those devices:
I have:
TL;DR:
While still running on standard riser with bifurcation card (and without 10G NIC) I found that system idles at C7, but the moment I boot TrueNAS VM it stops at C3. I had both passed through drivers at bifurcation card. As soon as I stop VM, system idles back at C7.
After some trial and error I found that for some reason passing through from bifurcated slot makes this happen. When I moved one of my storage SSDs to factory slot and passed only that drive - everything went to C6 (and even sometimes C7) just fine.
With new riser installed I confirmed this to be a case. I settled down by having one of boot and storage SSDs in factory slots, 2nd boot SSD in riser slot 1 (CPU) and 2nd storage ssd in riser slot 2 (PCH). This way BIOS still properly recognizes boot device in factory slot 1 and I pass through disks connected only to PCH slots.
When card is plugged in, system reports max C2. Looking at lspci output gives this:
03:00.0 Ethernet controller: Aquantia Corp. Device 80b1 (rev 02)
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
This is sus, as "unlimited" latency to my knowledge means once device enters state, it will never return from it.
Let's force ASPM enable on the card anyway:
root@pve:~# /opt/enable-aspm.sh 01:00.0 3
Device: 01:00.0 Ethernet controller: Aquantia Corp. Device 80b1 (rev 02)
0x50 : 0x42 --> 0x43 ... [SUCCESS] L1 and L0s
Device goes briefly to C7... aaaand hard lock. After some debugging I noticed in my boot scripts previously enabled L1 on PCIe port where this card is connected:
00:01.2 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x4) (rev 0a) (prog-if 00 [Normal decode])
LnkCap: Port #4, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <256ns, L1 <8us
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
After disabling it (PCIe port ASPM disabled, Card ASPM L1 Enabled)... system works fine and goes to C7 again?! I double checked that on stock M720 with factory riser - same happens. I don't know wh it works but it works. Everything is running stable now.
After I received the riser, during my initial tests I found something odd about PCIe passthrough of bifurcated slot. On the old riser I had both drives in that slot. With the new riser NIC and one SSD goes to that slot.
If I passed through this particular SSD, as soon as I started VM, NIC went down. I tested pass through for other slots and just this one had that issue.
Thanks for all the info! In my own system I'm currently running TrueNAS SCALE on bare metal, but I had been thinking of putting a Proxmox layer underneath. (TrueNAS works well enough as a VM host but Proxmox is much more powerful.) It's interesting that there's an interaction between passthrough and bifurcation, not terribly surprising but it seems like it "should" work.
I'm now sold out of the first batch of 20 boards! I'll ask the manufacturer about pricing for a follow-up batch. Anyone else who's interested in buying one, please let me know.
One more thought: I previously ran I5-8400 (non-T), now I have I9-9900T engineering sample QQC0. Behaviour is exactly the same on both. Move from 8400 to 9900T had no impact on my average power usage. That's with average load just slightly below 1.0
Combination of 2.5G to 10G upgrade (I previously had 2.5G m.2 card) and fixing C states ended in... lowering my average power consumption by roughly 3-4 watts. With a note that's 10G over SFP+ copper DAC which is the low power, low latency solution for 10G.
Interested in 2.
Update: I've ordered more boards! :) I'll reach out to everyone who's contacted me once they're ready to ship.
Hi @j4cbo, please count with me for one more board to EU. If there is anyone else interested in EU, I don't mind to join for a group buy.
thank you!
I wonder where @j4cbo ordered the first batch. Maybe it would be just cheaper to order a second batch directly to someone in EU ... assuming there's enough interest of course.
This is the option i was referring to. Collect the order, pay to j4cbo, order shipped directly to europe and from here we will manage somehow :)
I'd be interested in 2 boards, and I still believe this could me an interesting alternative...
I wonder where @j4cbo ordered the first batch. Maybe it would be just cheaper to order a second batch directly to someone in EU ... assuming there's enough interest of course.
This is the option i was referring to. Collect the order, pay to j4cbo, order shipped directly to europe and from here we will manage somehow :)
I'd be interested in 2 boards, and I still believe this could me an interesting alternative...
if @j4cbo agrees, I'm in and I don't mind to help with the distrubution...
Hey, I'd be interested in buying 2 if shipping to Australia isn't too much :)
Hey, I'd be interested in buying 2 if shipping to Australia isn't too much :)
@AndrewL82 I'm also interested in Australia, perhaps we can bundle shipping to save on costs
Hi, in the UK and interested in one. Thanks.
Awesome work @j4cbo! I would be interested in buying 3 or 6 of your risers once your new batch arrives. Shipment to the EU.
The new batch is here! :) Send me an email if you haven't already.
A few people have mentioned interest in buying one of these so I'm starting a thread to discuss. Price will be $50 each plus shipping.