search

geerlingguy / raspberry-pi-pcie-devices

Raspberry Pi PCI Express device compatibility database
http://pipci.jeffgeerling.com
GNU General Public License v3.0
1.6k stars 144 forks source link

Test DeskPi Super6C CM4 6-node Mini-ITX cluster board #425

Closed geerlingguy closed 2 years ago

geerlingguy commented 2 years ago

I just heard about the CM4 Cluster Mini ITX board on DWMzone:

raspberry-pi-cm4-cluster-mini-itx-board-6-rpi-cm4-supported

raspberry-pi-cm4-cluster-mini-itx-board-6-rpi-cm4-supported-1

It looks like it supports up to 6 CM4 modules, with an M.2 slot and microSD card slot available to each of the boards. It uses a 4-pin ATX CPU power plug, and has two RJ45 Gigabit Ethernet ports connected through an onboard switch. It looks like there is a USB 2.0 hub and two HDMI ports connected through to the first Pi, which is nice for cluster management / debugging purposes.

I just ordered one and will post testing notes here or to a new issue later.

joshuatam commented 2 years ago
  • The tolerances aren't great, my PCB even looked like it had impact on one of the corners, still works though.

I can confirm this, my PCB also looked like that, still can use btw.

image

michaelarmstrong commented 2 years ago

Just checked DeskPi website, they have released an official case for Super6c.

https://deskpi.com/collections/deskpi-super6c/products/deskpi-itx-case-kit-for-deskpi-super6c-raspberry-pi-cm4-cluster-mini-itx-board

image

Have just ordered it! Bit pricey for what it is, but will report back.

1201am commented 2 years ago

Looks like there is a new revision of DeskPi board....with LED light on network ports and USB header finally the way it should be. 20221003_195549 20221003_195613

geerlingguy commented 2 years ago

Nice! Looks like they're doing a great job taking feedback and adjusting a bit.

michaelarmstrong commented 2 years ago

My case arrived and it looks like the USB header is the wrong way around on my board so I can't plug it in! Anyone hazard a guess to confirm or denyy? usbheader

michaelarmstrong commented 2 years ago

Additionally I'm seeing packet loss of around 25% on all CM4's... wondering if anyone else experienced this.

althas316 commented 2 years ago

Has anyone ran EMMC on this board. I have had nothing but problems

Flashed 64-bit piOS lite - regularly fails to boot reporting end kernel panic - not syncing asynchronous error interrupt

Tried ubuntu lite and sticks on the rainbow boot screen

Spent the day flashing and re-flashing the emmc on this board and a carrier board and all the same results. Have no issues on the carrier board, they boot each and every time I am not a hugely knowledgeable person on this (that is half the reason to get this) so if anyone has encountered these issue would appreciate any feedback

rr4444 commented 1 year ago

@geerlingguy @michaelarmstrong @joshuatam

Amazing post, video and thread! I am now really thinking about one of these...

Do you guys have an idea whether:

1) there is space for NVME heatsinks underneath in the official or unofficial cases?

2) can the micro USBs used for mass storage, like SSDs, and would that be USB3?

3) any idea if Intel Optane NVME is RPI compatible? Struggling to find this out, as it seems like a very particular storage arch...

4) is the Deskpi AliExpress store legit? How on earth do problems get assessed? I'm worried I'll end up with an old board with upside USB problem or some other issue....

Any insights appreciated!!!!

danmanners commented 1 year ago

@rr4444 responding in order:

  1. Considering motherboard standoff screws are generally 6mm in height, as long as you have a heatsink that is no taller than ~2-3mm then yes. That being said, you're not going to have a ton of options for heatsinks that make any difference at that thickness, IMO.
  2. No, MicroUSB cannot be use for mass storage. Definitely USB2/3.
  3. No idea, but quick googling does not seem promising.
  4. No idea; I would buy directly from DeskPi's website personally, unless they don't ship to your location.
danmanners commented 1 year ago

@althas316 sorry, just seeing this now!!

I haven't tried any eMMC units on the board, but the fact that DeskPi is selling the eMMC units as related items, it'e be bizarre to not support them.

rr4444 commented 1 year ago

Many thanks @danmanners !!

@rr4444 responding in order:

  1. Considering motherboard standoff screws are generally 6mm in height, as long as you have a heatsink that is no taller than ~2-3mm then yes. That being said, you're not going to have a ton of options for heatsinks that make any difference at that thickness, IMO.

This was my concern.... I'm now wondering about using longer screws, which leads me to ask how tight the gap is at the top between the CM4 heatsinks and the official case?

Impossible to know without having one :(

  1. No, MicroUSB cannot be use for mass storage. Definitely USB2/3.

Just for flashing and stuff. Makes sense.

BTW can that be used as tty? This will actually be my first foray in rpis!

  1. No idea, but quick googling does not seem promising.

Booting from something else is ok though. So that link (which hadn't found myself, thank you!) is helpful, cos it confirms that the Rpi does see the Optane as NVME.

  1. No idea; I would buy directly from DeskPi's website personally, unless they don't ship to your location.

Looks official though....

Desk Pi Store https://a.aliexpress.com/_mKITP7u

I've asked on the Deskpi repo issues, but no answer.

rr4444 commented 1 year ago

@danmanners

@althas316 sorry, just seeing this now!!

I haven't tried any eMMC units on the board, but the fact that DeskPi is selling the eMMC units as related items, it's be bizarre to not support them.

The official deskpi site and other listings expressly say emmc is supported............

I hope so, that's what I have!

1201am commented 1 year ago

I have 6x CM4 modules with eMMC on my DeskPi installed and the OS is on eMMCs so it is definitely supported and works just fine.

I do have (6) 256GB NVMe SSDs as well installed for storage (not booting from these).

rr4444 commented 1 year ago

Additionally I'm seeing packet loss of around 25% on all CM4's... wondering if anyone else experienced this. @michaelarmstrong Where did you get yours from?

rr4444 commented 1 year ago

I have 6x CM4 modules with eMMC on my DeskPi installed and the OS is on eMMCs so it is definitely supported and works just fine.

I do have (6) 256GB NVMe SSDs as well installed for storage (not booting from these).

That's great news re eMMC @1201am , thank you! Which NVMEs do you use?

And how do your NVME temperatures behave?

I'm also wondering about how heatsinks fit in the official case... I was thinking about using longer screws, but then occurred to be that then the ports won't line up with the case :(

NVMEs need heatsinks, and I read about the WD Reds, for example, running at 80C! And that's without something like k8s etcd constantly hammering them

1201am commented 1 year ago

@rr4444: At first, I was testing it with Kioxia (Toshiba) 256GB SSD KBG40ZNV256G and I had nothing than problems: SC4s running too hot even idle, 1 or 2 network ports not working at all (not passing any traffic if all 6 SSDs are installed), SSDs running way too hot (hitting 70C with no I/O operations). I thought that the initial set of (6) SSDs os defective so I purchased (4) more Kioxia but the issue persisted.

If I remember correctly, the other brand/model that did not work was SK Hynix HFM256GD3JX013N, don't recall the model of the third one I tried as I was tired of testing and replacing SSDs. What I think is common between all 3 brands that caused the issue is that all 3 of them are single-chip packages and leverage Host Memory Buffer (HMB). The Samsung models utilize separate controller chip along with memory chips. I am not making any conclusions here, just sharing thoughts as I am not sure what is causing the issue here :-).

I had a few larger Samsung 970 EVO that I installed just to try and immediately everything started working as it should: no "dead" network ports, CM4 runs relatively cool (I had heatsinks on these), and SSDs are not hot even without cooling. So my final configuration has (6) Samsung PM981a 256GB (eBay item number 223708631693).

althas316 commented 1 year ago

@danmanners Thanks Danniel. After a lot of playing around, it was the nvme drives I had purchased. Looks like they were not compatible with the board.
I had bought Integral 250GB SSD NVME M.2 2280 PCIe Gen3x4 R-2000MB/s W-1200MB/s TLC M1 Solid State Drive, and these did not work

I had a spare Patriot P300 M.2 PCIe Gen 3 x4 128GB Low-Power Consumption SSD laying around, threw that on and no issues at all

Bought another 5 and is now very happily working away. All the CM4 are set up to boot from the nvme drives. Learning Ansible currently, then K3s, then docker, that's the plan

rr4444 commented 1 year ago

You'll want to learn Docker before any flavour of Kubernetes. Maybe even Docker Swarm too, which is v limited compared to Kube potential. But perfect for many use cases. Then K3S. You can't really understand Kube without understanding how containers, networking and APIs work.

Just my 2c to maximise your learning :)

adminy commented 1 year ago

There are actually no usable GPIO pins on this cluster board is there? ... What a waste considering how much potential GPIO pins probably lead nowhere on the cm4 boards.

knewt commented 11 months ago

Does anyone have any idea why NVMe drives wouldn't get detected on this? They're not showing up at all, which is needless to say rather frustrating!

knewt commented 11 months ago

Okay, this is bleh. So the drives I got are WD Blue SN580s, as I spotted they were discounted for Black Friday. Just nabbed a different drive from another system to try out, and it worked fine. That one.... it's an SN570 !!

So something about the newer drive isn't working on the super6c/cm4, grrr.

filipnavara commented 11 months ago

As a separate data point I can confirm that SN570 SSDs do work. I got two batches which were made in different factories and they are quite different to each other but both of them work.

knewt commented 11 months ago

Nice confirmation, thanks. Guess I'll be contacting WD tomorrow, but assuming they can't solve it, I suspect they won't be willing to swap my SN580s for SN570s. Though I'll try. Otherwise I guess I'll be having to return them all, gah.

1201am commented 11 months ago

Does anyone have any idea why NVMe drives wouldn't get detected on this? They're not showing up at all, which is needless to say rather frustrating!

Look at my posts from January, I spent many hours until I found that there is a compatibility issue with some SSDs. Samsung PM981a 256GB finally worked as it should.

althas316 commented 11 months ago

When I got mine, I bought 6 Integral M Series drives and none of them worked in the cluster board. I am using now Patriot P300 (P300P128GM28) and had no issues

On Wed, 6 Dec 2023 at 23:31, 1201am @.***> wrote:

Does anyone have any idea why NVMe drives wouldn't get detected on this? They're not showing up at all, which is needless to say rather frustrating!

Look at my posts from January, I spent many hours until I found that there is a compatibility issue with some SSDs. Samsung PM981a 256GB finally worked as it should.

— Reply to this email directly, view it on GitHub https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/425#issuecomment-1843855316, or unsubscribe https://github.com/notifications/unsubscribe-auth/APVMLVJ5O7USKTB6ZDQWS4LYID56NAVCNFSM5TWM7GJ2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBUGM4DKNJTGE3A . You are receiving this because you were mentioned.Message ID: @.***>

knewt commented 11 months ago

Ugh, this is ridiculous. With my SN570 500GB working fine, and the report from @filipnavara above, I grabbed some 2TB SN570s. And the one I've opened and tried right now isn't working! Tested in a little minipc x86 box that my 500GB is normally in, and the 2TB shows up in the bios fine there, but just as with the SN580s, nothing on the pcie bus at all on the CM4!

knewt commented 11 months ago

Might try some 1TB drives, see if they work. The 500GB actually shows up on the pcie but listed as 1TB from lspci, though nvme list reports the expected 500GB model. But who knows, maybe an indication that 1TB would work 🤞🏻

knewt commented 10 months ago

Well, picked up an SN570 1TB today (ordered more, but didn’t bother grabbing them yet just in case!), and it works 🧡

ETA: So yay for finally having a drive that works, but boo that the 2TB is somehow different in a way that doesn’t work on the CM4/Super6C

freesurr commented 6 months ago

I am interested in buying the Super6c with the case to create a Ceph storage cluster. But first they need to upgrade the switch, some management would be very nice. And I really hope Deskpi will also make some brackets for the case so you can hang it in a 19 inch server rack.

pcnate commented 6 months ago

I am interested in buying the Super6c with the case to create a Ceph storage cluster. But first they need to upgrade the switch, some management would be very nice. And I really hope Deskpi will also make some brackets for the case so you can hang it in a 19 inch server rack.

I just completed one. Hoping that it is not deprecated by the CM5 someday. I agree on the switch upgrade though I would like the ability to connect a 2.5Gig port. I'd also like at least 1 USB port exposed from each pi. This could allow use to run a USB cable to 6 printers or something. I suppose one could use a USB other IP setup but that is just more things.

Miguel-Rodrigues commented 6 months ago

Very interesting board. I wonder if using these M.2 board to USB C adapters and a couple of USB C hubs could be a workaround for the lack of connectivity. The only thing is you may need to make some holes on the back of the case to have access to the adapters IO, or even a rectangle cut to have easy access to the M.2 and SD card slots. Also a DC power jumper on each Pi would be great for easier hot-swap or manual disabling.

https://pt.aliexpress.com/item/1005005676467186.html https://pt.aliexpress.com/item/1005005871725446.html

pcnate commented 6 months ago

I don't think modifying the is that big of a deal. I do prefer to use the NVMe SSDs rather than the sdcard. An interesting thought those is to use an NVMe on the first CM4, and then the adapters on the rest and make CM4 #2-6 network boot from the first one and mount their own FS there over NFS?

Bandicoot commented 2 months ago

Has anyone ran EMMC on this board. I have had nothing but problems

Flashed 64-bit piOS lite - regularly fails to boot reporting end kernel panic - not syncing asynchronous error interrupt

[Late to the party but maybe this will help a future someone.] I had trouble with the board 3 slot [multiple CM4 boards] not taking a flash until I removed the µSD cards I had stuffed into the holders - fortunately there's enough room from the bottom access panel to get at them, even with my XL sized hands. No disassembly needed.

n.b. If you're running Ubuntu 22.04 LTS, you need to download and install the latest (1.8.5) version of the Rπ flasher as the 1.7.2 included with 22.04 LTS has a problem when it tries to mount the FAT32 partition. Fixed in 1.7.4, but moving forward to 1.8.5 gets you a slightly better UI for picking your image.