search

geerlingguy / raspberry-pi-pcie-devices

Raspberry Pi PCI Express device compatibility database
http://pipci.jeffgeerling.com
GNU General Public License v3.0
1.56k stars 142 forks source link

Test DeskPi Super6C CM4 6-node Mini-ITX cluster board #425

Closed geerlingguy closed 2 years ago

geerlingguy commented 2 years ago

I just heard about the CM4 Cluster Mini ITX board on DWMzone:

raspberry-pi-cm4-cluster-mini-itx-board-6-rpi-cm4-supported

raspberry-pi-cm4-cluster-mini-itx-board-6-rpi-cm4-supported-1

It looks like it supports up to 6 CM4 modules, with an M.2 slot and microSD card slot available to each of the boards. It uses a 4-pin ATX CPU power plug, and has two RJ45 Gigabit Ethernet ports connected through an onboard switch. It looks like there is a USB 2.0 hub and two HDMI ports connected through to the first Pi, which is nice for cluster management / debugging purposes.

I just ordered one and will post testing notes here or to a new issue later.

geerlingguy commented 2 years ago

Board is up on the site now: https://pipci.jeffgeerling.com/boards_cm/cm4-cluster-mini-itx.html

Changed the title for this issue so I can test it out soon.

geerlingguy commented 2 years ago

Guess what just arrived?

DSC07324

I believe @danmanners is getting one today too...

geerlingguy commented 2 years ago

Some notes from a tour around the board:

DSC07331

danmanners commented 2 years ago

PXL_20220427_203953015

I can confirm that the Chassis-connector power button and LED work as expected, and I've had one node boot up successfully.

Power supply shipped with my unit is a 19V 5.26A, or 100W (99.94w, but whatever) and my one Pi CM4 boots up. I haven't hooked up HDMI or network yet, but that's next on the list.

Can also confirm that the two Noctua Fans connected or operating as expected, albeit at full speed.

The unit powered off (with LEDs on the board on) draws ~2.0 watts according to my Kill-A-Watt, and powered on with a single CM4, Micro SD, and NVMe drive is drawing ~6.3W at peak and is idling at ~4.6W.

I'll be able to update more findings over the next few days.

EDIT: Just a note (not a gripe), the CM4 heatsink I'm using is definitely a little bit annoying to install with the unit. Not an issue, but definitely makes hot-swapping modules effectively impossible if this is installed in a chassis of any kind.

geerlingguy commented 2 years ago

@danmanners - Hmm... that's the same one I'm using. It looks like the little flat screws that hold the heatsink on can't but up against the risers in each of the four corners :(

D'oh!

danmanners commented 2 years ago

image

Migrated over my MicroSD and NVMe from one of my Waveshare BaseBoard (B) and have TalosOS up and operational, as well as HDMI via capture card.

image

At least according to my Ubnt EdgeSwitch, it doesn't see any additional MAC Addresses or management IPs, so my immediate guess is that Realtek chip is unmanaged.

tltangliang commented 2 years ago

Some notes from a tour around the board:

DSC07331

  • Ethernet switch is a Realtek RTL8370N.
  • Each Pi seems to have its own 12-pin header with GND/3V3 on one side, and nRPiBOOT, BT_nDisable, WL_nDisable, EEPROM, nWP, 5V, and GPIO18 down the left side.
  • Each Pi also has its own microSD jack for flashing on each slot
  • Each Pi also has a set of four LEDs. I'm guessing power/act, then Ethernet link/act, but not sure yet.
  • There's an 'Always ON' switch that seems to allow the board to always power all the Pis, or not... though I'm not sure. I'll have to see what that switch does exactly.
  • There are built in PWR_BTN and RESET momentary buttons.
  • There is a built-in front panel header, with PWR_LED connections, PWR_BTN, and RESET pins available
  • There are three fan headers—looks like they're PWM (they're all 4 pin)
  • There are two 4-pin POE1/POE2 headers near the Ethernet jacks, so I'm guessing there would be a way to add a PoE header. Not sure.
  • On the back side, there is a space for an M.2 2280 M-key device (e.g. NVMe SSD) and microSD card for each pi (though it doesn't follow a 1-6 order, so make sure you note which slot goes where!)
  • There's a chip labeled "6679GM 946118" on the back near the power input. Not sure what it is for.
  • The USB 2.0 ports are driven through a CoreChips SL2.1A USB 2.0 HUB TT0122A21.
  • Near the power/reset buttons, there's an STC Micro 15W408AS, with a 'Enhanced 8051 Central Processing Unit' inside. I'm guessing this is the MCU that controls power on the board.
  • On the top, near an oscillator and the Ethernet switch chip, there's an IC labeled "ATMCL908 24C08N SU27 D", not sure what it is though.

https://wiki.wisdpi.com/raspberry-pi-cm4/rpi-cm4-itx-cluster

geerlingguy commented 2 years ago

@tltangliang - Oh excellent! Thanks for posting that. Looks like the switch is basically unmanaged, and the two Ethernet ports are just hanging off it in addition to the 6 internal ports then.

That site also answers some other questions I had, like:

  1. The six green LEDs are 'ACT' LEDs for each of the individual Pis.
  2. The pinout of the ATX 12V input
  3. LED mapping for each of the Pis
  4. The 4-pin header by CM #1 is actually ARGB (controlled via GPIO pin 18). That's cool!

As @danmanners, myself, and some others get into testing this board, we may have more questions too. Is there a specific support forum or email we can contact? I know I got an email directly from one of the devs, and I'll share feedback there, but it would also be nice to know a central place so other users could also provide feedback or get guidance.

Edit: Looks like there's a forum but it's still being set up a bit. Is that the best place to direct users?

danmanners commented 2 years ago

I'll second everything @geerlingguy said.

Also going to see if I can control the addressable RGB via Kubernetes somehow, because why not, lol.

tltangliang commented 2 years ago

@tltangliang - Oh excellent! Thanks for posting that. Looks like the switch is basically unmanaged, and the two Ethernet ports are just hanging off it in addition to the 6 internal ports then.

That site also answers some other questions I had, like:

  1. The six green LEDs are 'ACT' LEDs for each of the individual Pis.
  2. The pinout of the ATX 12V input
  3. LED mapping for each of the Pis
  4. The 4-pin header by CM Test SATA adapter (I/O Crest 4 port Marvell 9215) #1 is actually ARGB (controlled via GPIO pin 18). That's cool!

As @danmanners, myself, and some others get into testing this board, we may have more questions too. Is there a specific support forum or email we can contact? I know I got an email directly from one of the devs, and I'll share feedback there, but it would also be nice to know a central place so other users could also provide feedback or get guidance.

Edit: Looks like there's a forum but it's still being set up a bit. Is that the best place to direct users?

I think the forum can be working now. https://forum.wisdpi.com/

geerlingguy commented 2 years ago

Hmm... this looks awfully similar to 52Pi's DeskPi Super6C, which retails for $199.

The Cluster Mini-ITX board is $200 on DWMZone and shipping was $40ish. Shipping is about the same on 52Pi. So the question is, who actually makes the board, and what's the board's most 'official' name? :D

Pi-Mania commented 2 years ago

Great Question. Let's find out..

Hmm... this looks awfully similar to 52Pi's DeskPi Super6C, which retails for $199.

The Cluster Mini-ITX board is $200 on DWMZone and shipping was $40ish. Shipping is about the same on 52Pi. So the question is, who actually makes the board, and what's the board's most 'official' name? :D

Pi-Mania commented 2 years ago

The ''official'' name is "DeskPi Super6C" DWMZone is an authorised agent

geerlingguy commented 2 years ago

Thanks for clarifying that! I guess we should update the name on the site, then, and also add a reference to both locations in the description.

drifterdave commented 2 years ago

Hey guys, would either of you by chance have a 3d Printer? I got this board, and designed an IO shield for the Super6c but don't have a 3D printer to try to print it out just yet.

Would you be willing to make sure all of the holes line up? As always, great work, I really appreciate the work you put into your videos.

thin_ITX_IO_Shield_Super6c.scad.txt

(It is labeled thin ITX, but there is a customizer that you can toggle Standard ATX or Thin ATX. The case I happen to be using is a Thin ITX case.)

adamfowleruk commented 2 years ago

FYI I've purchased a DeskPi Super6C board and am using it to test the SOQuartz module. Booted Manjaro ok, but need to rejig the USB driver. Initial details and picture on the SOQuartz issue here: https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/336#issuecomment-1190668987

danmanners commented 2 years ago

Hey guys, would either of you by chance have a 3d Printer? I got this board, and designed an IO shield for the Super6c but don't have a 3D printer to try to print it out just yet.

Would you be willing to make sure all of the holes line up? As always, great work, I really appreciate the work you put into your videos.

thin_ITX_IO_Shield_Super6c.scad.txt

(It is labeled thin ITX, but there is a customizer that you can toggle Standard ATX or Thin ATX. The case I happen to be using is a Thin ITX case.)

Hey @drifterdave; my apologies for the delay. I don't have a functional 3D printer, but I'm hoping to get one working in the next 2-3 months, so I should be able to validate everything for you then!

geerlingguy commented 2 years ago

Original URL was: https://pipci.jeffgeerling.com/boards_cm/cm4-cluster-mini-itx.html

New URL is: https://pipci.jeffgeerling.com/boards_cm/deskpi-super6c.html

geerlingguy commented 2 years ago

@drifterdave - I'm printing the shield as we speak (STL file attached). I'll let you know how it lines up!

DeskPi_Super6c_IO_Shield.stl.txt

drifterdave commented 2 years ago

@geerlingguy @danmanners Thank you so much! Hopefully it fits well and I can order one to be made :).

geerlingguy commented 2 years ago

@drifterdave - It's close but needs more tolerances; on my board one of the vertical HDMI ports is slanted a bit, and otherwise, almost everything looks like it needs to go down a bit to fit correctly. The power port is the furthest off though:

IMG_2361

I might mess with OpenSCAD and see if I can get the tolerances dialed in better. I feel like the QC on the finished port locations is not going to be as precise as their mechanical drawings!

drifterdave commented 2 years ago

Hmm very interesting, on my board the power jack is much closer to the edge than it seems yours is, I actually measured off the board as opposed to the mechanical drawings. I think you are right, tolerances are gonna be tough with the inconsistency, lol.

icaesarv commented 2 years ago

It doesnt have a GPIO as i can see? Is that correct?

icaesarv commented 2 years ago

@geerlingguy.. No video for it in a fully working Mode on your channel yet? 🥇

geerlingguy commented 2 years ago

@icaesarv - Working on that currently ;)

geerlingguy commented 2 years ago

Hmm very interesting, on my board the power jack is much closer to the edge than it seems yours is, I actually measured off the board as opposed to the mechanical drawings. I think you are right, tolerances are gonna be tough with the inconsistency, lol.

Heh... I noticed the solder joints on some of the points were a bit haphazard. All are good, but some of the ports are a little high/low—even the LEDs are all over the place. So yeah, I'm guessing everyone will need a little tweaking. I might just add a bunch of extra space (like +/-1mm in each dimension) and live with 'panel gaps'.

icaesarv commented 2 years ago

You should build the maximum out of that board, put 6 Raspberry CM 8Gb with eMMC & 6x 1TB SSDs, Watercooling all Processors or using 6 ICE Towers, power it with a decent power supply, clock the heck out of it and test its limits. That should be a very beasty NAS Server. Give the manufacturer your feedback on that setup and let them produce a casing for it, that can be used both as a desktop and a 19" server unit using Adapters. If it passes that test, it would be very useful as a NAS and in Home Automation ;)

geerlingguy commented 2 years ago

Working on a cluster configuration here: https://github.com/geerlingguy/deskpi-super6c-cluster

@drifterdave - After four iterations, I have the ports all lined up:

IMG_2390

Here are the relevant files:

drifterdave commented 2 years ago

Awesome thank you! I'm going to see if I can get a buddy to print it and see how well it fits mine.

drifterdave commented 2 years ago

V4 fits my case as well! Although the sides could be a little taller, it doesn't fit tightly without the motherboard behind it. Thank you for making the tweaks to make everything line up @geerlingguy ! image

geerlingguy commented 2 years ago

@drifterdave - Woah, that's a neat little setup! I was thinking of adding some sort of tiny retainer clip by adding a triangular shape that juts out on the frame that holds the io shield in the rectangle, just didn't get around to trying it out. It would be nice if it kind of 'popped in' instead of resting in place, because yes, mine drops down if you don't have the motherboard pressing it in.

icaesarv commented 2 years ago

V4 fits my case as well! Although the sides could be a little taller, it doesn't fit tightly without the motherboard behind it. Thank you for making the tweaks to make everything line up @geerlingguy ! image

Are those compute moduls heat thinks?

drifterdave commented 2 years ago

V4 fits my case as well! Although the sides could be a little taller, it doesn't fit tightly without the motherboard behind it. Thank you for making the tweaks to make everything line up @geerlingguy ! image

Are those compute moduls heat thinks?

It is yes, this one here

icaesarv commented 2 years ago

Watching what both of you are doing.. I know what my next Project is :D

fhemberger commented 2 years ago

@drifterdave What case are you using? Looks like the perfect fit …

drifterdave commented 2 years ago

@fhemberger It's the akasa Cypher it's pretty hard to find, but is such a great case, I have 3 of them from when intel was making Thin Mini ITX motherboards.

fhemberger commented 2 years ago

@drifterdave Thanks!

drifterdave commented 2 years ago

@drifterdave Thanks!

If you wanted one, looks like FrozenCPU has a few in stock, and I have ordered from them before.

(https://www.frozencpu.com/products/21372/cst-1440/Akasa_Cypher_Ultra-Compact_Thin_Mini-ITX_Case_-_VESA_Mountable_AK-ITX04-BK.html )

geerlingguy commented 2 years ago

Posted a video today: https://www.youtube.com/watch?v=ecdm3oA-QdQ

And here's the IO shield on Thingiverse: https://www.thingiverse.com/thing:5465766

paulwratt commented 2 years ago

Akasa Cypher Ultra-Compact Thin Mini-ITX Case - VESA Mountable (AK-ITX04-BK)

@drifterdave can you post your files for the 3D printed back-plate for this Mini-ITX case, thanks

With the dimensions being different from @geerlingguy regular sized back-plate, and me not having a 3D printer (and hence no software installed), I'd like to be able to upload it to PCB / 3D printing service ..

Cheers

Paul PS great to see both you guys with a nice little box for your Super6c - looking forward to mine ..

drifterdave commented 2 years ago

@paulwratt, @geerlingguy's thingiverse link actually includes the Thin-ITX version of the back plate. If you go to Customizer on thingaverse, there will be a dropdown for Standard or Thin-ITX.

gambcl commented 2 years ago

Does anyone know if this thing will auto power on after a power loss, or do you have to hit the power button every time?

I was hoping to build maybe 3 of these for a Kubernetes cluster (assuming I can eventually find some CM4 boards) but it would be a pain if the main board doesn't auto restart after a power cut.

gambcl commented 2 years ago

V4 fits my case as well! Although the sides could be a little taller, it doesn't fit tightly without the motherboard behind it. Thank you for making the tweaks to make everything line up @geerlingguy ! image

It looks like you have swapped out the stock 40mm fan with some Noctuas right? Can just about see them. How are they working out? Does the case have extra screw holes for the extra fans?

filipnavara commented 2 years ago

Does anyone know if this thing will auto power on after a power loss, or do you have to hit the power button every time?

There's a HW switch for that, so it's possible.

gambcl commented 2 years ago

Does anyone know if this thing will auto power on after a power loss, or do you have to hit the power button every time?

There's a HW switch for that, so it's possible.

Good to know, thanks.

drifterdave commented 2 years ago

image

@gambcl No screws in the case, but the fans fit very snugly under the edge of the case, so they don't move or vibrate at all.

paulwratt commented 1 year ago

right around 24days ago (a reply above) I was watching Wendel talk about 5 & 1/4 bays in his "IcyDock Stuff" video, and that night I got a Super6, which arived the day after my Axzez carrier board, and ended up look at 2 Inwin mini-desktop (BL040 & BL631) with 5 & 1/4 bay and 3x 3.5" internals

I ended up getting the Imwin BL631 (its easier to clean the outside - important in my use case) which comes with 300W TFX powersupply, and IcyDock Quinto MB344SPO (4x SSD + slim DVD) 5.25" bay, 2x SSD connected to Axzez, and one of the 4-pin connect to Super6C - I have the front off now with intent to add "second front panel (stuff)" ("stuff" because there can be only one power switch) - there is also just enough room on the backing plate area for both boards to be accessible in the normal manor (one flipped).

I plan to put 2 Exxos 10Gb or 12Gb 3.5" paired with the slim DVD + 2x SSD all connected to the Axzez (5x SATA), and the other 2 SSD on the Super6C CM4#1 via internal USB adapters (and that still leaves 1x 3.5" bay open .. for something ..)

I'll post pictures, but I cant use either atm, unless I take the 2Gb CM4 out of a PiKVM I got on special (for a similar glass panel Red Case AMD based multi-motherboard mATX RPi setup I got 3 months ago) :)

michaelarmstrong commented 1 year ago

I finally have one of these and wanted to share three very early experiences with the unit (1 hour of use) for others who are considering exploring it:

/dev/nvme0n1: Timing buffered disk reads: 1102 MB in 3.01 seconds = 366.66 MB/sec

michaelarmstrong commented 1 year ago
joshuatam commented 1 year ago

Just checked DeskPi website, they have released an official case for Super6c.

https://deskpi.com/collections/deskpi-super6c/products/deskpi-itx-case-kit-for-deskpi-super6c-raspberry-pi-cm4-cluster-mini-itx-board

image