geerlingguy / raspberry-pi-pcie-devices

Raspberry Pi PCI Express device compatibility database
http://pipci.jeffgeerling.com
GNU General Public License v3.0
1.56k stars 142 forks source link

Test Pineberry Pi's HatDrive! Top and Bottom NVMe HATs #559

Closed geerlingguy closed 10 months ago

geerlingguy commented 10 months ago

Pineberry Pi is a joint venture between Michał Gapiński and Mirosław Folejewski — both of whom have a history of making some pretty cool Pi products.

They just announced their "HatDrive!" Line of HATs (one for the top, one for the bottom) for the Raspberry Pi 5.

DSC04318

I have one of each and will be testing it, along with their impedance-controlled FFCs (they make two lengths):

I should also redirect the page https://pipci.jeffgeerling.com/hats/mirko-hat5m1-hat.html to these new HAT pages. See #550

geerlingguy commented 10 months ago

Redirect added, and pages are up here:

Bouni commented 10 months ago

Do you know if any RPi5 case that can be used with the HatDrive! Bottom? Pineberry has a picture on their website but it shows the HatDrive! Top variant I think.

zytegalaxy commented 10 months ago

I can't seem to find exact board outline dimensions for the bottom board. I was wondering if it would fit my enclosure...Is that published somewhere? I order a set btw and can verify that once they arrive.

geerlingguy commented 10 months ago

A few quite notes testing the Top Hat:

pi@pihat:~ $ sudo rpi-eeprom-update
*** UPDATE AVAILABLE ***
BOOTLOADER: update available
   CURRENT: Thu 28 Sep 10:24:57 UTC 2023 (1695896697)
    LATEST: Mon 30 Oct 16:45:10 UTC 2023 (1698684310)
   RELEASE: default (/lib/firmware/raspberrypi/bootloader-2712/default)
            Use raspi-config to change the release.

pi@pihat:~ $ sudo rpi-eeprom-update -d -a
*** INSTALLING EEPROM UPDATES ***

BOOTLOADER: update available
   CURRENT: Thu 28 Sep 10:24:57 UTC 2023 (1695896697)
    LATEST: Mon 30 Oct 16:45:10 UTC 2023 (1698684310)
   RELEASE: default (/lib/firmware/raspberrypi/bootloader-2712/default)
            Use raspi-config to change the release.
   CURRENT: Thu 28 Sep 10:24:57 UTC 2023 (1695896697)
    UPDATE: Mon 30 Oct 16:45:10 UTC 2023 (1698684310)
    BOOTFS: /boot/firmware
Using recovery.bin for EEPROM update

EEPROM updates pending. Please reboot to apply the update.
To cancel a pending update run "sudo rpi-eeprom-update -r".

pi@pihat:~ $ sudo reboot
geerlingguy commented 10 months ago

Another note: I could not get the Pi to boot off NVMe if I used rpi-clone. I was able to do that on an earlier revision of the Pi 5 bootloader with the Pi prototype board... but a LOT has changed since then. It would try to boot, then drop me into a BusyBox initramfs shell. Very strange.

It works best if you flash Pi OS directly to the NVMe SSD, using the Pi with a USB adapter, Pi Imager direct, or on another computer.

geerlingguy commented 10 months ago
Screenshot 2023-11-16 at 9 30 32 PM
geerlingguy commented 10 months ago

A few questions about the board that I have now that I've put in a few hours...

jack-ma commented 10 months ago

What is the pitch of the FPC cable connecting the PCIe signal? It looks like 0.5mm?

mikegapinski commented 10 months ago
  1. We'll stock it but I have not found a device that needs it yet. We are playing it safe in case someone needs it, that is why we have power monitoring in the first place.
  2. If you want to access the current monitoring chip via I2C (HatDrive! TOP only for now!!!):

dtparam=i2c_vc=on

You should see the INA with i2cdetect:

mgapinski@coralpi:/dev $ i2cdetect 0
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0.
I will probe address range 0x08-0x77.
Continue? [Y/n] y
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
40: 40 -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
50: 50 -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
70: -- -- -- -- -- -- -- --    
  1. We have ordered GPIO extenders for all of our TOP hats in the 1st batch.
  2. I am working on adapting a acrylic design (2 elements on top/bottom, sides exposed, it will also work with other boards we are developing now) , it won't be ready in 2 weeks but I'll drop you something once we are done. We are not dealing with enough volume to do plastic injection but a lot can change in a month.
Cristianistrate commented 10 months ago

The official active cooler can be used with the TOP Hat ?
Doe's significatively impact the cooling of the fan ?

mikegapinski commented 10 months ago

Yes, and it does not seem like it is having a huge impact. It did not throttle when building the kernel

ThomasKaiser commented 10 months ago

@geerlingguy as for your disk benchmark mentioned on X.

The 777 MB/s do not represent the device's capabilities since a) hdparm uses a rather small block size when benchmarking (it's a tool from last century originally meant to deal with spinning rust) and b) you already showed that ~900 MB/s are possible with Gen3 speeds on RPi 5.

So either testing methodology or SSD in question is to blame for these low numbers (most probably both) :)

As for the 'testing methodology'... PiBenchmarks' Storage.sh relies on hdparm (limited block size, in the past it was 128K, with more modern versions it looks like 1M) or dd with just 4K blocksize. Quick test with a SATA SSD should show the problem:

hdparm -Tt --direct /dev/sda

/dev/sda:
 Timing O_DIRECT cached reads:   754 MB in  2.00 seconds = 376.91 MB/sec
 Timing O_DIRECT disk reads: 1318 MB in  3.00 seconds = 439.28 MB/sec

dd if=/dev/zero of=test bs=4k count=130k conv=fsync
133120+0 records in
133120+0 records out
545259520 bytes (545 MB, 520 MiB) copied, 1.63406 s, 334 MB/s

Storage.sh tells us about 334 MB/s sequential write speeds and 439.28 MB/s sequential reads. While when testing with e.g. 16M blocksize we're at 490/520 MB/s in reality:

iozone -e -I -a -s 100M -r 1024k -r 16384k -i 0 -i 1

          kB  reclen    write  rewrite    read    reread
      102400    1024   107457   436871   441420   439296                                                                                  
      102400   16384   486231   493974   519500   521398

No idea how you measured your 'nearly 900 MB/sec' on the Pi 5... but if it was only with Storage.sh I would revisit this :)

geerlingguy commented 10 months ago

A few test results from Pineberry, of various SSDs:

Model PiBenchmarks.com score
Sabrent 500GB Rocket PCIe 4.0 2280 47002
Crucial P3 Plus 1TB 2280 40118
PNY CS1030 1TB 2280 42203
Lexar NM620 512GB Gen 3 NVMe 2280 44542
Western Digital SN530 512GB 45735
Samsung 970 Evo Plus 1TB 2280 27960

Edit: Also noticed they have a ton more up on PiBenchmarks.com: https://pibenchmarks.com/user/pineberrypi/

geerlingguy commented 10 months ago

No idea how you measured your 'nearly 900 MB/sec' on the Pi 5... but if it was only with Storage.sh I would revisit this :)

@ThomasKaiser - I only use PiBenchmarks.com script because it's a fun way of gamifying the testing—for my actual reporting/testing I use my own script: https://github.com/geerlingguy/pi-cluster/blob/master/benchmarks/disk-benchmark.sh

It runs fio at 1024k block size 4 GB test, then iozone 1024k and 4k random read and write tests. It's quick but effective for most cases. I am looking at expanding the benchmarks a little now that so many devices are reaching into "GB/sec" range, and trying to go for more gigabytes could help squash any additional caching that's happening and throwing off the results.

geerlingguy commented 10 months ago

The official active cooler can be used with the TOP Hat ?

Yep, see the video I just posted! https://www.youtube.com/watch?v=EXWu4SUsaY8

ThomasKaiser commented 10 months ago

It runs fio at 1024k block size 4 GB test

Well, just 1024k is IMO nothing that represents today's sequential access patterns any more. Even network protocols like AFP or SMB dynamically tune their internal blocksizes to more than this in a LAN environment for over a decade now. See my SSD example above: with 1M it's 430/440 MB/s write/read while it needs 16M to get closer to the real sequential transfer rates: 490/520 MB/s

And a write test 4GB in size will result with el cheapo SSDs in an amount of data written to fast cache and once the cache capacity is exceeded a drastic drop in write performance can be observed which – with a 4GB test size – will result in a MB/s number neither representing the 'cache write performance' nor the 'sustained write performance once cache is full' but a crude mixture.

Firing up fio with just 1GB data size but 4 times in a row generates both numbers and also an idea of how large the cache area of a specific SSD approximately is. Needless to say that once the cache area is exceeded the later iozone tests will also suffer from this and generate almost meaningless numbers (at least 4k random I/O) :)

AEW2015 commented 10 months ago

I want to try my FPGA load tester and see what the upper limit is. I can get about 25 Watts on the compute blade with the CM4. It was important to cool the CM4 to supply stable power otherwise the CM4 got to 80C very fast.

lijoseph commented 10 months ago

Does the pineberry hat resolve wifi issue with the prototype raspberry pi m.2 nvme hat?

mikegapinski commented 10 months ago

Jeff used the old eeprom firmware with the OG prototype HAT. It had tons of issues

balupton commented 9 months ago

Does the bottom nvme board fit within the raspi case, with the cooler on the top?

FriendlyNGeeks commented 9 months ago

Now you didn,t hear this from me, a birdie told me if you buy a spark fun qicc shim and bypass the voltage regulator you can i2c 5volts from pi to any accessories you want(pisugar, m.2[bottom] board, etc..)

DJJo14 commented 9 months ago

Maybe a stupid question but is it mounted with the nvme to the rpi, or the nvme to the table? Can you make a picture?

mikegapinski commented 9 months ago

@geerlingguy our docs with drive benchmarks are up on https://docs.pineberrypi.com/nvme-compatibility-list.

Some WD drives that did not work on the CM4 also fail on the Pi5, it is safe to assume most of them work.

MacGyverr commented 9 months ago

Couldn't find a case, so I made a case (for the bottom hat). https://www.thingiverse.com/thing:6382978 20231217_165106 20231217_165148

rjbrown99 commented 8 months ago

@geerlingguy our docs with drive benchmarks are up on https://docs.pineberrypi.com/nvme-compatibility-list.

Some WD drives that did not work on the CM4 also fail on the Pi5, it is safe to assume most of them work.

Protip for anyone buying one of these hats - read that link! I didn't do that ahead of time, bought a Western Digital SN770M, and wasted a bunch of time and effort trying to make it work.

EDIT - Ordered a Sabrent and it works just fine with the HatDrive Top. Here's the specific model if anyone is interested. https://www.amazon.com/dp/B0BQG6JCRP

This drive scored 38493 using the PiBenchmarks script linked above with dtparam=pciex1 (gen 2) and 49078 with dtparam=pciex1_gen=3.

tim-hilt commented 8 months ago

Where can I go to ask about hardware-issues related to the Pineberry Pi HatDrive! Top?

My Transcend 430s doesn't show up in lsblk or fdisk -l and I'm not sure, whether this is a hardware-incompatibility or just wrong configuration on my part (didn't have to deal with hardware-troubleshooting too much yet).

If this is a good place, I could post the journalctl-output related to pcie. I also don't want to spam inappropriately.

EDIT:

Taking a closer look at the ssd-connector and the socket on the hat made me realize that I might have bought the wrong model of ssd.

ghost commented 8 months ago

I got both hatdrive modules for my raspberry pi 5 but sometimes I have the problem, that the ssd is not recognized on boot. I use a Western Digital SN530 wich is running perfectly if the hat recognizes it. To solve the problem I have to bend the ribbon cable and then mostly the ssd is reconized. I think the cables are too thin or defective. It occurs with both cables - the long one from hatdrive bottom and the shorter one from hatdrive top. Have you guys here had similar experiences?

MacGyverr commented 8 months ago

Mine has been on since I got it, I've rebooted it multiple times. and I run Recalbox, but I upgraded the firmware and changed the boot order with the official OS first. I have no SD and just boot Recalbox from the M2. I didn't have an M2 to USB adapter so I booted from the SD and copied over the install image and ran the Raspberry Pi imager from the SD and installed the Recalbox image to the M2 that way. Has worked fine since.

geerlingguy commented 8 months ago

I never tested the full Bottom kit until today—a few notes:

There are also some 3D printable case designs appearing on Printables now! https://www.printables.com/search/models?q=hatdrive

grching commented 8 months ago

Just received my top and bottoms. It would be nice if the M.2 ssd could be screwed in from the top and not bottom, I hope that they will consider that in any future revisions. The supplied standoffs M2.5x6mm should be either M2.5x10mm or M2.5x6+6mm. This allows the bottom to then have extra stand offs on the bottom to keep the SSD off of the table/case or what ever you are using. Currently, for the bottom standoffs I am using M2.5x20+6mm nylon standoffs since the SSD has a heat sink.

for the bottom I have a Samsung 980 Pro + heatsink and for the top a micron 2400 2TB 2230.

The top came with a 28 pin gpio extender it would be nice if it included a 40 pin extender. Since I only have RPI 4's and RPI 5's I have no experience with the smaller gpios.

The installation went pretty smooth thanks to Jeff Geerling. Pineberry Pi did a wonderful job with the design of both hats.

Samsung 980 Pro Gen 3: Category Test Result
HDParm Disk Read 794.56 MB/sec
DD Disk Write 371 MB/s
Score: 45184

darkmanlv commented 8 months ago

tested 2 ssd with Hat Drive! Top on rpi5:

Union Memory not working, didn`t see nvme at all, size 2242

union_memory_2242

Kingston NV1, size 2280, is working, detectable, but size not correct

2280

will buy different 2242 size ssd to test...

shain commented 8 months ago

The HatDrive! Top works perfectly fine until I reboot the pi. Once I reboot or shutdown, the Nvme drive is no longer recognised until I unplug the pi from the power supply and boot it up again. This has happened with all 3 HatDrives that I have. Does anybody else have this issue? I am using WD SN530 Nvme drives.

mikegapinski commented 7 months ago

The HatDrive! Top works perfectly fine until I reboot the pi. Once I reboot or shutdown, the Nvme drive is no longer recognised until I unplug the pi from the power supply and boot it up again. This has happened with all 3 HatDrives that I have. Does anybody else have this issue? I am using WD SN530 Nvme drives.

WD drives are tricky - this particular issue with the SN530 is listed on our docs page in the NVME compatibility section

mathiasscheurer commented 7 months ago

I have similar problems (ssd not recognized at boot) with varoius drives. Unfortunately, my emails are ignored and my request to send new cables too... The drives work perfectly with the adapter from pimoroni.

mikegapinski commented 7 months ago

I have similar problems (ssd not recognized at boot) with varoius drives.

Unfortunately, my emails are ignored and my request to send new cables too...

The drives work perfectly with the adapter from pimoroni.

It is not feasible to send spare ribbon cables knowing the costs of international shipping. We have sent our backlog now, so you'll get a reply soon - my inbox is flooded at the moment.

mathiasscheurer commented 7 months ago

Hello Michal, I would have covered the costs, but of course it's fine this way - thank You very much!

shain commented 7 months ago

The HatDrive! Top works perfectly fine until I reboot the pi. Once I reboot or shutdown, the Nvme drive is no longer recognised until I unplug the pi from the power supply and boot it up again. This has happened with all 3 HatDrives that I have. Does anybody else have this issue? I am using WD SN530 Nvme drives.

WD drives are tricky - this particular issue with the SN530 is listed on our docs page in the NVME compatibility section

Thanks @mikegapinski, I have been able to get it to boot from an SN530 on one of the HatDrives without any issues. Do you have an idea what the root cause of the boot issue is? If it is an issue that can be fixed in the rpi-eeprom, I can give it a shot and release it for other HatDrive users who maybe facing this issue.

mikegapinski commented 7 months ago

The drive is not initialised in time and the bootloader doesn’t give. We don’t make the eeprom but there is one thing you can check - removing everything other than NVMe from the boot order. In theory it should reboot and retry.

RPI is constantly making changes to the bootloader so I expect those things to be ironed out in the future. We have decided to list the drives that are known to work to save people time on troubleshooting. We are also making some internal changes to the boards for future revisions but we don’t control the firmware on the pi itself

Wysyłane z aplikacji Outlook dla systemu iOShttps://aka.ms/o0ukef


Od: Shain Padmajan @.> Wysłane: Wednesday, January 17, 2024 9:48:37 AM Do: geerlingguy/raspberry-pi-pcie-devices @.> DW: Michał Gapiński @.>; Mention @.> Temat: Re: [geerlingguy/raspberry-pi-pcie-devices] Test Pineberry Pi's HatDrive! Top and Bottom NVMe HATs (Issue #559)

The HatDrive! Top works perfectly fine until I reboot the pi. Once I reboot or shutdown, the Nvme drive is no longer recognised until I unplug the pi from the power supply and boot it up again. This has happened with all 3 HatDrives that I have. Does anybody else have this issue? I am using WD SN530 Nvme drives.

WD drives are tricky - this particular issue with the SN530 is listed on our docs page in the NVME compatibility section

Thanks @mikegapinskihttps://github.com/mikegapinski, I have been able to get it to boot from an SN530 on one of the HatDrives without any issues. Do you have an idea what the root cause of the boot issue is? If it is an issue that can be fixed in the rpi-eepromhttps://github.com/raspberrypi/rpi-eeprom, I can give it a shot and release it for other HatDrive users who maybe facing this issue.

— Reply to this email directly, view it on GitHubhttps://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/559#issuecomment-1895350414, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAHDCB7OQTNDANR6V4KSVQTYO6F6LAVCNFSM6AAAAAA7NGHADKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJVGM2TANBRGQ. You are receiving this because you were mentioned.Message ID: @.***>

darkmanlv commented 7 months ago

bought sn740 today, size 2230, Pineberry HAT Top doesn`t detect at all

sn740

mikegapinski commented 7 months ago

All WD drives are very similar to one another, we have 3 different SKUs that use the same PCB. That is why I decided to update our docs a while ago to say that we do not recommend ANY Western Digital drives at the moment. I have two SN530 drives in our lab and one of them only works. Different revision it looks like, consumer drives do not have a fixed BOM so they change controllers/nand from batch to batch

darkmanlv commented 7 months ago

All WD drives are very similar to one another, we have 3 different SKUs that use the same PCB. That is why I decided to update our docs a while ago to say that we do not recommend ANY Western Digital drives at the moment. I have two SN530 drives in our lab and one of them only works. Different revision it looks like, consumer drives do not have a fixed BOM so they change controllers/nand from batch to batch

I think problem is in "pci gen4 x4", as i saw that sn520 wd drive working good with pineberry hat's, but it is "pci gen3 x2", already bought on ebay one of this 10, will test when will be received :)

Also maybe all drives with key M doesn`t work, because all pci gen4 x4 are without key b, only with key m, but older ones with pci gen3 x2 have b+m key....

image

mathiasscheurer commented 7 months ago

I will test some Samsung and Kingspec SSDs next week and I will report the results of my tests.

mikegapinski commented 7 months ago

Samsung devices are OK in general, but I've seen some complaints about Kingspec through our support channels. I don't know this brand at all, but we could test their lineup if they are popular

slagathor69 commented 7 months ago

@geerlingguy

How is the 5V power input on the 'Bottom' version of the board to be used? Will some sort of adapter cable be included?

@mikegapinski

  1. We'll stock it but I have not found a device that needs it yet. We are playing it safe in case someone needs it, that is why we have power monitoring in the first place.

I can't get Samsung 990 PRO w/ Heatsink PCIe 4.0 NVMe SSD 4TB to work with HatDrive! Bottom. This may be the first drive that needs the additional power, where can I order this?

pawelek-org commented 7 months ago

Hi Jeff & All!

I finally finished my Pi 5 setup with Pineberry Pi HatDrive! Bottom and Lexar NM620 256GB.

It took me a while because at first I got stuck with WD SN530. I couldn't force this little guy to boot but it seems this issue it's already mentioned on PineberryPi website. So I ordered Lexar NM620, tried again and it worked.

Thanks to Jeff's rpi-clone script I was able to transfer everything from microSD card to Lexar NVMe and finally boot from it.

I am pretty happy with this setup so far!

Here are some outputs using Storage.sh for LEXAR NM620 256GB.

PS. Greetings from Poland 🇵🇱

Score: 39051

     Category                  Test                      Result     
HDParm                    Disk Read                 432.48 MB/sec            
HDParm                    Cached Disk Read          347.03 MB/sec            
DD                        Disk Write                277 MB/s                 
FIO                       4k random read            108936 IOPS (435744 KB/s)
FIO                       4k random write           76992 IOPS (307969 KB/s) 
IOZone                    4k read                   135655 KB/s              
IOZone                    4k write                  169752 KB/s              
IOZone                    4k random read            69211 KB/s               
IOZone                    4k random write           183361 KB/s 

IMG_0569 IMG_0570

ghost commented 7 months ago

To copy the microSD with Raspberry Pi OS You can use "SD Card Copier" under "Accessoires" in the start menu.

pawelek-org commented 7 months ago

To copy the microSD with Raspberry Pi OS You can use "SD Card Copier" under "Accessoires" in the start menu.

Yes, but only using the GUI. I needed a shell script in Raspberry Pi OS Lite and rpi-clone worked just fine.

KombatPL commented 7 months ago

Might I suggest taking down WD SN530 from compatibility list - or at least mark it with red color or something... I just bought Pi5 + Pineberry NVME HAT + org. power supply + one of the first recommended drives (SN530) from the list only to have it not boot (of course the fault is mine cause I didn't read bottom of the page). Might save some frustration for future buyers. Found some old SK Hynix that works fortunately.

mikegapinski commented 7 months ago

Our docs say:

WD SN530 has been problematic for some users when it comes to NVMe boot, especially in the 2242 variant. Our test drive was a 2280 SKU and it worked OK but we generally recommend staying away from Western Digital drives completely.

Od: KombatPL @.> Data: piątek, 26 stycznia 2024 o 20:34 Do: geerlingguy/raspberry-pi-pcie-devices @.> DW: Michał Gapiński @.>, Mention @.> Temat: Re: [geerlingguy/raspberry-pi-pcie-devices] Test Pineberry Pi's HatDrive! Top and Bottom NVMe HATs (Issue #559)

Might I suggest taking down WD SN530 from compatibility listhttps://docs.pineberrypi.com/nvme-compatibility-list - or at least mark it with red color or something... I just bought Pi5 + Pineberry NVME HAT + org. power supply + one of the first recommended drives (SN530) from the list only to have it not boot (of course the fault is mine cause I didn't read bottom of the page). Might save some frustration for future buyers. Found some old SK Hynix that works fortunately.

— Reply to this email directly, view it on GitHubhttps://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/559#issuecomment-1912588000, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAHDCBYSVABRDN7DVHTJHGLYQQAK3AVCNFSM6AAAAAA7NGHADKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJSGU4DQMBQGA. You are receiving this because you were mentioned.Message ID: @.***>

KombatPL commented 7 months ago

I know I just got confused because this was one of the first positions on the compatibility list :) Like I said - fault is mine - I didn't read till the end before placing order (only read it after troubleshooting it for 2 days lol), but It could save someone else having to walk that path if it Was at least marked 'READ BELOW' :)