Test Pineberry Pi's HatDrive! Top and Bottom NVMe HATs

[geerlingguy]

Pineberry Pi is a joint venture between Michał Gapiński and Mirosław Folejewski — both of whom have a history of making some pretty cool Pi products.

They just announced their "HatDrive!" Line of HATs (one for the top, one for the bottom) for the Raspberry Pi 5.

I have one of each and will be testing it, along with their impedance-controlled FFCs (they make two lengths):

• HatDrive! Top for 2230/2242 M.2 E-key
• HatDrive! Bottom for 2230/2242/2280 M.2 E-key

I should also redirect the page https://pipci.jeffgeerling.com/hats/mirko-hat5m1-hat.html to these new HAT pages. See #550

[kitopopo]

Well then I will have to continue using my experiment to achieve stability en Gen3. Thank you

[mightymos]

Thanks @mikegapinski I tried setting Gen 2 and disabling ASPM but still getting AER corrected errors. I also just saw my first uncorrected (fatal) error so that is fun lol.

Apparently the aliexpress cable is claimed to be impedance controlled, so at worst I might have wasted some money. I can stick with the stock cable for now.

I still seem to require PCIE_PROBE to boot with 2/16/2024 firmware. Out of curiosity, since dtparam=nvme or similar is no longer explicitly required in config.txt where is it set and at what speed? How do I know that if it is set in config.txt that the setting there overrides whatever is apparently now default?

Thanks for the help. If there is anything else I can report let me know.

[mightymos]

Regarding the Kioxia KBG40ZNS drives I was able to attempt updating firmware with this tool (Windows only): https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=tknm3

However I apparently have the latest firmware already.

PHYSICALDRIVE2: Firmware download and activation command completed successfully.
[Old-revision:10410106] [New-revision:10410106]  (Same revision)
>>> Command success but firmware versions are same...
### Return code: 4000 ###

To summarize to /boot/firmware/config.txt added:

dtparam=pciex1_gen=2

To /boot/firmware/cmdline.txt added:

nvme_core.default_ps_max_latency_us=0 pcie_aspm=off

[vin100ross]

I just wanted to add my 2 cents. Note that my HAT is not from Pineberry, so this might or might not be relevant. However, to add some weight to @kitopopo's experiments, I also had several corrected AER errors with my setup. Disabling bluetooth/wifi on my Pi5 reduced them by more than 90% (the remaining errors might be from a nearby router). So radio interference and FPC shielding are definitely things worth looking at. The Aliexpress cable linked previously is also advertised as "shielded", and I have one arriving in a couple of days, so I'm curious to see how it goes.

[kitopopo]

I just wanted to add my 2 cents. Note that my HAT is not from Pineberry, so this might or might not be relevant. However, to add some weight to @kitopopo's experiments, I also had several corrected AER errors with my setup. Disabling bluetooth/wifi on my Pi5 reduced them by more than 90% (the remaining errors might be from a nearby router). So radio interference and FPC shielding are definitely things worth looking at. The Aliexpress cable linked previously is also advertised as "shielded", and I have one arriving in a couple of days, so I'm curious to see how it goes.

Dear @vin100ross , thanks for your answer.

I also have the router nearby, and by deactivating Bluetooth and Wi-Fi the errors improved. I'm sorry to tell you that in my case the 3cm ffc hose from the Asian store seems not to be very armored since I still have PCIe corrected errors.

I am preparing a new shield that I have thought of, at the moment it works with a 5cm flat cable and without errors, I hope my ideas will help you if you decide to prepare something similar. I hope to upload the photos today.

_For non-believers in physics and electricity here I leave the following:

The Faraday cage is a conductive structure that protects its interior from external electric fields. Named after the physicist Michael Faraday, this cage works by redistributing external electric charges along its surface, neutralizing electric fields inside.

Key features of a Faraday cage:

Conductive material: It must be made of a conductive material, such as metal. Galvanized steel is commonly used.

No large openings: The openings in the cage should be smaller than the wavelength of the radiation to be blocked.

Ground connection: The cage must be grounded to allow the discharge of accumulated electric charges.

Faraday cages are used in various applications, from laboratories to protect sensitive equipment to building construction for shielding against electromagnetic radiation. The main idea is to create an electromagnetically isolated environment inside the conductive structure._

Best regards

[mikegapinski]

@kitopopo @vin100ross Hmm. This is now something we can work with. In a decently isolated environment it is easy to create a test that is easy to reproduce. You can fire up a hotspot with hostapd on either 2.4ghz or 5ghz and connect to it from a different device. After that you can do large transfers and check the logs with a sensitive M.2 device (those MAXIO controllers are perfect).

I think you might have found a valid explanation why Gen 3 is not fully certified by RPI. The Wi-Fi antenna is located next to the FPC connector and using a impedance controlled FPC in a shape that is covering it directly (Pimoroni NVMe base?) could make things even worse.

We'll do a brainstorm and figure out if there is a way for us to create an even better FPC cable since it is just a flexible printed circuit.

[kitopopo]

Good afternoon,

This is another type of shielding that works better than the previous one since I can eliminate errors with gen3 using a simple 5cm fpc cable I have used copper tape or desoldering tape that is thicker and better than aluminum foil.

It is something provisional, I hope that @mikegapinski and his team manage to make a new FPC cable that solves these errors for everyone. Many of us will be interested in buying this new cable from Pineberry Pi.

I can't help more, best regards

[mikegapinski]

So... The official RPI HAT FPC cable is also not shielded, just impedance-controlled like ours. https://twitter.com/pineberrypi/status/1767244283025711559?s=61&t=Lh0E7mQyw_B9UkJjFAZW_Q

This makes me even more excited about trying to prototype this approach. It'll be expensive and the only thing we could try is Copper Layer Shields, but that would make the cable very stiff. Other more accessible shielding approaches would mean no impedance control and this is a no-no. You can read more here: https://blog.epectec.com/emi-and-rf-shielding-options-for-flexible-circuits

It might not be feasible for the HatDrive line but it could make more sense with something like the HatBRICK! Commander or uPCIty. Since we are already selling cables and they are very, very popular there could be a way to make a test batch at some point.

@geerlingguy another thing to play with that could be potentially improved, we didn't get any issues with 100mm on uPCIty but our office is in a basement and we usually use ethernet.

[vin100ross]

A quick Google shows wifi/Bluetooth interference with NVME on the Pi5 creeping up once in a while, so we're not the only ones it seems. I for one would buy into any solution you guys can come up with, even if it means a stiffer/thicker cable.

Anyway with aspm=off, and Bluetooth/wifi off, I went from a corrected error every minute on average to one every two days or so... Not perfect but manageable.

[kitopopo]

@vin100ross , In my case sometimes the errors remained stable for quite a long time. To speed up and reproduce the problem, I restarted my raspberry ,sometimes even restarting is not enough and it is necessary to turn off the power. If you look at the log after each boot with "journalctl -n 3000" you will see that they start to appear every 10 seconds, the way to solve it is to restart again and it is stable again with sporadic errors. The key is to shut down or restart. I think that at boot time is when the error occurs and there is no way to stop it. Greetings

[kitopopo]

Interesting and complicated armor blog, I hope you find a solution. I also prefer the cable to be a little more rigid but without errors. Thanks in advance

[matty67]

I use the HatDrive top. When I activate the option POWER_OFF_ON_HALT=1 and shutdown the raspberry, it still needs 0,8Watt and the ACT Led ist blinking 1 sec on and 1 sec off. Is there another option to poweroff the HatDrive completely?

[mikegapinski]

I use the HatDrive top. When I activate the option POWER_OFF_ON_HALT=1 and shutdown the raspberry, it still needs 0,8Watt and the ACT Led ist blinking 1 sec on and 1 sec off. Is there another option to poweroff the HatDrive completely?

HatDrive Top (2024/V2) or HatDrive Top Lite supports sleep, V1 was designed when this feature was not available (we started selling the board in early November and the spec/firmware support for HAT+ was released a month later.

HatDrive Bottom (2024/V3) that supports sleep is shipping for over 2.5 months now

[mightymos]

Well the Aliexpress cable did not appear to work at all...nothing showed up from lspci whereas drives would normally at least show up with stock cable. I believe my cable was seated properly.

I then returned to the stock cable and was able to write a lite image to an SN530 NVMe drive, which is listed as compatible - unlike my Kioxia drive. However rebooting the Pi would still not boot with the NVMe. I then booted via USB adapter with SSD drive and lspci did not show SN530 any longer despite it just successfully writing.

It seems like I am out of options for more experimenting within budget so would plan to boot by USB adapter in the future. I suppose the benefit here is that USB is fairly simply to swap to another system if needed. In any case I'm not sure if this helps anyone but has been my experience so far.