Test LSI 9440-8i (Intel MegaRAID Tri-Mode SAS3408)

[wallentx]

Starting issue for 9440-8i from https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/196#issuecomment-917255610

Where I'm at now: I realized the 9440-8i uses megaraid_sas, rather than mpt3sas. My Waveshare CM4-IO-POE-Box-B arrived, so I switched to that.

Built the kernel with advised details, and also patched megaraid_sas_fusion.c that shipped with the kernel, and booted.

Here's dmesg output:

Loading iSCSI transport class v2.0-870.
megasas: 07.714.04.00-rc1
pci 0000:00:00.0: enabling device (0000 -> 0002)
megaraid_sas 0000:01:00.0: enabling device (0000 -> 0002)
megaraid_sas 0000:01:00.0: BAR:0x0  BAR's base_addr(phys):0x0000000600200000  mapped virt_addr:0x(____ptrval____)
megaraid_sas 0000:01:00.0: Waiting for FW to come to ready state
megaraid_sas 0000:01:00.0: FW now in Ready state
megaraid_sas 0000:01:00.0: 63 bit DMA mask and 63 bit consistent mask
megaraid_sas 0000:01:00.0: firmware supports msix       : (128)
megaraid_sas 0000:01:00.0: requested/available msix 5/5                                       megaraid_sas 0000:01:00.0: current msix/online cpus     : (5/4)                               megaraid_sas 0000:01:00.0: RDPQ mode    : (enabled)
megaraid_sas 0000:01:00.0: Current firmware supports maximum commands: 1517      LDIO threshold: 0
megaraid_sas 0000:01:00.0: Configured max firmware commands: 1516
megaraid_sas 0000:01:00.0: Performance mode :Latency
megaraid_sas 0000:01:00.0: FW supports sync cache       : Yes
megaraid_sas 0000:01:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
megaraid_sas 0000:01:00.0: Init cmd return status FAILED for SCSI host 0
megaraid_sas 0000:01:00.0: Failed from megasas_init_fw 6460

And lspci output:

01:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID Tri-Mode SAS3408 (rev 01)
        Subsystem: Intel Corporation MegaRAID Tri-Mode SAS3408 (RAID Controller RSP3WD080E)
        Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 47
        Region 0: Memory at 600200000 (64-bit, prefetchable) [size=1M]
        Region 2: Memory at 600300000 (64-bit, prefetchable) [size=1M]
        Region 4: Memory at 600000000 (32-bit, non-prefetchable) [size=1M]                            Region 5: I/O ports at <unassigned> [disabled]
        [virtual] Expansion ROM at 600100000 [disabled] [size=1M]                                     Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)                    Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+                                            Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [70] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 1024 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited, L1 <64us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [b0] MSI-X: Enable- Count=128 Masked-
                Vector table: BAR=0 offset=00002000
                PBA: BAR=0 offset=00003000
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
        Capabilities: [148 v1] Power Budgeting <?>
        Capabilities: [158 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [168 v1] #19
        Capabilities: [254 v1] #16
        Capabilities: [284 v1] Vendor Specific Information: ID=0002 Rev=1 Len=100 <?>
        Capabilities: [384 v1] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
        Capabilities: [3bc v1] #15

And storcli64 output:

CLI Version = 007.1804.0000.0000 Apr 09, 2021
Operating system = Linux 5.10.63-v8+
Status Code = 0
Status = Success
Description = None

Number of Controllers = 0
Host Name = raspberrypi
Operating System  = Linux 5.10.63-v8+
StoreLib IT Version = 07.1803.0200.0000

I grabbed the 07.1803.0200.0000 drivers from the Intel site (they look to just be the same as Broadcom), and tried to compile.sh them, but was getting some errors that seem to be due to gcc8? So I've been compiling gcc10 on the pi for several hours now.. and forgive me if this is just be being dumb, but I shouldn't expect to be able to compile these drivers on arm64, right? You mentioned (somewhere) that you needed drivers that were arm64 compatible.. the drivers I downloaded had an ARM folder within the archive, so I'm assuming I'm on the right track..

[geerlingguy]

@wallentx - I know there was one patch that was required when I was trying to compile the ARM64 driver for Raspberry Pi OS. It was because of a bug with the way writeq() works on the Raspberry Pi 64-bit OS in particular: https://github.com/raspberrypi/linux/issues/4158

[wallentx]

Just taking a moment to share this link, which has cleared up so much confusion I've had since I entered the world of enterprise storage hardware: https://forums.servethehome.com/index.php?threads/broadcom-lsi-avago-hba-and-raid-controller-technical-discussion.24119/

Wow. Ok. For what I want to do in the end, I need to get some IT firmware on this card, and now I'm realizing that a 9305-16e that I returned wasn't broke..

[punnypenguins]

Dropping a nice photo of the card in this thread for reference—pilfered from this newegg listing.

.

[wallentx]

I've been stuck on this simply due to me not understanding how to interact with my JBOD using storcli. It's like I'm needing it in "IT mode", except that there isn't an IT mode with the tri-mode cards(?)

[geerlingguy]

storcli can be a little bit complicated, but some of the basics I remember:

sudo ./storcli64 /c0 show

Will show a detailed summary of the adapter and all the physical disks attached, along with attached 'enclosures' (if any).

Something like:

sudo ./storcli64 /c0 add vd r1 name=myraid1 drives=251:0-3 pdcache=default AWB ra direct Strip=64

Will add a virtual drive in raid 1 with four drives from enclosure 251 (0-3), named myraid1.

The StorCLI Reference Manual has everything, but it's a bit of a read :)

[mi-hol]

@wallentx have you had any luck with above hints?

[wallentx]

@wallentx have you had any luck with above hints?

@mi-hol Life got very busy, and I haven't budgeted time to get at this. I'll try to give it a go this week.

[geerlingguy]

I'm gonna call this card 'working', though please feel free to continue adding to the issue if you find anything new here. It's still linked from the card's page and is Google-able, I just want to close issues where the original task is accomplished ;)

[wallentx]

For the sake of completion, I did actually get a Cisco UCSC-9400-8E Brocade Broadcom 9400-8e working and hooked up to 3 jbods with an Nvidia Jetson Orin Nano. I had to crossflash it.