Can't enable RDMA: Windows 11 Pro client to ksmbd server

[cyfdecyf]

Please give me some help to make RDMA work between Windows 11 Pro client and ksmbd sever. I've tried using wireshark, compile and modify ksmbd souce code for dignosing for some days, but still can't figure out what's wrong.

Test setup:

• NICs: Two Mellanox ConnectX-4 Lx 25GbE card (both have 2 ports)
• Sever: Arch Linux VM (NIC passthrough to VM), kenel 6.5.5-arch1-1, ksmd-tools 3.4.9
  • Tried running ksmbd server dirctly on Proxmox VE host but can't make Win11 client use RDMA. So switched to Arch VM to use more recent kernel
• Client: Windows 11 Pro
  • SMB Direct optional feature enabled
  • DCB optional feature installed and then removed, tested both conditions
  • Also tried some NetQos related settings to enable PFC and ETS, but guess this should be unnecessary
• Client and server are directly connected with DAC.
  • Running nd_rping.exe on Windows and rping from Linux confirms that RDMA is working.

ksmbd's config has multi-channel enabled, relevent configs will be included at the end.

After Windows accessing a share on ksmbd sever, run following commands on windows:

PS C:\> Get-NetAdapterRdma

Name                      InterfaceDescription                     Enabled     Operational     PFC        ETS
----                      --------------------                     -------     -----------     ---        ---
CX4-2                     Mellanox ConnectX-4 Lx 10/25GbE 2-por... True        True            False      False
CX4-1                     Mellanox ConnectX-4 Lx 10/25GbE 2-por... True        True            False      False

PS C:\> Get-SmbClientNetworkInterface

Interface Index RSS Capable RDMA Capable Speed   IpAddresses                               Friendly Name
--------------- ----------- ------------ -----   -----------                               -------------
11              True        False        25 Gbps {fe80::b7b5:74be:2512:eb1e, 192.168.22.3} CX4-1
15              True        False        25 Gbps {fe80::daa3:67a9:666b:fbcf, 192.168.23.3} CX4-2

PS C:\> Get-SmbMultichannelConnection

Server Name  Selected Client IP    Server IP    Client Interface Index Server Interface Index Client RSS Capable Client RDMA Capable
-----------  -------- ---------    ---------    ---------------------- ---------------------- ------------------ -------------------
192.168.22.2 True     192.168.23.3 192.168.23.2 15                     3                      True               False
192.168.22.2 True     192.168.22.3 192.168.22.2 11                     4                      True               False

Multi-channel works, but RDMA is not used for SMB conection. Copying a 10GB file to server will trigger high CPU usage.

Here's a packet capture smb.pcap.gz and kernel log ksmbd.log.gz during windows making connection and accessing a shared folder.

Kernel log shows ksmbd: smb_direct: init RDMA listener, so the ksmbd kernel module has SMB_SERVER_SMBDIRECT enabled.

Client SMB negotiate request contains SMB2_RDMA_TRANSFORM_CAPABILITIES context, which is not sent if I connected with a non-RDMA capable NIC.

Negotiate Context: SMB2_RDMA_TRANSFORM_CAPABILITIES 
    Type: SMB2_RDMA_TRANSFORM_CAPABILITIES (0x0007)
    DataLength: 12
    Reserved: 00000000
    TransformCount: 2
    Reserved1: 0x0000
    Reserved2: 0x00000000
    RDMATransformId: Encryption (0x0001)
    RDMATransformId: Signing (0x0002)

ksmbd server's response does not have SMB2_RDMA_TRANSFORM_CAPABILITIES, but the SMB doc mentions the processing of this context and corresponding response:

• Building an SMB2_RDMA_TRANSFORM_CAPABILITIES negotiate response context:

  • If the server processed the SMB2_RDMA_TRANSFORM_CAPABILITIES negotiate request context, then the server MUST build an SMB2_RDMA_TRANSFORM_CAPABILITIES negotiate response context by setting the following:

  • If Connection.RDMATransformIds is empty, set TransformCount to 1 and set RDMATransformIds to SMB2_RDMA_TRANSFORM_NONE.

  • Otherwise, set RDMATransformIds to Connection.RDMATransformIds and set TransformCount to the number of elements in RDMATransformIds.

So I tried to modify ksmbd souce code and include SMB2_RDMA_TRANSFORM_CAPABILITIES in response. But this still dose not make windows to use RDMA for SMB connection. (If I saw #435 ealier, I would guess this is not necessary to make Windows client use RDMA.)

Finally, here's the excerpt of ksmbd.conf:

[global]
        bind interfaces only = no
        deadtime = 0
        guest account = nobody
        interfaces =
        ipc timeout = 0
        kerberos keytab file =
        kerberos service name =
        map to guest = never
        max active sessions = 1024
        max open files = 10000
        netbios name = KSMBD SERVER
        restrict anonymous = 0
        root directory =
        server max protocol = SMB3_11
        server min protocol = SMB2_10
        server multi channel support = yes
        server signing = disabled
        server string = SMB SERVER
        share:fake_fscaps = 64
        smb2 leases = no
        smb2 max credits = 8192
        smb2 max read = 4MB
        smb2 max trans = 1MB
        smb2 max write = 4MB
        smb3 encryption = no
        smbd max io size = 8MB
        tcp port = 445

        aio max threads = 100
        aio read size = 1
        aio write size = 1

[namjaejeon]

As I know, Windows pro version doesn't support smb-direct. So I could not work with windows 10 pro version before. Only windows workstation or enterprise(or server) support it ?

[cyfdecyf]

@namjaejeon Thanks for your reply!

I guess I maybe mis-guided by this Microsoft document

At least two computers running one or more of the following operating systems:

• Windows Server 2012 and later.
• Windows 10 Enterprise and later.
• Windows 10 Education and later.
• Windows 10 Pro and later.

Note

Windows 10 and Windows 11 family are restricted to client-only and can't act as an SMB Direct server.

There's indeed SMB direct option when installing optional features in Windows 11 Pro, so I think it should work as the document also says this. I'll try to use a different version of Windows to try enable RMDA to ksmbd server.

[namjaejeon]

I think that this seems to be old document. They remove high performance and new features like SMB Direct and refs filesystem in windows 10 pro. And Add them to new windows 10 pro S version. I am not sure I don't have tested all windows versions. Let me know if it doesn't still work on windows higher version.

[cyfdecyf]

Just tested with Windows 11 Pro workstation connecting to ksmbd server. It works.

• Get-SmbMultichannelConnection shows Client RDMA Capable True
• Transfering file indeed uses RDMA as ksmbd logs message like

  ksmbd: smb_direct: RDMA read, len 0x100000, needed credits 0x2
  ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
  ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
  ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
  ksmbd: smb_direct: returning rfc1002 length 144
  ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
  ksmbd: smb_direct: returning rfc1002 length 144

Thanks for your help @namjaejeon. I'm closing this issue as it's solved.

[namjaejeon]

@cyfdecyf Okay, Sound great! Thanks for your test:)

[cyfdecyf]

I noticed that while Client RDMA Capable is True, Client RSS Capable becomes False.

After using Disable-NetAdapterRmda to disable RDMA on all interfaces on windows, then Client RSS Capable becomes True.

Is this expected behavior?

[namjaejeon]

Yes, It's the same when I tested it. Please check whether RSS and RDMA can work together.