ksmbd is not configured to SMB-Direct automatically

[leehaoun]

I am currently working on creating an SMB Direct share between Windows and Linux. My working environment is as follows

SMB Server : Ubuntu 20.04 SMB Client : Windows 10 EnterPrise Adapter : Mellanox ConnectX-5 100Gbps

I check RDMA Option Property by using,

Get-SmbClientNetworkInterface -> RDMA Capable : True Get-NetAdapterRDMA -> Enable : True And i can check rdma traffics through nd_rping or nd_write_bw

i install ksmbd release 3.4.8 version and ksmbd-tools release 3.5.1 version

and type command

modprobe ksmbd ksmbd.mountd mount -o username=test //xxx.xx.xx.xx/MyShare /mnt/MyShare sudo ksmbd.control --debug rdma

ksmbd.conf [global] netbios name = SMBD server multi channel support = yes

[MyShare] ; share parameters path = /root/MyShare read only = no directory mask = 0777 create mask = 0777 guest ok = no ;force user = nobody ;force group = nobody ;force directory mode = 0777 ;force create mode = 0777

i can check xxx.xx.xx.xx/MyShare Folder on Windows 10 EnterPrise PC But, when i copy this folder Network Bandwidth is just 1Gbps and dmesg --follow show nothing. it seems to be operating on SMB2 rather than SMB-Direct.

and I add ksmbd_debug msg on this code

bool ksmbd_rdma_capable_netdev(struct net_device *netdev)
{

        struct smb_direct_device *smb_dev;
        int i;
        bool rdma_capable = false;

        read_lock(&smb_direct_device_lock);
        list_for_each_entry(smb_dev, &smb_direct_device_list, list) {
                for (i = 0; i < smb_dev->ib_dev->phys_port_cnt; i++) {
                        struct net_device *ndev;

                        ndev = smb_dev->ib_dev->ops.get_netdev(smb_dev->ib_dev,
                                                               i + 1);
                        if (!ndev)
                                continue;

                        if (ndev == netdev) {
                                dev_put(ndev);
                                rdma_capable = true;
                                goto out;
                        }
                        dev_put(ndev);
                }
        }
out:
        read_unlock(&smb_direct_device_lock);

        if (rdma_capable == false) {
                struct ib_device *ibdev;

                ibdev = ib_device_get_by_netdev(netdev, RDMA_DRIVER_UNKNOWN);
                if (ibdev) {
                        if (rdma_frwr_is_supported(&ibdev->attrs))
                                rdma_capable = true;
                        ib_device_put(ibdev);
                }
        }
        ksmbd_debug(RDMA, "rdma capable : %d\n", rdma_capable);
        return rdma_capable;
}

when i mount /mnt/MyShare , dmesg --follow show smb_direct: rdma capable : 0 (False)

How can i activate SMB-Direct??

[namjaejeon]

Can you check it on the latest git master of ksmbd instead of 3.4.8 release ? please get it using "git clone https://github.com/cifsd-team/ksmbd" command. As I remember, There is a related patch in the latest git master branch. Please try it first, And I will check if you still have the problem.

[leehaoun]

i clone master code, However, the bandwidth still does not reach 100Gbps, and there are no RDMA-related logs captured in dmesg

[ 7428.746940] ksmbd: smb_direct: init RDMA listener. cm_id=000000000f4a8e9a [ 7431.229635] CIFS: VFS: \30.0.0.1 has not responded in 180 seconds. Reconnecting... [ 7441.147329] CIFS: Attempting to mount \30.0.0.1\Test

[namjaejeon]

rdma capable will be set to "true" on the latest master ? And windows client switch Multichannel(tcp) to RDMA in the middle of write. Initially, data is transmitted via TCP and then switch to RDMA after a certain period of time.

[leehaoun]

rdma capable still false. Even after copying the image and waiting for a long time, it doesn't seem to transition to RDMA [10912.028596] ksmbd: smb_direct: rdma capable : 0 [10912.028604] ksmbd: smb_direct: rdma capable : 0 [10912.028606] ksmbd: smb_direct: rdma capable : 0 [10912.029330] ksmbd: smb_direct: rdma capable : 0 [10912.030008] ksmbd: smb_direct: rdma capable : 0 [10912.030010] ksmbd: smb_direct: rdma capable : 0 [10912.030012] ksmbd: smb_direct: rdma capable : 0 [10912.030014] ksmbd: smb_direct: rdma capable : 0

[namjaejeon]

It should be true first to use RDMA.

[leehaoun]

static bool rdma_frwr_is_supported(struct ib_device_attr *attrs)
{
    if (!(attrs->device_cap_flags & IB_DEVICE_MEM_MGT_EXTENSIONS))
    {
        ksmbd_debug(RDMA, "cap_flags : %d\n", attrs->device_cap_flags);
        ksmbd_debug(RDMA, "MGT_EXTENSION : %d\n", IB_DEVICE_MEM_MGT_EXTENSIONS);
        return false;
    }
    if (attrs->max_fast_reg_page_list_len == 0)
    {
        ksmbd_debug(RDMA, "reg_page_list_len : %d\n", attrs->max_fast_reg_page_list_len);
        return false;
    }
    return true;
}

When logging like above, i can show this log. [630.283591] ksmbd: smb_direct: reg_page_list_len: 0

[namjaejeon]

I don't test ConnectX-5 before. Recently someone have checked RDMA of ksmbd work fine with ConnectX-4. okay, maybe your NIC don't support FMR.. What will happen if your force to set rdma_capable as true ?

[namjaejeon]

and let me know kernel version ? "uname -r"

[leehaoun]

kernel version is 5.15.0-88-generic

and i remove installed MLNX Driver , and can checked RDMA turn on.

but, BandWidth is still 1 Gbps when i send 129GB img. and when i send 30GB img, it down to 200Mbps

--ksmbd dmesg--

[ 1247.389556] ksmbd: smb_direct: Sending smb (RDMA): smb_len=80
[ 1247.389559] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=80 remaining_data_length=0
[ 1247.389570] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1247.390124] ksmbd: smb_direct: Sending smb (RDMA): smb_len=80
[ 1247.390127] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=80 remaining_data_length=0
[ 1247.390137] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1247.390936] ksmbd: smb_direct: Sending smb (RDMA): smb_len=80
[ 1247.390942] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=80 remaining_data_length=0
[ 1247.390958] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1247.395198] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1247.395208] ksmbd: smb_direct: returning rfc1002 length 144
[ 1247.395211] ksmbd: smb_direct: returning to thread data_read=144 reassembly_data_length=0 first_entry_offset=0
[ 1247.395217] ksmbd: smb_direct: wait_event on more data
[ 1247.395222] ksmbd: RDMA r/w request 0x0: token 0x6dd8, length 0xfffc0
[ 1247.395224] ksmbd: RDMA r/w request 0x1: token 0x6bd6, length 0x40
[ 1247.395388] ksmbd: smb_direct: RDMA read, len 0x0, needed credits 0x2
[ 1247.395816] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1247.395827] ksmbd: smb_direct: returning rfc1002 length 144
[ 1247.395830] ksmbd: smb_direct: returning to thread data_read=144 reassembly_data_length=0 first_entry_offset=0
[ 1247.395833] ksmbd: smb_direct: wait_event on more data
[ 1247.395837] ksmbd: RDMA r/w request 0x0: token 0x84ef, length 0xfffc0
[ 1247.395839] ksmbd: RDMA r/w request 0x1: token 0x8bf6, length 0x40
[ 1247.395925] ksmbd: smb_direct: RDMA read, len 0x0, needed credits 0x2
[ 1247.396332] ksmbd: smb_direct: Sending smb (RDMA): smb_len=80
[ 1247.396336] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=80 remaining_data_length=0
[ 1247.396348] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1247.396958] ksmbd: smb_direct: Sending smb (RDMA): smb_len=80
[ 1247.396961] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=80 remaining_data_length=0
[ 1247.396972] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1247.422594] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1247.422609] ksmbd: smb_direct: returning rfc1002 length 144
[ 1247.422613] ksmbd: smb_direct: returning to thread data_read=144 reassembly_data_length=0 first_entry_offset=0
[ 1247.422617] ksmbd: smb_direct: wait_event on more data
[ 1247.422625] ksmbd: RDMA r/w request 0x0: token 0x6dd8, length 0xfffc0
[ 1247.422627] ksmbd: RDMA r/w request 0x1: token 0x6bd6, length 0x40
[ 1247.422697] ksmbd: smb_direct: RDMA read, len 0x0, needed credits 0x2
[ 1247.423092] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1247.423103] ksmbd: smb_direct: returning rfc1002 length 144
[ 1247.423106] ksmbd: smb_direct: returning to thread data_read=144 reassembly_data_length=0 first_entry_offset=0
[ 1247.423109] ksmbd: smb_direct: wait_event on more data
[ 1247.423113] ksmbd: RDMA r/w request 0x0: token 0x84ef, length 0xfffc0
[ 1247.423115] ksmbd: RDMA r/w request 0x1: token 0x8bf6, length 0x40
[ 1247.423208] ksmbd: smb_direct: RDMA read, len 0x0, needed credits 0x2
[ 1247.423549] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1247.423563] ksmbd: smb_direct: returning rfc1002 length 144
[ 1247.423566] ksmbd: smb_direct: returning to thread data_read=144 reassembly_data_length=0 first_entry_offset=0
[ 1247.423570] ksmbd: smb_direct: wait_event on more data
[ 1247.423578] ksmbd: RDMA r/w request 0x0: token 0x67d2, length 0xfffc0
[ 1247.423581] ksmbd: RDMA r/w request 0x1: token 0x66d1, length 0x40
[ 1247.423590] ksmbd: smb_direct: Sending smb (RDMA): smb_len=80
[ 1247.423594] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=80 remaining_data_length=0
[ 1247.423604] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1247.423762] ksmbd: smb_direct: RDMA read, len 0x0, needed credits 0x2
[ 1247.424026] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1247.424034] ksmbd: smb_direct: returning rfc1002 length 144
[ 1247.424037] ksmbd: smb_direct: returning to thread data_read=144 reassembly_data_length=0 first_entry_offset=0
[ 1247.424041] ksmbd: smb_direct: wait_event on more data
[ 1247.424045] ksmbd: RDMA r/w request 0x0: token 0x6dd8, length 0xfffc0
[ 1247.424047] ksmbd: RDMA r/w request 0x1: token 0x6bd6, length 0x40
[ 1247.424108] ksmbd: smb_direct: RDMA read, len 0x0, needed credits 0x2
[ 1247.424217] ksmbd: smb_direct: Sending smb (RDMA): smb_len=80
[ 1247.424221] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=80 remaining_data_length=0
[ 1247.424238] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1247.424502] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1247.424510] ksmbd: smb_direct: returning rfc1002 length 144
[ 1247.424514] ksmbd: smb_direct: returning to thread data_read=144 reassembly_data_length=0 first_entry_offset=0
[ 1247.424516] ksmbd: smb_direct: wait_event on more data
[ 1247.424520] ksmbd: RDMA r/w request 0x0: token 0x84ef, length 0xfffc0
[ 1247.424521] ksmbd: RDMA r/w request 0x1: token 0x8bf6, length 0x40
[ 1247.424610] ksmbd: smb_direct: RDMA read, len 0x0, needed credits 0x2
[ 1247.424954] ksmbd: smb_direct: Sending smb (RDMA): smb_len=80
[ 1247.424958] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=80 remaining_data_length=0
[ 1247.424969] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1247.424974] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1247.424987] ksmbd: smb_direct: returning rfc1002 length 144
[ 1247.424991] ksmbd: smb_direct: returning to thread data_read=144 reassembly_data_length=0 first_entry_offset=0
[ 1247.424995] ksmbd: smb_direct: wait_event on more data
[ 1247.425001] ksmbd: RDMA r/w request 0x0: token 0x65d0, length 0xfffc0
[ 1247.425004] ksmbd: RDMA r/w request 0x1: token 0x69d4, length 0x40
[ 1247.425190] ksmbd: smb_direct: RDMA read, len 0x0, needed credits 0x2
[ 1247.425662] ksmbd: smb_direct: Sending smb (RDMA): smb_len=80
[ 1247.425665] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=80 remaining_data_length=0
[ 1247.425676] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1247.426290] ksmbd: smb_direct: Sending smb (RDMA): smb_len=80
[ 1247.426294] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=80 remaining_data_length=0
[ 1247.426307] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1247.427099] ksmbd: smb_direct: Sending smb (RDMA): smb_len=80
[ 1247.427103] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=80 remaining_data_length=0
[ 1247.427117] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1247.431660] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1247.431671] ksmbd: smb_direct: returning rfc1002 length 144
[ 1247.431675] ksmbd: smb_direct: returning to thread data_read=144 reassembly_data_length=0 first_entry_offset=0
[ 1247.431680] ksmbd: smb_direct: wait_event on more data
[ 1247.431685] ksmbd: RDMA r/w request 0x0: token 0x65d0, length 0xfffc0
[ 1247.431688] ksmbd: RDMA r/w request 0x1: token 0x69d4, length 0x40
[ 1247.431763] ksmbd: smb_direct: RDMA read, len 0x0, needed credits 0x2
[ 1247.432680] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1247.432691] ksmbd: smb_direct: returning rfc1002 length 144
[ 1247.432694] ksmbd: smb_direct: returning to thread data_read=144 reassembly_data_length=0 first_entry_offset=0
[ 1247.432696] ksmbd: smb_direct: wait_event on more data
[ 1247.432698] ksmbd: smb_direct: Sending smb (RDMA): smb_len=80
[ 1247.432701] ksmbd: RDMA r/w request 0x0: token 0x84ef, length 0xfffc0
[ 1247.432705] ksmbd: RDMA r/w request 0x1: token 0x8bf6, length 0x40
[ 1247.432704] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=80 remaining_data_length=0
[ 1247.432720] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1247.432797] ksmbd: smb_direct: RDMA read, len 0x0, needed credits 0x2
[ 1247.434423] ksmbd: smb_direct: Sending smb (RDMA): smb_len=80
[ 1247.434425] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=80 remaining_data_length=0
[ 1247.434437] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1247.459184] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1247.459199] ksmbd: smb_direct: returning rfc1002 length 144
[ 1247.459203] ksmbd: smb_direct: returning to thread data_read=144 reassembly_data_length=0 first_entry_offset=0
[ 1247.459207] ksmbd: smb_direct: wait_event on more data
[ 1247.459215] ksmbd: RDMA r/w request 0x0: token 0x65d0, length 0xfffc0
[ 1247.459217] ksmbd: RDMA r/w request 0x1: token 0x69d4, length 0x40
[ 1247.459292] ksmbd: smb_direct: RDMA read, len 0x0, needed credits 0x2
[ 1247.459647] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1247.459657] ksmbd: smb_direct: returning rfc1002 length 144
[ 1247.459660] ksmbd: smb_direct: returning to thread data_read=144 reassembly_data_length=0 first_entry_offset=0
[ 1247.459663] ksmbd: smb_direct: wait_event on more data
[ 1247.459667] ksmbd: RDMA r/w request 0x0: token 0x84ef, length 0xfffc0
[ 1247.459668] ksmbd: RDMA r/w request 0x1: token 0x8bf6, length 0x40
[ 1247.459762] ksmbd: smb_direct: RDMA read, len 0x0, needed credits 0x2
[ 1247.460081] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1247.460094] ksmbd: smb_direct: returning rfc1002 length 144
[ 1247.460097] ksmbd: smb_direct: returning to thread data_read=144 reassembly_data_length=0 first_entry_offset=0
[ 1247.460101] ksmbd: smb_direct: wait_event on more data
[ 1247.460107] ksmbd: RDMA r/w request 0x0: token 0x6dd8, length 0xfffc0
[ 1247.460109] ksmbd: RDMA r/w request 0x1: token 0x6bd6, length 0x40
[ 1247.460175] ksmbd: smb_direct: RDMA read, len 0x0, needed credits 0x2
[ 1247.460178] ksmbd: smb_direct: Sending smb (RDMA): smb_len=80
[ 1247.460181] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=80 remaining_data_length=0
[ 1247.460192] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1247.460513] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1247.460521] ksmbd: smb_direct: returning rfc1002 length 144
[ 1247.460524] ksmbd: smb_direct: returning to thread data_read=144 reassembly_data_length=0 first_entry_offset=0
[ 1247.460528] ksmbd: smb_direct: wait_event on more data
[ 1247.460532] ksmbd: RDMA r/w request 0x0: token 0x65d0, length 0xfffc0
[ 1247.460534] ksmbd: RDMA r/w request 0x1: token 0x69d4, length 0x40
[ 1247.460605] ksmbd: smb_direct: RDMA read, len 0x0, needed credits 0x2
[ 1247.460805] ksmbd: smb_direct: Sending smb (RDMA): smb_len=80
[ 1247.460808] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=80 remaining_data_length=0
[ 1247.460819] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1247.460947] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1247.460957] ksmbd: smb_direct: returning rfc1002 length 144
[ 1247.460960] ksmbd: smb_direct: returning to thread data_read=144 reassembly_data_length=0 first_entry_offset=0
[ 1247.460963] ksmbd: smb_direct: wait_event on more data
[ 1247.460966] ksmbd: RDMA r/w request 0x0: token 0x8af5, length 0xfffc0
[ 1247.460967] ksmbd: RDMA r/w request 0x1: token 0x84ef, length 0x40
[ 1247.461066] ksmbd: smb_direct: RDMA read, len 0x0, needed credits 0x2
[ 1247.461380] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1247.461397] ksmbd: smb_direct: returning rfc1002 length 144
[ 1247.461404] ksmbd: smb_direct: returning to thread data_read=144 reassembly_data_length=0 first_entry_offset=0
[ 1247.461410] ksmbd: smb_direct: wait_event on more data
[ 1247.461418] ksmbd: RDMA r/w request 0x0: token 0x67d2, length 0xfffc0
[ 1247.461422] ksmbd: RDMA r/w request 0x1: token 0x66d1, length 0x40
[ 1247.461521] ksmbd: smb_direct: Sending smb (RDMA): smb_len=80
[ 1247.461520] ksmbd: smb_direct: RDMA read, len 0x0, needed credits 0x2
[ 1247.461526] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=80 remaining_data_length=0
[ 1247.461538] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1247.462232] ksmbd: smb_direct: Sending smb (RDMA): smb_len=80
[ 1247.462236] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=80 remaining_data_length=0
[ 1247.462247] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1247.462885] ksmbd: smb_direct: Sending smb (RDMA): smb_len=80
[ 1247.462888] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=80 remaining_data_length=0
[ 1247.462901] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1247.463683] ksmbd: smb_direct: Sending smb (RDMA): smb_len=80
[ 1247.463687] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=80 remaining_data_length=0
[ 1247.463700] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1247.467808] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1247.467820] ksmbd: smb_direct: returning rfc1002 length 144
[ 1247.467824] ksmbd: smb_direct: returning to thread data_read=144 reassembly_data_length=0 first_entry_offset=0
[ 1247.467829] ksmbd: smb_direct: wait_event on more data
[ 1247.467834] ksmbd: RDMA r/w request 0x0: token 0x67d2, length 0xfffc0
[ 1247.467837] ksmbd: RDMA r/w request 0x1: token 0x66d1, length 0x40
[ 1247.468036] ksmbd: smb_direct: RDMA read, len 0x0, needed credits 0x2
[ 1247.468842] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1247.468853] ksmbd: smb_direct: returning rfc1002 length 144
[ 1247.468856] ksmbd: smb_direct: returning to thread data_read=144 reassembly_data_length=0 first_entry_offset=0
[ 1247.468859] ksmbd: smb_direct: wait_event on more data
[ 1247.468865] ksmbd: RDMA r/w request 0x0: token 0x8af5, length 0xfffc0
[ 1247.468867] ksmbd: RDMA r/w request 0x1: token 0x84ef, length 0x40
[ 1247.468978] ksmbd: smb_direct: Sending smb (RDMA): smb_len=80
[ 1247.468979] ksmbd: smb_direct: RDMA read, len 0x0, needed credits 0x2
[ 1247.468982] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=80 remaining_data_length=0
[ 1247.468996] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1247.469784] ksmbd: smb_direct: Sending smb (RDMA): smb_len=80
[ 1247.469787] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=80 remaining_data_length=0
[ 1247.469798] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1247.485588] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1247.485603] ksmbd: smb_direct: returning rfc1002 length 144
[ 1247.485606] ksmbd: smb_direct: returning to thread data_read=144 reassembly_data_length=0 first_entry_offset=0
[ 1247.485610] ksmbd: smb_direct: wait_event on more data
[ 1247.485619] ksmbd: RDMA r/w request 0x0: token 0x67d2, length 0xfffc0
[ 1247.485622] ksmbd: RDMA r/w request 0x1: token 0x66d1, length 0x40
[ 1247.485696] ksmbd: smb_direct: RDMA read, len 0x0, needed credits 0x2
[ 1247.486530] ksmbd: smb_direct: Sending smb (RDMA): smb_len=80
[ 1247.486533] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=80 remaining_data_length=0
[ 1247.486545] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1247.486766] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1247.486773] ksmbd: smb_direct: returning rfc1002 length 104
[ 1247.486776] ksmbd: smb_direct: returning to thread data_read=104 reassembly_data_length=0 first_entry_offset=0
[ 1247.486780] ksmbd: smb_direct: wait_event on more data
[ 1247.488731] ksmbd: smb_direct: Sending smb (RDMA): smb_len=66
[ 1247.488734] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=66 remaining_data_length=0
[ 1247.488744] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1247.488883] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1247.488889] ksmbd: smb_direct: returning rfc1002 length 168
[ 1247.488892] ksmbd: smb_direct: returning to thread data_read=168 reassembly_data_length=0 first_entry_offset=0
[ 1247.488895] ksmbd: smb_direct: wait_event on more data
[ 1247.488903] ksmbd: smb_direct: Sending smb (RDMA): smb_len=80
[ 1247.488905] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=80 remaining_data_length=0
[ 1247.488914] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1247.494434] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1247.494441] ksmbd: smb_direct: returning rfc1002 length 104
[ 1247.494444] ksmbd: smb_direct: returning to thread data_read=104 reassembly_data_length=0 first_entry_offset=0
[ 1247.494446] ksmbd: smb_direct: wait_event on more data
[ 1247.494450] ksmbd: smb_direct: Sending smb (RDMA): smb_len=128
[ 1247.494451] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=128 remaining_data_length=0
[ 1247.494461] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1247.494576] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1247.494582] ksmbd: smb_direct: returning rfc1002 length 104
[ 1247.494584] ksmbd: smb_direct: returning to thread data_read=104 reassembly_data_length=0 first_entry_offset=0
[ 1247.494586] ksmbd: smb_direct: wait_event on more data
[ 1247.495008] ksmbd: smb_direct: Sending smb (RDMA): smb_len=92
[ 1247.495010] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=92 remaining_data_length=0
[ 1247.495020] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1247.495573] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1247.495578] ksmbd: smb_direct: returning rfc1002 length 88
[ 1247.495581] ksmbd: smb_direct: returning to thread data_read=88 reassembly_data_length=0 first_entry_offset=0
[ 1247.495584] ksmbd: smb_direct: wait_event on more data
[ 1247.495590] ksmbd: smb_direct: Sending smb (RDMA): smb_len=124
[ 1247.495592] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=124 remaining_data_length=0
[ 1247.495601] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1248.792865] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1248.792883] ksmbd: smb_direct: returning rfc1002 length 240
[ 1248.792889] ksmbd: smb_direct: returning to thread data_read=240 reassembly_data_length=0 first_entry_offset=0
[ 1248.792893] ksmbd: smb_direct: wait_event on more data
[ 1248.792936] ksmbd: smb_direct: Sending smb (RDMA): smb_len=240
[ 1248.792939] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=240 remaining_data_length=0
[ 1248.792951] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1248.793323] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1248.793331] ksmbd: smb_direct: returning rfc1002 length 104
[ 1248.793334] ksmbd: smb_direct: returning to thread data_read=104 reassembly_data_length=0 first_entry_offset=0
[ 1248.793338] ksmbd: smb_direct: wait_event on more data
[ 1248.793567] ksmbd: smb_direct: Sending smb (RDMA): smb_len=92
[ 1248.793570] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=92 remaining_data_length=0
[ 1248.793581] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1248.794195] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1248.794202] ksmbd: smb_direct: returning rfc1002 length 128
[ 1248.794205] ksmbd: smb_direct: returning to thread data_read=128 reassembly_data_length=0 first_entry_offset=0
[ 1248.794209] ksmbd: smb_direct: wait_event on more data
[ 1248.794213] ksmbd: RDMA r/w request 0x0: token 0x66d1, length 0x1000
[ 1248.794226] ksmbd: smb_direct: RDMA write, len 0x0, needed credits 0x1
[ 1248.794246] ksmbd: smb_direct: Sending smb (RDMA): smb_len=80
[ 1248.794248] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=80 remaining_data_length=0
[ 1248.794259] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1248.794406] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1248.794415] ksmbd: smb_direct: returning rfc1002 length 104
[ 1248.794420] ksmbd: smb_direct: returning to thread data_read=104 reassembly_data_length=0 first_entry_offset=0
[ 1248.794426] ksmbd: smb_direct: wait_event on more data
[ 1248.794442] ksmbd: smb_direct: Sending smb (RDMA): smb_len=66
[ 1248.794445] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=66 remaining_data_length=0
[ 1248.794457] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1248.861684] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1248.861698] ksmbd: smb_direct: returning rfc1002 length 97
[ 1248.861701] ksmbd: smb_direct: returning to thread data_read=97 reassembly_data_length=0 first_entry_offset=0
[ 1248.861705] ksmbd: smb_direct: wait_event on more data
[ 1248.861710] ksmbd: smb_direct: Sending smb (RDMA): smb_len=66
[ 1248.861713] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=66 remaining_data_length=0
[ 1248.861724] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1248.870490] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1248.870502] ksmbd: smb_direct: returning rfc1002 length 176
[ 1248.870505] ksmbd: smb_direct: returning to thread data_read=176 reassembly_data_length=0 first_entry_offset=0
[ 1248.870508] ksmbd: smb_direct: wait_event on more data
[ 1248.870527] ksmbd: smb_direct: Sending smb (RDMA): smb_len=240
[ 1248.870530] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=240 remaining_data_length=0
[ 1248.870541] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1248.870883] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1248.870891] ksmbd: smb_direct: returning rfc1002 length 202
[ 1248.870894] ksmbd: smb_direct: returning to thread data_read=202 reassembly_data_length=0 first_entry_offset=0
[ 1248.870898] ksmbd: smb_direct: wait_event on more data
[ 1248.871118] ksmbd: smb_direct: Sending smb (RDMA): smb_len=624
[ 1248.871121] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=624 remaining_data_length=0
[ 1248.871133] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1248.871391] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1248.871398] ksmbd: smb_direct: returning rfc1002 length 88
[ 1248.871401] ksmbd: smb_direct: returning to thread data_read=88 reassembly_data_length=0 first_entry_offset=0
[ 1248.871405] ksmbd: smb_direct: wait_event on more data
[ 1248.871412] ksmbd: smb_direct: Sending smb (RDMA): smb_len=124
[ 1248.871414] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=124 remaining_data_length=0
[ 1248.871425] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1248.871541] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1248.871548] ksmbd: smb_direct: returning rfc1002 length 104
[ 1248.871551] ksmbd: smb_direct: returning to thread data_read=104 reassembly_data_length=0 first_entry_offset=0
[ 1248.871555] ksmbd: smb_direct: wait_event on more data
[ 1248.871559] ksmbd: smb_direct: Sending smb (RDMA): smb_len=128
[ 1248.871561] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=128 remaining_data_length=0
[ 1248.871572] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1248.876804] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1248.876814] ksmbd: smb_direct: returning rfc1002 length 88
[ 1248.876817] ksmbd: smb_direct: returning to thread data_read=88 reassembly_data_length=0 first_entry_offset=0
[ 1248.876821] ksmbd: smb_direct: wait_event on more data
[ 1248.876842] ksmbd: smb_direct: Sending smb (RDMA): smb_len=124
[ 1248.876845] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=124 remaining_data_length=0
[ 1248.876856] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1248.877970] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1248.877978] ksmbd: smb_direct: returning rfc1002 length 232
[ 1248.877981] ksmbd: smb_direct: returning to thread data_read=232 reassembly_data_length=0 first_entry_offset=0
[ 1248.877985] ksmbd: smb_direct: wait_event on more data
[ 1248.877999] ksmbd: smb_direct: Sending smb (RDMA): smb_len=240
[ 1248.878001] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=240 remaining_data_length=0
[ 1248.878013] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1248.878398] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1248.878412] ksmbd: smb_direct: returning rfc1002 length 96
[ 1248.878418] ksmbd: smb_direct: returning to thread data_read=96 reassembly_data_length=0 first_entry_offset=0
[ 1248.878425] ksmbd: smb_direct: wait_event on more data
[ 1248.878431] ksmbd: smb_direct: Sending smb (RDMA): smb_len=73
[ 1248.878433] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=73 remaining_data_length=0
[ 1248.878446] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1248.878809] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1248.878819] ksmbd: smb_direct: returning rfc1002 length 96
[ 1248.878822] ksmbd: smb_direct: returning to thread data_read=96 reassembly_data_length=0 first_entry_offset=0
[ 1248.878826] ksmbd: smb_direct: wait_event on more data
[ 1248.878830] ksmbd: smb_direct: Sending smb (RDMA): smb_len=73
[ 1248.878832] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=73 remaining_data_length=0
[ 1248.878844] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1248.879187] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1248.879195] ksmbd: smb_direct: returning rfc1002 length 88
[ 1248.879198] ksmbd: smb_direct: returning to thread data_read=88 reassembly_data_length=0 first_entry_offset=0
[ 1248.879202] ksmbd: smb_direct: wait_event on more data
[ 1248.879211] ksmbd: smb_direct: Sending smb (RDMA): smb_len=124
[ 1248.879213] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=124 remaining_data_length=0
[ 1248.879225] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1248.879497] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1248.879504] ksmbd: smb_direct: returning rfc1002 length 88
[ 1248.879507] ksmbd: smb_direct: returning to thread data_read=88 reassembly_data_length=0 first_entry_offset=0
[ 1248.879511] ksmbd: smb_direct: wait_event on more data
[ 1248.879516] ksmbd: smb_direct: Sending smb (RDMA): smb_len=124
[ 1248.879519] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=124 remaining_data_length=0
[ 1248.879530] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1248.880496] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1248.880504] ksmbd: smb_direct: returning rfc1002 length 176
[ 1248.880506] ksmbd: smb_direct: returning to thread data_read=176 reassembly_data_length=0 first_entry_offset=0
[ 1248.880510] ksmbd: smb_direct: wait_event on more data
[ 1248.880539] ksmbd: smb_direct: Sending smb (RDMA): smb_len=240
[ 1248.880541] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=240 remaining_data_length=0
[ 1248.880552] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1250.922174] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1250.922196] ksmbd: smb_direct: returning rfc1002 length 232
[ 1250.922205] ksmbd: smb_direct: returning to thread data_read=232 reassembly_data_length=0 first_entry_offset=0
[ 1250.922211] ksmbd: smb_direct: wait_event on more data
[ 1250.922259] ksmbd: smb_direct: Sending smb (RDMA): smb_len=240
[ 1250.922264] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=240 remaining_data_length=0
[ 1250.922280] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1250.922615] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1250.922629] ksmbd: smb_direct: returning rfc1002 length 88
[ 1250.922632] ksmbd: smb_direct: returning to thread data_read=88 reassembly_data_length=0 first_entry_offset=0
[ 1250.922637] ksmbd: smb_direct: wait_event on more data
[ 1250.922645] ksmbd: smb_direct: Sending smb (RDMA): smb_len=124
[ 1250.922648] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=124 remaining_data_length=0
[ 1250.922660] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1250.923388] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1250.923398] ksmbd: smb_direct: returning rfc1002 length 232
[ 1250.923402] ksmbd: smb_direct: returning to thread data_read=232 reassembly_data_length=0 first_entry_offset=0
[ 1250.923406] ksmbd: smb_direct: wait_event on more data
[ 1250.923419] ksmbd: smb_direct: Sending smb (RDMA): smb_len=240
[ 1250.923421] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=240 remaining_data_length=0
[ 1250.923433] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1250.923648] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1250.923656] ksmbd: smb_direct: returning rfc1002 length 96
[ 1250.923660] ksmbd: smb_direct: returning to thread data_read=96 reassembly_data_length=0 first_entry_offset=0
[ 1250.923664] ksmbd: smb_direct: wait_event on more data
[ 1250.923668] ksmbd: smb_direct: Sending smb (RDMA): smb_len=73
[ 1250.923670] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=73 remaining_data_length=0
[ 1250.923682] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1250.923943] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1250.923952] ksmbd: smb_direct: returning rfc1002 length 248
[ 1250.923955] ksmbd: smb_direct: returning to thread data_read=248 reassembly_data_length=0 first_entry_offset=0
[ 1250.923959] ksmbd: smb_direct: wait_event on more data
[ 1250.924646] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1250.924665] ksmbd: smb_direct: returning rfc1002 length 88
[ 1250.924671] ksmbd: smb_direct: returning to thread data_read=88 reassembly_data_length=0 first_entry_offset=0
[ 1250.924674] ksmbd: smb_direct: Sending smb (RDMA): smb_len=264
[ 1250.924678] ksmbd: smb_direct: wait_event on more data
[ 1250.924681] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=264 remaining_data_length=0
[ 1250.924698] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1250.924700] ksmbd: smb_direct: Sending smb (RDMA): smb_len=124
[ 1250.924708] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=124 remaining_data_length=0
[ 1250.924726] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1250.925068] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1250.925078] ksmbd: smb_direct: returning rfc1002 length 88
[ 1250.925081] ksmbd: smb_direct: returning to thread data_read=88 reassembly_data_length=0 first_entry_offset=0
[ 1250.925086] ksmbd: smb_direct: wait_event on more data
[ 1250.925096] ksmbd: smb_direct: Sending smb (RDMA): smb_len=124
[ 1250.925098] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=124 remaining_data_length=0
[ 1250.925110] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1250.925580] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1250.925588] ksmbd: smb_direct: returning rfc1002 length 248
[ 1250.925591] ksmbd: smb_direct: returning to thread data_read=248 reassembly_data_length=0 first_entry_offset=0
[ 1250.925596] ksmbd: smb_direct: wait_event on more data
[ 1250.925643] ksmbd: smb_direct: Sending smb (RDMA): smb_len=73
[ 1250.925645] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=73 remaining_data_length=0
[ 1250.925657] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1250.926412] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1250.926427] ksmbd: smb_direct: returning rfc1002 length 232
[ 1250.926432] ksmbd: smb_direct: returning to thread data_read=232 reassembly_data_length=0 first_entry_offset=0
[ 1250.926439] ksmbd: smb_direct: wait_event on more data
[ 1250.926456] ksmbd: smb_direct: Sending smb (RDMA): smb_len=240
[ 1250.926461] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=240 remaining_data_length=0
[ 1250.926477] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1250.926874] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1250.926884] ksmbd: smb_direct: returning rfc1002 length 202
[ 1250.926888] ksmbd: smb_direct: returning to thread data_read=202 reassembly_data_length=0 first_entry_offset=0
[ 1250.926893] ksmbd: smb_direct: wait_event on more data
[ 1250.927179] ksmbd: smb_direct: Sending smb (RDMA): smb_len=504
[ 1250.927183] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=504 remaining_data_length=0
[ 1250.927197] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 1250.927471] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 1250.927481] ksmbd: smb_direct: returning rfc1002 length 88
[ 1250.927484] ksmbd: smb_direct: returning to thread data_read=88 reassembly_data_length=0 first_entry_offset=0
[ 1250.927489] ksmbd: smb_direct: wait_event on more data
[ 1250.927494] ksmbd: smb_direct: Sending smb (RDMA): smb_len=124
[ 1250.927496] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=124 remaining_data_length=0
[ 1250.927508] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0

It seems that RDMA traffic is confirmed through Perfmon, and judging by the fact that the CPU load does not significantly increase, it appears that RDMA has been successfully turned on. However, the bandwidth is still in the range of 1-2 Gbps, and during transmission, it drops to 2-300 Mbps. Small-sized files consistently maintain a speed of 2-300 Mbps. How can i Get 100Gbps BandWidth??

[namjaejeon]

Sorry, Could you please explain why rdma does not work before ?

[leehaoun]

Because Mellanox driver is already installed. (23.10-0.5.5.0 ver) When i install the Mellanox driver, an "ofd_kernel" folder is created in the usr/src directory. In that state, if i use the 'mount' command in Linux to share a folder with Windows, rdma_capable shows as false, and data communication occurs via SMB2 instead of RDMA.

so i uninstall the driver, delete the ofd_kernel folder, and reinstall ksmbd, rdma_capable switches to true.

However, there is currently an issue where the bandwidth is not returning to normal.

[namjaejeon]

What is disk type ? SSD(or NVME)? HDD ? You need to consider the bottle neck of disk speed, not network.

[leehaoun]

i use NVME SSD(970-EVO)

[namjaejeon]

@leehaoun It is built-in to both windows client and linux server ? and You need to turn debug print of rdma off when measuring the performance.

It seems that RDMA traffic is confirmed through Perfmon, and judging by the fact that the CPU load does not significantly increase, it appears that RDMA has been successfully turned on. However, the bandwidth is still in the range of 1-2 Gbps, and during transmission, it drops to 2-300 Mbps. Small-sized files consistently maintain a speed of 2-300 Mbps. How can i Get 100Gbps BandWidth??

@hcbwiz Hi Yufan, Is there an any advice for @leehaoun 's question ? I remember You achive optimal performance on ksmbd's RDMA.

[leehaoun]

Yes, the SSD is installed on both sides of the motherboard in NVME format. Additionally, instead of copying images with a mouse, I tested it using CrystalDiskMark. While the write performance did not meet the expected level, the read speed seems to be around 9000MB/s, approximately 72Gbps in bandwidth. Can you suggest a reliable testing method?

[namjaejeon]

@leehaoun Can you share your SSD read/write performance CrystalDiskMark(not throughout ksmbd/windows smb client) ?

[leehaoun]

SSD Local READ/WRITE

SSD RDMA READ/WRITE

Because there is no CrystalMark program on Linux, I tested it with a similar program called kdiskmark under the same conditions. It seems that the speed of reading and writing to the local SSD is much slower.

[namjaejeon]

As I pointed out, local ssd speed will be a bottleneck. To get the maximum performance on your RDMA NIC, the local disk speed must be faster than 100Gbps

[hcbwiz]

@leehaoun 　　How much RAM do you install in your linux server?　 　　KSMBD access local filesystem via page cache (buffer io). 　　 　you can check something first. 　 　step 1. use fio or dd with direct io mode to test local disk. 　step 2. If the Read IO performance is okay by step 1: 　 In your windows client, test READ I/O first (copy file from ksmbd to windows) 　step 3. for write io (copy fie from windows to ksmbd), you should have large system RAM and tinning some kernel VM parameters for buffering IO. 　 Generally, you should have very fast local disk (NVME raid), and make linux kernel flush page cache as fast as possible. 　 otherwise, you would get about only half write throughput of this local ssd.

[leehaoun]

i reinstalled my ununtu os to NVME-SSD. BandWidth Read : 3300 MIB/S , Write : 2200MIB/S

Dragging and dropping a simple image file wasn't enough to achieve sufficient bandwidth. I was getting a speed of around 1GB/s, similar to before. I changed my testing method and used C# code to create multiple streams in parallel with BinaryWrite to write to the ksmbd network drive in order to test the bandwidth. below is my code.

It took 3.2 seconds to simultaneously transmit 300 buffers of size 5000*5000. That means transmitting 7.14 gigabytes in 3.2 seconds, which is about 2.23 gigabytes per second.

Even when I varied the size and number of buffers, the results were almost the same.

Transmitting at 2.23 gigabytes per second translates to approximately 17.85 gigabits per second in one second. This seems to indicate a bandwidth almost similar to the 2248MB/s at which the OS is installed.

I'll retest this when a faster SSD with better bandwidth is available.

It seems like the initial questions have been resolved, so I'll go ahead and close the ISSUE.