hcbwiz commented 2 years ago

Hi, I used Intel X722 to play smb direct, and encountered some issues.

 The max_send_sge of Intel X722 is 3. Then I changed some settings:

          #define SMB_DIRECT_MAX_SEND_SGES  3
          static int smb_direct_max_send_size = 4096;

However, smb_direct_post_send_data() might show the error:

else if (sg_cnt + msg->num_sge > SMB_DIRECT_MAX_SEND_SGES - 1) {
                      pr_err("buffer not fitted into sges\n");

In my environment, sg_cnt = 1, msg->num_sge = 2

 I  have no idea why the size of "sg array" is (SMB_DIRECT_MAX_SEND_SGES - 1)

hclee commented 2 years ago

Hi @hcbwiz

I have no idea why the size of "sg array" is (SMB_DIRECT_MAX_SEND_SGES - 1)

Because smb direct header should be mapped. Could you decrease smb_direct_max_send_size and test it agian?

hclee commented 2 years ago

else if (sg_cnt + msg->num_sge > SMB_DIRECT_MAX_SEND_SGES - 1) {

And "-1" should be removed. I will send the patch to fix it tomorrow.

hcbwiz commented 2 years ago

Hi, Hyunchul Lee

After decreasing smb_direct_max_send_size to 2048, I got other issues: ksmbd: smb_direct: consider lowering send_credit_target = 255, or max_outstanding_rw_ops = 8 ksmbd: smb_direct: Possible CQE overrun, device reporting max_cqe 1048575 max_qp_wr 895

Then I changed the related setting to that: smb_direct_send_credit_target = 128 smb_direct_max_outstanding_rw_ops = 4

However, there were another error on transferring data: ksmbd: smb_direct: failed to init rdma_rw_ctx: -11

Yufan Chen

Hyunchul Lee @.***> 於 2021年8月25日週三下午8:04寫道：

else if (sg_cnt + msg->num_sge > SMB_DIRECT_MAX_SEND_SGES - 1) {

And "-1" should be removed. I will send the patch to fix it tomorrow.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cifsd-team/ksmbd/issues/533#issuecomment-905439938, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIFDJVXGIN5NPILI4TDGH5TT6TL5XANCNFSM5CYPVR2A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

hclee commented 2 years ago

@hcbwiz

Could you apply the following patch and test it again? https://github.com/hclee/ksmbd/commit/f4821163c5d45368c1e453b6490a842e5f807a43

hcbwiz commented 2 years ago

Hi,

I will try it. Thanks

BTW, i forgot mention that i used windows 2016 server as the client

I was testing smb direct over roce on mellanox 100g hba.

It also got the same error when i transfered data to the ksmbd server: ksmbd: smb_direct: failed to init rdma_rw_ctx: -11

I observed the windows client sent "MaxSendSize = 1364" at negotiation.

Is it related the issue?

Hyunchul Lee @.***> 於 2021年8月27日週五 16:32 寫道：

@hcbwiz https://github.com/hcbwiz

Could you apply the following patch and test it again? @.*** https://github.com/hclee/ksmbd/commit/f4821163c5d45368c1e453b6490a842e5f807a43

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cifsd-team/ksmbd/issues/533#issuecomment-907028338, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIFDJVRLM6XIYW6TMAN2SLDT65ESVANCNFSM5CYPVR2A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

hcbwiz commented 2 years ago

Hi, Hyunchul Lee,

@.*** https://github.com/hclee/ksmbd/commit/f4821163c5d45368c1e453b6490a842e5f807a43 fixes the issue on intel x722 and mellanox connectx-5

ksmbd: smb_direct: failed to init rdma_rw_ctx: -11*

However, I encountered another issue on these two hardwares:

On Intel x722: [ 195.918999] i40iw_cqp_ce_handler: opcode = 0x1 maj_err_code = 0xffff min_err_code = 0x800a [ 195.922431] RDMA CM event. cm_id=00000000ea1400cb event=disconnected (10) [ 195.922433] ksmbd: smb_direct: read/write error. opcode = 2, status = WR flushed(5) [ 195.924203] ksmbd: smb_direct: disconnected [ 195.928358] ksmbd: Failed to send message: -107

On Mellanox ConnectX-5: [ 489.694677] infiniband mlx5_0: dump_cqe:272:(pid 6): dump error cqe [ 489.695745] 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 489.696766] 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 489.697810] 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 489.698859] 00000030: 00 00 00 00 00 00 88 13 10 00 00 49 00 08 e2 d2 [ 489.699914] ksmbd: smb_direct: read/write error. opcode = 0, status = remote access error(10) [ 489.700396] RDMA CM event. cm_id=00000000e5abce98 event=disconnected (10) [ 489.701834] ksmbd: smb_direct: read/write error. opcode = 0, status = WR flushed(5) [ 489.703394] ksmbd: smb_direct: disconnected

Then, I tried to lower these values: smb_direct_max_fragmented_recv_size = 512 1024; smb_direct_max_read_write_size = 512 1024;

Now, everything works well so far with your patch.. Thanks.

Best Regards, Yufan Chen

Note: What the settings I changed,

For Intel X722:

define SMB_DIRECT_MAX_SEND_SGES 3

 static int smb_direct_max_send_size = 4096;
 static int smb_direct_max_fragmented_recv_size = 512 * 1024;
 static int smb_direct_max_read_write_size = 512 * 1024;

For Mellanox ConnectX-5

define SMB_DIRECT_PORT 445

 static int smb_direct_max_fragmented_recv_size = 512 * 1024;
 static int smb_direct_max_read_write_size = 512 * 1024;

YuFan Chen @.***> 於 2021年8月28日週六下午12:32寫道：

Hi,

I will try it. Thanks

BTW, i forgot mention that i used windows 2016 server as the client

I was testing smb direct over roce on mellanox 100g hba.

It also got the same error when i transfered data to the ksmbd server: ksmbd: smb_direct: failed to init rdma_rw_ctx: -11

I observed the windows client sent "MaxSendSize = 1364" at negotiation.

Is it related the issue?

Hyunchul Lee @.***> 於 2021年8月27日週五 16:32 寫道：

@hcbwiz https://github.com/hcbwiz

Could you apply the following patch and test it again? @.*** https://github.com/hclee/ksmbd/commit/f4821163c5d45368c1e453b6490a842e5f807a43

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cifsd-team/ksmbd/issues/533#issuecomment-907028338, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIFDJVRLM6XIYW6TMAN2SLDT65ESVANCNFSM5CYPVR2A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

hclee commented 2 years ago

I will check it at tonight because I am on vacation.

hclee commented 2 years ago

@hcbwiz,

[ 489.699914] ksmbd: smb_direct: read/write error. opcode = 0, status = remote access error(10)

We try to connect with Windows client and know this issue, and are fixing it. As you did, decreasing smb_direct_max_read_write_size is helpful.

What the settings I changed,

We will refine the auto configuration and enable users to change these parameters without compiling.

Thank you for your test! We will let you know when we are done.

namjaejeon commented 2 years ago

@hcbwiz Is it true that ksmbd with iWARP NICs should open 5445 port instead of 445 ?

hcbwiz commented 2 years ago

@namjaejeon

According to Implementing_SMBDirect_for_CIFS windows uses port 445 for RoCE and port 5445 for iwarp.

The SMBD client implementation in linux kernel uses port 5445 first. If port 5445 doesn't work, it will try port 445 again.

My windows client has ROCEv2 NIC only. I didn't test the windows client with an iWARP NIC

namjaejeon commented 2 years ago

What the settings I changed, For Intel X722:

define SMB_DIRECT_MAX_SEND_SGES 3
static int smb_direct_max_send_size = 4096;
static int smb_direct_max_fragmented_recv_size = 512 * 1024;
static int smb_direct_max_read_write_size = 512 * 1024;

If so, What kind of client is the result of testing ?

hcbwiz commented 2 years ago

What the settings I changed, For Intel X722:

define SMB_DIRECT_MAX_SEND_SGES 3

static int smb_direct_max_send_size = 4096; static int smb_direct_max_fragmented_recv_size = 512 1024; static int smb_direct_max_read_write_size = 512 1024;

If so, What kind of client is the result of testing ?

This is another linux client with onboard Intel X722

namjaejeon commented 2 years ago

@hcbwiz Yufan, I have a few question regarding to max_read_write setting on your device.

You told us that ksmbd RDMA work with the below setting on your Mellanox ConnectX-5.
```
 #define SMB_DIRECT_PORT 445
 static int smb_direct_max_fragmented_recv_size = 512 * 1024;
 static int smb_direct_max_read_write_size = 512 * 1024;
```
Is it fixed on the latest #master ? or still need to change setting ?
When you setting the below, does ksmbd rdma work on Intel X722 ?

diff --git a/transport_rdma.c b/transport_rdma.c
index b71c373..61b35e9 100644
--- a/transport_rdma.c
+++ b/transport_rdma.c
@@ -80,7 +80,7 @@ static int smb_direct_max_fragmented_recv_size = 1024 * 1024;
 /*  The maximum single-message size which can be received */
 static int smb_direct_max_receive_size = 8192;

-static int smb_direct_max_read_write_size = 1048512;
+static int smb_direct_max_read_write_size = 512 * 1024 - 64;

 static int smb_direct_max_outstanding_rw_ops = 8;

Can you show me max_fast_reg_page_list_len value on Intel X722 ?


@@ -2077,6 +2078,7 @@ static int smb_direct_ib_client_add(struct ib_device *ib_dev)
    if (ib_dev->node_type != RDMA_NODE_IB_CA)
            smb_direct_port = SMB_DIRECT_PORT_IWARP;

pr_err("attrs.max_fast_reg_page_list_len : %d\n", ib_dev->attrs.max_fast_reg_page_list_len);

hcbwiz commented 2 years ago

@hcbwiz Yufan, I have a few question regarding to max_read_write setting on your device.
1. You told us that ksmbd RDMA work with the below setting on your Mellanox ConnectX-5.
     #define SMB_DIRECT_PORT 445
     static int smb_direct_max_fragmented_recv_size = 512 * 1024;
     static int smb_direct_max_read_write_size = 512 * 1024;
Is it fixed on the latest #master ? or still need to change setting ?

Yes, the latest master branch can works without changing these settings for Mellanox ConnectX-5.

2. When you setting the below, does ksmbd rdma work on Intel X722 ?

diff --git a/transport_rdma.c b/transport_rdma.c
index b71c373..61b35e9 100644
--- a/transport_rdma.c
+++ b/transport_rdma.c
@@ -80,7 +80,7 @@ static int smb_direct_max_fragmented_recv_size = 1024 * 1024;
 /*  The maximum single-message size which can be received */
 static int smb_direct_max_receive_size = 8192;

-static int smb_direct_max_read_write_size = 1048512;
+static int smb_direct_max_read_write_size = 512 * 1024 - 64;

 static int smb_direct_max_outstanding_rw_ops = 8;

3. Can you show me max_fast_reg_page_list_len value on Intel X722 ?

@@ -2077,6 +2078,7 @@ static int smb_direct_ib_client_add(struct ib_device *ib_dev)
        if (ib_dev->node_type != RDMA_NODE_IB_CA)
                smb_direct_port = SMB_DIRECT_PORT_IWARP;

+       pr_err("attrs.max_fast_reg_page_list_len : %d\n", ib_dev->attrs.max_fast_reg_page_list_len);

I will rebuild the environment and let you know the result.

namjaejeon commented 2 years ago

@hcbwiz Okay, Thank you for your help!

hcbwiz commented 2 years ago

2. When you setting the below, does ksmbd rdma work on Intel X722 ?

diff --git a/transport_rdma.c b/transport_rdma.c
index b71c373..61b35e9 100644
--- a/transport_rdma.c
+++ b/transport_rdma.c
@@ -80,7 +80,7 @@ static int smb_direct_max_fragmented_recv_size = 1024 * 1024;
 /*  The maximum single-message size which can be received */
 static int smb_direct_max_receive_size = 8192;

-static int smb_direct_max_read_write_size = 1048512;
+static int smb_direct_max_read_write_size = 512 * 1024 - 64;

 static int smb_direct_max_outstanding_rw_ops = 8;

The max_send_sge of Intel X722 is 3. Thus, I change these settings:

#define SMB_DIRECT_MAX_SEND_SGES               3
static int smb_direct_max_send_size = 4096;
static int smb_direct_max_read_write_size = 512 * 1024 - 64;

3. Can you show me max_fast_reg_page_list_len value on Intel X722 ?

@@ -2077,6 +2078,7 @@ static int smb_direct_ib_client_add(struct ib_device *ib_dev)
        if (ib_dev->node_type != RDMA_NODE_IB_CA)
                smb_direct_port = SMB_DIRECT_PORT_IWARP;

+       pr_err("attrs.max_fast_reg_page_list_len : %d\n", ib_dev->attrs.max_fast_reg_page_list_len);

It showes that:

ksmbd: smb_direct: attrs.max_fast_reg_page_list_len : 512

However, my linux cifs client cannot mount the folder via RDMA. It is okay with TCP.

The log of CIFS client:

[ 5861.251190] CIFS: Attempting to mount \\172.5.5.2\mnt
[ 5861.256164] CIFS: VFS: RDMA transport established
[ 5863.130566] CIFS: VFS: smbd_post_send:815 ib_post_send failed rc=-22
[ 5864.333754] CIFS: VFS: \\172.5.5.2 Error -11 sending data on socket to server
[ 5864.333934] CIFS: VFS: \\172.5.5.2 Error -11 sending data on socket to server
[ 5864.334078] CIFS: VFS: \\172.5.5.2 Error -11 sending data on socket to server
[ 5864.334096] CIFS: VFS: smbd_recv_buf:1889 disconnected
[ 5864.334221] CIFS: VFS: \\172.5.5.2 Error -11 sending data on socket to server
[ 5864.370388] CIFS: VFS: RDMA transport re-established
[ 5865.046819] CIFS: VFS: smbd_post_send:815 ib_post_send failed rc=-22
[ 5865.356396] CIFS: VFS: \\172.5.5.2 Error -11 sending data on socket to server
[ 5865.356587] CIFS: VFS: \\172.5.5.2 Error -11 sending data on socket to server
[ 5865.356745] CIFS: VFS: \\172.5.5.2 Error -11 sending data on socket to server
[ 5865.356870] CIFS: VFS: smbd_recv_buf:1889 disconnected
[ 5865.356889] CIFS: VFS: \\172.5.5.2 cifs_put_smb_ses: Session Logoff failure rc=-11
[ 5865.357219] CIFS: VFS: cifs_mount failed w/return code = -11
[ 5865.396708] CIFS: VFS: smbd_create_id:579 rdma_resolve_addr timeout rc: -512
[ 5865.396872] CIFS: VFS: smbd_create_id:579 rdma_resolve_addr timeout rc: -512

the log of KSMBD:

[ 4819.244440] ksmbd: smb_direct: attrs.max_fast_reg_page_list_len : 512
[ 4840.233200] ksmbd: smb_direct: Received connection request. cm_id=000000001b4faa66
[ 4840.233204] ksmbd: smb_direct: Waiting for SMB_DIRECT negotiate request
[ 4840.233211] ksmbd: smb_direct: RDMA CM event. cm_id=000000001b4faa66 event=established (9)
[ 4840.233766] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 4840.233836] ksmbd: smb_direct: MinVersion: 256, MaxVersion: 256, CreditRequested: 255, MaxSendSize: 1364, MaxRecvSize: 8192, MaxFragmentedSize: 1048576
[ 4840.233872] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 4840.233888] ksmbd: smb_direct: wait_event on more data
[ 4840.234066] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 4840.236300] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 4840.236366] ksmbd: smb_direct: returning rfc1002 length 224
[ 4840.236373] ksmbd: RFC1002 header 224 bytes
[ 4840.236379] ksmbd: smb_direct: returning to thread data_read=224 reassembly_data_length=0 first_entry_offset=0
[ 4840.236396] ksmbd: no length check for command
[ 4840.236398] ksmbd: SMB2 data length 0 offset 0
[ 4840.236427] ksmbd: SMB2 len 100
[ 4840.236430] ksmbd: cli req padded more than expected. Length 224 not 100 for cmd:0 mid:0
[ 4840.236436] ksmbd: client requested dialect 0x311
[ 4840.236438] ksmbd: selected SMB 3.1.1 dialect
[ 4840.236441] ksmbd: conn->dialect 0x311
[ 4840.236444] ksmbd: Received negotiate request
[ 4840.236446] ksmbd: decoding 4 negotiate contexts
[ 4840.236449] ksmbd: deassemble SMB2_PREAUTH_INTEGRITY_CAPABILITIES context
[ 4840.236451] ksmbd: deassemble SMB2_ENCRYPTION_CAPABILITIES context
[ 4840.236453] ksmbd: deassemble SMB2_NETNAME_NEGOTIATE_CONTEXT_ID context
[ 4840.236454] ksmbd: deassemble SMB2_POSIX_EXTENSIONS_AVAILABLE context
[ 4840.236483] ksmbd: assemble SMB2_PREAUTH_INTEGRITY_CAPABILITIES context
[ 4840.236489] ksmbd: assemble SMB2_POSIX_EXTENSIONS_AVAILABLE context
[ 4840.236492] ksmbd: negotiate context offset 208, count 2
[ 4840.236495] ksmbd: credits: requested[10] granted[1] total_granted[1]
[ 4840.236506] ksmbd: smb_direct: Sending smb (RDMA): smb_len=280
[ 4840.236511] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=280 remaining_data_length=0
[ 4840.236524] ksmbd: smb_direct: wait_event on more data
[ 4840.236561] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 4840.236708] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 4840.236721] ksmbd: smb_direct: returning rfc1002 length 120
[ 4840.236725] ksmbd: RFC1002 header 120 bytes
[ 4840.236729] ksmbd: smb_direct: returning to thread data_read=120 reassembly_data_length=0 first_entry_offset=0
[ 4840.236743] ksmbd: SMB2 data length 32 offset 88
[ 4840.236747] ksmbd: SMB2 len 120
[ 4840.236750] ksmbd: Received request for session setup
[ 4840.236762] ksmbd: negotiate phase
[ 4840.236769] ksmbd: NTLMSSP SecurityBufferLength 108
[ 4840.236772] ksmbd: credits: requested[130] granted[130] total_granted[130]
[ 4840.236782] ksmbd: smb_direct: Sending smb (RDMA): smb_len=180
[ 4840.236786] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=180 remaining_data_length=0
[ 4840.236797] ksmbd: smb_direct: wait_event on more data
[ 4840.236813] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 4840.236872] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 4840.236882] ksmbd: smb_direct: returning rfc1002 length 258
[ 4840.236886] ksmbd: RFC1002 header 258 bytes
[ 4840.236889] ksmbd: smb_direct: returning to thread data_read=258 reassembly_data_length=0 first_entry_offset=0
[ 4840.236895] ksmbd: SMB2 data length 170 offset 88
[ 4840.236898] ksmbd: SMB2 len 258
[ 4840.236900] ksmbd: Received request for session setup
[ 4840.236908] ksmbd: authenticate phase
[ 4840.236912] ksmbd: session setup request for user test
[ 4840.236929] ksmbd: smb_direct: wait_event on more data
[ 4840.237301] ksmbd: decode_ntlmssp_authenticate_blob dname
[ 4840.237345] ksmbd: dumping generated AES signing keys
[ 4840.237347] ksmbd: Session Id    1
[ 4840.237349] ksmbd: Session Key   8c 6e b3 12 8b 54 66 3d ee db 06 fd 87 04 8a e4
[ 4840.237353] ksmbd: Signing Key   22 34 2c 47 50 79 93 df d3 72 5d 21 ce 8e b1 f5
[ 4840.237357] ksmbd: credits: requested[130] granted[130] total_granted[259]
[ 4840.237370] ksmbd: smb_direct: Sending smb (RDMA): smb_len=73
[ 4840.237373] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=73 remaining_data_length=0
[ 4840.237413] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 4840.237578] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 4840.237589] ksmbd: smb_direct: returning rfc1002 length 106
[ 4840.237592] ksmbd: RFC1002 header 106 bytes
[ 4840.237595] ksmbd: smb_direct: returning to thread data_read=106 reassembly_data_length=0 first_entry_offset=0
[ 4840.237602] ksmbd: skip to check tree connect request
[ 4840.237604] ksmbd: SMB2 data length 32 offset 72
[ 4840.237607] ksmbd: SMB2 len 104
[ 4840.237608] ksmbd: cli req padded more than expected. Length 106 not 104 for cmd:3 mid:3
[ 4840.237618] ksmbd: tree connect request for tree IPC$ treename \\172.5.5.2\IPC$
[ 4840.237630] ksmbd: smb_direct: wait_event on more data
[ 4840.237763] ksmbd: IPC share path request
[ 4840.237766] ksmbd: credits: requested[64] granted[64] total_granted[322]
[ 4840.237771] ksmbd: smb_direct: Sending smb (RDMA): smb_len=80
[ 4840.237774] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=80 remaining_data_length=0
[ 4840.237799] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 4840.237837] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 4840.237846] ksmbd: smb_direct: returning rfc1002 length 104
[ 4840.237850] ksmbd: RFC1002 header 104 bytes
[ 4840.237853] ksmbd: smb_direct: returning to thread data_read=104 reassembly_data_length=0 first_entry_offset=0
[ 4840.237860] ksmbd: skip to check tree connect request
[ 4840.237861] ksmbd: SMB2 data length 30 offset 72
[ 4840.237864] ksmbd: SMB2 len 102
[ 4840.237869] ksmbd: tree connect request for tree mnt treename \\172.5.5.2\mnt
[ 4840.237879] ksmbd: smb_direct: wait_event on more data
[ 4840.237991] ksmbd: credits: requested[64] granted[64] total_granted[385]
[ 4840.237996] ksmbd: smb_direct: Sending smb (RDMA): smb_len=80
[ 4840.237999] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=80 remaining_data_length=0
[ 4840.238022] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 4843.313658] ksmbd: smb_direct: RDMA CM event. cm_id=000000001b4faa66 event=disconnected (10)
[ 4843.313726] ksmbd: smb_direct: disconnected
[ 4843.313871] ksmbd: smb_direct: Disconnecting cm_id=000000001b4faa66
[ 4843.313877] ksmbd: smb_direct: wait for all send posted to IB to finish
[ 4843.330030] ksmbd: smb_direct: drain the reassembly queue
[ 4843.347393] ksmbd: smb_direct: Received connection request. cm_id=00000000f3bfd62c
[ 4843.347398] ksmbd: smb_direct: Waiting for SMB_DIRECT negotiate request
[ 4843.347402] ksmbd: smb_direct: RDMA CM event. cm_id=00000000f3bfd62c event=established (9)
[ 4843.347922] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 4843.347997] ksmbd: smb_direct: MinVersion: 256, MaxVersion: 256, CreditRequested: 255, MaxSendSize: 1364, MaxRecvSize: 8192, MaxFragmentedSize: 1048576
[ 4843.348032] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 4843.348046] ksmbd: smb_direct: wait_event on more data
[ 4843.348247] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 4843.350356] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 4843.350423] ksmbd: smb_direct: returning rfc1002 length 224
[ 4843.350430] ksmbd: RFC1002 header 224 bytes
[ 4843.350435] ksmbd: smb_direct: returning to thread data_read=224 reassembly_data_length=0 first_entry_offset=0
[ 4843.350450] ksmbd: no length check for command
[ 4843.350452] ksmbd: SMB2 data length 0 offset 0
[ 4843.350455] ksmbd: SMB2 len 100
[ 4843.350457] ksmbd: cli req padded more than expected. Length 224 not 100 for cmd:0 mid:0
[ 4843.350462] ksmbd: client requested dialect 0x311
[ 4843.350465] ksmbd: selected SMB 3.1.1 dialect
[ 4843.350467] ksmbd: conn->dialect 0x311
[ 4843.350469] ksmbd: Received negotiate request
[ 4843.350472] ksmbd: decoding 4 negotiate contexts
[ 4843.350474] ksmbd: deassemble SMB2_PREAUTH_INTEGRITY_CAPABILITIES context
[ 4843.350476] ksmbd: deassemble SMB2_ENCRYPTION_CAPABILITIES context
[ 4843.350478] ksmbd: deassemble SMB2_NETNAME_NEGOTIATE_CONTEXT_ID context
[ 4843.350480] ksmbd: deassemble SMB2_POSIX_EXTENSIONS_AVAILABLE context
[ 4843.350490] ksmbd: assemble SMB2_PREAUTH_INTEGRITY_CAPABILITIES context
[ 4843.350494] ksmbd: assemble SMB2_POSIX_EXTENSIONS_AVAILABLE context
[ 4843.350496] ksmbd: negotiate context offset 208, count 2
[ 4843.350500] ksmbd: credits: requested[10] granted[1] total_granted[1]
[ 4843.350510] ksmbd: smb_direct: Sending smb (RDMA): smb_len=280
[ 4843.350515] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=280 remaining_data_length=0
[ 4843.350527] ksmbd: smb_direct: wait_event on more data
[ 4843.350543] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 4843.350648] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 4843.350659] ksmbd: smb_direct: returning rfc1002 length 120
[ 4843.350663] ksmbd: RFC1002 header 120 bytes
[ 4843.350666] ksmbd: smb_direct: returning to thread data_read=120 reassembly_data_length=0 first_entry_offset=0
[ 4843.350673] ksmbd: SMB2 data length 32 offset 88
[ 4843.350676] ksmbd: SMB2 len 120
[ 4843.350679] ksmbd: Received request for session setup
[ 4843.350689] ksmbd: negotiate phase
[ 4843.350694] ksmbd: NTLMSSP SecurityBufferLength 108
[ 4843.350696] ksmbd: credits: requested[130] granted[130] total_granted[130]
[ 4843.350706] ksmbd: smb_direct: Sending smb (RDMA): smb_len=180
[ 4843.350709] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=180 remaining_data_length=0
[ 4843.350717] ksmbd: smb_direct: wait_event on more data
[ 4843.350734] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 4843.350795] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 4843.350804] ksmbd: smb_direct: returning rfc1002 length 258
[ 4843.350808] ksmbd: RFC1002 header 258 bytes
[ 4843.350811] ksmbd: smb_direct: returning to thread data_read=258 reassembly_data_length=0 first_entry_offset=0
[ 4843.350818] ksmbd: SMB2 data length 170 offset 88
[ 4843.350820] ksmbd: SMB2 len 258
[ 4843.350822] ksmbd: Received request for session setup
[ 4843.350830] ksmbd: authenticate phase
[ 4843.350833] ksmbd: session setup request for user test
[ 4843.350850] ksmbd: smb_direct: wait_event on more data
[ 4843.351053] ksmbd: decode_ntlmssp_authenticate_blob dname
[ 4843.351079] ksmbd: dumping generated AES signing keys
[ 4843.351080] ksmbd: Session Id    1
[ 4843.351082] ksmbd: Session Key   fc 9c c0 be 26 aa 98 06 54 3d f9 9d f0 3a 12 95
[ 4843.351086] ksmbd: Signing Key   69 37 5c 87 1a 2c 9d 53 4b b1 66 dc 6e 51 62 bb
[ 4843.351089] ksmbd: credits: requested[130] granted[130] total_granted[259]
[ 4843.351097] ksmbd: smb_direct: Sending smb (RDMA): smb_len=73
[ 4843.351100] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=73 remaining_data_length=0
[ 4843.351124] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 4843.351195] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 4843.351205] ksmbd: smb_direct: returning rfc1002 length 106
[ 4843.351209] ksmbd: RFC1002 header 106 bytes
[ 4843.351211] ksmbd: smb_direct: returning to thread data_read=106 reassembly_data_length=0 first_entry_offset=0
[ 4843.351219] ksmbd: skip to check tree connect request
[ 4843.351220] ksmbd: SMB2 data length 32 offset 72
[ 4843.351223] ksmbd: SMB2 len 104
[ 4843.351225] ksmbd: cli req padded more than expected. Length 106 not 104 for cmd:3 mid:3
[ 4843.351233] ksmbd: tree connect request for tree IPC$ treename \\172.5.5.2\IPC$
[ 4843.351244] ksmbd: smb_direct: wait_event on more data
[ 4843.351585] ksmbd: IPC share path request
[ 4843.351587] ksmbd: credits: requested[64] granted[64] total_granted[322]
[ 4843.351593] ksmbd: smb_direct: Sending smb (RDMA): smb_len=80
[ 4843.351596] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=80 remaining_data_length=0
[ 4843.351622] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 4843.351706] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 4843.351716] ksmbd: smb_direct: returning rfc1002 length 104
[ 4843.351720] ksmbd: RFC1002 header 104 bytes
[ 4843.351723] ksmbd: smb_direct: returning to thread data_read=104 reassembly_data_length=0 first_entry_offset=0
[ 4843.351729] ksmbd: skip to check tree connect request
[ 4843.351731] ksmbd: SMB2 data length 30 offset 72
[ 4843.351733] ksmbd: SMB2 len 102
[ 4843.351739] ksmbd: tree connect request for tree mnt treename \\172.5.5.2\mnt
[ 4843.351749] ksmbd: smb_direct: wait_event on more data
[ 4843.351749] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 4843.351762] ksmbd: smb_direct: returning rfc1002 length 152
[ 4843.351766] ksmbd: RFC1002 header 152 bytes
[ 4843.351768] ksmbd: smb_direct: returning to thread data_read=152 reassembly_data_length=0 first_entry_offset=0
[ 4843.351784] ksmbd: smb_direct: wait_event on more data
[ 4843.352679] ksmbd: SMB2 data length 32 offset 120
[ 4843.352682] ksmbd: SMB2 len 152
[ 4843.352685] ksmbd: credits: requested[10] granted[10] total_granted[331]
[ 4843.352688] ksmbd: smb_direct: Sending smb (RDMA): smb_len=73
[ 4843.352691] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=73 remaining_data_length=0
[ 4843.352701] ksmbd: credits: requested[64] granted[64] total_granted[394]
[ 4843.352715] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 4843.352742] ksmbd: smb_direct: Sending smb (RDMA): smb_len=80
[ 4843.352745] ksmbd: smb_direct: credits_requested=255 credits_granted=0 data_offset=24 data_length=80 remaining_data_length=0
[ 4843.352768] ksmbd: smb_direct: Send completed. status='success (0)', opcode=0
[ 4843.352795] ksmbd: smb_direct: Recv completed. status='success (0)', opcode=128
[ 4844.336424] ksmbd: smb_direct: RDMA CM event. cm_id=00000000f3bfd62c event=disconnected (10)
[ 4844.336493] ksmbd: smb_direct: disconnected
[ 4844.336634] ksmbd: smb_direct: Disconnecting cm_id=00000000f3bfd62c
[ 4844.336640] ksmbd: smb_direct: wait for all send posted to IB to finish
[ 4844.350023] ksmbd: smb_direct: drain the reassembly queue

namjaejeon commented 2 years ago

@hcbwiz Thanks! Can you show max_send_sge and max_recv_sge from device info ?

        pr_err("max_send_sge : %d\n", ib_dev->attrs.max_send_sge);
        pr_err("max_recv_sge : %d\n", ib_dev->attrs.max_recv_sge);

Thanks!

hcbwiz commented 2 years ago

@hcbwiz Thanks! Can you show max_send_sge and max_recv_sge from device info ?

        pr_err("max_send_sge : %d\n", ib_dev->attrs.max_send_sge);
        pr_err("max_recv_sge : %d\n", ib_dev->attrs.max_recv_sge);

Thanks!

[ 8729.363786] ksmbd: smb_direct: attrs.max_fast_reg_page_list_len : 512
[ 8729.363932] ksmbd: smb_direct: max_send_sge : 3
[ 8729.364064] ksmbd: smb_direct: max_recv_sge : 3

namjaejeon commented 2 years ago

@hcbwiz Okay, Thanks!

namjaejeon commented 2 years ago

@hcbwiz I have understood RDMA doesn't work with linux and windows client although the below change is applied. Right ? you told me it can work with linux cifs client before ?

#define SMB_DIRECT_MAX_SEND_SGES               3
static int smb_direct_max_send_size = 4096;
static int smb_direct_max_read_write_size = 512 * 1024 - 64;

hcbwiz commented 2 years ago

@hcbwiz I have understood RDMA doesn't work with linux and windows client although the below change is applied. Right ? you told me it can work with linux cifs client before ?
#define SMB_DIRECT_MAX_SEND_SGES               3
static int smb_direct_max_send_size = 4096;
static int smb_direct_max_read_write_size = 512 * 1024 - 64;

The kernel in the linux cifs client has been updated to 5.15 The original kernel version (sorry, I forget what the version is..) with the changed in the ksmbd:

#define SMB_DIRECT_MAX_SEND_SGES 3
static int smb_direct_max_send_size = 4096;
static int smb_direct_max_fragmented_recv_size = 512 * 1024;
static int smb_direct_max_read_write_size = 512 * 1024;

hcbwiz commented 2 years ago

In the linux CIFS client:

[ 5863.130566] CIFS: VFS: smbd_post_send:815 ib_post_send failed rc=-22

The error code is -EINVAL.

Then I check the code and print out some debug information.

smbd_post_send_sgl() in fs/cifs/smbdirect.c:

/* Fill in the packet data payload */
num_sgs = sgl ? sg_nents(sgl) : 0;

The sgl is NULL.. hmm... I'm digging what the problem is.

hcbwiz commented 2 years ago

@namjaejeon I had got something wrong before.

I checked the code in linux CIFS client:

  smbd_post_send_sgl() uses  1 segment for the header

  SMB2_open_init() generates a request using 3 segments at at least.

It needs 4 segments totally. However. Intel X722 only supports max_send_sge = 3.

namjaejeon commented 2 years ago

@hcbwiz Okay, Thanks for your check!

cifsd-team / ksmbd

SMB Direct (iwarp) over Intel X722 #533

Best Regards, Yufan Chen

define SMB_DIRECT_MAX_SEND_SGES 3

define SMB_DIRECT_PORT 445

define SMB_DIRECT_MAX_SEND_SGES 3

define SMB_DIRECT_MAX_SEND_SGES 3