Open darkwrat opened 11 months ago
Okay. windows client seems send wrong signature. I think that you turn signing feature on windows client or ksmbd.
Can you show me your smb.conf file ? and can you answer about the following question ? You told me problem happen with a couple of windows clients.
There are only two windows clients, and the issue was observed on both of them at different times. The share is mapped as a network drive, the drive stops working after some random time with the mentioned error, but can be restored by unmapping and remapping.
Windows 10 21H2 build 19044.3570 on both machines.
I don't have a separate smb.conf file, but ksmbd.conf is provided below. ksmbd-tools from 52a3955 built and make install'ed on the server machine. I am able to apply test patches to the tools and kernel if needed.
# cat /usr/local/etc/ksmbd/ksmbd.conf
; see ksmbd.conf(5) for details
[part0]
; share parameters
force group = ftp
force user = ftp
path = /mnt/part0
read only = no
Can you give me wireshark dump(or tcpdump) that captured patckets on problem situation to me ?
Can you give me wireshark dump(or tcpdump) that captured patckets on problem situation to me ?
Sure. Will capture once I can observe the problem again. Might take some time, can't pinpoint a way to reproduce it yet.
Really thanks for your help! I am trying to reproduce it, But it is not easy to reproduce it. If you find some pattern for this issue, It will be really helpful to fix this issue.
Please tell me what tcpdump filters should I apply, or if you want the unfiltered dump.
Stranged.. I unzip it using gunzip. But I can not open it using wireshark...
Please check the checksum of the file. I've double checked, this should work.
m.galaganov@xps ~ % sha256sum ksmbd_20231020.pcap.gz
0fe78da621841eb28e83048f3923e5141f9183a6f7f48d29e52f104f3cffb5e1 ksmbd_20231020.pcap.gz
m.galaganov@xps ~ % gunzip ksmbd_20231020.pcap.gz
m.galaganov@xps ~ % tcpdump -nn -r ksmbd_20231020.pcap |head -n5
reading from file ksmbd_20231020.pcap, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144
Warning: interface names might be incorrect
13:14:58.054649 vethIjhHJ5 In IP 192.168.1.234.49698 > 192.168.1.150.445: Flags [P.], seq 210913679:210913797, ack 508876861, win 65534, length 118
13:14:58.054649 ? In IP 192.168.1.234.49698 > 192.168.1.150.445: Flags [P.], seq 0:118, ack 1, win 65534, length 118
13:14:58.054702 ? Out IP 192.168.1.150.445 > 192.168.1.234.49698: Flags [.], ack 118, win 5328, length 0
13:14:58.054712 vethIjhHJ5 Out IP 192.168.1.150.445 > 192.168.1.234.49698: Flags [.], ack 118, win 5328, length 0
13:14:58.054828 ? Out IP 192.168.1.150.445 > 192.168.1.234.49698: Flags [P.], seq 1:78, ack 118, win 5328, length 77
tcpdump: Unable to write output: Broken pipe
m.galaganov@xps ~ % tcpdump --version
tcpdump version 4.99.4
libpcap version 1.10.4 (with TPACKET_V3)
OpenSSL 3.0.9 30 May 2023
m.galaganov@xps ~ % wireshark --version |head -n1
Wireshark 4.0.8 (Git commit 81696bb74857).
I'll leave tcp stream hexdump below, just in case.
00000000 00 00 00 72 fe 53 4d 42 40 00 01 00 00 00 00 00 ...r.SMB @.......
00000010 03 00 01 00 18 00 00 00 00 00 00 00 f4 49 02 00 ........ .....I..
00000020 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
00000030 00 00 00 00 7d 8f 20 8a 91 01 e4 24 0c 4a f5 69 ....}. . ...$.J.i
00000040 da 24 a1 14 09 00 00 00 48 00 2a 00 5c 00 5c 00 .$...... H.*.\.\.
00000050 31 00 39 00 32 00 2e 00 31 00 36 00 38 00 2e 00 1.9.2... 1.6.8...
00000060 31 00 2e 00 31 00 35 00 30 00 5c 00 70 00 61 00 1...1.5. 0.\.p.a.
00000070 72 00 74 00 30 00 r.t.0.
00000000 00 00 00 49 fe 53 4d 42 40 00 01 00 22 00 00 c0 ...I.SMB @..."...
00000010 03 00 01 00 09 00 00 00 00 00 00 00 f4 49 02 00 ........ .....I..
00000020 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
00000030 00 00 00 00 7c 15 bd 59 22 53 2c 29 c1 7f fe ab ....|..Y "S,)....
00000040 2f af 0a 8c 09 00 00 00 00 00 00 00 00 /....... .....
00000076 00 00 00 72 fe 53 4d 42 40 00 01 00 00 00 00 00 ...r.SMB @.......
00000086 03 00 01 00 18 00 00 00 00 00 00 00 f5 49 02 00 ........ .....I..
00000096 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
000000A6 00 00 00 00 0d 5b 00 2d 4d c2 da 8d 38 b8 7d fe .....[.- M...8.}.
000000B6 41 e8 76 a6 09 00 00 00 48 00 2a 00 5c 00 5c 00 A.v..... H.*.\.\.
000000C6 31 00 39 00 32 00 2e 00 31 00 36 00 38 00 2e 00 1.9.2... 1.6.8...
000000D6 31 00 2e 00 31 00 35 00 30 00 5c 00 70 00 61 00 1...1.5. 0.\.p.a.
000000E6 72 00 74 00 30 00 r.t.0.
0000004D 00 00 00 49 fe 53 4d 42 40 00 01 00 22 00 00 c0 ...I.SMB @..."...
0000005D 03 00 01 00 09 00 00 00 00 00 00 00 f5 49 02 00 ........ .....I..
0000006D 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
0000007D 00 00 00 00 18 53 f7 8f 73 df 3d 32 50 94 84 d7 .....S.. s.=2P...
0000008D 2b 83 89 e2 09 00 00 00 00 00 00 00 00 +....... .....
000000EC 00 00 00 72 fe 53 4d 42 40 00 01 00 00 00 00 00 ...r.SMB @.......
000000FC 03 00 01 00 18 00 00 00 00 00 00 00 f6 49 02 00 ........ .....I..
0000010C 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
0000011C 00 00 00 00 b3 10 af cb 7d 22 44 fb da 02 45 23 ........ }"D...E#
0000012C c9 34 94 0c 09 00 00 00 48 00 2a 00 5c 00 5c 00 .4...... H.*.\.\.
0000013C 31 00 39 00 32 00 2e 00 31 00 36 00 38 00 2e 00 1.9.2... 1.6.8...
0000014C 31 00 2e 00 31 00 35 00 30 00 5c 00 70 00 61 00 1...1.5. 0.\.p.a.
0000015C 72 00 74 00 30 00 r.t.0.
0000009A 00 00 00 49 fe 53 4d 42 40 00 01 00 22 00 00 c0 ...I.SMB @..."...
000000AA 03 00 01 00 09 00 00 00 00 00 00 00 f6 49 02 00 ........ .....I..
000000BA 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
000000CA 00 00 00 00 5b f4 04 9b f2 0f 96 f3 85 b5 75 cb ....[... ......u.
000000DA 0a 83 b7 84 09 00 00 00 00 00 00 00 00 ........ .....
00000162 00 00 00 72 fe 53 4d 42 40 00 01 00 00 00 00 00 ...r.SMB @.......
00000172 03 00 01 00 18 00 00 00 00 00 00 00 f7 49 02 00 ........ .....I..
00000182 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
00000192 00 00 00 00 4b 5c 36 4d 57 e5 f7 0f 9c bd 74 04 ....K\6M W.....t.
000001A2 cd 99 49 ce 09 00 00 00 48 00 2a 00 5c 00 5c 00 ..I..... H.*.\.\.
000001B2 31 00 39 00 32 00 2e 00 31 00 36 00 38 00 2e 00 1.9.2... 1.6.8...
000001C2 31 00 2e 00 31 00 35 00 30 00 5c 00 70 00 61 00 1...1.5. 0.\.p.a.
000001D2 72 00 74 00 30 00 r.t.0.
000000E7 00 00 00 49 fe 53 4d 42 40 00 01 00 22 00 00 c0 ...I.SMB @..."...
000000F7 03 00 01 00 09 00 00 00 00 00 00 00 f7 49 02 00 ........ .....I..
00000107 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
00000117 00 00 00 00 84 8b 6c 7b a3 cd 7b a7 ff 0d 9b c8 ......l{ ..{.....
00000127 3f d1 48 76 09 00 00 00 00 00 00 00 00 ?.Hv.... .....
000001D8 00 00 00 72 fe 53 4d 42 40 00 01 00 00 00 00 00 ...r.SMB @.......
000001E8 03 00 01 00 18 00 00 00 00 00 00 00 f8 49 02 00 ........ .....I..
000001F8 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
00000208 00 00 00 00 0f 82 61 2d d2 71 42 13 c0 11 77 10 ......a- .qB...w.
00000218 b6 48 ca 85 09 00 00 00 48 00 2a 00 5c 00 5c 00 .H...... H.*.\.\.
00000228 31 00 39 00 32 00 2e 00 31 00 36 00 38 00 2e 00 1.9.2... 1.6.8...
00000238 31 00 2e 00 31 00 35 00 30 00 5c 00 70 00 61 00 1...1.5. 0.\.p.a.
00000248 72 00 74 00 30 00 r.t.0.
00000134 00 00 00 49 fe 53 4d 42 40 00 01 00 22 00 00 c0 ...I.SMB @..."...
00000144 03 00 01 00 09 00 00 00 00 00 00 00 f8 49 02 00 ........ .....I..
00000154 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
00000164 00 00 00 00 45 4e b3 56 39 27 67 54 92 94 84 8f ....EN.V 9'gT....
00000174 92 2a 6b 5e 09 00 00 00 00 00 00 00 00 .*k^.... .....
0000024E 00 00 00 72 fe 53 4d 42 40 00 01 00 00 00 00 00 ...r.SMB @.......
0000025E 03 00 01 00 18 00 00 00 00 00 00 00 f9 49 02 00 ........ .....I..
0000026E 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
0000027E 00 00 00 00 21 21 01 8b af 9b f1 63 10 c9 4b a0 ....!!.. ...c..K.
0000028E 94 ae cf 63 09 00 00 00 48 00 2a 00 5c 00 5c 00 ...c.... H.*.\.\.
0000029E 31 00 39 00 32 00 2e 00 31 00 36 00 38 00 2e 00 1.9.2... 1.6.8...
000002AE 31 00 2e 00 31 00 35 00 30 00 5c 00 70 00 61 00 1...1.5. 0.\.p.a.
000002BE 72 00 74 00 30 00 r.t.0.
00000181 00 00 00 49 fe 53 4d 42 40 00 01 00 22 00 00 c0 ...I.SMB @..."...
00000191 03 00 01 00 09 00 00 00 00 00 00 00 f9 49 02 00 ........ .....I..
000001A1 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
000001B1 00 00 00 00 79 31 9d 3a cd 31 b9 a9 ad 87 87 e2 ....y1.: .1......
000001C1 bb ac 9b a6 09 00 00 00 00 00 00 00 00 ........ .....
000002C4 00 00 00 72 fe 53 4d 42 40 00 01 00 00 00 00 00 ...r.SMB @.......
000002D4 03 00 01 00 18 00 00 00 00 00 00 00 fa 49 02 00 ........ .....I..
000002E4 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
000002F4 00 00 00 00 6f 7a c0 4d 62 cd 41 4f f9 60 44 61 ....oz.M b.AO.`Da
00000304 67 e8 51 56 09 00 00 00 48 00 2a 00 5c 00 5c 00 g.QV.... H.*.\.\.
00000314 31 00 39 00 32 00 2e 00 31 00 36 00 38 00 2e 00 1.9.2... 1.6.8...
00000324 31 00 2e 00 31 00 35 00 30 00 5c 00 70 00 61 00 1...1.5. 0.\.p.a.
00000334 72 00 74 00 30 00 r.t.0.
000001CE 00 00 00 49 fe 53 4d 42 40 00 01 00 22 00 00 c0 ...I.SMB @..."...
000001DE 03 00 01 00 09 00 00 00 00 00 00 00 fa 49 02 00 ........ .....I..
000001EE 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
000001FE 00 00 00 00 89 ab 43 3d 8b ee b6 c9 f3 08 69 16 ......C= ......i.
0000020E 43 a4 19 ae 09 00 00 00 00 00 00 00 00 C....... .....
0000033A 00 00 00 72 fe 53 4d 42 40 00 01 00 00 00 00 00 ...r.SMB @.......
0000034A 03 00 01 00 18 00 00 00 00 00 00 00 fb 49 02 00 ........ .....I..
0000035A 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
0000036A 00 00 00 00 98 90 85 8e 8e 5d 07 6d af fa 18 be ........ .].m....
0000037A fb 40 80 d3 09 00 00 00 48 00 2a 00 5c 00 5c 00 .@...... H.*.\.\.
0000038A 31 00 39 00 32 00 2e 00 31 00 36 00 38 00 2e 00 1.9.2... 1.6.8...
0000039A 31 00 2e 00 31 00 35 00 30 00 5c 00 70 00 61 00 1...1.5. 0.\.p.a.
000003AA 72 00 74 00 30 00 r.t.0.
0000021B 00 00 00 49 fe 53 4d 42 40 00 01 00 22 00 00 c0 ...I.SMB @..."...
0000022B 03 00 01 00 09 00 00 00 00 00 00 00 fb 49 02 00 ........ .....I..
0000023B 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
0000024B 00 00 00 00 cb c1 93 ce 5a d5 7d ea e0 40 cf 27 ........ Z.}..@.'
0000025B 03 59 f9 24 09 00 00 00 00 00 00 00 00 .Y.$.... .....
000003B0 00 00 00 72 fe 53 4d 42 40 00 01 00 00 00 00 00 ...r.SMB @.......
000003C0 03 00 01 00 18 00 00 00 00 00 00 00 fc 49 02 00 ........ .....I..
000003D0 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
000003E0 00 00 00 00 88 64 ad ab 89 6c db 05 02 b9 c7 b0 .....d.. .l......
000003F0 07 82 65 3c 09 00 00 00 48 00 2a 00 5c 00 5c 00 ..e<.... H.*.\.\.
00000400 31 00 39 00 32 00 2e 00 31 00 36 00 38 00 2e 00 1.9.2... 1.6.8...
00000410 31 00 2e 00 31 00 35 00 30 00 5c 00 70 00 61 00 1...1.5. 0.\.p.a.
00000420 72 00 74 00 30 00 r.t.0.
00000268 00 00 00 49 fe 53 4d 42 40 00 01 00 22 00 00 c0 ...I.SMB @..."...
00000278 03 00 01 00 09 00 00 00 00 00 00 00 fc 49 02 00 ........ .....I..
00000288 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
00000298 00 00 00 00 ed b6 8d 24 08 4c 86 e6 e5 3c 19 77 .......$ .L...<.w
000002A8 dc b4 39 e5 09 00 00 00 00 00 00 00 00 ..9..... .....
00000426 00 00 00 72 fe 53 4d 42 40 00 01 00 00 00 00 00 ...r.SMB @.......
00000436 03 00 01 00 18 00 00 00 00 00 00 00 fd 49 02 00 ........ .....I..
00000446 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
00000456 00 00 00 00 23 1a 33 89 8b d8 00 7f fb 4b 84 0c ....#.3. .....K..
00000466 51 3f b6 bf 09 00 00 00 48 00 2a 00 5c 00 5c 00 Q?...... H.*.\.\.
00000476 31 00 39 00 32 00 2e 00 31 00 36 00 38 00 2e 00 1.9.2... 1.6.8...
00000486 31 00 2e 00 31 00 35 00 30 00 5c 00 70 00 61 00 1...1.5. 0.\.p.a.
00000496 72 00 74 00 30 00 r.t.0.
000002B5 00 00 00 49 fe 53 4d 42 40 00 01 00 22 00 00 c0 ...I.SMB @..."...
000002C5 03 00 01 00 09 00 00 00 00 00 00 00 fd 49 02 00 ........ .....I..
000002D5 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
000002E5 00 00 00 00 ac ef 84 a7 b0 dc 3d ac 83 f8 58 41 ........ ..=...XA
000002F5 b2 4b 46 2c 09 00 00 00 00 00 00 00 00 .KF,.... .....
0000049C 00 00 00 72 fe 53 4d 42 40 00 01 00 00 00 00 00 ...r.SMB @.......
000004AC 03 00 01 00 18 00 00 00 00 00 00 00 fe 49 02 00 ........ .....I..
000004BC 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
000004CC 00 00 00 00 50 6f b9 f2 c3 47 d8 30 96 12 05 56 ....Po.. .G.0...V
000004DC 0e d7 6f 6c 09 00 00 00 48 00 2a 00 5c 00 5c 00 ..ol.... H.*.\.\.
000004EC 31 00 39 00 32 00 2e 00 31 00 36 00 38 00 2e 00 1.9.2... 1.6.8...
000004FC 31 00 2e 00 31 00 35 00 30 00 5c 00 70 00 61 00 1...1.5. 0.\.p.a.
0000050C 72 00 74 00 30 00 r.t.0.
00000302 00 00 00 49 fe 53 4d 42 40 00 01 00 22 00 00 c0 ...I.SMB @..."...
00000312 03 00 01 00 09 00 00 00 00 00 00 00 fe 49 02 00 ........ .....I..
00000322 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
00000332 00 00 00 00 aa 59 6c d8 dc fb 59 68 ec 88 f3 cd .....Yl. ..Yh....
00000342 8c 0c 5b 5e 09 00 00 00 00 00 00 00 00 ..[^.... .....
00000512 00 00 00 72 fe 53 4d 42 40 00 01 00 00 00 00 00 ...r.SMB @.......
00000522 03 00 01 00 18 00 00 00 00 00 00 00 ff 49 02 00 ........ .....I..
00000532 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
00000542 00 00 00 00 ac bd 3f ba 56 9a d8 a3 b6 18 2b 31 ......?. V.....+1
00000552 47 66 1a 83 09 00 00 00 48 00 2a 00 5c 00 5c 00 Gf...... H.*.\.\.
00000562 31 00 39 00 32 00 2e 00 31 00 36 00 38 00 2e 00 1.9.2... 1.6.8...
00000572 31 00 2e 00 31 00 35 00 30 00 5c 00 70 00 61 00 1...1.5. 0.\.p.a.
00000582 72 00 74 00 30 00 r.t.0.
0000034F 00 00 00 49 fe 53 4d 42 40 00 01 00 22 00 00 c0 ...I.SMB @..."...
0000035F 03 00 01 00 09 00 00 00 00 00 00 00 ff 49 02 00 ........ .....I..
0000036F 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
0000037F 00 00 00 00 93 a3 25 b7 72 96 fd 23 6f 20 70 05 ......%. r..#o p.
0000038F 34 fe 21 ef 09 00 00 00 00 00 00 00 00 4.!..... .....
00000588 00 00 00 72 fe 53 4d 42 40 00 01 00 00 00 00 00 ...r.SMB @.......
00000598 03 00 01 00 18 00 00 00 00 00 00 00 00 4a 02 00 ........ .....J..
000005A8 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
000005B8 00 00 00 00 92 14 2d 23 dd c3 ef 24 99 f8 2e 5b ......-# ...$...[
000005C8 61 2f cf 9c 09 00 00 00 48 00 2a 00 5c 00 5c 00 a/...... H.*.\.\.
000005D8 31 00 39 00 32 00 2e 00 31 00 36 00 38 00 2e 00 1.9.2... 1.6.8...
000005E8 31 00 2e 00 31 00 35 00 30 00 5c 00 70 00 61 00 1...1.5. 0.\.p.a.
000005F8 72 00 74 00 30 00 r.t.0.
0000039C 00 00 00 49 fe 53 4d 42 40 00 01 00 22 00 00 c0 ...I.SMB @..."...
000003AC 03 00 01 00 09 00 00 00 00 00 00 00 00 4a 02 00 ........ .....J..
000003BC 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
000003CC 00 00 00 00 6a 27 37 fb 9a 22 15 5e b0 74 96 cb ....j'7. .".^.t..
000003DC 7d 5c 8d b4 09 00 00 00 00 00 00 00 00 }\...... .....
000005FE 00 00 00 72 fe 53 4d 42 40 00 01 00 00 00 00 00 ...r.SMB @.......
0000060E 03 00 01 00 18 00 00 00 00 00 00 00 01 4a 02 00 ........ .....J..
0000061E 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
0000062E 00 00 00 00 58 c1 45 14 73 e4 4b 66 5e e5 a4 93 ....X.E. s.Kf^...
0000063E 13 b9 15 fb 09 00 00 00 48 00 2a 00 5c 00 5c 00 ........ H.*.\.\.
0000064E 31 00 39 00 32 00 2e 00 31 00 36 00 38 00 2e 00 1.9.2... 1.6.8...
0000065E 31 00 2e 00 31 00 35 00 30 00 5c 00 70 00 61 00 1...1.5. 0.\.p.a.
0000066E 72 00 74 00 30 00 r.t.0.
000003E9 00 00 00 49 fe 53 4d 42 40 00 01 00 22 00 00 c0 ...I.SMB @..."...
000003F9 03 00 01 00 09 00 00 00 00 00 00 00 01 4a 02 00 ........ .....J..
00000409 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
00000419 00 00 00 00 57 2f 97 ab 0d c9 d2 9c 52 3b 37 30 ....W/.. ....R;70
00000429 f5 77 0f 18 09 00 00 00 00 00 00 00 00 .w...... .....
00000674 00 00 00 72 fe 53 4d 42 40 00 01 00 00 00 00 00 ...r.SMB @.......
00000684 03 00 01 00 18 00 00 00 00 00 00 00 02 4a 02 00 ........ .....J..
00000694 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
000006A4 00 00 00 00 89 59 aa 33 29 57 23 dc 3d 47 21 81 .....Y.3 )W#.=G!.
000006B4 54 89 0e 04 09 00 00 00 48 00 2a 00 5c 00 5c 00 T....... H.*.\.\.
000006C4 31 00 39 00 32 00 2e 00 31 00 36 00 38 00 2e 00 1.9.2... 1.6.8...
000006D4 31 00 2e 00 31 00 35 00 30 00 5c 00 70 00 61 00 1...1.5. 0.\.p.a.
000006E4 72 00 74 00 30 00 r.t.0.
00000436 00 00 00 49 fe 53 4d 42 40 00 01 00 22 00 00 c0 ...I.SMB @..."...
00000446 03 00 01 00 09 00 00 00 00 00 00 00 02 4a 02 00 ........ .....J..
00000456 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
00000466 00 00 00 00 5a 06 1c b6 05 fe 7e e2 42 7c d3 8f ....Z... ..~.B|..
00000476 15 23 4c 2c 09 00 00 00 00 00 00 00 00 .#L,.... .....
000006EA 00 00 00 72 fe 53 4d 42 40 00 01 00 00 00 00 00 ...r.SMB @.......
000006FA 03 00 01 00 18 00 00 00 00 00 00 00 03 4a 02 00 ........ .....J..
0000070A 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
0000071A 00 00 00 00 ad 7f 78 e3 3d d0 e2 d5 25 38 56 1f ......x. =...%8V.
0000072A d0 0c 9b c4 09 00 00 00 48 00 2a 00 5c 00 5c 00 ........ H.*.\.\.
0000073A 31 00 39 00 32 00 2e 00 31 00 36 00 38 00 2e 00 1.9.2... 1.6.8...
0000074A 31 00 2e 00 31 00 35 00 30 00 5c 00 70 00 61 00 1...1.5. 0.\.p.a.
0000075A 72 00 74 00 30 00 r.t.0.
00000483 00 00 00 49 fe 53 4d 42 40 00 01 00 22 00 00 c0 ...I.SMB @..."...
00000493 03 00 01 00 09 00 00 00 00 00 00 00 03 4a 02 00 ........ .....J..
000004A3 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
000004B3 00 00 00 00 59 81 bb 82 5b 3e 82 bb 52 49 61 25 ....Y... [>..RIa%
000004C3 87 95 6a 7a 09 00 00 00 00 00 00 00 00 ..jz.... .....
00000760 00 00 00 72 fe 53 4d 42 40 00 01 00 00 00 00 00 ...r.SMB @.......
00000770 03 00 01 00 18 00 00 00 00 00 00 00 04 4a 02 00 ........ .....J..
00000780 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
00000790 00 00 00 00 bf f1 15 b6 60 2e 3f 37 8d ba 5c 6a ........ `.?7..\j
000007A0 77 f3 47 4d 09 00 00 00 48 00 2a 00 5c 00 5c 00 w.GM.... H.*.\.\.
000007B0 31 00 39 00 32 00 2e 00 31 00 36 00 38 00 2e 00 1.9.2... 1.6.8...
000007C0 31 00 2e 00 31 00 35 00 30 00 5c 00 70 00 61 00 1...1.5. 0.\.p.a.
000007D0 72 00 74 00 30 00 r.t.0.
000004D0 00 00 00 49 fe 53 4d 42 40 00 01 00 22 00 00 c0 ...I.SMB @..."...
000004E0 03 00 01 00 09 00 00 00 00 00 00 00 04 4a 02 00 ........ .....J..
000004F0 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
00000500 00 00 00 00 a1 7c a1 7f b6 6c 98 5c 37 e7 a0 4c .....|.. .l.\7..L
00000510 b9 8a 42 2a 09 00 00 00 00 00 00 00 00 ..B*.... .....
000007D6 00 00 00 72 fe 53 4d 42 40 00 01 00 00 00 00 00 ...r.SMB @.......
000007E6 03 00 01 00 18 00 00 00 00 00 00 00 05 4a 02 00 ........ .....J..
000007F6 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
00000806 00 00 00 00 9d ca 27 ba 28 8a 71 37 38 d3 1d 77 ......'. (.q78..w
00000816 ef 9b 1e 5b 09 00 00 00 48 00 2a 00 5c 00 5c 00 ...[.... H.*.\.\.
00000826 31 00 39 00 32 00 2e 00 31 00 36 00 38 00 2e 00 1.9.2... 1.6.8...
00000836 31 00 2e 00 31 00 35 00 30 00 5c 00 70 00 61 00 1...1.5. 0.\.p.a.
00000846 72 00 74 00 30 00 r.t.0.
0000051D 00 00 00 49 fe 53 4d 42 40 00 01 00 22 00 00 c0 ...I.SMB @..."...
0000052D 03 00 01 00 09 00 00 00 00 00 00 00 05 4a 02 00 ........ .....J..
0000053D 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
0000054D 00 00 00 00 78 68 c3 cf f9 60 24 f3 73 a4 59 4c ....xh.. .`$.s.YL
0000055D 22 9f da 5b 09 00 00 00 00 00 00 00 00 "..[.... .....
0000084C 00 00 00 72 fe 53 4d 42 40 00 01 00 00 00 00 00 ...r.SMB @.......
0000085C 03 00 01 00 18 00 00 00 00 00 00 00 06 4a 02 00 ........ .....J..
0000086C 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
0000087C 00 00 00 00 d3 d5 f0 a6 ee 77 41 cd 93 5b fa 39 ........ .wA..[.9
0000088C dc 3a 80 bd 09 00 00 00 48 00 2a 00 5c 00 5c 00 .:...... H.*.\.\.
0000089C 31 00 39 00 32 00 2e 00 31 00 36 00 38 00 2e 00 1.9.2... 1.6.8...
000008AC 31 00 2e 00 31 00 35 00 30 00 5c 00 70 00 61 00 1...1.5. 0.\.p.a.
000008BC 72 00 74 00 30 00 r.t.0.
0000056A 00 00 00 49 fe 53 4d 42 40 00 01 00 22 00 00 c0 ...I.SMB @..."...
0000057A 03 00 01 00 09 00 00 00 00 00 00 00 06 4a 02 00 ........ .....J..
0000058A 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
0000059A 00 00 00 00 c4 c5 7f f9 d2 bd 6c c7 29 42 43 75 ........ ..l.)BCu
000005AA 7e a6 c3 8d 09 00 00 00 00 00 00 00 00 ~....... .....
000008C2 00 00 00 72 fe 53 4d 42 40 00 01 00 00 00 00 00 ...r.SMB @.......
000008D2 03 00 01 00 18 00 00 00 00 00 00 00 07 4a 02 00 ........ .....J..
000008E2 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
000008F2 00 00 00 00 ef 68 e8 c6 c9 13 df 1c 8f 11 b2 9b .....h.. ........
00000902 62 ed 03 19 09 00 00 00 48 00 2a 00 5c 00 5c 00 b....... H.*.\.\.
00000912 31 00 39 00 32 00 2e 00 31 00 36 00 38 00 2e 00 1.9.2... 1.6.8...
00000922 31 00 2e 00 31 00 35 00 30 00 5c 00 70 00 61 00 1...1.5. 0.\.p.a.
00000932 72 00 74 00 30 00 r.t.0.
000005B7 00 00 00 49 fe 53 4d 42 40 00 01 00 22 00 00 c0 ...I.SMB @..."...
000005C7 03 00 01 00 09 00 00 00 00 00 00 00 07 4a 02 00 ........ .....J..
000005D7 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
000005E7 00 00 00 00 3a 1f 63 fc 09 05 e1 18 58 ca 49 ef ....:.c. ....X.I.
000005F7 0d 41 81 4b 09 00 00 00 00 00 00 00 00 .A.K.... .....
00000938 00 00 00 72 fe 53 4d 42 40 00 01 00 00 00 00 00 ...r.SMB @.......
00000948 03 00 01 00 18 00 00 00 00 00 00 00 08 4a 02 00 ........ .....J..
00000958 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
00000968 00 00 00 00 54 7e 6d a3 b9 a7 90 8f aa f1 3c 64 ....T~m. ......<d
00000978 f0 e6 bd e4 09 00 00 00 48 00 2a 00 5c 00 5c 00 ........ H.*.\.\.
00000988 31 00 39 00 32 00 2e 00 31 00 36 00 38 00 2e 00 1.9.2... 1.6.8...
00000998 31 00 2e 00 31 00 35 00 30 00 5c 00 70 00 61 00 1...1.5. 0.\.p.a.
000009A8 72 00 74 00 30 00 r.t.0.
00000604 00 00 00 49 fe 53 4d 42 40 00 01 00 22 00 00 c0 ...I.SMB @..."...
00000614 03 00 01 00 09 00 00 00 00 00 00 00 08 4a 02 00 ........ .....J..
00000624 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
00000634 00 00 00 00 88 d1 12 f5 5a a3 34 46 b2 3f e6 48 ........ Z.4F.?.H
00000644 6f 9e d5 de 09 00 00 00 00 00 00 00 00 o....... .....
000009AE 00 00 00 72 fe 53 4d 42 40 00 01 00 00 00 00 00 ...r.SMB @.......
000009BE 03 00 01 00 18 00 00 00 00 00 00 00 09 4a 02 00 ........ .....J..
000009CE 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
000009DE 00 00 00 00 61 23 87 ac 3f 88 1e 5f 7a cd a1 7a ....a#.. ?.._z..z
000009EE 3b aa 29 6e 09 00 00 00 48 00 2a 00 5c 00 5c 00 ;.)n.... H.*.\.\.
000009FE 31 00 39 00 32 00 2e 00 31 00 36 00 38 00 2e 00 1.9.2... 1.6.8...
00000A0E 31 00 2e 00 31 00 35 00 30 00 5c 00 70 00 61 00 1...1.5. 0.\.p.a.
00000A1E 72 00 74 00 30 00 r.t.0.
00000651 00 00 00 49 fe 53 4d 42 40 00 01 00 22 00 00 c0 ...I.SMB @..."...
00000661 03 00 01 00 09 00 00 00 00 00 00 00 09 4a 02 00 ........ .....J..
00000671 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
00000681 00 00 00 00 f0 da 42 3b bf c4 4d fa bf 7e 3b c6 ......B; ..M..~;.
00000691 40 27 be 6f 09 00 00 00 00 00 00 00 00 @'.o.... .....
00000A24 00 00 00 72 fe 53 4d 42 40 00 01 00 00 00 00 00 ...r.SMB @.......
00000A34 03 00 01 00 18 00 00 00 00 00 00 00 0a 4a 02 00 ........ .....J..
00000A44 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
00000A54 00 00 00 00 a2 a2 d5 b6 0a d3 a6 2d 8d 16 59 03 ........ ...-..Y.
00000A64 4b b0 96 30 09 00 00 00 48 00 2a 00 5c 00 5c 00 K..0.... H.*.\.\.
00000A74 31 00 39 00 32 00 2e 00 31 00 36 00 38 00 2e 00 1.9.2... 1.6.8...
00000A84 31 00 2e 00 31 00 35 00 30 00 5c 00 70 00 61 00 1...1.5. 0.\.p.a.
00000A94 72 00 74 00 30 00 r.t.0.
0000069E 00 00 00 49 fe 53 4d 42 40 00 01 00 22 00 00 c0 ...I.SMB @..."...
000006AE 03 00 01 00 09 00 00 00 00 00 00 00 0a 4a 02 00 ........ .....J..
000006BE 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
000006CE 00 00 00 00 64 9e e9 59 1d 3e 74 2d b6 11 27 50 ....d..Y .>t-..'P
000006DE ee 58 82 a8 09 00 00 00 00 00 00 00 00 .X...... .....
00000A9A 00 00 00 72 fe 53 4d 42 40 00 01 00 00 00 00 00 ...r.SMB @.......
00000AAA 03 00 01 00 18 00 00 00 00 00 00 00 0b 4a 02 00 ........ .....J..
00000ABA 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
00000ACA 00 00 00 00 b3 e8 d0 79 ce 35 6b cf df 0a 50 8c .......y .5k...P.
00000ADA cf 6e 08 08 09 00 00 00 48 00 2a 00 5c 00 5c 00 .n...... H.*.\.\.
00000AEA 31 00 39 00 32 00 2e 00 31 00 36 00 38 00 2e 00 1.9.2... 1.6.8...
00000AFA 31 00 2e 00 31 00 35 00 30 00 5c 00 70 00 61 00 1...1.5. 0.\.p.a.
00000B0A 72 00 74 00 30 00 r.t.0.
000006EB 00 00 00 49 fe 53 4d 42 40 00 01 00 22 00 00 c0 ...I.SMB @..."...
000006FB 03 00 01 00 09 00 00 00 00 00 00 00 0b 4a 02 00 ........ .....J..
0000070B 00 00 00 00 ff fe 00 00 00 00 00 00 07 00 00 00 ........ ........
0000071B 00 00 00 00 ef a1 2e a4 c6 86 47 79 54 fb fb b6 ........ ..GyT...
0000072B 83 47 3a 8e 09 00 00 00 00 00 00 00 00 .G:..... .....
wireshark on linux (version 4.0.6) opens the file without issue (directly, or after gunzip).
The packet dump is a series of Tree Connect Request messages from the client, with the server responding STATUS_ACCESS_DENIED, which is what is expected if the signature is wrong.
The TCP checksum on response packets is incorrect, but that may be because of checksum offload. Responses from the server are also systematically retransmitted which suggests some packet loss is occurring.
@darkwrat Did you do the capture on the server ? How are clients connected to the server ?
Did you do the capture on the server ?
Yes, indeed. I still have the issue present, can collect more data before restoring.
How are clients connected to the server ?
These are excerpts from nmcli on the server. A windows client has a X520-DA2 card installed and is connected via SFP+ cable directly to the server's built in card. The server bridges this link with 1Gbps from a router, and gets DHCP from the bridge. The second client usually connects over wifi, but that machine is off now. Perhaps the bridge may be the source of "retransmissions" in the dump? It was captured with -i any.
enp3s0f1: connected to Ethernet connection 2
"Intel X552 SFP+"
ethernet (ixgbe), E0:D5:5E:5C:BA:93, hw, sriov, mtu 1500
master nm-bridge
enp6s0: connected to Ethernet connection 1
"Intel I210"
ethernet (igb), E0:D5:5E:5C:BA:94, hw, mtu 1500
master nm-bridge
nm-bridge: connected to Bridge connection 1
"nm-bridge"
....
inet4 192.168.1.150/24
The TCP checksum on response packets is incorrect, but that may be because of checksum offload. Responses from the server are also systematically retransmitted which suggests some packet loss is occurring.
@mmakassikis Really good catch! How did you know that ?
@namjaejeon Wireshark defaults to "don't compute TCP checksums" because it can be costly cpu-wise for big pcaps, and with more and more NICs supporting checksum offloading it doesn't even make sense. It may not even mean anything here.
On the other hand, retransmissions indicate something is happening. IIRC, either a retransmission timeout expired, or duplicate acks have been received (both of which indicate packet loss).
@darkwrat
Can you share the server's interface statistics ? "ip -s link show dev
On the client, do you still get the ERROR_ALREADY_ASSIGNED error ?
Can you share the server's interface statistics ?
enp3s0f1 is the one connected to the winows box. I think retransmissions are bogus because the same packet is captured twice -- once on the bridge member and once on the bridge itself. I've seen this before on vlan interfaces.
[root@whip ~]# ip -s link show dev enp3s0f1
5: enp3s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master nm-bridge state UP mode DEFAULT group default qlen 1000
link/ether e0:d5:5e:5c:ba:93 brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped missed mcast
18319285171 44826358 0 344 0 245034
TX: bytes packets errors dropped carrier collsns
23890759234 36432401 0 0 0 0
[root@whip ~]# ip -s link show dev enp6s0
2: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master nm-bridge state UP mode DEFAULT group default qlen 1000
link/ether e0:d5:5e:5c:ba:94 brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped missed mcast
6684896751 24384015 0 0 0 39888
TX: bytes packets errors dropped carrier collsns
19004603204 43803519 0 0 0 0
[root@whip ~]# ip -s link show dev nm-bridge
6: nm-bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether e0:d5:5e:5c:ba:93 brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped missed mcast
1035288139 2208612 0 0 0 283881
TX: bytes packets errors dropped carrier collsns
18285664532 1085805 0 0 0 0
On the client, do you still get the ERROR_ALREADY_ASSIGNED error ?
Yes, in the popup messagebox when trying to open the network disk. Also I tried doing Z: in cmd, and get "Incorrect signature." (or "invalid signature", i have to lookup translations of error texts..)
So ISTM the problem is two-way, the client cannot verify server's signature, and the server cannot verify client's signature. It happens rarely and randomly, and gets stuck there. Perhaps the root cause could be found in sess_key corruption? I'm inclined to make some debug patches for the kernel with more logging and warns, and run them until the next occurrence of the issue.
@namjaejeon any advice on the above? should anything of interest be included?
enp3s0f1 is the one connected to the winows box. I think retransmissions are bogus because the same packet is captured twice -- once on the bridge member and once on the bridge itself. I've seen this before on vlan interfaces.
Ah yes, that may confuse wireshark. Can you share a capture done with "-i enp3s0f1" rather "-i any" ?
It seems smb3 multichannel can bind multiple tcp connections even if there's only a single NIC. Can you try disabling multichannel on the client with the following powershell command ?
Set-SmbClientConfiguration -EnableMultiChannel $false
Can you share a capture done with "-i enp3s0f1" rather "-i any" ?
Done. ksmbd_20231020_3.pcap.gz
Set-SmbClientConfiguration -EnableMultiChannel $false
Executed, the first messagebox was "Access denied", then ERROR_ALREADY_ASSIGNED again.
Done.
Thanks. You were right: it went from all segments being retransmitted to basically none.
Executed, the first messagebox was "Access denied", then ERROR_ALREADY_ASSIGNED again.
Did you unmap/remap the drive ?
I'm not sure when the parameter is applied. Normally, the steps are:
In the packet dump, the endpoints are in step 3, so it's either:
Did you unmap/remap the drive ?
If I do this, the problem could be gone, and I'll probably have to wait for a week to observe it again :) I'll unmap/remap now, if we don't need more pcaps.
In the packet dump, the endpoints are in step 3
Oh yes, sorry, the capture was already done at that point when I changed the EnableMultiChannel setting. I'll try to keep tcpdump running for long time to capture the entire session.
I'll try to keep tcpdump running for long time to capture the entire session.
beware of long running tcpdump, as you could end up filling all of your RAM (which may happen very fast if you transfer a few large files from/to the server)
beware of long running tcpdump, as you could end up filling all of your RAM (which may happen very fast if you transfer a few large files from/to the server)
Thank you for the heads up. I hope won't be an issue, the server has 256 GB of RAM.
Writing a tcpdump filter to match 0xfe534d42 in packet body takes more time than I expected though )
http://ams2.trail5.net/smb2_20231025.pcap.gz
# sha256sum smb2_20231025.pcap.gz
b6bfbc69f2e1cc7e69703cea85e39407bcf52d2ab96963389b1c163e0b05b3e9 smb2_20231025.pcap.gz
4 GB file, captured as tcpdump -i enp3s0f1 -w /mnt/part0/smb2_20231025.pcap 'tcp[((tcp[12:1] & 0xf0) >> 2) + 4:4] = 0xfe534d42'
Must contain the moment when issue appeared.
This 20 MB chunk should be more digestible and possibly complete.
Hello,
I have ksmbd running on linux 6.6-rc5, and a couple of windows clients. Sometimes clients cannot reconnect to the share with ERROR_ALREADY_ASSIGNED, and dmesg contains:
Please advise on what debug should I collect, and also tell me if this is not a correct place to report this issue.
Thanks, Max