search

jborean93 / smbprotocol

Python SMBv2 and v3 Client
MIT License
322 stars 73 forks source link

Trouble connecting with username and password #289

Closed mxmlnkn closed 1 month ago

mxmlnkn commented 1 month ago

Hello,

I tried to set up a simple server and access it with:

python3 -m pip install impacket
wget 'https://github.com/fortra/impacket/raw/27e7e7478df5d3d3bb12923055a7d8b614825ff4/examples/smbserver.py'
python3 smbserver.py -smb2support -username username -password password -ip 127.0.0.1 -port 8445 test "$PWD"
sudo apt install smbclient
smbclient --user=username --password=password --port 8445 -c ls //127.0.0.1/test  # Lists all files as expected

However, I am unable to get it to work with smbprotocol. I tried to follow the example in the ReadMe:

import smbclient
smbclient.register_session(server="127.0.0.1", username="username", password="password", port=8445)

But, I only get:

Traceback (most recent call last):
  File "/home/user/.local/lib/python3.10/site-packages/smbprotocol/session.py", line 282, in connect
    out_token = context.step(in_token)
  File "/home/user/.local/lib/python3.10/site-packages/spnego/_negotiate.py", line 161, in step
    out_mic = self._step_spnego_mic(in_mic=mech_list_mic)
  File "/home/user/.local/lib/python3.10/site-packages/spnego/_negotiate.py", line 304, in _step_spnego_mic
    out_mic = self.sign(pack_mech_type_list(self._mech_list))
  File "/home/user/.local/lib/python3.10/site-packages/spnego/_negotiate.py", line 389, in sign
    return self._context.sign(data, qop=qop)
  File "/home/user/.local/lib/python3.10/site-packages/spnego/_ntlm.py", line 843, in sign
    return sign(
  File "/home/user/.local/lib/python3.10/site-packages/spnego/_ntlm_raw/security.py", line 69, in sign
    raise OperationNotAvailableError(context_msg="Signing without integrity.")
spnego.exceptions.OperationNotAvailableError: SpnegoError (16): Operation not supported or available, Context: Signing without integrity.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/user/.local/lib/python3.10/site-packages/smbclient/_pool.py", line 422, in register_session
    session.connect()
  File "/home/user/.local/lib/python3.10/site-packages/smbprotocol/session.py", line 284, in connect
    raise SMBAuthenticationError(f"Failed to authenticate with server: {err}") from err
smbprotocol.exceptions.SMBAuthenticationError: Failed to authenticate with server: SpnegoError (16): Operation not supported or available, Context: Signing without integrity.

Out of desperation and without understanding, I also tried with auth_protocol="ntlm" given to register_session but only got:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/user/.local/lib/python3.10/site-packages/smbclient/_pool.py", line 422, in register_session
    session.connect()
  File "/home/user/.local/lib/python3.10/site-packages/smbprotocol/session.py", line 395, in connect
    self.connection.verify_signature(response, self.session_id, force=True)
  File "/home/user/.local/lib/python3.10/site-packages/smbprotocol/connection.py", line 1176, in verify_signature
    raise SMBException(
smbprotocol.exceptions.SMBException: Server message signature could not be verified: 2ab9616591d29fd57dee6b366f6fb63f != 67d1568b493ad52f9b4f8ee13c60a1fd
Exception in thread msg_worker-127.0.0.1:8445:
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.10/site-packages/smbprotocol/connection.py", line 1369, in _process_message_thread
    self.verify_signature(header, session_id)
  File "/home/user/.local/lib/python3.10/site-packages/smbprotocol/connection.py", line 1163, in verify_signature
    raise SMBException(f"Failed to find session {session_id} for message verification")
smbprotocol.exceptions.SMBException: Failed to find session 0 for message verification

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/smbprotocol/connection.py", line 1406, in _process_message_thread
    self.disconnect(False)
  File "/home/user/.local/lib/python3.10/site-packages/smbprotocol/connection.py", line 957, in disconnect
    self._t_worker.join(timeout=2)
  File "/usr/lib/python3.10/threading.py", line 1093, in join
    raise RuntimeError("cannot join current thread")
RuntimeError: cannot join current thread
Failed to close connection 127.0.0.1:8445
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.10/site-packages/smbclient/_pool.py", line 445, in reset_connection_cache
    connection.disconnect()
  File "/home/user/.local/lib/python3.10/site-packages/smbprotocol/connection.py", line 952, in disconnect
    session.disconnect(True, timeout=timeout)
  File "/home/user/.local/lib/python3.10/site-packages/smbprotocol/session.py", line 421, in disconnect
    res = self.connection.receive(request, timeout=timeout)
  File "/home/user/.local/lib/python3.10/site-packages/smbprotocol/connection.py", line 1028, in receive
    self._check_worker_running()  # The worker may have failed while waiting for the response, check again
  File "/home/user/.local/lib/python3.10/site-packages/smbprotocol/connection.py", line 1184, in _check_worker_running
    raise self._t_exc
  File "/home/user/.local/lib/python3.10/site-packages/smbprotocol/connection.py", line 1369, in _process_message_thread
    self.verify_signature(header, session_id)
  File "/home/user/.local/lib/python3.10/site-packages/smbprotocol/connection.py", line 1163, in verify_signature
    raise SMBException(f"Failed to find session {session_id} for message verification")
smbprotocol.exceptions.SMBException: Failed to find session 0 for message verification
jborean93 commented 1 month ago

The problem here is that impacket performs NTLM authentication but does not offer and signing capabilities so the Negotiate wrapped NTLM token is unable to sign the mech list MIC. The other problem is SMB signing also requires this feature from the NTLM auth so that will fail once it gets there.

What you can try to use plain NTLM auth (without the Negotiate wrapping) and disable signing is by setting:

smbclient.register_session(…, auth_protocol='ntlm', require_signing=False)
mxmlnkn commented 1 month ago 
import smbclient
smbclient.register_session(
    server="127.0.0.1", username="username", password="password", port=8445, 
    auth_protocol="ntlm", require_signing=False)

results in:

Exception in thread msg_worker-127.0.0.1:8445:
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.10/site-packages/smbprotocol/connection.py", line 1369, in _process_message_thread
    self.verify_signature(header, session_id)
  File "/home/user/.local/lib/python3.10/site-packages/smbprotocol/connection.py", line 1163, in verify_signature
    raise SMBException(f"Failed to find session {session_id} for message verification")
smbprotocol.exceptions.SMBException: Failed to find session 0 for message verification

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/smbprotocol/connection.py", line 1406, in _process_message_thread
    self.disconnect(False)
  File "/home/user/.local/lib/python3.10/site-packages/smbprotocol/connection.py", line 957, in disconnect
    self._t_worker.join(timeout=2)
  File "/usr/lib/python3.10/threading.py", line 1093, in join
    raise RuntimeError("cannot join current thread")
RuntimeError: cannot join current thread
Failed to close connection 127.0.0.1:8445
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.10/site-packages/smbclient/_pool.py", line 445, in reset_connection_cache
    connection.disconnect()
  File "/home/user/.local/lib/python3.10/site-packages/smbprotocol/connection.py", line 952, in disconnect
    session.disconnect(True, timeout=timeout)
  File "/home/user/.local/lib/python3.10/site-packages/smbprotocol/session.py", line 421, in disconnect
    res = self.connection.receive(request, timeout=timeout)
  File "/home/user/.local/lib/python3.10/site-packages/smbprotocol/connection.py", line 1028, in receive
    self._check_worker_running()  # The worker may have failed while waiting for the response, check again
  File "/home/user/.local/lib/python3.10/site-packages/smbprotocol/connection.py", line 1184, in _check_worker_running
    raise self._t_exc
  File "/home/user/.local/lib/python3.10/site-packages/smbprotocol/connection.py", line 1369, in _process_message_thread
    self.verify_signature(header, session_id)
  File "/home/user/.local/lib/python3.10/site-packages/smbprotocol/connection.py", line 1163, in verify_signature
    raise SMBException(f"Failed to find session {session_id} for message verification")
smbprotocol.exceptions.SMBException: Failed to find session 0 for message verification

I am also wondering why it works out of the box with the system-insalled smbclient 4.15.13-Ubuntu. Has it signing disabled by default?

mxmlnkn commented 1 month ago

I have "fixed" it like this:

--- /home/user/.local/lib/python3.10/site-packages/smbprotocol/connection.py    2024-10-04 09:06:52.183148226 +0200
+++ /home/user/.local/lib/python3.10/site-packages/smbprotocol/connection.py    2024-10-04 09:06:23.911091124 +0200
@@ -1155,6 +1155,7 @@
             or not flags.has_flag(Smb2Flags.SMB2_FLAGS_SIGNED)
             or status == NtStatus.STATUS_PENDING
             or command == Commands.SMB2_SESSION_SETUP
+            or not self.require_signing
         ):
             return

I'm not sure if what I am doing is right and whether it should be merged, but seeing that the system smbclient works out of the box, I'd want to see it also work similarly easily with smbprotocol.

import smbclient
smbclient.register_session(
    server="127.0.0.1", username="username", password="password", 
    port=8445, auth_protocol="ntlm", require_signing=False, encrypt=False)
print(smbclient.listdir("127.0.0.1//test/", port=8445))'

This now prints folders successfully. Although, I was confused that I had to respecify the port (and IP) again. Is there some object-oriented API with which I don't have to respecify connection details for each command?

I am also confused about this error message in my initial post:

smbprotocol.exceptions.SMBException: Server message signature could not be verified: 
2ab9616591d29fd57dee6b366f6fb63f != 67d1568b493ad52f9b4f8ee13c60a1fd

It implies to me that there is indeed some signing going on and working but failing.

jborean93 commented 1 month ago

I am also wondering why it works out of the box with the system-insalled smbclient 4.15.13-Ubuntu. Has it signing disabled by default?

smbclient the executable is unrelated to this library, it has different defaults to why this library has. There's a good chance that for backwards compatibility it doesn't require signing by default whereas I've explicitly made signing required by default.

Although, I was confused that I had to respecify the port (and IP) again. Is there some object-oriented API with which I don't have to respecify connection details for each command?

I've had a look at the code and unfortunately it does not seem to be possible, you must always specify the port kwarg if you aren't using the default of 445. It might be a good idea to allow specifying the port in the path like r'\\server:8445\share' but I'll have to think about it a bit more.

I'll have to setup a test server to try out impacket to see why it doesn't work and what needs to be done to allow disabling the signing request.

jborean93 commented 1 month ago

I've opened https://github.com/fortra/impacket/pull/1826 to fix up the mechListMIC side when using the default negotiate auth but need to spend some more time as to why the signature cannot be validated which seems to be unrelated.

jborean93 commented 1 month ago

The invalid signature is due to the Impacket server clearing out the session id in the logoff response packet making our worker fail to find the session. It does this because their code sets the "Uid" (which is the SessionId) to 0 on receiving the logoff request.

https://github.com/fortra/impacket/blob/65b774ded17a79f1041397202852eab0c24cd039/impacket/smbserver.py#L3818

Because this is cleared before the response is built the final value contains 0 in the Session Id field and that session doesn't exist.

image

I also realised that require_signing=False doesn't disable signing, it just doesn't require it. I'm reluctant to try and workaround what I would consider an issue with the server implementation. They should be setting the Session Id properly just like other implementations. But I'll have a think about it and whether I'll add in a workaround to avoid this.

mxmlnkn commented 1 month ago

I'm reluctant to try and workaround what I would consider an issue with the server implementation.

I guess with your explanation one could also file this as a bug in impacket. I just thought that this was a bug here because it worked with smbclient. I'm also not very fixed on impacket. I just needed something for local and CI testing for ratarmount that could be set up quickly and without root privileges. Impacket came up via Google and was much easier to set up than the fully-fledged samba package.

jborean93 commented 1 month ago

So I've done two things on Impacket

jborean93 commented 1 month ago

I've opened https://github.com/jborean93/smbprotocol/pull/291 which adds the workaround for the session id lookup. I've tested it locally but happy if you wish to try it out yourself.

mxmlnkn commented 1 month ago

I've opened #291 which adds the workaround for the session id lookup. I've tested it locally but happy if you wish to try it out yourself.

Thank you so much for fixing this in impacket and smbprotocol!

Setup:

python3 -m pip install --user --force-reinstall \
    'git+https://github.com/jborean93/impacket.git@smb-ntlm-flags#egginfo=impacket'
python3 -m pip install --user --force-reinstall \
    'git+https://github.com/jborean93/smbprotocol.git@impacket-session-di#egginfo=smbprotocol'

mkdir /tmp/sambashare
echo foo > /tmp/sambashare/bar
echo bar > /tmp/sambashare/foo
echo mimi > /tmp/sambashare/mimi

wget 'https://github.com/fortra/impacket/raw/27e7e7478df5d3d3bb12923055a7d8b614825ff4/examples/smbserver.py'
python3 smbserver.py -smb2support -username username -password password -ip 127.0.0.1 -port 8445 test /tmp/sambashare

Access via Python:

import smbclient
server = "127.0.0.1"
port = 8445
smbclient.register_session(
    server=server, username="username", password="password", port=port,
    auth_protocol='ntlm', require_signing=False,
)
print(smbclient.listdir(f"//{server}/test", port=port))
for name in ['mimi', 'foo', 'bar']:
    with smbclient.open_file(f"//{server}/test/{name}", port=port, mode="rb") as file:
        print(f"Name: {name} -> Contents: {file.read()}")

Output:

['mimi']
Name: mimi -> Contents: b'mimi\n'
Name: foo -> Contents: b'bar\n'
Name: bar -> Contents: b'foo\n'

It seems to work mostly, but for some reason, listdir only returns a single file instead of all three? But, the files themselves can be opened.

Test with smbclient:

sudo apt install smbclient
smbclient --user=username --password=password --port 8445 -c ls //127.0.0.1/test
  mimi                               AN        5  Tue Oct  8 11:19:21 2024
  bar                                AN        4  Tue Oct  8 11:17:51 2024
  foo                                AN        4  Tue Oct  8 11:17:54 2024
jborean93 commented 1 month ago

I’ll have to debug the listdir response to see why it’s not showing all three entries. Thanks for confirming the others work for you.

jborean93 commented 1 month ago

It's another bug in Impacket's smbserver, the result does not set the next entry offset value in the query response payload and thus the client thinks there is only 1 result. I've opened https://github.com/fortra/impacket/pull/1831 which fixes the problem for me and the return result contains the expected values.

The smbclient binary is either using a different query info class that is unaffected or they use a different logic for parsing the result payload. In smbprotocol's case we are following the MS specs so I wouldn't want to change the parsing logic.

mxmlnkn commented 1 month ago

Urgh, I'm really sorry about all those bugs originating from a foreign project. I would have thought that SMB was sufficiently specified to avoid such issues between different client and server implementations. I guess smbclient is just too lenient with misbehaving servers. Thank you yet again.

I can confirm that it works with:

python3 -m pip install --user --force-reinstall \
    'git+https://github.com/mxmlnkn/impacket.git@fixes#egginfo=impacket'

Output:

['bar', 'foo', 'mimi']
Name: mimi -> Contents: b'mimi\n'
Name: foo -> Contents: b'bar\n'
Name: bar -> Contents: b'foo\n'
jborean93 commented 1 month ago

That's all good, I think Impacket is more focused on netsec like work so it's not too surprising it has some edge cases. The fact that it works in general as an SMB server is pretty impressive and coming across these bugs is helpful for the entire ecosystem. It's fun to challenge my assumptions and also to fix these things so it's an overall positive :)

mxmlnkn commented 1 month ago

I am hesistant to mention this, but for completeness sake: While my example above still works with all the workarounds, I get the signature failure again when trying smbclient.stat instead of listdir.

import smbclient
server = "127.0.0.1"
port = 8445
smbclient.ClientConfig(auth_protocol='ntlm', require_signing=False)
smbclient.register_session(
    server=server, username="username", password="password", port=port,
    auth_protocol='ntlm', require_signing=False,
)
print(smbclient.listdir(f"//{server}/test", port=port))  # Works
print(smbclient.stat(f"//{server}/test", port=port))  # Error

Output:

['mimi', 'bar', 'foo']
Exception in thread msg_worker-127.0.0.1:8445:
Traceback (most recent call last):
  File "~/.local/lib/python3.12/site-packages/smbprotocol/connection.py", line 1373, in _process_message_thread
    self.verify_signature(header, session_id)
  File "~/.local/lib/python3.12/site-packages/smbprotocol/connection.py", line 1176, in verify_signature
    raise SMBException(
smbprotocol.exceptions.SMBException: Server message signature could not be verified: 2c71c3516494658c9ec0e9be0c2f91b4 != fef82f7840703aa13afe781897a5c043

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.12/threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "~/.local/lib/python3.12/site-packages/smbprotocol/connection.py", line 1409, in _process_message_thread
    self.disconnect(False)
  File "~/.local/lib/python3.12/site-packages/smbprotocol/connection.py", line 957, in disconnect
    self._t_worker.join(timeout=2)
  File "/usr/lib/python3.12/threading.py", line 1144, in join
    raise RuntimeError("cannot join current thread")
RuntimeError: cannot join current thread
Traceback (most recent call last):
  File "/media/f/Beruf/ratarmount/test-smb.py", line 10, in <module>
    print(smbclient.stat(f"//{server}/test", port=port))  # Error
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/smbclient/_os.py", line 600, in stat
    with SMBFileTransaction(raw) as transaction:
  File "~/.local/lib/python3.12/site-packages/smbclient/_io.py", line 260, in __exit__
    self.commit()
  File "~/.local/lib/python3.12/site-packages/smbclient/_io.py", line 296, in commit
    res = func(requests[idx])
          ^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/smbprotocol/open.py", line 1168, in _create_response
    response = self.connection.receive(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/smbprotocol/connection.py", line 1028, in receive
    self._check_worker_running()  # The worker may have failed while waiting for the response, check again
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/smbprotocol/connection.py", line 1184, in _check_worker_running
    raise self._t_exc
  File "~/.local/lib/python3.12/site-packages/smbprotocol/connection.py", line 1373, in _process_message_thread
    self.verify_signature(header, session_id)
  File "~/.local/lib/python3.12/site-packages/smbprotocol/connection.py", line 1176, in verify_signature
    raise SMBException(
smbprotocol.exceptions.SMBException: Server message signature could not be verified: 2c71c3516494658c9ec0e9be0c2f91b4 != fef82f7840703aa13afe781897a5c043
jborean93 commented 1 month ago

The first problem is due to Impacket and how it generates the signatures for a compound response. I've sent a PR to fix up their logic so the correct signatures are generated https://github.com/fortra/impacket/pull/1834. Unfortunately even after fixing that there are still some further errors

Traceback (most recent call last):
  File "/Users/jborean/dev/smbprotocol/client.py", line 20, in <module>
    res = smbclient.stat(r"\\localhost\test", port=8445)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jborean/dev/smbprotocol/src/smbclient/_os.py", line 600, in stat
    with SMBFileTransaction(raw) as transaction:
  File "/Users/jborean/dev/smbprotocol/src/smbclient/_io.py", line 260, in __exit__
    self.commit()
  File "/Users/jborean/dev/smbprotocol/src/smbclient/_io.py", line 296, in commit
    res = func(requests[idx])
          ^^^^^^^^^^^^^^^^^^^
  File "/Users/jborean/dev/smbprotocol/src/smbclient/_io.py", line 209, in _receive_resp
    return query_resp.parse_buffer(info_class)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jborean/dev/smbprotocol/src/smbprotocol/open.py", line 932, in parse_buffer
    file_obj.unpack(buffer)
  File "/Users/jborean/dev/smbprotocol/src/smbprotocol/structure.py", line 121, in unpack
    mem = field.unpack(mem)
          ^^^^^^^^^^^^^^^^^
  File "/Users/jborean/dev/smbprotocol/src/smbprotocol/structure.py", line 219, in unpack
    self.set_value(data[0:size])
  File "/Users/jborean/dev/smbprotocol/src/smbprotocol/structure.py", line 206, in set_value
    parsed_value = self._parse_value(value)
                   ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jborean/dev/smbprotocol/src/smbprotocol/structure.py", line 352, in _parse_value
    int_value = struct.unpack(struct_string, value)[0]
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
struct.error: unpack requires a buffer of 2 bytes

The buffer problem here comes from the FileStandardInformation with the reserved field. This is documented in MS-FSCC FileStandardInformation but Impacket does not set these last two bytes and thus the buffer is 2 bytes short of unpacking. I can certainly add logic to have a "default" value for this case but even with that it won't fix stat with the Impacket server.

This is because Impacket does not support 2 of the file info classes that stat uses (FileBasicInformation and FileAttributeTagInformation) so even when fixing our client to properly parse that response the function will fail later on with:

Traceback (most recent call last):
  File "/Users/jborean/dev/smbprotocol/client.py", line 20, in <module>
    res = smbclient.stat(r"\\localhost\test", port=8445)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jborean/dev/smbprotocol/src/smbclient/_os.py", line 600, in stat
    with SMBFileTransaction(raw) as transaction:
  File "/Users/jborean/dev/smbprotocol/src/smbclient/_io.py", line 260, in __exit__
    self.commit()
  File "/Users/jborean/dev/smbprotocol/src/smbclient/_io.py", line 349, in commit
    raise failures[0]
smbprotocol.exceptions.SMBOSError: [Error 0] [NtStatus 0xc00000bb] Unknown NtStatus error returned 'STATUS_NOT_SUPPORTED': '\\localhost\test'

I'll add in the logic to handle a "default" value for this particular buffer but until Impacket implements those 2 info classes (and our bugfixes) it won't work with the stat call.

An alternative to this is to use the scandir function and use the properties in the smb_info attribute:

for entry in smbclient.scandir(r"\\localhost\test", port=8445):
    print(entry.smb_info)

This is supported by Impacket and is more efficient than listing the directory with listdir then calling stat on each entry and the listing and getting the attributes are done in one call. Keep in mind entry.stat() or any of the other methods which also calls stat() internally will still fail.

jborean93 commented 1 month ago

Re-opening to deal with FileStandardInformation and the smaller buffer response

jborean93 commented 1 month ago

Ok to sum up this issue: