register_connection fails in some conditions when netname_id is set to the IP of the samba server

I have a docker container that has both python3/smbprotocol and samba in it. I updated it recently (last update was back in June 2021) and the nature by which the connection sets up has broken.

I fire up 2 containers, one running my python script and the other running the samba service. The first container connects to the second by finding its IP from docker (using docker inspect) and this has worked up until now.

This is the minimal code to reproduce by running from the first container (which is not running the smbd service):

root@9d969100be68:/opt# python3 -c 'import smbclient; smbclient.register_session("192.168.80.9", "simulator", "simulator"); print(smbclient.listdir("\\192.168.80.9\data"))'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/smbclient/_pool.py", line 374, in register_session
    connection.connect(timeout=connection_timeout)
  File "/usr/local/lib/python3.6/dist-packages/smbprotocol/connection.py", line 799, in connect
    smb_response = self._send_smb2_negotiate(dialect, timeout, enc_algos, sign_algos)
  File "/usr/local/lib/python3.6/dist-packages/smbprotocol/connection.py", line 1497, in _send_smb2_negotiate
    response = self.receive(request, timeout=timeout)
  File "/usr/local/lib/python3.6/dist-packages/smbprotocol/connection.py", line 1006, in receive
    raise SMBResponseException(response)
smbprotocol.exceptions.InvalidParameter: Received unexpected status from the server: An invalid parameter was passed to a service or function. (3221225485) STATUS_INVALID_PARAMETER: 0xc000000d

I ran this from the container that the samba smbd service is running in and it works fine:

root@509d41ece260:/opt# python3 -c 'import smbclient; smbclient.register_session("127.0.0.1", "simulator", "simulator"); exit; print(smbclient.listdir("\\127.0.0.1\data"))'
['.simulator', '1_2021112320441637700279_SomeUser', '1_2021112321081637701722_SomeUser', '2_2021112321311637703061_SomeUser', '1_2021112322441637707480_SomeUser', '2_2021112916311638203503_SomeUser', '
9_2021112923041638227087_SomeUser', '1_2021120219011638471682_SomeUser', '1_2021120219331638473588_SomeUser', '1_2021120319231638559401_SomeUser', '1000_2021120323151638573310_SomeUser', '1001_20211203
23191638573577_SomeUser', '1_2021120616351638808504_SomeUser', '1_2021121717561639763815_SomeUser', '1002_2021122120281640118531_SomeUser', '1002_2021122120501640119801_SomeUser', '1002_202112212103164
0120604_SomeUser', '1002_2021122121141640121277_SomeUser']

The two docker containers are based on the same image, which means the smbprotocol version and python environment is identical between the two.

I did a packet capture and noticed that the SMB2_NETNAME_NEGOTIATE_CONTEXT_ID is being set to the server value passed to register_connection. When it's set to the IP address 192.168.80.9 the connection fails with that STATUS_INVALID_PARAMETER, but when it's set to 127.0.0.1 it succeeds. I used socat from the first container to forward localhost port 445 to the second container's IP and tried connecting to 127.0.0.1 again in the first container and it succeeded.

So next, I did another packet capture this time using samba's own smbclient and noticed it doesn't set the SMB2_NETNAME_NEGOTIATE_CONTEXT_ID negotiation parameter at all... so I commented out the netname_id field from the _send_smb2_negotiate method of connection.py (around line 1490) from the neg_req['negotiate_context_list'] list. This fixed my connection issues and I've added a hack to my Dockerfile to fix my situation temporarily:

RUN find /opt/conda /usr/lib/python3.6/dist-packages/smbprotocol -name connection.py | grep smbprotocol | xargs sed -i 's/netname_id,/#netname_id,/'

(I use both Ubuntu's Bionic and conda's python3 so I patch them both)

I noticed issue #89 is similar to this, but may not actually be (in that issue the authentication type was forced to ntlm which seemed to fix their connectivity issue).

Hmm that's interesting, the SMB2_NETNAME_NEGOTIATE_CONTEXT_ID field is meant to be used as an identifier for things like load balancers or other network inspection tools as a way to identify the true target of the request. It's not meant to be parsed by the SMB server in the negotiation phase. It even says that it shouldn't be processed on the server side in the MS-SMB2 docs https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/b39f253e-4963-40df-8dff-2f9040ebbeb1

For each context in the received NegotiateContextList, if the context is SMB2_NETNAME_NEGOTIATE_CONTEXT_ID or any negotiate context other than SMB2_PREAUTH_INTEGRITY_CAPABILITIES, SMB2_COMPRESSION_CAPABILITIES, SMB2_RDMA_TRANSFORM_CAPABILITIES, SMB2_SIGNING_CAPABILITIES, SMB2_TRANSPORT_CAPABILITIES, or SMB2_ENCRYPTION_CAPABILITIES, the server MUST ignore the negotiate context.

While I don't doubt that that this negotiate context is causing your issue, especially since commenting it out gets it working again, I'm not sure what the proper next steps should be. To me it looks like a bug on the Samba side where it's either processing this context in a particular way, or it's failing because it has a strict set of contexts that it can receive. If either of these scenarios is the case then the proper fix would be to have Samba update their code. Failure to do so will result in other client that send this context id (newer Windows hosts) failing to negotiate with that server. Unfortunately this would require some more investigation before coming to a final conclusion.

So next, I did another packet capture this time using samba's own smbclient and noticed it doesn't set the SMB2_NETNAME_NEGOTIATE_CONTEXT_ID negotiation parameter at all

The reason why smbclient doesn't send this is because it's a fairly recent addition to the SMB protocol so not many clients have been updated to start sending it. At least on Windows you need Windows 10 1903 or Server 2022 for it to start being sent by those clients.

I noticed issue #89 is similar to this, but may not actually be (in that issue the authentication type was forced to ntlm which seemed to fix their connectivity issue).

In that particular case the failure was happening on the next phase of the connection. It was also reported before I added the netname context (October 2021) so it wouldn't be related. Unfortunately STATUS_INVALID_PARAMETER is a nice catch all for the server received something it didn't like. It doesn't really help narrow down what exactly that was :(

Thanks for the excellent investigation work, it's been very helpful trying to narrow down what the problem is. Is it possible to share some more information so I can hopefully replicate it on my end? I'm hoping to find out:

What the Samba version is on your container
Potentially what Linux distro the container is based on and how it's installing Samba

If I can install Samba at the same version you are on then hopefully I can replicate your problem and potentially track down the bug in the code somewhere.

Ah, so you mean really recent! Yeah, I agree it's probably a hard solve.

Could I suggest making these "optional" connection parameters optional by an argument to the register_connection method? Let the software decide whether anything more than the base protocol features are necessary so that there's a path moving forward for situations like this? I know that's full of caveats too, and almost certainly could be a footgun at some point where features are just not available due to copy/pasta coding...

As for the versions of the software involved:

root@2180bff0cf11:/opt# smbd --version
Version 4.7.6-Ubuntu
root@2180bff0cf11:/opt# conda --version
conda 4.11.0
root@2180bff0cf11:/opt# python3 --version
Python 3.9.5
root@2180bff0cf11:/opt# grep Version /opt/conda/lib/python3.9/site-packages/smbprotocol-1.8.3.dist-info/METADATA
Metadata-Version: 2.1
Version: 1.8.3
root@2180bff0cf11:/opt# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.6 LTS"

Here's the relevant sections from my Dockerfile:

FROM continuumio/miniconda3:4.10.3 AS miniconda_image

FROM ubuntu:bionic AS base

RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y samba smbclient

# build environment #
ENV LANG=C.UTF-8 LC_ALL=C.UTF-8
ENV PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

# install miniconda #
COPY --from=miniconda_image /opt/conda /opt/conda
# dependencies #
RUN conda install pip && \
  pip install smbprotocol
# hacky patch to disable netname_id in smb protocol negotiation: https://github.com/jborean93/smbprotocol/issues/158 #
RUN find /opt/conda -name connection.py | grep smbprotocol | xargs sed -i 's/netname_id,/#netname_id,/'

# setup samba for simulator #
COPY smb.conf /etc/samba
COPY smb_usermap.txt /etc/samba/usermap.txt
RUN (echo simulator; echo simulator) | smbpasswd -as root

VOLUME /instrument

# run samba server #
CMD /usr/bin/smbd -F

My smb.conf looks like:

[global]
   workgroup = WORKGROUP
   syslog = 0
   server role = standalone server
   passdb backend = tdbsam
   obey pam restrictions = yes
   map to guest = bad user
   usershare allow guests = no
   username map = /etc/samba/usermap.txt

[data]
   path = /instrument
   browseable = yes
   read only = no
   valid users = root @users

The usermap.txt is:

root = minit

Thanks for your response and attention!

I cannot seem to replicate this on my end, either by using your exact image or my own custom setup. Both a request from a Windows Server 2022 image and also from my localhost to the docker image work just fine. I even set up a fake hostname alias to test out when the value is completely unrelated to the Samba container.

Even when testing your setup of communication between the containers I found it still passed the negotiation step when an IP was in the netname context

I honestly cannot imagine why my setup is working but yours fails. Are you sure your Samba host is the one that's returning an error and nothing else is responding? One thing I did note was your Dockerfile wouldn't work for me as it was. I had to change the CMD value to /usr/sbin/smbd -F as that binary wasn't in bin. I don't know why yours would still work but it's just something interesting I found when testing all this out.

Hey, I apologize for the typo in the /usr/sbin/smbd, I typed from memory as I tried to minimalize the Dockerfile (it has a lot of extra software packages, and a bunch of specific stuff to the software that I'm working with in it.

Unfortunately I threw away the image that was affected in favor of the patched container... when I remove the patch, it continues to work. I tried to re-create the conditions and bring in all the software from both apt and pip to see if it was something conflicting there, and I cannot get this to reproduce. All the versions look as they were when I grabbed the versions yesterday.

I'm baffled and extraordinarily apologetic as I know these issues are already pretty hard to reproduce and troubleshoot.

Glad to hear it's working again for you. I can't explain why it started working for you, looking at the changelog there have been some recent changes to the package but nothing that seems related or was in the last 2 or so days. Considering it's now working for you I'll close off the issue. If it does appear again I'm happy to reinvestigate to try and understand it a bit better.

I'm baffled and extraordinarily apologetic as I know these issues are already pretty hard to reproduce and troubleshoot.

That's all good, the information you've shared has been great and helps to make tracking down these issues. If anything it's yet another thing to keep in mind if I start seeing more reports around this and I now have something more to start with.

jborean93 / smbprotocol

register_connection fails in some conditions when netname_id is set to the IP of the samba server #158