STATUS_ACCESS_DENIED when write by dfs path before read

[mrmaxi]

continue #85 If write file by DFS path that not in referall cache yet, it cause STATUS_ACCESS_DENIED exception on my environment. If I read file first - all successful. If I try write after cache is expired it cause STATUS_ACCESS_DENIED too.

For example:

import smbclient
import time
smbclient.open_file(r'\\mycompany.com\DFS\share\1.TXT').read()                       # file 1.TXT need exist
smbclient.open_file(r'\\mycompany.com\DFS\share\2.TXT', 'w').write('123456')         # all right, 123456 > 2.TXT
time.sleep(61)
smbclient.open_file(r'\\mycompany.com\DFS\share\2.TXT', 'w').write('777777')         # cause STATUS_ACCESS_DENIED exception

As we known in #85: For unknown reasons (maybe version of Windows Server), requests with different desired_access (mode: write instead of read) for DFS-share by original path don't cause PathNotCovered, that trigger _resolve_dfs, so it cause STATUS_ACCESS_DENIED

Now i allways read first before write, but in several case it cause exception still. It happens because sometimes cache is expired exactly before i write, but after read.

Suggested solution: remember that path if DFS if it discovered once, and than when referral was expired only refresh it.

[jborean93]

I've commented on this a bit more under https://github.com/jborean93/smbprotocol/pull/171#issuecomment-1218909187.

Ultimately this seems to be a problem with the DFS server not returning the expected error codes as part of the DFS protocol specification. There are workarounds in place, like setting smbclient.ClientConfig(domain_controller='...') to ensure smbclient knows the requested path is a DFS path rather than trying to do it after sending the request that fails. For this particular case, once a referral has expired it should no longer be considered which is what is happening.

[mrmaxi]

Hello Jordan. Nice to hear you! Offcourse it's a problem with my DFS server, I know it, but it remains (.

But you are wrong, there is no workarounds. I use smbclient.ClientConfig(domain_controller='...') but it doesn't help in this case (in different of #85, where it helps). I'm sorry i wasn't say it at first during writing the issue. But in this case the behaviour is the same with smbclient.ClientConfig(domain_controller='...') and without both.

But i think that I found the problem! smpclient._pool.get_smb_tree line 297:

        if domain_referral:
            # Use the dc hint as the source for the root referral request
            ipc_tree = get_smb_tree(u"\\%s\\IPC$" % domain_referral.dc_hint, **get_kwargs)[0]
            referral_response = dfs_request(ipc_tree, "\\%s\\%s" % (path_split[0], path_split[1]))

in my case path_split[0], path_split[1] = \\mycompany.com\\DFS and cause STATUS_ACCES_DENIED later but if i try *path_split[0], path_split[1]*, path_split[2] = \\mycompany.com\\DFS\\share that really stored at host mydfshost.mycompany.com\share - it work fine!

get_smb_tree(\\mycompany.com\\DFS) -> mydfsroot.mycompany.com get_smb_tree(\\mycompany.com\\DFS\share) -> mydfshost.mycompany.com

[jborean93]

But in this case the behaviour is the same with smbclient.ClientConfig(domain_controller='...') and without both.

Ah my apologies, I assumed this was still the same problem as before.

I guess at this point I need to try and understand your environment a bit better because I do not understand how the current code would fail. It is meant to first enumerate the root DFS servers by asking for DFS server with the path \mycompany.com\DFS. From there it should try and open \\{referral_response.server}\share\1.TXT and if share is a DFS link to another folder it will fail with STATUS_PATH_NOT_COVERED. This error has the client then send another DFS request but now for \domain.test\DFS\share\1.TXT which returns the correct root path of \\mydfshost.mycompany.com\share\1.TXT and the subsequent connection works.

To replicate this scenario I have a domain called domain.test, in there I have a DFS namespace of DFS and inside that namespace I have a link server2022 that points to \\server2022.domain.test\c$.

This is what happens when I run

import smbclient

smbclient.ClientConfig(domain_controller="dc01.domain.test")

with smbclient.open_file(r"\\domain.test\DFS\server2022\temp\test.txt", mode="w") as fd:
    fd.write("test")

• ClientConfig(domain_controller='dc01.domain.test') will have the client send a DFS request for "" to \\dc01.domain.test\IPC$
  • This returns \DOMAIN and \domain.test
• The client sees the file to open is in \domain.test and sends a DFS request for "\domain.test" to \\dc01.domain.test\IPC$ to enumerate the domain controllers in the domain and select the best one
  • This returns \DC01.domain.test which is correct as I only have 1 DC
• The client then sends a DFS request for "\domain.test\DFS" to \\DC01.domain.test\IPC$ to get a list of DFS root servers
  • This returns \server2019.domain.test\dfs which is correct as it's my only root DFS server
• The client rewrites the path so it's now trying to connect to \\server2019.domain.test\dfs\server2022\temp\test.txt
• This request fails with STATUS_PATH_NOT_COVERED because server2022 isn't on the DFS root but linked to another server
• The client sends a DFS request for "\domain.test\DFS\server2022\temp\test.txt" to \\server2019.domain.test\IPC$ to get the link target
  • This returns \server2022.domain.test\c$
• The client rewrites the path yet again so it's now trying to connect to \\server2022.domain.test\c$\temp\test.txt
• This works and can write the file

So while your workaround to adjust the DFS Request to do \\mycompany.com\DFS\share gives you the proper result it shouldn't be necessary as the request to open the file on the DFS root initial should have resulted in STATUS_PATH_NOT_FOUND rather than STATUS_ACCESS_DENIED.

If you have enough permissions you should be able to get a view of the DFS namespace of your environment and see if it matches up with my test environment or whether I'm missing something.

Feel free to send through the latest Wireshark capture from a scenario that replicates your problem and one where the workaround fixes it.

[mrmaxi]

I'l try to describe my environment. We have 3 dc: node2, node3, node4 We have 5 dfs root: node2, node3, node4, fs-u40, fs-p24 All servers are Windows Server 2012 R2 ( Dfs path \mycompany.com\DFS\Report$ lead to target \fs-u40\Report$ I pass domain_controller - node4:

import smbclient

smbclient.ClientConfig(domain_controller="node4.mycompany.com")

smbclient.open_file(r"\\mycompany.com\DFS\Report$\1.txt", mode="w").write("123456")

When I try to require READ access all work exactly as you wrote. But when I try to require WRITE access, my "broken" dc (i try every, node2/node3/node4 - result the same) return STATUS_ACCESS_DENIED instead of STATUS_PATH_NOT_COVERED

• The client then sends a DFS request for "\mycompany.com\DFS" to \node4.mycompany.com\IPC$ to get a list of DFS root servers It returns refferal:

  dfs_path='\\mycompany.com\\DFS'
  target_hint.target_path='\\FS-U40\\DFS' (or \\NODE4\DFS or \\NODE2\DFS ...)

• The client rewrites the path so it's now trying to connect to \\FS-U40\DFS\Report$\1.txt
• The client register_session to '\FS-U40\DFS', and get_smb_tree return tree \\FS-U40\DFS and file_path='Report$\1.txt'
• when it try to open this file with desired write access request fails with STATUS_ACCESS_DENIED

[jborean93]

Thanks, I’ll try and set up a domain environment with Server 2012 R2 and see if I can get the same problem. Does it work if you try opening a write fd on either host directly and not through DFS?

[mrmaxi]

Yes, if i write directly to server share without using DFS everything works fine:

import smbclient

smbclient.ClientConfig(domain_controller='node4.mycompany.com',)

smbclient.open_file(r'\\fs-u40\Report$\1.TXT', 'w').write('123456')

If I add into smbclient._pool.get_smb_tree after DFS root resolution request to DFS root host for referral to dfs end share:

    if len(path_split) > 2:
        # Use dfs root referral hint as source for the end share request
        ipc_tree = get_smb_tree(u"\\%s\\IPC$" % referral.target_hint.target_path, **get_kwargs)[0]
        referral_response = dfs_request(ipc_tree, "\\%s\\%s\\%s" % (path_split[0], path_split[1], path_split[2]))
        client_config.cache_referral(referral_response)
        referral = client_config.lookup_referral(path_split)
        if not referral:
            raise ObjectPathNotFound()

everything works!

What do you think about it? I know about dfs not enough. Can shares be connected not to dfs root but to other shares connected to dfs root?

[mrmaxi]

Some additions.

if I require file from destination server (it is dfs root in addition) FS-U40 by using DFS name: \\fs-u40\DFS\Report$\1.TXT

it doesn't work at all

import smbclient
smbclient.ClientConfig(domain_controller='node4.mycompany.com')
smbclient.open_file(r'\\fs-u40\Report$\1.TXT', 'r').read()                 # cause exception 'NoneType' object is not iterable
smbclient.open_file(r'\\fs-u40\Report$\1.TXT', 'w').write('123456')        # cause STATUS_ACCESS_DENIED

Traceback (most recent call last):
  File "ttt.py", line 9, in <module>
    smbclient.open_file(r'\\fs-u40\DFS\Report$\2.TXT', 'r')
  File "\lib\site-packages\smbclient\_os.py", line 372, in open_file
    raw_fd.open()
  File "\lib\site-packages\smbclient\_io.py", line 468, in open
    transaction.commit()
  File "\lib\site-packages\smbclient\_io.py", line 327, in commit
    for smb_open in _resolve_dfs(self.raw):
  File "\lib\site-packages\smbclient\_io.py", line 162, in _resolve_dfs
    for target in info:
TypeError: 'NoneType' object is not iterable

Case with READ access isn't suprising because client_config.cache_referral in _io.py doesn't return anything, and row with lookup_referral is commented:

158:    info = client_config.cache_referral(referral)
159:    # info = client_config.lookup_referral([p for p in raw_path.split("\\") if p])
160:    connection_kwargs = getattr(raw_io, '_%s__kwargs' % type(raw_io).__name__, {})
161:
162:    for target in info:

if I uncomment row 159 with lookup_referral (or add return in cache_referral) - read from \\fs-u40\DFS\Report$\1.TXT works fine.

And then everything as expected:

• if I read first then write - works
• if i write first - STATUS_ACCESS_DENIED

[jborean93]

What do you think about it?

I understand why it might work for your use case but I need to look further into the DFS specs to see what should be done. I'll also have to see what Windows does in cases like this.

For that NoneType object is not iterable are you running the latest version of smbprotocol (1.9.0)? It should have been fixed with https://github.com/jborean93/smbprotocol/pull/149. Also are you saying if you connect directly to \\fs-u40\DFS\Report$\1.TXT (with your commented changed) with write it fails with STATUS_ACCESS_DENIED? Does fs-p42 also fail when connecting directly?

[mrmaxi]

I was sure that use 1.9.0, but it seems like I wasn't. Now I reinstall 1.9.0 from source and there isn't error with NoneType. But the rest is the same: write access first cause STATUS_ACCESS_DENIED.

smbclient.open_file(r'\\mycompany.com\DFS\Report$\1.TXT', 'w').write('123456')      # STATUS_ACCESS_DENIED
smbclient.open_file(r'\\fs-u40\DFS\Report$\1.TXT', 'w').write('123456')             # STATUS_ACCESS_DENIED
smbclient.open_file(r'\\fs-p24\DFS\Report$\1.TXT', 'w').write('123456')             # STATUS_ACCESS_DENIED
smbclient.open_file(r'\\fs-u40\Report$\1.TXT', 'w').write('123456')                 # pass
smbclient.open_file(r'\\fs-p24\Report$\1.TXT', 'w').write('123457')                 # pass

All work If I execute next code in cmd:

copy 1.txt \\fs-p24\DFS\Report$

[mrmaxi]

As we see in https://docs.microsoft.com/en-us/windows-server/storage/dfs-namespaces/dfs-overview it is possible to exist several levels of folders inside dfs namespace root with folder-targets inside.

I offer next solution: Change SMBFileTransaction.commit to resolve_dfs without waiting STATUS_PATH_NOT_COVERED if we realize that current tree is dfs share, because after resolving it'l be a direct share to target server

_io.py

318:                 try:
319:                     # if tree is dfs share, resolve it without waiting STATUS_PATH_NOT_COVERED
320:                     if self.raw.fd.tree_connect.is_dfs_share:
321:                         raise PathNotCovered()
322:                     res = func(requests[idx])

[jborean93]

I still cannot replicate this problem even after setting up a test environment with Server 2012 R2 only. While potentially we could have a check in there to do a referral request for the full link this isn't what is meant to happen, or even what I see happen on Windows. Using Referral Process for Domain-based Namespaces as the workflow of what's meant to happen

1. A user attempts to access a link in a domain-based namespace, such as by typing \Contoso.com\Public\Software in the Run dialog box.

In this case you are trying to access \\mycompany.com\DFS\Report$\1.TXT

2. The client checks its domain cache for an existing domain name referral for the Contoso.com domain. If this referral is in the domain cache, the client proceeds to step 3. If no domain name referral is in the domain cache, the client connects to the IPC$ shared folder of the active domain controller in the context of the LocalSystem account and sends a domain name referral request with a null string. The domain controller returns the list of local and trusted domains to the client.

If smbclient.ClientConfig(domain_controller='...') is set there should be an entry for mycompany.com that tells the client this path is a DFS path and will continue onto step 3 as normal.

3. The client checks its referral cache for the longest prefix match from a previous referral. This might be a root referral (\Contoso.com\Public) or a link referral (\Contoso.com\Public\Software). If the client finds a valid entry, it goes to step 5 or 6 depending on the type of the referral. If not, the client continues to the next step.

Right now there is no referral cache that will match so will continue onto step 4.

4. The client checks its domain cache for an existing domain controller referral for the Contoso.com domain. If this referral is in the cache, the client proceeds to step 5. If no domain controller referral is in the domain cache, the client connects to the IPC$ shared folder of the active domain controller in the context of the LocalSystem account and sends a domain controller referral request containing the appropriate domain name (Contoso.com). The domain controller returns the list of domain controllers in the Contoso.com domain. The domain controllers in the clients site are at the top of the list. If least-expensive target selection is enabled, domain controllers outside of the targets site are sorted by lowest cost. If same-site target selection is enabled, DFS ignores this setting and lists the remaining domain controllers in random order.

For the first call the client knows about mycompany.com but it doesn't know the list of domain controllers for that domain. It will connect to the IPC$ share of the configured domain controller and send a domain controller referral to enumerate the domain controllers of the domain. Once done it moves onto step 5.

5. The client checks its referral cache for an existing domain-based root referral. If this referral is in the cache, the client proceeds to step 6. If no domain-based root referral exists in the referral cache, the client connects to the IPC$ shared folder of the active domain controller in the context of the LocalSystem account and requests a domain-based root referral. The domain controller determines the clients site and returns a list of root targets. By default, the root targets in the clients site are at the top of the list, followed by the remaining root targets in random order. If least-expensive target selection is enabled, the remaining root targets are ordered by lowest cost. If same-site target selection is enabled, only root servers in the clients site are listed in the referral.

On the first request there will be no cached referrals so will once again connect to the IPC$ share of the domain controller and ask for a list of DFS root servers that can service requests for \\mycompany.com\DFS. This is the code you see below and is the reason why the request only asks for \\mycompany.com\DFS as it's just trying to get the root servers not resolve the full path.

https://github.com/jborean93/smbprotocol/blob/73d0d00e6948b57e3927d428bf6fc020b6102b0a/src/smbclient/_pool.py#L289

It uses this root DFS server to know what to translate \\mycompany.com\DFS to for future requests to the DFS namespace.

6. The client chooses the first root target in the domain-based root referral. The client connects to the root server and navigates the subfolders under the root folder. When a client encounters a link folder, the root server sends a STATUS_PATH_NOT_COVERED status message to the client, indicating that this is a link folder that requires redirection.

This is the problem here, the client is connecting to \\dfs-root-server\DFS\Report$\1.TXT but instead of returning STATUS_PATH_NOT_COVERED as expected it's returning STATUS_ACCESS_DENIED. This is the part that is confusing me as it's going against all documentation I can find on DFS, even from Microsoft themselves. The MS-DFSC (DFS specific protocol) doesn't event mention this error code at all. Getting back STATUS_PATH_NOT_COVERED means the client will do a DFS referral request to resolve the link target but because it's not that error code the client is returning back access denied.

The crux of this problem is figuring out why this is happening as I don't wish to blindly add a fix for something I do not understand. Even when testing this particular scenario I see Windows acting the same way as smbclient does. For example I run the following to clear all the caches and then write a file on a DFS share

dfsutil cache domain flush
dfsutil cache referral flush
dfsutil cache provider flush
Set-Content \\smb.test\dfs\test\test.txt text

Clearing the domain cache will result in the client doing a DFS referral request for "" (domain referral request that returns \SMB and \smb.test

Accessing \smb.test\DFS\* will have the client do a DFS root referral request for \smb.test\dfs to figure out the root DFS servers, which in this case is simply \SERVER2019.smb.test\DFS (ignore the name this is actually a Server 2012 R2 host).

Now the client knows the DFS root server for \\smb.test\DFS it will connect to that tree \\SERVER2019.smb.test\DFS.

It is connected to the tree and will try and access the file directly on the root DFS server but it returns STATUS_PATH_NOT_COVERED. One thing I have noticed is that Windows uses a special filename and a flag that denotes this being a DFS operation. The smbclient does not do that but even if the filename is a normal request like smbclient does it still returns STATUS_PATH_NOT_COVERED. Maybe this could be the cause but I don't know why I can't replicate it the error right now with a similar environment.

Because of the STATUS_PATH_NOT_COVERED error the client connects to the IPC$ tree of the DFS root server and issues a DFS link referral request to resolve the full path.

Through this info it now knows the final path to connect to is \\dfshost2.smb.share\share\test.txt which is connects to successfully

The only difference between what smbclient does and Windows is the initial request for \\dfs-root\DFS\Report$\1.TXT. On Windows it flags the header to include SMB2_FLAGS_DFS_OPERATIONS and the filename is mycompany.com\DFS\Report$\1.TXT and not just Report$\1.TXT that smbclient does. It is possible this is the cause of the problem but even with this different behaviour I cannot see the access denied error with smbclient in a similar environment.

I'm at a loss as to what to try, I'll see if I can try and change the code to do the initial request in a similar way to see if it fixes your problem but until then I don't really have anything else to try or recommend. The only other thing you could do is create 2 separate network captures for doing the smbclient way that fails and the Windows way that works (including the dfsutil cache ... flush commands above.

[jborean93]

The changes in https://github.com/jborean93/smbprotocol/pull/190 change the behaviour to mark an open operation on a DFS share like Windows does. It would be great if you could test it out and see if it helps in your situation in any way.

[mrmaxi]

The changes in #190 change the behaviour to mark an open operation on a DFS share like Windows does. It would be great if you could test it out and see if it helps in your situation in any way.

It works perfect now! For full path request dfs root server returns STATUS_PATH_NOT_COVERED and smbclient resolves the dfs link by standard way.

smbclient.open_file(r'\\mycompany.com\DFS\Report$\1.TXT', 'w').write('123456')    # pass
smbclient.open_file(r'\\fs-u40\DFS\Report$\1.TXT', 'w').write('123456')               # pass
smbclient.open_file(r'\\fs-p24\DFS\Report$\1.TXT', 'w').write('123456')               # pass
smbclient.open_file(r'\\fs-u40\Report$\1.TXT', 'w').write('123456')                   # pass
smbclient.open_file(r'\\fs-p24\Report$\1.TXT', 'w').write('123457')                   # pass
smbclient.open_file(r'\\node4\DFS\Report$\1.TXT', 'w').write('123456')                # pass

Thank you a lot!

[jborean93]

Awesome, still no idea why I'm not seeing the problem but I'm at least glad it's now working for you. I'll try and fix up the rename operation stuff in that PR as the current logic isn't liking what I've changed but at least I know what the problem is here.

[mrmaxi]

Small remark about #190 - I was fail when try to install it through python setup.py install because it create UNKNOWN-0.0.0-py3.8.egg I just copy smbclient and smbprotocol folders into my site-packages to test it

[jborean93]

You are better off using pip to install the package from source, python -m pip install . or python -m pip install -e . (if you want an editable install). This makes your installation step agnostic around the build/packaging tool used and rely on the pyproject.toml metadata. It also future proofs you when installing packages that aren't based on setuptools. You can even generate the sdist and/or wheel directly if you have the build package installed by doing python -m build when in the project dir.

I've also just pushed one extra change around the rename operations. It would be good to see if the rename work you were originally trying to do works or does not with the changes there.

[mrmaxi]

I'cant remember what's the problem with rename operations.

smbclient.rename(r'\\mycompany.com\DFS\Report$\1.TXT', r'\\mycompany.com\DFS\Report$\2.TXT')

It works excelent in 1.9.0 and 1.10.0 both

[jborean93]

No worries, as long as it doesn't fail due to the changes in #190 then I'm happy. Thanks for sticking through with this one and sharing all the information you could.

Fixed by https://github.com/jborean93/smbprotocol/pull/190

[mrmaxi]

raname with dst_raw.fd.file_name work rename with dst_raw._fd_path fail with:

Traceback (most recent call last):
  File "C:/APPS/ttt.py", line 9, in <module>
    smbclient.rename(r'\\mycompany.com\DFS\Report$\1.TXT', r'\\mycompany.com\DFS\Report$\2.TXT')
  File "c:\apps\smbprotocol\src\smbclient\_os.py", line 485, in rename
    _rename_information(src, dst, replace_if_exists=False, **kwargs)
  File "c:\apps\smbprotocol\src\smbclient\_os.py", line 1135, in _rename_information
    set_info(transaction, file_rename)
  File "c:\apps\smbprotocol\src\smbclient\_io.py", line 258, in __exit__
    self.commit()
  File "c:\apps\smbprotocol\src\smbclient\_io.py", line 339, in commit
    raise failures[0]
smbprotocol.exceptions.SMBOSError: [Error 2] [NtStatus 0xc000003a] No such file or directory: '\\mycompany.com\DFS\Report$\1.TXT'

[jborean93]

https://github.com/jborean93/smbprotocol/pull/191 modified it yet again after testing things out a bit more. It will continue to use dst_raw.fd.file_name but remove the DFS prefix if it's a DFS share and set the required header flag.

[mrmaxi]

Thanks, now it works fine at least in my environement!