jborean93 / smbprotocol

Python SMBv2 and v3 Client
MIT License
318 stars 73 forks source link

Randomly getting STATUS_INVALID_DEVICE_REQUEST for every smbclient calls #154

Closed Wolfcast closed 2 years ago

Wolfcast commented 2 years ago

I use your library to connect to a Windows shared drive and everything works fine for a couple of hours and then everything stop working and all calls (open_file, remove, stat...) raise an SMBOSError with STATUS_INVALID_DEVICE_REQUEST:

[Error 0] [NtStatus 0xc0000010] Unknown NtStatus error returned 'STATUS_INVALID_DEVICE_REQUEST': '\\my.server.domain.com\Apps\AA\MyApp\My_App_DEV_Docs\test.txt'

I tried anything I could think of (sending credentials at each request instead of register_session, always deleting the session before registering, encryption, no encryption, smbclient.ClientConfig(require_secure_negotiate=False)...) but nothing helps.

What can cause this erratic behaviour? Are there logs anywhere with more details?

jborean93 commented 2 years ago

I can’t say I’ve seen this behaviour before so I’m not sure what exactly it could be. Usually just resetting the connection is enough to workaround these types of problems until the root cause is found.

I would first look into the SMB server logs on the endpoint to see why it might be returning that status code. Do you find restarting the process is enough to get it working again or does it require more work. Do you know if these endpoints are behind a DFS server?

Wolfcast commented 2 years ago

And how do you reset the connection usually?

I will try to have the logs checked on the server but it's another team that manage that and it will be tough to have them look that for me :)

I found no solution to have it work again. It just start to work again by itself... I will try to get more information and get back.

jborean93 commented 2 years ago

And how do you reset the connection usually?

Running smbclient.reset_connection_cache() is enough to clear out the connections the client has altogether. The only thing that is never really reset in this case is the client GUID that helps the server identify common clients. This could indicate the server is blocking all future connections to that GUID itself. You can do the following to rule out this scenario

import uuid

import smbclient

... Do work until it fails

smbclient.reset_connection_cache()

# Override the client GUID to pretend to be a new client
smbclient.ClientConfig(client_guid=uuid.uuid4())

... See if it starts working again

Starting a brand new process will ensure everything is fresh as well and will rule out whether it's something linked to the process or some server end problem.

When reading through the MS-SMB2 protocol docs I can only find references to STATUS_INVALID_DEVICE_REQUEST when it comes to the server processing IOCTL requests. This is not really a standard operation so I'm unsure what would be the cause of it in your case. I'm afraid without server side logs debugging this is going to be difficult.

Wolfcast commented 2 years ago

Just a quick update since I'm still with my team on this issue. I learned that the UNC path that I was accessing via SMB is under a DFS (I don't have more details). Could it be the reason? BTW I added an auto retry feature to my calls and before the last try I do a reset like shown above. It help a bit (I think) but still sometinmes I get STATUS_INVALID_DEVICE_REQUEST.

Do I have something special to do if under a DFS?

jborean93 commented 2 years ago

Could it be the reason?

Most likely, as I mentioned in the previous comment the only time I could find references to a server returning STATUS_INVALID_DEVICE_REQUEST is in an IOCTL response. DFS referrals are done using IOCTL so it looks like it's a problem happening with the DFS referral process. Specifically the MS-SMB2 docs for processing an IOCTL request docs states

The server SHOULD<353> fail the request with STATUS_NOT_SUPPORTED when an FSCTL is not allowed on the server, and SHOULD<354> fail the request with STATUS_INVALID_DEVICE_REQUEST when the FSCTL is allowed, but is not supported on the file system on which the file or directory handle specified by the FSCTL exists, as specified in [MS-FSCC] section 2.2.

There some other cases around named pipe operations and VHD resiliancy but it sounds like DFS might be the cause here.

Do I have something special to do if under a DFS?

You shouldn't have to do anything special, it should just work. Unfortunately without being able to reproduce it myself I'm very limited with what I can try out and suggest.

Are you able to try out or clarify some of these:

jborean93 commented 2 years ago

Closing due to no response, if you are still having troubles here and can answer the questions above I can try and take a closer look.