smbclient.shutil.copy*() should fallback if server side SMB copy isn't available

[jammann]

I'm experiencing STATUS_NOT_SUPPORTED: 0xc00000bb errors on some smbclient.shutil file operations (copy mostly) very similar to the ones described in #72 . I tried setting smbclient.ClientConfig(require_secure_negotiate=False) but this didn't help.

I presume the errors happens because some of our company SMB servers (Hitachi NAS) are pretending to support SMB 3, but apparently don't fully do so. I wanted to try to downgrade the high level client to only use SMB_2_1_0, but failed to find a way how to do this?

It seems easy with the low level API with connection.connect(Dialects.SMB_2_1_0), but how do I do the same with the smbclient high level API?

[adiroiban]

I am not sure this is supported... by a quick read of the code, I don't see how you can pass the dialect.

[jborean93]

Unfortunately there is no way of doing so with the high level API. Typically this shouldn't be needed at all and might be a sign that this library is doing something incorrect that causing a failure. Does Windows work normally when talking to the NAS? Can you potentially get a network capture of both smbprotocol and Windows so I could try and find some differences.

I would prefer to fix the problem in this library (if there is one) rather than having people downgrade to older and more insecure dialects.

[jammann]

Yes, from Windows a regular copy works. I tried copy(), copy2() and copyfiles() from smbconnection.shutil, and all of them fail the same way.

Regular open_file()/read/write operations from smbclient work without problems.

I will try to get a network capture from Windows, but I doubt I will be able to (it's a locked down corporate environment). Looking at the trace from Linux/smbprotocol I did not see any obvious problems.

I just know that these Hitachi NAS are not always 100% compliant when it comes to SMB standards. Or to put it better - the NAS have so many obscure knobs to configure things, that our NAS team sometimes has troubles finding the right combination. They already informed me that due to issues with some other SMB client (not smbprotocol) they had to configure the NAS to support at most SMB_2_1_0. This is also what I see in the in the DEBUG logs

2023-08-08 23:43:58.436 [MainThread] INFO: smbprotocol.connection: Negotiating with SMB2 protocol with highest client dialect of: SMB_3_1_1 {connection|||||| - |}
2023-08-08 23:43:58.436 [MainThread] INFO: smbprotocol.connection: Sending SMB2 Negotiate message {connection|||||| - |}
2023-08-08 23:43:58.438 [MainThread] INFO: smbprotocol.connection: Receiving SMB2 Negotiate response {connection|||||| - |}
2023-08-08 23:43:58.438 [MainThread] INFO: smbprotocol.connection: Negotiated dialect: (528) SMB_2_1_0 {connection|||||| - |}

It's going to take me some time to get a capture and have it reviewed to be able to post it here, but I'll try

Thanks for your quick and insightful response!

[jborean93]

If the error is around copying only it could be that the NAS doesn't support the copyfile mechanism which utilises the server side copy IOCTL operation. The shutil copy(), copy2(), copyfiles(), and by extension copytree() methods all call the underlying copyfile() method if it sees the src and dst paths as being on the same server/tree.

What you can do to test this out is to see if shutil.copyfileobj works. This utilises the manual reading and writing using the fd object Python provides rather than rely on server side copying, for example:

import smbclient
from smbclient.shutilimport copyfileobj

src = smbclient.open_file("...", mode="rb")
dst = smbclient.open-file("...", mode="wb")

# You can also just use shutil.copyfileobj as well for this
copyfileobj(src, dst)

If this is truly the case, the smbclient.shutil copy functions should be updated to handle the STATUS_NOT_SUPPORTED error to revert back to this less efficient copy mechanism for you. If it isn't it would be good to know what works and what doesn't work. It might help to narrow down the scenario and figure out what is failing.

[jammann]

Hi Jordon

Sure enough that is exactly the problem! copy'ing with copyfileobj() works :-)

Once again I'm just blown away by the speed and quality of your support! I wished we'd get just a 10th of that quality on some subscriptions that the company is paying tens of thousands for...

I'll take it up with our NAS team, if there is any way to enable that mechanism on the server side. Sometimes they just need a little nudge to get things going.

I'm closing this ticket, the answer is perfect for me!

[jborean93]

Once again I'm just blown away by the speed and quality of your support! I wished we'd get just a 10th of that quality on some subscriptions that the company is paying tens of thousands for...

Thanks you for your kind words, I really do appreciate it.

I'll take it up with our NAS team, if there is any way to enable that mechanism on the server side. Sometimes they just need a little nudge to get things going.

It depends on whether the NAS software has implemented the custom feature or not. If it's there it would need to be enabled, if not it would be up to the manufacturer to implement it for you.

[jborean93]

I am going to keep this open because the copy methods in shutil should probably catch this error and manually fallback for you. I'll look at implementing it in another release at some point (can't guarantee when though sorry).