[Discussion] SMB Server not granting credits

[skhare-r7]

Hello

I'm running a series of calls using SMBJ on a server, and eventually I'm hitting this exception:

Exception: com.hierynomus.smbj.common.SMBRuntimeException: Not enough credits (0 available) to hand out 1 sequence numbers

• The chain of events is as follows:

  1. SMB2 connection is established, credits are granted.
  2. SMB2 requests / responses are exchanged, usually receiving the number of credits that are requested.
  3. After many transactions (~2000 request/responses), something happens, credits granted drop to 0. From this point, no credits are ever granted.
  4. Since we have 512 credits banked, we make 512 requests after this point, requesting more and more credits. Each request receives a prompt response.
  5. Eventually, we run out of credits and see the exception above.

• Steps attempted to fix this (unsuccessfully)

  1. Added a delay between our requests (1 sec) when our banked credits are below a threshold (128). Effectively makes it so our last 128 requests are spread out over 2 minutes. We still do not receive any credits back and cannot recover.
  2. Attempted to send request despite not having credits (assuming perhaps miscalculation of credits between server and client). This does not work, the server resets the connection on using a credit that was not granted.

• Steps attempted to fix this (successfully)

  1. Increasing the SMB2CreditsMax from 2K (default on Windows 10) to 8K does delay the issue
  2. Reconnecting to the server with a new connection immediately works.

• A few more details about this issue

  1. It is difficult, but 100% reproducible
  2. It is seen on Windows 7 and Windows 10 hosts, which seem to have enough memory to handle these calls
  3. There are no Warning / Error events on the Windows Event Log (from SMB2 - Performance, Diagnostics, etc.)
  4. The Server / SMB service does not seem to show any apparent issues (freezing, crashing)

Any ideas what is going on here? Have you encountered anything like this in your testing?

[hierynomus]

Have not encountered this. From what you describe it seems that windows has some artificial limit in the total amount of credits that are granted to a single client. Can you check whether it always halts at exactly the same amount of total credits?

[skhare-r7]

I don't believe it is a total number of credits issue, since I've not been able to reproduce this by flooding requests in a loop. This might be some limitation on the Windows side. For now, it seems the only possible workaround is to reconnect the session.

[hierynomus]

Do you get this with all Windows versions? Can you submit a reproduction test case, which we can run?

[hierynomus]

@skhare-r7 Any update here? I did indeed mean that there might be an artificial limit on the windows (remote) side, not in SMBJ.

[smb311]

Hi @hierynomus @skhare-r7 did you ever make any progress with this? I have seen this happening in production systems (not SMBJ) and am wondering if anyone ever got some info from the Windows team about this issue.

[hierynomus]

@smb311 If you've got more info I'd be happy to hear it, but I have not seen this occur myself unfortunately.

But from what you say, you also see this appearing without using smbj?

[ErwanLegrand]

I have seen this happening with Samba's smbclient.

According to the specs, the server must make sure the clients' credits do not drop to zero: https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/2e366edb-b006-47e7-aa94-ef6f71043ced

Thus, this looks like a server bug. Now, one might wonder why MS clients do not appear to trigger it.

[skpatro93]

@hierynomus @skhare-r7 is this issue resolved as I am also facing this issue.

[rokkakasu]

Hi santhosh,

To an Smb share you can open only limited number of connections at a time. Not granting credit is a warning you can ignore. Once the previous connections are closed Smb server will allow new connection.

Try to reduce the frequency of opening connections. Issue is not with smbj library but with the Smb server you have.

Thanks R Ramarajan

On Tue, 5 Jul, 2022, 11:25 am santosh kumar, @.***> wrote:

@hierynomus https://github.com/hierynomus @skhare-r7 https://github.com/skhare-r7 is this issue resolved as I am also facing this issue.

— Reply to this email directly, view it on GitHub https://github.com/hierynomus/smbj/issues/270#issuecomment-1174646194, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIJCEGSIMDHCW64QNZ44J2TVSPE6RANCNFSM4EI5QGTA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[zacharybest]

Was doing some testing using a MuleSoft SMB connector, which uses SMBJ.

In working with infrastructure, we found that a smart firewall was interpreting the traffic as a DDOS attack. In this case, the traffic would be blocked, but the error we were receiving was Not enough credits (0 available) to hand out 1 sequence numbers

[akshay-pokle]

Thanks @zacharybest, worked for me. One of the firewalls was blocking the requests interpreting the traffic as a DDOS attack. Had to add the Mule CIDR range as allowed IPs in the firewall.

[zacharybest]

Thanks @akshay-pokle for the comment. I forgot I posted here and should probably post an update. I am glad it worked out for you!

In our situation, while the firewall was confirmed to reproduce the issue, the firewall was not the cause of the intermittent issue we were experiencing. Ultimately, we decided to pursue an alternative to SMB and I am no longer looking into the cause.

For anyone deciding to pursue this further, a takeaway is that this credit message can occur when no communication with the server is occurring. For example, firewalls blocking the communication.

While I'm no longer pursuing this issue, the OP states the issue can be reproduced by sending a high volume of requests. Firewalls are confirmed to reproduce it (and smart firewalls can potentially view a high volume of requests as DDOS or some other threat), but given recreation scenario, I'd guess it could also occur in other situations where internal banked credits are consumed, while at the same time an issue occurs impacting the application's ability to communicate with the smb server. Ex: app maxing available connections, broken socket, etc.