Suspect the cheques do not get cashed out

[crtahlin]

⚠️ Support requests in an issue-format will be closed immediately. For support, go to Swarm's Discord.

Context

Bee used for uploading has a relatively large amount of BZZ in its chequebook. But the "Available Uncommited balance" is negative. It seems there has not been cheques cashout for this Bee in a while, I suspect the cashouts are not even happening, as other Bees also have no decrease in the chequebooks for a while.

Summary

Not sure, but this seems to be the situation for a while.

Expected behavior

I expect the chequebook of an active (uploader bee) to be drained steadily, daily.

Actual behavior

Explained in the context.

Steps to reproduce

Use a Bee actively and monitor its chequebook.

Possible solution

[ldeffenb]

It is up to the cheque beneficiary to decide when to cash out. The recipient (beneficiary) of the cheque pays the gas cost for the transaction, so naturally they won't cash small cheques. I just work with the availableBalance rather than looking at the totalBalance (references to the /chequebook/balance debug API response).

Have you cashed your cheques recently? I know I rarely do!

[crtahlin]

What you say makes sense. But I was under the impression that this is set in the bee config file, with these settings:

If the nodes are currently not cashing out cheques, because it needs to be done manually, there is a lot of debt accumulating.

I presume also that because of that, the nodes doing a lot of uploading / downloading are being blocklisted (blacklisted?) as they are in debt, yet the debt is not settled. Hence they need to find other nodes to work with, that have not yet blocklisted them, wasting time searching and switching. And maybe eventually running out of options?

Or maybe it does not work that way at all and none of the thresholds are respected? Do they need to be uncommented in the settings file and are not used by default?

But the way I see it for a healthy performing Swarm, the nodes should be cashing our regularly.

[ldeffenb]

It is not the cheque issuer's responsibility for the cheques after they are issued. It's just like writing cheques on your own checking account, you give it to the recipient, record it in your chequebook ledger, reduce your available balance, and keep going with life. But you don't (and the bee node shouldn't) write more cheques than your available balance can handle. That way, all of the recipients can cash their cheques whenever they want to and be assured that they won't bounce.

So, there really isn't debt accumulating in the swarm. What I find interesting is the negative number in your uncommitted balance. Can you issue the following command against your bee node and post what it says:

curl http://localhost:1635/chequebook/balance | jq adjusting the IP:port appropriately.

The configuration parameters you highlighted govern when cheques are issued, not when they are cashed. Cashing is a manually-initiated operation as it spends gas for the transactions as I said before.

AFAIK, bee doesn't blocklist nodes based on an imbalance of swap exchanges. The connections will continue, but swap-generating traffic may not flow to nodes for which cheques would be required if the sending node doesn't have enough available balance in it's chequebook.

However, there are also "pseudo" or time settlements in the protocol that provide for a small-ish amount of swap exchange for "free", meaning that small portions of the swap imbalance between nodes is written off or decremented. This feature is what allows ultra-light nodes to retrieve data from the swarm without even having a chequebook. As I understand it, if the pseudo settlement is exceeded, a node will simply not communicate with a heavily imbalanced (meaning, I owe you too much) peer but will keep the connection alive and use it once sufficient time has gone by. This allows for retrieval from the swarm at reduced rates, but with no crypto-financial expense.

So the thresholds ARE used, cheques ARE issued, but swap DOES occur even if a node's chequebook available balance has gone (close) to zero. I have a few nodes that have done this on many occasions. I see their swap exchanges slow down until I deposit more sBZZ (this is testnet) into their chequebooks. Then the swap transfer picks back up to speed and cheques are again issued to the peers. In the testnet, few if any nodes are actually cashing those cheques, but my node continues working as expected, provided that I keep a fairly positive available balance in my chequebook.

So I hope you see that the swarm will continue to be healthy, regardless of the existence of uncashed cheques.

PS. You can see a lot of this swap compensation activity, both pseudo and actual cheques, if you run your nodes with --verbosity 5 debug logs enabled. I don't recall the actual required level, but when learning how swap compensation works, I ran them all with debug logging enabled.

[crtahlin]

The requested output:

curl http://localhost:1635/chequebook/balance | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    76  100    76    0     0   1267      0 --:--:-- --:--:-- --:--:--  1288
{
  "totalBalance": "562813830999974700",
  "availableBalance": "-27986000082600"
}

Thank you for the explanation. I thought it worked a bit differently, still have to think about this a bit more.

[ldeffenb]

"totalBalance": "562813830999974700", "availableBalance": "-27986000082600"

Hm, I really didn't think that availableBalance should be able to go negative. I'll have to go look at the source again sometime to see how that might be possible. There may be a concurrency issue there where two goroutines both see enough available, but in total they drive it negative once they actually complete.

I'm interested in this because I've long been considering a hack that will allow me to throttle originating swap traffic (read: downloading data for /stewardship checks) so that my node chooses alternate peers for retrieval while skiing close to the edge without issuing swap cheques at all.

But the bottom line is that your availableBalance is your responsibility. It is unaffected by when/if your peers cash your issued cheques. When they do cash the cheques, your totalBalance will go down, but availableBalance will remain the same. If you deposit more funds into your chequebook, both your total and available balances will increase and your node will automatically go back to full swap performance for your uploads or downloads.