Idea: Deep pruning for dust and empty accounts

[My1]

So we have the big issue of having a metric ton of accounts with the spam attack now we have about 25 and a half MILLION accounts/frontier blocks, which is getting annoyingly close to half of the total block count of 66 Million when I started out running a node, I have no idea how many accounts there were exactly pre-spam but what I read on the discord it should be several Million. The issue with that is that this means that not only you cannot prune these away with the pruning measures that are supposed to come soon as the frontier level pruning only clears out the past history of accounts, but also that all these accounts likely make it very annoying to get stuff like voting weights properly and all.

So I have an Idea that tries to tackle both. (not only that but this also helps a bit with the issue of single use addresses)

new tables for dust and empty accounts, where you add the frontier hash but ax the following fields:

• parent hash
• balance (empty accounts only, on dust a lower size can be used tho)
• rep (depending on need only empty or also dust)
• link
• work (if that is even stored past confirmation on pruned nodes)

for empty accounts that means we only need to keep 3 things:

• account address/key (for obvious reasons)
• frontier hash (to have the origin to sign toward)
• signature (obvious as well)

this means that the account only stores 128 bytes instead of the 216 according to the Docs, that would lead to savings of 88 bytes per block or almost 41%. If we assume about half the nano accounts are empty that's about 2,1GB (scratch that, see below) less in raw block storage (no idea about any implications for the database storage tho).

accounts that are not empty but very low like for example smaller than 2^64 raw could get their balance as an 8 byte number instead and possibly still ax out the rep field if the number is too insignificant still saving 80 bytes per block in terms of raw storage. if we keep these dust accounts out of the voting as well that might also increase performance of meta-operations like getting weights and all as there would be vastly less accounts to consider.

[clemahieu]

Is there a reason to keep the signature when we don't have the content's that's been signed? We can't verify it against anything.

[My1]

Is there a reason to keep the signature when we don't have the content's that's been signed? We can't verify it against anything.

that's why the frontier hash itself is kept rather than the biggest part of the content. also future blocks need the parent hash which would be the current frontier hash, honestly I am not extremely deep in it all but sure if a node can see its own storage as trusted and has other ways of making sure that block data isnt accidentially or otherwise modifiedm the sig can be dropped too, and sure if we can ax the signature, a thing I honestly would have never thought about, the savings would be another massive 64 bytes per block.

really good idea, if the signature can be practically dropped then that might work too. I thought to at least keep the hash and sig so the hash itself is pinned to the account and all.

[My1]

Adding for completeness, @qwahzi had the pretty cool Idea of adding tables with the most common dust amounts for example if there are a few million accounts with 1 raw, that would save 8 bytes per account so maybe not too much currently, the bigger issue is that spammers who want to be annoying can easily get around this by sending out differing amounts of dust anyway, which wouldnt be much different considering you can get like 0.000123 NANO from a faucet which is still over 10^26 raw to play around with to entirely screw this, also entirely unlike empty accounts, you generally dont tend to have millions of accounts with the same amount of dust legitimately, while zero on the other hand due to single use addresses and people exiting nano wanting to get rid of everything are a fairly normal thing in legitimate traffic.

[ctkristek]

Hello all. I am rather new here and I am really enjoying the amazing work you all are doing. My knowledge about nano, and more specifically the "dust" issue is limited, so please let me know if I am missing something here. It seems like issue right now is that nodes are having to process "dust" transactions which are so small in size, that spammers are able to generate millions of them at almost no cost. Working with that understanding of the issue (which I am sure is rudimentary at best), what would be the down side to setting a minimum transaction size. Importantly, this would not be a fee. Think of it more like walking into a gas station to buy a soda, and having to pay cash because there is a minimum for credit/debit cards. Those gas stations likely had a similar issue to solve, the fee/hassle for processing such a small transaction just wasn't worth it.

Now, i completely understand that Atom is built on the basis of being a fee free crypto--which I think is awesome. And thats why i am proposing a minimum transaction size, as opposed to a fee. We could even let each node decide what their minimum transaction size would be, kind of like how some place will accept a credit card for a $2 purchase while others wont accept under $5. This would be a more free market type of approach and allow the user running a node to determine the transaction size they are willing to process. Alternatively, a much simpler solution would be to set a blanket minimum transaction amount. Maybe it could be some proportion of market cap, current price, and volume (just some variable price that takes into account the current state of the coin).

From an business/economic stand point, I think this would make sense. People sending "dust" are not the customers/users that the platform should be catering to in my view. We want people who are exchange real amounts of Atom amongst each other. By setting something even as low as a 10 cent minimum transaction amount, would that not solve the dust issue? It would make it significantly more expensive to run these spam attacks.

This is just a thought. I would love to learn more about this and Im sure I am oversimplifying this quite a bit. Maybe this is already implemented and I am misunderstanding the dust problem.

Cheers,

[ctkristek]

Oops. I see now that the issue is account balance and storing that. Would the same work for requiring a minimum account balance?

[My1]

@ctkristek the problem with dust is not necessarily the transaction count as the basic pruning comes at least to passive nodes (aka those that dont vote), which means you only need to store the frontier (and unconfirmed stuff for the backlog) block of every account, as the frontier has all the info an account could literally ever need for getting the current information of it due to the state block design.

the obvious limitation tho is that while this definitely helps fold down long chains like binance or the nano faucets, spammers can just make more accounts, which obviously doesnt get helped by the pruning as the pruned node still has to store and notably process all those accounts at enough stages, which might be even worse than just storing some extra blocks.

The Idea of this proposal is to make it that accounts that are empty (which happen legitimately often enough) and accounts with VERY low amounts of dust (2^64 raw is about 18 µnano, so 18 millionths of a small nano, which already is a millionth of what most consider 1 NANO, yes the unit naming sux, I agree) could not only be pruned into smaller things by just axing off things that are not needed for knowing the base state of the account, but also potentially throwing out the accounts out of the voting calculations etc, so nodes do not waste time processing them, but only keep around the account address/key, the frontier hash (to keep the chain going) and if the account is not empty, the balance. that way nodes can care more about the accounts that have more meaningful amounts of nano in them.

---

I am against minimum transaction amounts as one problem is that a minimum amount would kinda defeat the purpose of having nano so highly divisible. having a minimum balance is something I have seen in stellar used as a method to stop account spam but it is also very annoying since it binds the nano up and makes it worse to spend as that min balance can be seen as a fee as it's stuck in there and not only would this mean that to open an account the sender needs to send that minimum amount, but also that the reciever cannot use that balance. a miminum amount for non-zero accounts might be interesting though, so an account must be either above the minimum, or empty, however, this number would need to change constantly depending on how much nano is worth relative to the amounts legit users (including those in poorer regions) have and transact, also this would be an entirely different issue from what I am proposing here.

---

for general spam attacks and getting them less annoying there are v22 with the new prio system as well as other proposals like bounded backlog or only allowing new blocks to be added if the current block is confirmed.

[B3g]

I don't know if it would be possible, or if someone had proposed it in the past, but making it more difficult to open an account with something like a super POW, I guess it would limit spam attacks and some ddos attacks?

[My1]

while that is certainly an idea it's its whole own issue, and with v22 seeming to kill PoW to make nano TRULY feeless and all, not sure if that would check out.

[B3g]

you can see if it is possible to make the first transaction of a new account difficult, perhaps giving them a pre-established low priority perhaps even of 1 day (an example), because the ease of opening new accounts makes an attacker very agile in the network

[My1]

I dont know anything about this but that should really be its own issue because it doesnt directly relate to the deep pruning itself.

[My1]

okay after revisiting my calculations I have honestly no idea how I arrived at the 2,1GB number but scratch that. taking some data from my node we have about 120 429 000 blocks, for convenience let's just say 120,5 Million. each block per the aforementioned blocks is 216 bytes so 120,5 Million blocks * 216 bytes/block / (1024^3 Bytes/GiB) = 24,2GiB raw block space in total (which obviously is less than a real database, but at least puts some ballpark numbers in to start with.

thanks to Trashman aka @mistakia I even got some actual stats to have some actual calculations with rather than just assumptions. https://discord.com/channels/370266023905198083/370266023905198085/849764427259314206

Number of accounts:
0: 7 354 323
<2^64: 24 868 632 -> (non-empty <2^64 = 17 514 309
>2^64: 631 368
-> total: 25 500 000

Total amount of Nano in <2^64 accounts (frontier balance only):
0.0000000043 Nano (TLDR: nothing)

so there are 3 main stats here: 1) my assumption with half of the accounts empty was off, okay. 2) 8byte accounts amount to basically no change in anything regarding vote weights, also even if all 17,5 Million had exactly the max of 2^64-1 raw or basically a bit under 18,5 µnano (that is a millionth of a small nano) we still get only 323,75 small nano which is still nothing in terms of vote weight. 3) the number of 8 byte non-empty accounts is massive, if you include empty it gets even crazier.

so I would say if we entirely ax voting weight and rep calculations for 8byte and empty accounts, I would not be surprised if this had a positive effect on the performance of nodes. (I'll make another issue for that), but on a more interesting node even if my empty accounts estimate was off, I was not expecting that many 8byte accounts to be entirely honest. so some new pruning calcs if we ax voting and signatures:

a frontier pruned node would have 25 500 000 blocks (plus pending plus backlog, which we'll ignore for the sake of convenience) which makes 5,13 GiB of plain block storage, which already is just a little more than 20% of a full block storage.

However if we now go with deep pruning we could save

7 354 323 152 bytes (8 work, 16 bytes balance, 32 link, 32 rep, 64 sig, parent hash gets replaced with frontier hash so no changes for the other numbers) = 1,04 GiB from empty accounts as well as 17 514 309 144 bytes (same as before but we keep 8 bytes of balance instead of removing it completely) = 2,35GiB

for a total deep pruning savings of about 3,4GiB, which is 66% of the already pruned db. (in raw blockspace)